I'm brown, born Muslim, grew up in a Muslim country (U.A.E), an Indian national, med school in India, moved in my late 20s to the UK (Nottingham), now doing a PhD, turned permanent resident this year (a proud moment for me). Applying for citizenship next year, so I don't count yet. It's clear the thrust of OP's question is to garner how many white Brits are afraid of the brown influx. So of course this is biased. But do with it what you will. It might be interesting.
Also, there's numbers out there that will tell you what things are really like, but the OP seems to want "feelings". I'll give you mine.
I go to London almost every month, often twice a month. It's my favorite city in the world. I go there for two reasons predominantly: 1) concerts and 2) to visit my brother who works there. He's an IT consultant, struggling to get on the permanent residence ladder.
OP's question has no simple answer. At a rock concert, I'm very often the only brown person there if it's a lesser known artist, and I still feel very much like a sore thumb at bigger events as well. Not afraid/apprehensive at all, but I do feel different. I certainly represent a miniscule, even negligible minority there. I've had to take some courses in the "rougher" parts of London to prepare for licensing exams, but never felt unsafe. I have relatives in Seven Sisters (Tottenham Spurs area), which is a bit rough, but your mileage may vary. I have no idea where these "no-go" zones are. I literally hadn't even heard the term used until Sherdog, and I've been an immigrant here since 2011. I've never been afraid for my life from an imminent terrorist attack, and I spend most of my fun time at movies and concerts.
Where I live (Nottingham), there's dodgy areas, sure, but you try to avoid them on the way to where you're going. I do the same when I visit an American city. Where areas are more diverse, both in London and Nottingham, I fucking love it. It's a million times more vibrant. I remember going to Brixton, London, in 2011 and loving how crazy it was, stalls and markets and whatnot. Now it's gentrified, which has its pluses but is no longer a place I'll go to with some buddies. In Nottingham, my (white) girlfriend, made it a point to cycle through the more diverse areas once every day to get some food - it's just more interesting: there's Polish, Caribbean, Indian and Pakistani, Lebanese, Arabic, pubs all in the same area.
As of now, there's a little bit of everything for someone like me. I'm a movie and music nut, so I tend to gravitate towards white spaces by virtue of having those interests. I tend to be the only brown guy in a lot of these situations, but to be honest, I actually find white strangers more awkward around me, than me around them (this DOES NOT happen in the US, white people just talk to me; clearly a cultural thing).
Among white people I see regularly, they tend to be friendly but not really curious about me or my background. Some of it is their not knowing whether they might say something wrong, but largely, at work for example, I genuinely think they couldn't give a shit where I'm from. A disinterest. This is fine, honestly. I tend to end up with white people who DO want to know things like that (like the girl I'm seeing) and are well-travelled etc., but for the love of Christ, how the hell is it that I, and say my brother, keep up so easily with the names of counties, travel within the country so much more, attend Christmas parties, sometimes two or three just for work, but most British people don't know that the name of my national language is Hindi and not Hindu/Indian? I understand it is for us to fit in but my countrymen have been coming here since the 70s in droves!
I think it is underestimated how much people like me love this country. We know so much about it because it's provided us freedoms and somehow stuck with Enlightenment values while our own countries can often feel foreign (when my brother or I are in an altercation in India, we provide a Hindu name and not a Muslim one, for fear of being beaten to our last breaths - no that is NOT overly dramatic, look up what's going on there with the BJP/Hindu Nationalism). We respect it, we bring our own values while embracing the freedoms the UK provides. How about y'all think of it the same way? Ask us where we're from, come to our homes, if you see me at a movie or a gig, strike up a chat. Know that we like being here loads, and love your ideas and freedoms. Largely you do get this, but good God has Brexit brought about the worst in those that don't.
The counterargument is: well, you're highly educated, you're the ones we WANT! You and your brother are always welcome, and have always been.
Really?
I will likely have my name on five publications in high impact journals by the end of my PhD, during which I simultaneously worked as the on-call doctor of the research facility. None of those studies would have run without me. It's clinical research in exercise, muscle physiology, diabetes, metabolic syndrome; some of those are crippling the NHS. Despite contributions of that sort, I spent close to 7,000 GBP of my own money on various visa applications, my permanent residence application, immigration lawyer's fees, etc. I switched contracts twice, a stress most couldn't grapple with if they knew that not having a visa in place each time would find them back in what the WR considers "shithole" India in a matter of weeks, despite my hard work, contributions to national insurance, and a wonderful new "immigration surcharge" being levied on skilled immigrants so they can use the NHS.
My brother is unable to get on the PR ladder despite having worked here for almost as long as I have. He is now going through the Canadian PR route as a backup while he puts in MBA applications, because staying in the UK on a work permit has not helped him build up any capital in terms of getting a PR. He might as well try getting into UCL/Oxford and see if he can get a scholarship, get educated some more. Yes, it can be that hard even if you're smart and sharp. And the Canadian PR process that supposedly just lets people in because they're brown? It takes nearly a year, sometimes more for someone as qualified as my brother to navigate. He's been at it for three months now. A lot of folks can't deal with the paperwork and will get a lawyer. Many simply won't try because they don't have the money. There's a substantial number of hoops to jump through, though it's still a million times better than the States. I think the problem with the US system is that it's not more like the Canadian one (righties going to love this).
You want people like us. You should be getting us in there, in reasonable numbers, with all the vetting that requires. What's happening at the minute in the US with skilled immigration is beyond idiotic. Instead, my ex-gf (Indian), who is an IT consultant in LA, has had two "Requests For Evidence" on her H1B this past year, for barely a decent reason. Guess what? She's looking at the Canadian PR as well.
The larger idea is: quit this nonsense. We're not trying to overrun your precious country. We're here because it's precious to us. When you go on about refugees and muslims, you do realise people like me have muslim parents who are observant and who we see once a year, sometimes because we've been too busy with research that will contribute to your well-being when you're older? That we know observant muslim doctors who likely were up for hours trying to get a cannula into your sick grandmother for the worst wages of any doctor, probably in the Western world?
Are there shitty elements? Of course. But not once have I met a radical Muslim in my time here. Like a lot of communities, they keep to themselves. I think they could work harder to think of themselves like you, sure. But that's happening as well, all the time.
We should be embracing a nationalism, but the melting pot kind. The kind that reflects the advances in the UK in light of your inclusiveness. I feel very included, and I guarantee your life has been made better by my predecessors. Why not focus on that instead of, "Have you noticed the brownies recently?"