The problem that you point out is true, but probably not because of what you have stated.
Both sides in the referendum campaigned on negative agendas. As a result, the people in the exit camp were campaigning against Europe, rather than for a particular model of Britain's future. It thus became a wide church, covering a range of people who are out-and-out xenophobic, all the way to those who are prepared to live in a global society, just not one controlled by Europe. There really was no one issue other that Europe itself that the voters could agree on.
And for the campaign that wanted to stay, they really did overplay the fear card, claiming consequences once we had left that are unlikely to pan out to the extreme cases thay envisaged, so much so that many people just didn't believe it.
As a result of these broad campaigns, we have a situation where even if you just look at the leavers, whatever deal is chosen eventually will upset significant numbers of people. It's unavoidable. There is no way there can be a solution that will satisfy a majority of the UK population.
The referendum was just not thought out properly, and was framed in a way that, because David Cameron thought he could not lose, did not actually ask what kind of Britain people wanted.
I would like to have seen a third option of "kick it down the road for a few years". I believe that most people would have voted for that rather than stay in or leave, which would have shown Europe that the UK was not happy with the way things were going, and put them on notice that we really could leave.
I myself have no problem with working with people from other races, creeds and colours. I have no problem with them living and working here as long as they are doing something that cannot be provided by the local workforce (but please see below about my views on training).
My view was that I no longer wanted to be shackled to a group of countries that were becoming inward looking and so bureaucratic that it was going to become impossible to achieve any change, while the rest of the world was moving in a different direction. So I voted to leave. I did not want to be part of a United States of Europe, which I believe is the direction it is going.
I understood that in the short term, there would be economic costs, but my belief is that within 10 years, as long as we don't end up with some membership-lite deal, the UK will pull ahead of Europe.
Where I do have a problem is with a predilection for businesses to employ people from abroad in preference to training our own young (or even older) people to do the jobs. This not specifically a European issue, but was complicated by free movement within Europe. The lack of training upsets me greatly, as I want to invest in future generations. As I look at it, it should be a no-brainer to make sure we have a adequately trained and experienced native workforce. We have to do this for the sake of our children and children's children.