"the cultural effect of having sizable numbers of people in the country who clearly reject our laws, customs and mores and wish to impose theirs on the rest of us aren't helping matters"
Not sure how much this is true or that I agree with it, however the groups that this is mostly associated with are the Middle Eastern refugees from Syria, Iraq and Afghanistan etc. Primarily Muslim (not that I have anything against peaceful Islamic beliefs) countries, whose entire cultural history is quite different to ours, unlike the EU member states (like Poland where most of the EU economic migrants in the UK come from) who's cultural backgrounds are to some extent similar to ours.
Refugees coming into the UK from these Middle Eastern countries have very little to do with the EU, although we have an agreement in place with the EU to take a certain number of them this is to ensure that they are spread fairly across the EU member states and other member states actually take in as many or more than we do, if for no other reason than that their country is physically larger.
If we were not in the EU we would still be taking in these refugees, possibly in even greater numbers, because we take them in out of a caring and humanitarian nature (after bombing the shit out of their countries) not because we have to because the EU makes us.