I have no problem with them living and working here as long as they are doing something that cannot be provided by the local workforce
To look at it another way -- why pick the nation state as the arbitrary regional border that defines how happy you are to have people working around you? What about the UK regions? Or counties?
(This is a variation on the counterargument to the ""unaccountable faceless Brussels bureaucrats" point, namely that the same argument could be applied to, say, Gloucestershire being largely controlled by unaccountable, faceless Westminster bureaucrats. And why should we Foresters pay attention to those fancy book-larnin' types at Shire Hall? Let's take back our Borough Council from County control! And so on and so forth.)