A false argument
If we do leave (in my view, we should stay), what is to prevent those workers from still working in the UK? Nothing from the EU side of things.
It would only be an issue if a post-EU UK government refused work visas for those it currently allows in under the EU’s freedom of movement rules.
And why would they do that?