Let's face it, they're stuffed.
The problem is that all governments try and develop policies on the fly, to try and tackle symptoms of a particular (often temporary) problem. Then they get totally snarled up in the unforseen consequences (unforseen by them, everyone else spots them). I'm just waiting for some loony in the Home Office to demand that babies get a passport before they're born (with ultrasound scan photo), to prove they have right of entry to the country!
They need to go back to principles, which are based on who is allowed to ENTER the country.
1) British subjects do not need the permission of anyone to leave the country.
2) British subjects only need to show their passport to enter the country
3) non-British people entering the country need permission and need to identify themselves (passport, optional visa)
4) non-British people can leave the country whenever they like.
So develop some border controls based on the principles. Be consistent. If everyone entering and leaving needs to be checked, then close the border (again) between N.Ireland and RoI.