Re: "they failed to keep accurate data that was relevant to these people's cases"
Indeed, this is what makes me really cross.
The default position now simply being, "the computer says no" while paper records are destroyed. Coupled with the computer enhanced position of "absence of evidence is now evidence of absence" it will be forever impossible for people with partial documentation or documents "not on the approved list" to ever get through the process.
Immigration status will often not be checked, or even asked about for decades after people arrive, especially if they arrive as a child and it doesn't come up until an employer checks.
The home office also take every opportunity to charge people fees for doing their own job, while losing files and documentation in "Archives" and starting the whole process again. This then creates its own inefficiency and effort for the home office, while also preventing people from getting a job and paying tax.
If the threshold is now computer generated, I would expect it to be almost impossible to work effectively for another 20-30 years as data required now simply does not exist.