Not so simples.
Your name and postcode are out there on numerous marketing databases. As for a poll card number match, I do not now what my poll card number is. It is not a number I have ever needed to write down. So i expect introducing this as a requirement would disenfranchise 99% of the electorate.
OK, that last is fixable, but then there would be a DB of poll card numbers, which would get hacked sooner rather than later.
What, you don't think they'd resort to impersonation and hacking?