Not really, when you get your polling card they write the cards serial number down next to your name, and when counting the votes they write down the cards serial number and it's vote, this is all then stored in a database. So the electoral commission know who you voted for.
Not quite to not at all.
A ballot paper has two parts; a receipt part which is kept by officials which gets the voter's registration number written upon it. The part you are handed to express your vote. Both parts have the ballot paper's number encoded on it as a sequence of punched holes.
The receipt part which officials retain and the part with the vote are both kept but never married-up unless there needs to be an investigation into the vote. They not only need to be married-up but need to be matched to the registration list to determine who actually cast that vote. All three things are needed to identify a vote, just two isn't enough.
While it can be done it is not done by rote, rarely done at all. There is no correlating data stored in any database so political parties, the council, and electoral commission, do not know who you voted for.
It would be possible to undertake such a task to determine who voted for a particular candidate being recalled but it's an awful lot of work and would identify who voted for whom, worse, it would be easy to tell who voted for the candidate if only those received recall voting invites.
You would need to send invites to everyone, with either a 'can' or 'cannot' vote card, or have to marry-up this and previous votes to only count those entitled to recall. It may also encourage those not entitled to vote to commit fraud to do so.
It's much easier and cheaper to just allow everyone to vote in a recall and, as those elected are meant to represent everyone, including those who didn't vote for them, plus those who made some kind of protest vote safe in the knowledge they would be elected anyway, it's not unreasonable everyone should get a say on whether they are doing a good job or should be recalled..