Re: Audit the source? Nobody *runs* the source, they run binaries
"with a new version, the auditing needs to be all over again"
That is why you have an automated process, one where the agreed compilers and build environment are used and you can check that the binary coming out of the audit system matches the download version for a given code release.
Then your review of the source code changes is a meaningful activity.
But until the code has been independently audited by cryptographic experts (ideally not from the USA, etc, where there is a justifiable suspicion of court-ordered tampering) it is hard to trust the system, even as compiled from source, not to have either a foolish or deliberate flaw that makes the security much less than the password.
"a TrueCrypt virus. One that only attacks that particular program and inserts a backdoor into installed copies"
Really, you don't think that a simple key logger to grab the password would be easier and more deniable? If your machine has been compromised, even by a user-space program for your account, then ANYTHING you do from then onwards is, by definition, insecure.