Unless you're reading every line of code how is this any safer than downloading binaries from repositories? Your security is dependent on other people in both cases.
I agree with you where, as I call it, 'hands on' security is concerned. If there's a small programming error in a program somewhere then it's quite unlikely that anyone will notice. I hate to drag up disasters from the past, but the Debian OpenSSL debacle really showcased this. This goes double in my opinion because the changes were done by a Debian package maintainer and were also applied to the encrypting engine. I'd like to think that the engine is the first place people would look for issues.
However, there is definitely a sense of security to be found in compiling your own software, in my opinion of course. I'm a FreeBSD user and all the servers I run have been compiled from scratch (so, the base OS has been reconfigured and compiled) as well as all the software I use on it (Ports collection).
Here's the thing: at the very least you can see for yourself that the software compiles, and without too many intrusive "hacks". What if a package maintainer or developer found something odd and instead of fixing the programming error they simply used other compile flags to "make it work"? Wouldn't be the first time, and this approach is a sure way to check for it.
However, that's still in the eye of the beholder.
The main security advantage can be found in customizing. If you only use default settings then any intruder will know exactly where to look and how to accomplish certain tasks. But the moment you change those defaults to fit your own environment this task can sometimes become a lot harder. At the very least you'll stall or stop automated processes such as rootkits.