Nexus 4 owner here - caveat emptor
It has major build quality issues. The camera is truly rubbish. The screen is very obviously inaccurate. The fake chrome edges wear away very fast. The glass back was a huge mistake. (I've been lucky so far, but who knows?) The battery life is depressing, though Android 4.3 did seem to make life a bit better.
The Nexus 5 appears to fix many of these issues, though obviously nobody's had long to look at it yet. If you want the best Android there is, I'd wait. I'm with you on killing carrier phones, though - one reason I bought the Nexus 4 was it gave me a way out. None of these things are show-stoppers (and it is great value given the performance and screensize), but they're all annoying. As I hate OEM Android, I'm considering switching to iPhone - for all Apple's flaws, they do get hardware right.