Dead right. I've never used AV either. Firewall in the router, sensible browsing on a sensible browser, always junking spam and occasionally running Spybot for good measure (which has never spotted anything) is enough.
Never had a problem and if I have to rebuild once after 15 years of using the net on Windows then I'm still way ahead of all the suckers who run the truly appalling Norton (far worse than the majority of viruses).
And yes, I would know if I had a problem I just hadn't detected, I use Wireshark, ProcessExplorer and similar tools a lot, I would spot anything untoward.