I think you found the nub of the problem:
"If one good one gets traction in the market..."
For all that it sucks, MS Exchange sucks less than the other solutions for most businesses. I'm working in a shop that uses one of those *nix-based IMAP mail solutions. It's a mess and they are looking to move it to the cloud to save money. Originally it was going to be an in-house conversion to Exchange, but somebody didn't plan properly and at the last minute a reviewer pointed out a hole in the plan that killed it outright. No official word but I think they suddenly realized they hadn't planned for backup storage and nobody thought through trying to back up the user's local archives. I think they've finally gotten the server situation stabilized after standing up some new mail servers (upgrades had been on hold because they were supposed to be replaced by the new system).
The other obstacle is that Exchange has a good calendar client. We use an Oracle solution that is kludged to the mail servers. The whole thing sucks. They can't even tell me what the password rules are so we can inform users when they set their mail passwords. Just every so often when a user changes their mail password, calendar doesn't update. And don't get me started on the user interface. It's something only a command line guru could love, only it has a graphics interface and thus manages to offend both camps.
You are right, especially about the AV, archive, and data retention requirements. Current mail solutions all suck for these items and if someone can produce a solution at a price point businesses can swallow they will be the next Microsoft.