Reading slush (unsolicited manuscripts of novels) is generally a terrible experience. There's an article from Teresa Nielsen Hayden (an editor at Tor) that lists the reasons things get rejected:
Herewith, the rough breakdown of manuscript characteristics, from most to least obvious rejections:
1. Author is functionally illiterate.
2. Author has submitted some variety of literature we don’t publish: poetry, religious revelation, political rant, illustrated fanfic, etc.
3. Author has a serious neurochemical disorder, puts all important words into capital letters, and would type out to the margins if MSWord would let him.
4. Author is on bad terms with the Muse of Language. Parts of speech are not what they should be. Confusion-of-motion problems inadvertently generate hideous images. Words are supplanted by their similar-sounding cousins: towed the line, deep-seeded, dire straights, nearly penultimate, incentiary, reeking havoc, hare’s breath escape, plaintiff melody, viscous/vicious, causal/casual, clamoured to her feet, a shutter went through her body, his body went ridged, empirical storm troopers, ex-patriot Englishmen, et cetera.
5. Author can write basic sentences, but not string them together in any way that adds up to paragraphs.
6. Author has a moderate neurochemical disorder and can’t tell when he or she has changed the subject. This greatly facilitates composition, but is hard on comprehension.
7. Author can write passable paragraphs, and has a sufficiently functional plot that readers would notice if you shuffled the chapters into a different order. However, the story and the manner of its telling are alike hackneyed, dull, and pointless.
(At this point, you have eliminated 60-75% of your submissions. Almost all the reading-and-thinking time will be spent on the remaining fraction.)
8. It’s nice that the author is working on his/her problems, but the process would be better served by seeing a shrink than by writing novels.
9. Nobody but the author is ever going to care about this dull, flaccid, underperforming book.
10. The book has an engaging plot. Trouble is, it’s not the author’s, and everybody’s already seen that movie/read that book/collected that comic.
(You have now eliminated 95-99% of the submissions.)
11. Someone could publish this book, but we don’t see why it should be us.
12. Author is talented, but has written the wrong book.
13. It’s a good book, but the house isn’t going to get behind it, so if you buy it, it’ll just get lost in the shuffle.
14. Buy this book.
If there was an filter on self-publishing that kicked out everything from 1-8, then I'd be happy to take chance on hitting a few 9s and 10s to get to the 11s and 13s that the editors are rejecting.
The problem that self-publishing should solve is the 13s (good novel; not commercial in the current market) can get published and find their market; mostly that will be only 5,000 readers, which isn't enough to be profitable for a publisher, but is still plenty for a self-publisher. Sometimes, the editor will be wrong and it will really sell (e.g. Dr Debra above).
The problem is that if you're picking through self-published stuff more or less at random, then you hit lots of 1-7s, or if you have a "wisdom of crowds" filtering system, then those often get gamed by spammers.
Add to that the problem that good novel-length writing needs real editing ("you've forgotten what happened to this character" "I've tried to lay out the events on a calendar/map and they don't work" - "this character had an unexplained personality transplant between book one and their reappearance in book four") as well as proof-reading and copy-editing, which, yes, self-published novelists can buy in, either by the hour or for a percentage, but most don't.