Don't panic about the 4.6 million overnight figure. Doctor Who's overnights have been dropping steadily since 2007, not because it's shit and no-one watches it, but due to the increase in "record then play back later" equipment. All TV drama's overnights are down for the same reason, and all are experiencing increases in "watch later" viewers. Who is no exception.
Back in 2005 8% of Doctor Who viewers recorded Chris Eccleston's series and watched it later. By the time Peter Capaldi came along, that figure was almost 30%. But the final figure is always the same - between 7 and 8 million watch Doctor Who nowadays, just less of them on the night. I'm a Who fan, but if I'm going somewhere on Saturday or watching something else, I'll record it for later too (though I still like the little buzz of watching it "live"). It's no biggie, and I'm sure most of the country feels the same now.
Anyway, the X Factor is always going to beat Who on the night, because there's not much point in saving X Factor to watch later in the week, as you'll miss the chance to phone in and vote. Doctor Who (thank Christ) doesn't have a phone in.