We can answer that question today with laptops - why not plug into a TV and pick up the bluetooth controller?
As someone who prefers PC games anyway, I do wonder this. I think the answer falls into:
1. Can't be bothered to plug it into TV, want something just sitting there already plugged in.
2. Like the kind of games released on consoles.
3. Prefer that you can buy a specific console and be guaranteed to have games run on it for years, rather than having to worry about specs and upgrading.
4. Games are far more likely to be written to support controllers, that every console ships with.
1 would still apply. 2 would apply far more so - whilst there are plenty of fun games for phones, these tend to be a very different style than those on consoles. And although there are more free games, there are also fewer games made with large budgets.
For 3, whilst phones don't have the widly different specs of customisable PCs, there are large numbers of different models, as well as an upgrade cycle far faster than even PCs.
4 would apply even more so - how many games are written for bluetooth controllers, as opposed to written for pure touch that the majority of people will be using on a phone?
We might see that the technology delivered by the phone market could be used by manufacturers to create their own competing consoles, without having to roll their own technology and OS - e.g., as seen with the Ouya. Potentially this could lead to the market increasing in size (more choice, lower costs) rather than reducing.