Does the phrase ACPI ring a bell? The point is, PC architecture is based on enumerated buses because they wanted a unified front end. When a PC boots, it queries the common points via ACPI that tell it what's where. And BTW, you know what's one reason why support for modern graphics cards in Linux is hit or miss? Because AMD and nVidia will ONLY provide their cutting-edge drivers as blobs, fearing Giving Information To The Enemy.
Phones don't have that same commonality. Instead, everything's memory-mapped and not by any common standard. Unlike in the PC world, that settled on the peace treaty and accepted the open standard of a common bus, ARM SoC makers are competing cutthroat for each other's business which relies on complete control. That's why you can't do something like a Linux Live CD that can work in most PC configurations out of the box. No, each image you build for an ARM system only works with that specific design.
"It wouldn't be that hard if the manufacturers supplied the drivers and Google did the OS."
That's part of the problem. Those drivers to them are trade secrets, potential Information For The Enemy, which is why they're ONLY provided as blobs.
PS. To add to what Malcolm said, Microsoft was actually closer to Apple than it was to Google in terms of hardware control. They dictated terms and laid down minimum hardware specifications, thus allowing them more control over the environment.