"Device drivers will never ever go through Google. Even if they did it would not help because they wouldn't be getting the source code."
They wouldn't need the source code. Just the blobs and the interface will do. With that level of control, they can do their darndest to work around recalcitrance.
"How would Google be responsible for exploits in other vendor's drivers?"
What if the exploit is in Android itself? Stagefright is an exploit in Android itself, for example. And some of the exploits are in the kernel, meaning it CAN'T be taken out of the system partition (because PID 0 essentially IS the system). If something worse than Stagefright comes along and pwns a million phones and is traced to the Android baseline, that stuff belongs solely to Google, meaning they're now liable (because no one else controls the code). That's the dilemma Google faces. They MUST gain control or they're going to face civil and probably even CRIMINAL liability (because something worse than Stagefright is a matter of WHEN, not IF).