"Well, if it resembles auto-pilot systems (such as those on the Airbus), the correct fall-back would be manual control by the driver"
Yes, and look how well that worked out for AF447 after all!
See that is the problem, if it can't cope near-perfectly with anything on the roads your screwed. You won't be sitting there with full concentration all the time "just in case" - otherwise you might as well be driving. And in the event of an unhanded exception as car has seconds to impact, not the minute or two the startled pilots of AF447 had.