Software isn't hardware
Software simulation is a good way to weed out the most egregious bugs, but tacking on "AI" and thinking that it is perfect is a recipe for bad things.
When I was working on rockets, we started with software sims, progressed to a "hardware in the loop" mock up and then sub-system (engines, etc), full up hardware on the ground (or held down) and then actual launches (and returns. I was working on that long before SpaceX).
There are too many variables to rely on just one tool as gospel. Real hardware has tolerances just like an ideal op-amp used as a stand in when designing an audio circuit doesn't predict the real work performance of something out of the Analog Devices catalog.
There should be a Potemkin village qualifying course that AV's must pass that contains lots of random real world scenarios. Is an AV going to panic stop and roll if a tumbleweed is blown across the road? How about a ball in a residential neighborhood. Will an advertising sign with a person on it cause an AV to wait thinking it's a person that is going to cross the road? What happens if a sensor is blocked by mud or hit with an errant sprinkler? Will a rough road cause too much jitter in a sensor input for the car to make any sense of the data?
AI testing is only as good as the programmers. The real world is very good at throwing up combinations of issues that might not ever occur to a person or group.
Biting the hand that feeds IT
