Re: "But then, you get to, like, 10 million training examples, it becomes incredible"
The aim of getting autonomous vehicles to outperform humans is a low bar.
Yes, but it's still not an easy problem, because of the really, really, really long tail – the vast array of improbable cases.
How many cases of "driving at highway speed and a significant part falls off the vehicle in front" are in Tesla's corpus? That's happened to me a couple of times. How many of "a significant part just fell off this vehicle"? I towed a Jeep once that had a wheel come off after the axle sheared, due to a manufacturing defect. How many of "driving down the highway and there's a vehicle on its side in the passing lane, facing back down the highway"? That's happened to me three times, once at night in a heavy snowstorm. How many of "oil slick on a hill on a curving country road" – I had that once. How many of "some random dude trying to direct traffic around a truck that's double-parked on a city street"?
Have Teslas FSD'd over Hardknott Pass, or other roads with sufficient grade that you can't see the road surface in front of you? I've done a few of those. (Actually I don't know where Tesla camera mounts are; maybe this isn't an issue.)
Humans are rubbish at maintaining vigilance, but surprisingly adaptable to novel situations.