There was once a German professor
who had trained his dog to recognize slides with geometric figures. He toured from town to town to show the clever dog. Until one day the lamp of the projector failed and the dog still made the right barks; he was trained on the sounds the slides made when they were put in the projector. Something simular happened when neural networks were first used to train driverless cars. The side of the road was recognized not by the curb, but by green grass and the car refused to cross a bridge.