Re: ability to tell the difference depends on 3 things
My experience is with "home cinema" situations. Most of the cheap rigs you can buy for that have small speakers for the directional channels—the surround sound—and a subwoofer. But you can hear when all the bass is coming from the front, rather than from the same direction as the high-frequency element.
My father's hearing was pretty bad, but having background noises coming from different directions than the character speech made a big difference. Even just putting the normal TV sound through a Dolby Analogue Surround decoder can help (It makes me wonder what some of the "bad" TV sound really is.)
Anyway, most stereo recordings position sound sources merely by relative volume. They don't bother with phase. It's a trick. You don't hear anything now of binaural stereo, which did try to capture the phase differences. The modern digital surround sound does stand a chance of using that data, but speaker design tends to ignore frequency-related phase shifts.
Your show-off hi-fi audiophile with the big speakers isn't exactly wasting money, not until he gets into exotic speaker cables, but he may be missing out on a lot by sticking with such an ancient sound recording standard. Stereo sound is still faking it. So might a 5:1 recording, but it doesn't have to throw away so much data