After all talking hands-free should logically be not much different than carrying on a conversation with a passenger
Compressed audio necessarily involves a delay. If this is <50ms, you won't notice.
But when the delay is ~150ms, the conversation is significantly distracting.
Next time you make a GSM call, think about the delay you're experiencing. It's not 50ms...