That's why it needs to be a scaled response:
On the Whitelist > Straight through -
not on Whitelist > Challenge of some description -
Fail challenge > No connection.
The problem will arise when bad actors get hold of voice recognition software that can beat the challenge and/or certain scams are profitable enough to justify using 3rd world call centres "Hello this is Microsoft calling we have detected a problem with your computer".