Its also the destination
I used to live in South Tottenham, a rather insalubrious area of London.
From an evening out in the City or West End it was almost impossible to get a (licensed, black) cab home - as soon as I gave my destination they would refuse.
I took to saying I wanted a cab to Stamford Hill and then directing them through the backstreets to my flat.
More than one driver hurled abuse at me when he realised where we were. I am white as were many, but not all, of the abusive cab drivers.