I disagree. If it bothered them, they would be justified in saying "I don't feel comfortable doing that" - whether male, female or non-binary.
They would indeed be justified. But some (a lot) of people don't enjoy being put in the position of having to say that to their employer (who, after all, they rely on financially).
The whole point is that you shouldn't be put in that position in the first place.
Your position is like claiming it's fine to say "'ere love, fancy a fuck?" to every woman you meet because she's allowed to say "no". That's really not how this works