Now I don't feel so bad about having 9 different SSH keys on my computer, each with a unique password.
Luckily an SSH key password is entered entirely locally and into an SSH Agent. So, simply no (remote) password sniffing possible.
But, of course. Even with keys, if your keys are stolen it could be game over regardless.