It probably is hard if you don't have an ear for it. Because the rule isn't too obvious. And the consensus is if you're not sure don't use it.
And since it's so often omitted or misused not too many people will have an ear for it.
It's easier of there's a clear preposition first - of whom to whom with whom etc.
This works, but note the comments at the end.
And yes, I know I started some sentences with and.
Because I wanted to.