Re: I just cannot see what the actual issue is
You're right, google are not a mafia.
When a real mafia like group gains access to this data then everyone you know, meet or have communicated with will be known to them, everywhere you've been will be known to them, everything you've looked at will be known to them.
If they decide you're worth their attention what can you do about it?