you could apply that thinking to any variable
I could, for instance, be a male with straight A* at school and college and with a 1st class honours degree and hence be much more responsible than one of the local Bogans. We may both share an interest in higly modified cars, under the currently commonly accepted status quo I would be penalised simply for having a modified car because he will not be responsible with his.
This is the way the insurance system works and so it should. having a modified vehicle does not prove that I am a better or worse driver than anyone else but staticially there is a connection between modified vehicle and higher accident rates. Being a male does not prove that I am a bad driver but statistically, assuming all other factors are the same, I am more likely to have an accident than a female.
The end extension of your logic is that they (insurance companies) should not penalise me for the poor behaviour of others. Even if I am 19 year old male on a provisional license driving a highly modified needlessly powerful japnese import. Because I might just be the exception to the rule.
"The fundamental problem is that being male doesn't inherently mean you'll be a worse driver"
I absolutely agree that being male does not inherently make you a worse driver. But it does inherently make you more likely to take risks than a female paticulalry if you are a young male. this in turn makes you more likley to be involved in an accident.
Where ever there is a clear statistical variation based on a variable whose value can be easily and clearly measured for all policy holders then the insurance companies should be adjusting premiums to reflect that. Regardless of whether that variable be gender, age, vehicle modification, claim history, garaging, primary purpose. It is a fact that males are more likely to have an accident than females, whatever the reason. It is not discrimination to adjust premiums to reflect this.