"the historical association white = good*, black = bad"
I'm quite sure it comes from a very ancient experience that day/light are safe, dark/night are dangerous because you can't see dangers waiting for you (unless you're a cat).
The problems came when religions tried to give a supernatural explanation to ancient fears, and turned them into "personae" - thus the evil, dangerous one often live in the dark and are black while the good, helpful ones live in the light and are white (it also suited best fair-skinned people, of course).
Which in turn probably created a justification for racism especially against "black" people.
PS: left handed people have been discriminated, and often children exhibiting the behaviour were forced in different ways to become right-handed until recent times.