Wunis I know what you mean, but I think there are more nuances than that. A supremacy movement, carried out exclusively by the Altmer, essentially subjugated the Nord population of Skyrim. I don't know if you've ever played Half-Life 2, but I think the Nords see the Imperials similarly to how we see Dr. Breen, in that they "sold out" to the invading party in exchange for power over their brethren.
So from this we see the foundation for their distrust of both the Imperials and the Altmer, as well as, to some degree, the Khajiit and Bosmer, whose homelands are client states of the Aldmeri dominion. The Khajiit are also distrusted due to the tendancy of many, at least outside of Elsweyr, to become thieves or assassins due to their superior stealth abilities. The Bosmer are known in Skyrim for the Wild Hunt which killed King Borgas, and plunged Skyrim into a 50-year civil war.
Another thing is that during times of war, it's very common for the ethnic majority of a country to become untrusting of others. The most obvious example of this was during World War II, not just in Germany, but also in Britain and other Allied countries. Again, in the Cold War, there was a strong ingroup / outgroup mentality, though this time it was based on ideology, not race. These ingroup / outgroup distinctions are deeply ingrained in humans and other primates, and just about any pack animals. They're part of our biology, though they can be overcome.
Please note that I'm not justifying any of these prejudices, I'm simply explaining my interpretation of them, and why I don't think the classic line of "The Stormcloaks are just racist" is in any way an indication of the entire story. All I want is for people to sit down and think about the nuances of the cultural attitudes in Skyrim.