@pawl scooped me up from Bolter and Chainsword over this topic. I've sometimes stumbled across people who describe Warhammer 40'000 itself as something nefarious, which has always struck me as incredibly narrow-minded and an odd interpretation to say the least. Ergo, this is my attempt to describe 40k from a workshop perspective of writers' overarching visions, knowledge and inspirations as implicitly evident in their work and explicitly explained in interviews and White Dwarf designers' notes.
The references to Star Wars is in no way a smear attempt toward that good setting, but a comparison with a more well-known setting which most people can recognize. Star Wars is strong in its simplicity, but the simplicity itself also narrow down what we can expect to see from that setting. Both Star Wars and 40k are most enjoyable (and everyone who is the least bit interested in the former should check out the masterful four episode Clone Wars finale post haste):
On Inspiration and Historical Reference:
What is Warhammer 40'000?
40k is a smörgåsbord of a setting, where all manner of concepts rub shoulders: From emotionless killer robots, barbarian hordes and space bugs, to fallen empires, technologically advanced upstarts, religious fanatics and cruel pirates. 40k is rich aesthetically, and sports a vast written background which is meant to give players a universe in which they can dream big and craft their own nooks and corners, whether it be by creating their own Marine Chapters, Hive Fleets or Craftworlds complete with named characters and backstories.
The setting of 40k has been crafted lovingly by many hands and minds, some more learned and skilled than others. Poorer writers tend to end up with an impression of "Waaagh! The Emperor!" or "Look at the glorious heroes in their big pauldrons!", yet good writers never fail to display the deep flaws of humanity in the dark future. A great deal of ambiguity has been consciously injected into the setting, where the Emperor of Man can be viewed both as a divine saviour and a ruthless, bloodthirsty, powermongering mass murderer and tyrant.
One of the very best aspects of 40k has always been that the evil empire is its protagonist, and much of the setting is seen through its propaganda lense: This is leaps and bounds ahead of the more childishly black and white worldbuilding in Star Wars, for instance (certainly a great setting in its own right when it is done well). In Star Wars, we would never be shown benevolent sides of the Galactic Empire; and neither would the dark side of the Force ever be a fundamentally integral part of what it means to be alive, rather than just a corruption on the pure light side.
By contrast to Star Wars: In 40k, the Imperium is ridiculously oppressive and cruel, and its rotten stagnation may have doomed mankind, yet from another perspective it is also the sole remaining strong shield of humanity - incidentally brought about by the Great Crusade crushing all less extreme alternatives (an indication that the original vision for the setting has not been lost by later writers). Likewise in 40k, the daemonic Chaos gods are spawned by emotions which are inseparable from what it means to be alive.
Above all, 40k is a bonkers fun take on the most depraved aspects of human history, all exaggerated to such ludicrous heights that 40k has always achieved being its own parody. This is why religious wars, gladiator battles, human sacrifice, slavery, pogroms, G.U.L.A.G. labour camps, the Inquisition, horrible slum conditions, fascism, Stalinist purges, genocides, brutal repression, starvation, crazed sects, gang warfare, plagues and witch hunts all feature so heavily in the material.
It's a setting we love to visit for fun and dark humour, but not a place we'd ever want to live in.
It's all harmless fiction, and a great sweep of imaginary worldbuilding. Enjoy the ride!
On Inspiration and Historical Reference:
What is the historical basis of Warhammer 40'000?
40k draws heavily upon a great number of science fiction works which preceded it, not least Dune, the Foundation series and AD 2000 comics. This is part of its enduring success as a smörgåsbord setting of science fantasy, where the best parts of existing works were included to make a greater whole. The founders of 40k were an unusually learned bunch, and this knowledge enfathomed not only fiction, but also included a thorough understanding of history, mythology and archaeology. It shows in their creation: Just like its fantasy counterpart, Warhammer 40'000 is steeped in learning and historical references, often carried lightly and with a parodic touch of exaggeration as befits the best of British humour.
The entire premise of the 40k setting is that of a golden age long since gone to hell, of a descent into darkness, barbarity, ignorance and fanaticism. This is heavily based upon the decline and fall of the Roman empire (both in the west and east) and the Dark Ages in general. The late antique inspirations are obvious:
The human Imperium is beset on all sides by an increasing number of lethal enemies, while its bloated bureaucracy has swollen to an unprecedented size. It is an oppressive tyranny, with religious fervour on the rise as doomsday seems nigh. The once-all-conquering Imperium is mired in bickering and rotten stagnation, even as its professional armies (in spite of the general decline) still fight its ever-increasing foes with great strength and efficiency. Despite the societal decay and lack of innovation, one of the Imperium's core strengths lie in its sheer stubbornness and institutional ability to organize and muster resources; a very Roman trait.
As its wars changed from conquest to defence, its armies reorganized, and its Legions were broken down into smaller units. And so a doomed empire seemingly claw itself back from the brink of collapse time and time again, even as its internal strength slowly erodes away, much like the Roman empire did both in late antiquity and following its disastrous loss of all eastern provinces bar the Anatolian Themes.
These traits of late antiquity are mixed up with a decentralized feudalism/warlordism of the European Middle Ages, complete with knightly orders, an Inquisition and a very powerful (but often divided) Ecclesiarchy. The theme of fanatic religious wars are likewise sprung primarily from the Medieval centuries, inspired by those of both large Abrahamic religions (and including emperor Heraclius' last chance campaign against the Sassanid Persians), although the Catholic basis is clearly the strongest inspiration. The inspiration for plagues, starvation, witch hunts, colonization attempts, pogroms, slavery and alchemical mysticism are particularly drawn from the Early Modern period, while the sheer cruelty and harshness of state oppression is plucked from all over known history.
The 18th century has likewise lent inspiration not for rational philosophy and science, but for corrupt and decadent nobles, and for spaceship combat largely based on ships of the line during the age of sail. The 19th century is a large source of inspiration for 40k, with its overpopulation, polluted industry and miserable urban conditions of working and living.
The 20th century is unsurprisingly a strong basis of inspiration behind 40k: Its unravelling balance of power along with mass mobilization, dark deeds and different branches of extreme thought gave rise to a tide of bloodletting, hatred (both of classes and peoples) and artificial starvation, ranging from the conventional warfare of both World Wars in particular; through a long list of mass murder and genocide; to chemical warfare, massive purges, labour and extermination camps, barrier troops, human wave tactics, terror bombing, commissars, secret police torture and weapons of mass destruction.
While the concept of the Imperium as a theocratic dictatorship is largely sprung from an imaginative rethinking of the Middle Ages for purposes of fiction, the concept of the Imperium as a bloodthirsty police state mainly stems from the 20th century, with special mention to Soviet Chekists for pioneering this field. Likewise, the unhinged willingness of some 20th century scientists to experiment on living humans rings strongly with the sheer inhumanity of the grim darkness of the far future, as does the 20th century willingness of certain armies to make lab rats out of their own troops for atomic tests. Excessively deadly special forces training is likewise something mainly derived from the 1900s, although accounts of ancient Spartans likewise have a prominent place. Finally, 20th century Soviet stagnation rhymes well with that of the ancient Romans as a source of bleak inspiration for the grim darkness of 40k.
There is much more to be said in detail for the historical basis behind 40k, yet one thing is for sure: If there is anything dark and dramatic in human history, the kind of stuff which makes for the most harrowing and thrilling of stories, you can be sure that Warhammer 40'000 stands ready to be inspired and make good use of it for the purpose of crafting over-the-top fiction, as bonkers as it is majestic in scope.
People who like Star Wars and 40k tend to share a love for history in common. Indeed both settings play on strings of mythological themes and historical foundations: Both Anakin and Horus ironically caused a dark future to come about because they attempted to avoid it at all cost, in a narrative twist familiar from a great number of myths and legends, including Enkidu's faulty dream interpretation in the Epic of Gilgamesh or any number of folk tales documented in Herodotus' Histories.
If history has anything to teach us, then it is that the human condition ultimately is a tragedy. Good storytellers have long been aware of this, and the stories and worldbuilding found in Warhammer 40'000 is no exception. And so the far future is painted in dark colours, as a grand tragedy and unfolding catastrophe. The historical basis of 40k constitutes no small part of its grim darkness.