On October 22, 1844 the world did not end. This was a heartbreaking blow for the thousands of evangelical followers of preacher William Miller of upstate New York, who had calculated the exact End of Days from his mathematical study of the Bible. Many of the Millerites as they were known had gladly cast off worldly responsibilities such as planting next year’s crop and gathered out of doors to confidently await their unencumbered ascent into heaven - some draped in special robes fashioned for the journey. The fateful day came and went without any particular celestial event other than a passing thunderstorm; the lack of cleansing brimstone washing over the planet bitterly remembered only as The Great Disappointment. 
People tend to hang tightly to their views of reality, even in the face of such irrefutable facts as an absent Armageddon. Miller’s perplexed followers convened a conference in the state capital the following year to debate the possible reasons for nothing-of-note happening on October 22. Like Star Trek fans arguing over the proper conjugation of Klingon verbs, the remaining Millerites labored over whether the wicked faced eternal torment or mere annihilation. Other theological sticking points centered around ritual foot-washing or value the Supreme Being placed on shaving one’s head. One item notably absent from the agenda was the question of whether William Miller was possibly just full of shit.
Somewhat confused but undaunted, Miller’s followers split into roughly ten different factions but by no means did they abandon a worldview that married unwavering fundamentalist faith, bold real-world prediction and utter absence of evidence. Over 22 million people today subscribe to some form of Miller’s doctrine including the 14 million Seventh Day Adventists worldwide. Dozens of splinter beliefs cleaved over issues of divine etiquette including the Branch Davidians, almost all of whom perished in the shootout with the US government in Waco Texas in 1993.
Judge not lest ye be judged, as the good book says. Who is to say what people should and should not believe? A small tribe in Vanuatu believed that Prince Phillip was a living deity. Local custom holds that a mountain spirit from Vanuatu flew over the ocean to marry a powerful woman. This volcano god was foretold to be tall and have pale skin, further corroborated when the royal couple visited the island in the 1974. Hearing about the cult, the handlers for the Queen’s husband deigned to send the villagers a signed photograph, now worshipped as relic. In return they gifted their god with a traditional pig-killing club.
Which brings us to the subject of AI safety. Experts believe that value alignment between humanity and AI is critical to avoid undesirable outcomes involving something with vastly more agency than ourselves. How would a Millerite, or a Vanuatu tribesman, or a Cardi B fan describe core human virtues?
Beyond the enormous variance in human worldviews, how might a super intelligent AI discern our values - assuming it was even interested in such a thing? Likely by looking through our underwear drawer. Rather than obediently referring to some list of lofty ideals from panel of experts, an AI would instead be able to draw its own conclusions on the human condition by scraping zettabytes of raw information from social media and every other digital source on the planet.
Do you think most people would be unconditionally proud of their unedited thoughts and deeds if scrutinized by an omnipotent third party? Or the conduct of our elected or unelected leaders? Income disparity, human trafficking, exploitation of animals and a degraded environment might also stand in empirical contrast to our stated values. An AI would quickly be able to compile and assess the entire human zeitgeist, warts and all, and make decisions accordingly. In the proverbial last words of many a Darwin Award winner, "what could go wrong?"
Since there will be no sweeping of secrets under a lumpy carpet, clarifying and hopefully polishing authentic human values in the short window available is therefore fundamental for AI safety. So is somehow instilling in our AI offspring the ability and desire to experience compassion - we might be in need of it.
William Miller might have been off on the math but nonetheless, judgement day could soon be close at hand.