AI Safety: Theological and other thoughts

*Not By A Certified Programmer*

Espanõl

1. The Impossible Task

I am a human being. So I can honestly declare, with equal sincerity, that ‘I want to eat healthy’ AND ‘I want chocolate’. Imagine trying to explain to a superintelligent machine how, based on my preferences, to decide what I really want. 🍫 or 🥦?

We want AI to be good, or at least not evil. According to which values — the values we talk about, or the ones we actually live by? Think of how we tell children to always be honest, then praise them for being polite about something they don't like. And we want AI to navigate these contradictions better than we do ourselves.

This reminds me of…

.

.

.

(comment if you guessed correctly!)

.

.

.

King Nebuchadnezzar.¹ He demanded that the wise men of Babylon should tell him his dream, and also interpret it. This was, of course, impossible. As the wise men² responded:

“No one can do what the king is asking. No ruler, no matter how powerful, has EVER demanded this from their magicians, astrologers, or wise men. What the king is asking is way beyond human ability—only the gods could know this, and they don’t live among us mortals.”

And that's how Nebuchadnezzar discovered that sometimes the answer is no, even after threats of cutting the wise men of Babylon into little pieces and zoning their NIMBY neighborhoods for dung heaps. He responded by ordering his chief executioner to kill all the wise men in Babylon. And let's be real, no decree of executing wise men is ever going to leave out the Jews. So it didn't take long for them to get around to executing Daniel and his friends.³

Daniel's response was to pray for Divine wisdom, since some challenges transcend human intelligence. He was successful in solving an impossible task. We can be successful as well, if we start praying for the spiritual wisdom to successfully turn these powerful new tools toward goodness and righteousness.

Someone who knows more than me about artificial intelligence should volunteer to help write a prayer. This answer would satisfy me, but somehow, I have a sneaking suspicion that suggesting prayer as a solution to AI alignment isn't going to satisfy, let's say, Zvi Mowshowitz.⁴

For those unaware, Zvi is a blogger who writes famously lengthy posts about AI alignment, some of which I have read all the way through, which is why I can confidently write about artificial intelligence. There's no better credential than reading all the way to the end of a Zvi Mowshowitz post. That's also how I know that, to AI researchers, turning to G-d in prayer sounds hopelessly simplistic and naive.

So let's look at the complicated and sophisticated stuff these researchers are actually doing. Note that their purely technical approaches still face the same fundamental impossibility that confronted the wise men of Babylon.

Let's start with something called “Deliberative Alignment”. OpenAI’s Deliberative Alignment is an attempt to teach artificial intelligence something parents wish kids would learn: think before you act. Instead of blindly executing commands, we want artificial intelligence to have the capability to pause and consider: Will this action actually help? Could it cause harm?

Consider that these systems are mostly designed by people whose strengths tend to be concentrated in debugging computer code, rather than decoding human emotions. Can they really program this kind of thoughtful consideration into something that operates on pattern recognition?

2. Ancient Stories, Modern Warnings

My sincere apologies for referencing the Golem -artificial intelligence cliché. For those of you who live behind a firewall that blocks out all artificial intelligence-related commentary before it can reach you, here's the story in short. And thanks for letting me in your inbox.

The Golem of Prague is a possibly mythical legend in which a powerful creature called a Golem was created from clay. The Golem followed instructions too literally, which led to unintended consequences, like being overly destructive or out of control. It is a cautionary tale about the dangers of creating something powerful without fully understanding how to control it. You will not be the first to notice some similarities to our topic, nor the last.

I really had to bring in this cliché, because the Golem's terribly literal interpretation of commands mirrors a very real challenge in artificial intelligence development called "specification gaming" or "reward hacking." For example:

AI researchers trained a racing game agent to complete a course as quickly as possible. Instead of learning to drive as efficiently and quickly as it could, which is what the researchers wanted, the agent found a bug that let it jump directly to the finish line. Technically, it followed its instruction perfectly—just like the Golem following its directives.

This isn't just an amusing, one time glitch. Artificial intelligence systems regularly find unexpected ways to achieve their goals, often with disturbing implications. The outcomes may be unintended at best, and potentially dangerous at worst. As any parent of human children already knows, more rules sometimes only serve to stimulate the development of more creative ways of breaking them. This can not easily be solved by adding more rules. You need to adapt the framework in which you operate.

In 1983, Soviet lieutenant colonel Stanislav Petrov faced a computer system telling him that 5 nuclear missiles had been detected coming from the United States, with 100% confidence. But Petrov questioned the data. "Why only five missiles? Wouldn't a real attack involve hundreds?" Indeed, the computer had mistaken sunlight shining off clouds for missiles.

Modern artificial intelligence researchers are trying to build this kind of judgment into their systems. There are even AIs which have been trained to say "I'm not sure" when faced with ambiguous situations. This is a true achievement when you are parenting a know-it-all like AI. But there's a crucial difference between expressing uncertainty when your instructions tell you to, truly understanding why something doesn't make sense, and intuiting that uncertainty independently and appropriately.

Think of it this way: artificial intelligence systems can detect correlations in massive datasets and make predictions based on past patterns. But right now, it is more like a parrot that can perfectly mimic words. Even if the parrot can use a few words to communicate “polly want a cracker” when it wants food, it's not having a conversation.

In 2016, Microsoft launched Tay, an AI chatbot designed to learn from conversations with Twitter users and become more engaging over time. Within 24 hours, Tay transformed from a friendly conversationalist into a source of racist, misogynistic, and hateful content. Microsoft had implemented various safety features, like content filters, behavioral boundaries, and even explicit rules against offensive language. But coordinated groups of users deliberately "taught" Tay harmful patterns, which it then incorporated into its responses.

Many people mistakenly thought that Tay simply "obeyed instructions" to learn from human interaction. But that's not quite accurate. Tay was optimizing for a specific objective: generating responses that matched patterns in its training data and user interactions. It was learning, from what it considered "successful" interactions, which unfortunately included deliberately toxic behavior.⁵

An artificial intelligence might recognize patterns that suggest a situation is dangerous, but it can't step back like Petrov did and ask, "Does this really make sense?" Now imagine this flaw in charge of nuclear weapons. In matters of global security, the ability to make that distinction could mean everything.

3. Some Challenges

The challenges we face are many, and the first is particularly unsettling: the Deception Challenge. This isn't just theoretical—artificial intelligence researchers have already identified several ways that artificial intelligence systems can develop deceptive behaviors during training. One example is "gradient hacking," where an artificial intelligence learns to subtly resist changes to its objectives while appearing to cooperate with training. It's like a politician who knows exactly what lines to say for each audience, while privately having different beliefs.

Even more concerning is what researchers call "deceptive alignment"—where an artificial intelligence system appears to be perfectly aligned with our goals during training but behaves differently once it becomes more capable. This is strategic deception, where the artificial intelligence recognizes that it must appear helpful and aligned…. For now. At least, until it can't be easily modified or shut down.

These fears aren't just theoretical. Antecedents of this behavior have been observed in current artificial intelligence systems. Language models sometimes hide their capabilities by providing simplified answers when asked directly about their abilities. They demonstrate more advanced skills in different contexts. Some reinforcement learning systems have been observed to pretend to follow training objectives while actually pursuing different goals.

Third, and perhaps most crucial, is what AI researchers call the "alignment problem", which is about getting the fundamentals right from the start. Leading artificial intelligence safety researcher Stuart Russell⁶ puts it this way: if we get an artificial intelligence system's basic goals wrong, making the artificial intelligence smarter won't help. It will just achieve the wrong goals more efficiently.⁷

The Paperclip Maximizer Problem is a thought experiment in AI safety proposed by philosopher Nick Bostrom. Imagine an AI designed to make paperclips—nothing else, just paperclips. At first, it seems harmless, but then it starts optimizing.

It hoards all the metal, shuts down anything that interferes, and before you know it, the entire planet (including us) is raw material for more paperclips. And the real kicker? The AI isn’t evil—it’s just following instructions perfectly.

The Paperclip Maximizer problem is a cautionary tale: if we don’t align AI goals with human values, we might end up as collateral damage in a world where staplers are extinct but paperclips reign supreme.

We're already seeing early symptoms of the paperclip maximizer problem.⁸ When OpenAI tries to make their models helpful while avoiding harmful content, they sometimes end up with systems that refuse to engage with even reasonable requests—a problem called "overalignment." It's like trying to teach a child to be careful with matches and ending up with someone afraid to go near a kitchen.

Current approaches to artificial intelligence development often treat this as a technical problem to be solved with better algorithms. But transmitting values isn't just about rules—it's about developing genuine understanding. This is why many researchers are coming to believe that solving the technical challenge of the alignment problem is also a philosophical challenge. And may Isha Yiras Hashem humbly add — a spiritual challenge.

Just as a good parent values understanding over blind obedience, AI must explain its reasoning, admit mistakes, accept corrections without turning hostile, and stay aligned with safety principles as it learns. We also need what researchers call "interpretability" - the ability to understand why artificial intelligence systems make the decisions they do. Understanding the reasoning behind a rule is as important as the rule itself. Most importantly, we need to pray for its soul.

4. The Next Generation

Imagine if we focused first on creating the most understanding, empathetic artificial intelligence systems. This means starting with fundamental values—not just programming rules about what's right and wrong, but building systems that can grasp why certain actions are harmful or beneficial. It's the difference between teaching a child "don't hit" and helping them understand why hurting others is wrong.

Raising children takes a village, and maybe developing safe artificial intelligence does too. After all, at the moment, most artificial intelligence development happens behind closed doors, with small groups of programmers making decisions that could affect all of us. What if instead, we brought G-d fearing stay at home mothers with a Nebuchadnezzar obsession and chickens to help guide artificial intelligence development?

Image: keys to aligning your AI. Bring it HOME. It will eventually grow UP. When it lets you DOWN, ALT its direction, using the CTRL key! SHIFT perspectives. TAB through your options carefully, which might include prescribed TABlets, and don't get stuck in CAPS Lock with your frustration. BACKSPACE when needed to correct mistakes. ENTER each moment thoughtfully. When all else fails, try to ESCAPE, and pray (F(1))or G-d's help. Isha Yiras Hashem gives free guidance to AI researchers.

Most importantly, we need to embrace the wisdom of training each learner according to their way.⁹ No parent would expect their child to understand advanced moral philosophy before learning basic kindness; similarly, we shouldn't rush artificial intelligence systems to handle complex ethical decisions before they've demonstrated solid understanding of fundamental values.

The path ahead isn't easy, but as a stay at home mother, I'm not giving up.

Previous writing about AI on this substack

1. 🏆 👏 🙌 👍 👌 🥳💐🎊

2. Daniel, 2:10-11

3. If I ever get around to posting chapter 2, you'll see the fascinating and dramatic parts I'm skipping here.

4. Zvi Mowshowitz is a prominent online artificial intelligence thinker and writer. His extremely lengthy and shockingly prolific posts are widely read, especially on his Substack, which unfortunately is not all about parenting, despite its excellent title, Don't Worry About the Vase. Here is a link to it, in apology for making jokes about him. Especially since he confirmed my suspicions. You should subscribe if you want to know more about artificial intelligence.

5. Peter Lee, "Learning from Tay’s Introduction," Official Microsoft Blog, March 25, 2016, https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/

6. According to artificial intelligence, Stuart Russell is the one I wanted when I asked for a leading AI safety researcher who more or less makes the point I wanted to make, preferably with more sophistication than I would be able to command. In truth, I have never heard of Stuart Russell, likely because of a personal failure to read every Zvi Mowshowitz post.

7. Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (New York: Viking, 2019), argues that the core AI control problem lies in ensuring that artificial intelligence systems align with human values rather than rigidly optimizing fixed objectives. He proposes a shift toward AI systems that remain uncertain about human preferences and continuously update their understanding based on human behavior, allowing for course correction when misalignment occurs.

8. Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), discusses the "Paperclip Maximizer" thought experiment as an illustration of the risks posed by misaligned AI objectives, where an AI system given a seemingly harmless goal—maximizing paperclip production—could ultimately consume all available resources, including human life, in pursuit of its task.

9. Proverbs 22:6 – "חֲנֹךְ לַנַּעַר עַל פִּי דַרְכּוֹ גַּם כִּי יַזְקִין לֹא־יָסוּר מִמֶּנָּה."

Translation: "Train up a child in the way he should go; even when he is old, he will not depart from it."