In the epicenter of AI, technologists reject Pope Leo’s warnings about the new technology

When on Monday Pope Leo XIV. presented a 42,300-word open letter to the world’s 1.4 billion Catholics calling for protection against the rise of artificial intelligence, joined by Christopher Olah, co-founder of Anthropic, one of the leading AI companies.

As Leo urged corporate executives, government regulators and other citizens of the world to protect humanity from the dangers of artificial intelligence, Leo included Mr. Olah as a symbol of the dialogue he hopes to foster between leaders of the spiritual and technological worlds.

But for Jeremy Nixon, Monday’s meeting at the Vatican showed that the two worlds are far from connected. While the pope said artificial intelligence is not fundamentally human, Mr. Nixon, a well-connected figure in the frenetic Bay Area artificial intelligence scene, argued that Mr. Olah’s remarks he seemed to suggest otherwise.

“They’re not having a dialogue,” Mr. Nixon said during an interview at AGI House, a San Francisco “hacker house” with deep ties to many of the people who helped create the artificial intelligence technologies discussed in the pope’s encyclical. “Their perspectives are different.”

The difference between a humanist’s view of the risks of AI and a technologist’s dream of what it might become is something that has long been debated in Mr. Nixon’s community. “That’s why the community exists,” Mr. Nixon said. “It’s its basic purpose.

Mr. Nixon, 33, is one of the founders of AGI House, which is named after Silicon Valley’s headlong pursuit of “artificial general intelligence,” a hypothetical machine that can do everything a human brain can do.

“We’re named after this moment in history,” he said of his seven-room group home. Over the years it has been home to a rotating collection of researchers, entrepreneurs and philosophers.

Mr. Nixon understands more than most the technology coming out of Silicon Valley and the attitudes of the people who create it. In the late 2010s and early 2020s, when Silicon Valley began developing the technologies that power chatbots, Mr. Nixon worked in Google’s central AI lab. He later founded AGI House with Andrej Karpathy, who was an early employee of OpenAI, oversaw self-driving technologies at Tesla, and recently joined Anthropic.

Mr. Nixon said the papal encyclical may mean something to the world’s Catholics, but he doubted it would have an effect on Silicon Valley. The only reason Silicon Valley even paid attention to the event, he said, was because Leo had invited Mr. Olah to speak.

To Mr. Nixon, the Pope’s open letter sounded like one of many government policy documents that have been shared between think tanks and regulators in the past few years. “The church doesn’t seem to have thought deeply about what their independent view of AI is,” he said. “It’s not clear they could do it even if they tried.”

He added: “They couldn’t take a position on it because they don’t understand it.”

Reaction to the encyclical from across Silicon Valley was relatively muted.

David Sacks, a Silicon Valley investor who served as White House AI czar, pushed back to the Pope’s call for additional regulation of AI. “If we give governments a lot of power over the development of artificial intelligence in the name of security, how do we prevent it from being used to censor, monitor and control citizens – as Orwell predicted in 1984?” he said.

Jack Dorsey, the co-founder of Twitter who now runs the financial company Block, he welcomed the Pope’s challenge so that the foundations of AI—patents, data, and technological infrastructure—are shared with everyone, not just a relative few.

The Vatican did not respond to a request for comment.

Overlooking San Francisco, toward the Golden Gate Bridge to the north, the AGI House hosts ad hoc seminars on the philosophies driving Silicon Valley’s AI push and “hackathons” in which researchers from leading labs explore the boundaries of the technologies they’re building.

In the early afternoon after the Pope’s encyclical, dozens of folding chairs were lined up across the living room. But on this Memorial Day, the only figures in the room were two jet-black mannequins standing against the wall.

The founders of AGI House first moved into a $68 million mansion in Hillsborough, California, south of San Francisco. They wanted to create a group house where researchers and engineers would gather to discuss new ideas and start new companies. Mr. Nixon built a second house in San Francisco near the top of Twin Peaks, overlooking the city from its highest hills.

Many Anthropic and OpenAI founders and prominent researchers joined the first meetings at AGI House. Mr. Nixon is now the founder and chief executive of a start-up called the Infinity Artificial Intelligence Institute, which seeks to automate the creation of AI.

Mr. Nixon said he encountered a generation of scientists who shunned traditional religion in favor of technology. After growing up with books like “The God Delusion” — in which evolutionary biologist Richard Dawkins portrayed God as a false belief at odds with empirical evidence — he and his peers saw AI as an alternative that was more real and far more powerful.

AI started solving math problems people have been fighting for decadeshe said, and soon he would cure diseases in the same way. “Practically speaking, it will achieve results that many religions claim their deities would be able to achieve,” he said.

This is an increasingly common belief among researchers in Silicon Valley. They insist they are on the way to building a more powerful species – or even a new God.

“People are saying matter-of-factly that they want to build a machine god,” said Rayan Krishnan, chief executive of Vals AI, a San Francisco company that tracks the performance of the latest artificial intelligence technologies. “They’re not saying it ironically or jokingly. They’re saying it as a matter of fact.”

In his encyclical entitled “Magnifica Humanitas” or “Magnificent Humanity”, Leo called for guarantees to protect humanity. He emphasized that people must continue to play a vital role in the workplace.

“A society that guarantees employment to only a small part of the population, despite having a high level of technical development, risks exposing many to forced inactivity,” he wrote. “This creates a paradox of material progress and anthropological regression that undermines the foundations of a just and stable social peace.”

Work, he added, is not just a way to earn money. It is “a requirement of the human condition, a normal path to maturity, development, and personal fulfillment.”

He also used the biblical story of the Tower of Babel — in which a group of people who speak a single unified language aim to build a tower that reaches to the heavens so they can exercise their power — as a way of warning Silicon Valley about the pitfalls of trying to outdo God.

His strongest message was that artificial intelligence is fundamentally not human because it can only mimic certain parts of human intelligence and behavior. “So-called artificial intelligences do not have experiences, do not possess bodies, feel pleasure or pain, do not mature through relationships, and do not know from within what love, work, friendship, or responsibility means,” he wrote.

But in a seven-minute speech at the Vatican, Mr Olah, the co-founder of Anthropic, described a technology that may have advanced beyond a simple machine.

“What’s actually going on in them?” he said. “We find structures that reflect the results of human neuroscience. We find evidence of introspection. We find internal states that functionally reflect joy, contentment, fear, sadness, and restlessness.”

Mr Nixon thought Mr Olah’s speech was far more spiritual than the words spoken by the Pope, although Mr Olah said it was not necessarily based on science. Although Mr Olah hinted that chatbots could have a working life of their own, Mr Nixon said they were still very dependent on humans to do anything, as Mr Olah noted.

“AI is still driven by people trying to make money or solve some mathematical problem or what have you,” Mr Nixon said.

But what is clear, he added, is that AI researchers are trying to build technologies that have jobs, feel pleasure and pain, and exhibit all sorts of qualities that match and even exceed the qualities that make us human. He believes it could happen within ten years.

He laughed at the Pope’s invocation of the Tower of Babel. While at Google, the company’s Babel project built translation technology that essentially allowed everyone to speak the same language.

On Monday evening, he and others in his circle gathered in the living room of AGI House to discuss the encyclical. They agreed that Leo’s open letter would have little influence on Silicon Valley, he said, but wondered whether Silicon Valley could influence the pope — if the Vatican could use AI technologies to “create a New Jerusalem.”

“AI and its capabilities represent something like the Second Coming,” he said.