The Pope’s AI Guy Is Worried About ‘Cartel’ Behavior Among Big Labs

Recently, warnings from researchers inside the world’s largest artificial intelligence labs about the rapid pace of development have led to panic about the possibility that the technology could extinguish humanity. Those warnings are predicated on the idea that—propelled by ferocious competition and a rivalry between the US and China—labs are barreling toward superintelligent models whose abilities will far surpass those of humans.

But according to Father Paolo Benanti, a priest who advises the Catholic Church on AI, the handwaving over superintelligence distracts from the need for open debate about how to constrain the companies building the technology. By casting the problem as so complex that only they can solve it, he says, the big labs exclude everyone else from the conversation.

“Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square.” Benanti tells WIRED. “We need democratic regulation.”

In an essay published on the heels of the safety warnings, Anthropic CEO Dario Amodei called for new AI models to undergo mandatory third-party evaluations prior to release, and some sort of crackdown on distillation, a process where rival labs train their own AI on the outputs of proprietary models. Others have argued for a total ban on recursive self-improvement, where AI uses its advanced coding capabilities to help develop future iterations of itself.

Benanti is less interested in restricting methods for developing AI, which he fears could quash innovation, and more so in establishing standards that ensure models cannot slip from human control. Whatever shape those standards take, he says, they should not be designed by the labs.

“The narrative of the frontier labs is self-interested, to gain or maintain a position in the market,” says Benanti. When a small group of companies dictates the rules under which an industry operates, he says, “this has a really clear name: a cartel.”

It’s difficult to know how seriously to take grand pronouncements about the capabilities of the latest and future AI models, says Benanti, owing to the overlapping economic, political, and personal incentives at play.

While both OpenAI and Anthropic have supported calls for a slowdown in development, they’re also preparing to go public, and there are many billions of dollars at stake for whoever captures the market. The animosity between the US and China, where the most advanced AI models are being developed, also complicates the picture.

Anthropic declined to comment. OpenAI and Google DeepMind did not respond to requests for comment.

Benanti also takes issue with the way certain corners of the AI industry talk about superintelligence, with the kind of reverence typically reserved for a god: as sophisticated beyond our understanding, practically omnipotent. (OpenAI CEO Sam Altman once said he aspires to build a “magic intelligence in the sky.”) That kind of framing, Benanti says, threatens to skew the discourse about the technology.

“Are we talking about some-thing, or some-one? This is the real difference. It’s not a matter of faith, it’s a matter of mathematics,” he says. “We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation [going on].”

By raising questions about what separates man and machine, and how to shield future generations from the negative effects of technology, the advent of AI has the potential to create “a new role for religions in public debate,” says Benanti. “Living the challenge of the time is part of the mission of the Church.”

Though Benanti is a member of the priesthood, he does not formally represent the Holy See, the governing body of the Catholic Church. Instead, he counsels the church in his capacity as a scholar of philosophy, theology, and technology. Benanti tells WIRED he uses AI to help create tools and materials for his university students, but declined to specify how Pope Leo XIV engages with the technology, if at all. (Benanti’s staff say he is bound by an “oath of secrecy.”)

In May, Pope Leo published his first-ever encyclical, a letter that applies religious teachings to the problems of the day and clarifies the position of the Church. He chose artificial intelligence as its subject.

In the letter, the Pope warned against the technocratic logic that runs through the AI industry, which presents “efficiency as the ultimate measure of value,” and casts human beings as “a project to be optimized.” He also called for international collaboration between rival superpowers and AI companies, who he said must set aside the “desire to secure geopolitical or commercial dominance” to ensure that the technology is developed in service of the common good. In one particularly prescient section, he warned of “the danger of humanity becoming a victim of its own achievements.”

The flap over superintelligence and existential risk threatens to distract from the ambitions laid out in the encyclical, says Benanti, by encouraging people to forfeit their say in the direction the technology should take to a small group of mathematicians, computer scientists, and executives at the big labs.

“The first thing is, we need to have brakes, and a wheel,” says Benanti. “That means a significant position of control for human beings over the machine.”