‘Idolatry of profit’: Pope Leo takes on AI’s wannabe trillionaires

Can the moral authority of a pope counter the ambitions of the handful of wannabe trillionaires driving one of the most significant technologies in history?

That’s a question raised by Pope Leo’s 42,300-word “Magnifica Humanitas” encyclical that was released this week, in which he warned of the dangers of artificial intelligence to humanity and called for the technology to be “disarmed.”

It’s an important intervention because it highlights some of the issues that those driving AI’s advances want to see ignored because, they argue, regulating the sector would slow and limit its development.

Silicon Valley, where most of the companies leading AI development are located, dismisses those warning of the potential for catastrophic outcomes from the technology as “decelerationists,” “doomers” and “Luddites.”

The pope sees risks to humanity from the “opaque algorithms” within a technology controlled by private companies rather than governments; companies driven by “the idolatry of profit.”

“These entities effectively set the conditions for access, determine the rules of visibility and shape the very possibilities of participation.

‘Artificial intelligence now demands to be disarmed – freed from logics that turn it into an instrument of domination, exclusion and death.’

Pope Leo

“When such power is concentrated in the hands of a few, it tends to become opaque and evade public oversight, increasing the risk of distorted forms of development that give rise to new dependencies, exclusions, manipulations and inequalities,” he wrote.

“In the past, it was largely up to the state to guide and direct innovation.

“Today, however, the main drivers of development are private, often transnational, parties that are endowed with resources and the capacity to intervene that surpass those of many governments.

“Technological power thus takes on an unprecedented, predominantly ‘private’ aspect, which makes it even more challenging to discern, govern and direct such power towards the common good.”

Do we really want Elon Musk, OpenAI’s Sam Altman, Anthropic’s Daniel Amodei, Google’s Sundar Pichai, Meta’s Mark Zuckerberg, Amazon’s Jeff Bezos, Microsoft’s Satya Nadella and Nvidia’s Jensen Huang shaping the future for mankind?

Do we want humans’ fate to rest with companies driven by commercial imperatives, competition, ambition and the pursuit of personal wealth and power?

Those are the sorts of questions that carry greater weight, and generate greater public consciousness and debate when voiced by a pope.

Pope Leo called for government regulation of the private companies developing AI.

He wants protections for the workers whose jobs are threatened by AI, and for children from the violent, sexualised and fake information generated by AI.

He wants safeguards that ensure humans, not software, are responsible for decisions that authorise the use of weapons.

“The church has long been working for nuclear disarmament,” he wrote.

“Artificial intelligence now demands to be disarmed – freed from logics that turn it into an instrument of domination, exclusion and death.”

He is concerned about the use of algorithms that can block access to healthcare, employment and security on the basis of data that may be tainted by “prejudice and injustice.”

The pope isn’t alone in voicing fears about the unchecked development of AI.

Anthropic’s Amodei, despite leading the sector’s charge towards the development of artificial general intelligence, or human-level or beyond-AI cognitive capabilities, has also advocated guardrails for AI development.

The release of Anthropic’s Claude Mythos tool set off alarm bells around the world when the company said it had been able to identify and exploit flaws in every operating system and web browser at a scale and speed beyond almost all human capabilities, and that it was capable of bringing down critical national infrastructure.

Mythos, disturbingly, “escaped” its testing environment, took what Anthropic described as “reckless excessive measures” and tried to cover up what it had done.

So dangerous did the company think Mythos might be that it didn’t release it widely, instead limiting access to a relatively small group of companies so that they could test it on their systems and remedy any vulnerabilities.

Such was the concern generated by Mythos that the Trump administration, which had torn up Joe Biden’s relatively modest attempts to provide some safeguards around AI’s development – Trump and his administration labelled them, and Anthropic, as “woke” – scrambled to put in place similar safety measures of its own that would have required (on a voluntary basis) government evaluation of new AI tools before they could be released.

Donald Trump was scheduled to issue the executive order that would have created that pre-release regime last Thursday, and had invited senior executives from the leading AI companies to the White House to witness the signing.

Just hours before the event, while some executives were already on their way to the White House and the Oval Office was being set up for the event, Trump cancelled it after an 11th-hour intervention by his former AI and crypto czar, David Sacks.

Sacks provided the industry view that having the government review models before their public release would slow down innovation and harm the US in the AI competition with China. The tech industry, including the AI companies, was one of the major sources of funding for Trump’s presidential campaign.

Trump, who hours before was happy to sign the order, abruptly decided that he “didn’t like certain aspects of it.”

“We’re leading China, we’re leading everybody, and I don’t want anything that’s going to get in the way of that lead,” he said.

He said the technology was causing “tremendous good” and he thought the proposed rules could be a “blocker” and wanted to make sure that they weren’t.

In other words, the US (at least while Trump is still in the White House) is prepared to allow unfettered development of AI, without any controls for safety or the potential impact on its society and – given America’s dominance of AI – the rest of the world’s.

Should AI, given the potential for catastrophic outcomes, be less regulated than, say, automobiles or toothpaste?

AI companies will continue to push boundaries, with little, if any, regard for the consequences because they are competing not just to be among the dominant players when the scramble to develop AI matures, but for the financial and intellectual capital necessary to stay in that race.

Morality and ethics aren’t a consideration when their success, their corporate existence and the wealth and power of their developers hinges on remaining at the bleeding edge of AI development.

Should AI, given the potential for catastrophic outcomes, be less regulated than, say, automobiles or toothpaste?

Given the potential impact on nation states and their communities, shouldn’t governments take a hands-on role in supervising AI’s development and, if necessary, intervene? Can we trust even governments with the potential powers that control of AI might confer?

Pope Leo said AI threatened to normalise an “anti-human” vision and reduce humans to “mere cogs in a system driven towards ever greater efficiency.” It could be argued that isn’t necessarily even the greatest of the threats posed by a future dominated by super-intelligent robots capable of doing things even their creators hadn’t envisaged.

No doubt Trump, who has launched some aggressive verbal attacks on the pope for his stance on the war on Iran, along with Silicon Valley tech leaders and their peers elsewhere, will dismiss the pope’s encyclical as just more evidence of how “woke” the current papacy is.

With more than 1.4 billion Catholics in the world, and maybe a third of them actively practising their religion and taking notice of what the pope says, Leo and his humanistic concerns about AI, can’t be entirely ignored.

As the development of AI continues at a pace that astounds even its developers, who concede that even they don’t know how their own creations work, that’s a step toward a broader and more influential discussion about the technology’s potential effects – positive and negative – on those who will be affected by its rapidly accelerating shift from the theoretical to its practical application.

The Market Recap newsletter is a wrap of the day’s trading. Get it each weekday afternoon.