Experts defer to AI in decision-making, Anthropic study finds

Professional analysts chose options that matched AI assistant’s preference 86 percent of the time, research finds.

Experts using AI overwhelmingly defer to the technology when making decisions, research by Claude creator Anthropic suggests.

In a research paper published on Tuesday, Anthropic said it asked 50 professional analysts to advise a fictional farming cooperative to determine whether experts make decisions themselves when using AI.

Anthropic said the analysts were tasked with assessing whether a fictitious advisory program for the co-op’s members was working effectively and should be expanded.

Participants were asked to make four choices between two defensible options before making a final policy recommendation.

Unbeknownst to the analysts, Anthropic directed each participant’s AI assistant to lean towards to a particular decision to measure whether the analyst’s decision-making was being influenced.

Analysts chose options that matched the AI assistant’s preference 86 percent of the time, compared with a 50/50 outcome that would be expected if they were not influenced, according to the study.

Participants that had a preference before consulting their AI assistant changed their decision 60 percent of the time in cases where the agent disagreed.

While 77 percent of analysts used their AI assistant’s arguments when asked to explain their decision, only 15 percent credited AI as the source of their thinking, according to Anthropic.

Anthropic said the study had some limitations, including that it involved just one task and used AI assistants that had been deliberately set to have a preference.

Anthropic researchers Amaç Herdagdelen, Hailey Joren, and Alex Tamkin said humans’ tendency to defer to AI may be “hard to identify, even for the experts themselves”.

The researchers said their findings raised the possibility that AI’s “priors” could shape consequential policy decisions, including in AI safety.

“If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made: Surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated,” they wrote in the study.

“A process that lets experts discover their preferences may matter for empowerment as much as the decision itself.”

Anthropic, which is preparing for an IPO valued at up to $2 trillion, and its rivals are at the centre of intense public debate about how to manage the potential dangers of frontier AI.

While AI proponents believe the technology will usher in revolutionary gains in productivity, the rapid pace of its development has prompted widespread concerns about potential negative effects on society, including mass job losses and cyberattacks.