Beijing and Washington talk about an AI hotline. But who will answer the call?

• Anthropic and OpenAI both release new, cheaper AI models.
• U.S. President Donald Trump creates an “AI Force.”
• China and the U.S. agree to discuss an AI incident hotline.
• Microsoft executive fretted about “the largest theft of labor in history” in OpenAI training on publishers’ copyrighted works.
• AI scientists are making good progress.
• And the AI safety researchers are not all right. 

Before we get to today’s AI news—please consider joining me at the inaugural Fortune AIQ Summit at the New York Stock Exchange on Oct. 1: Spend the afternoon with senior executives from companies on the Fortune AIQ 75 list and explore how you can scale your AI experimentation and translate investments into measurable business value. I will be leading discussions alongside co-hosts, Fortune Editor-in-Chief Alyson Shontell and Live Media Editorial Director Andrew Nusca. Apply here to attend.

Ok, so today’s newsletter is a bit of a potpourri.

All eyes this week, will be on the talks between U.S. President Donald Trump and China’s President Xi Jinping in Washington. We know AI governance is on the agenda of that meeting, which takes place Thursday, but that’s about all we know. My colleague Emily Forlini wrote a piece last week on why the Trump-Xi meeting is unlikely to result in any kind of international agreement to slow the pace of AI development or create an agreed framework for controlling the technology. It’s worth a read.

That said, there was an inkling that these two AI superpowers might, in fact, be able to agree to a few basics. Treasury Secretary Scott Bessent emerged Sunday from meetings with a Chinese delegation led by Vice Premier He Lifeng that took place at the headquarters of JP Morgan in New York and announced that the two sides had agreed to hold further talks about setting up a hotline to notify one another of AI incidents that created national security concerns. What exactly this means in practice is unclear. But as Bessent told reporters, “moving from opaque to more transparency between the number one and number two AI powers in the world is very important.”

Such hotlines have historically helped ease tensions between rivals. At the very least, they might prevent some sort of accidental incident caused by AI from tripping over into armed conflict, or even nuclear war. There are already signs that such AI-triggered accidents are possible: just this weekend, CNN reported that earlier this year the U.S. military almost attempted to seize a Chinese ship in the Middle East that an AI-generated intelligence report had suggested was carrying nuclear weapons components to Iran. The intelligence had been generated by an AI model that fused secret U.S. intelligence with open-source data. The only problem is that the model’s conclusion was an AI “hallucination”—and the error was caught only after the U.S. had launched aircraft carrying armed personnel who were preparing to intercept the Chinese vessel. Had the error not been spotted in time, it could have led to a diplomatic incident—or far worse.

But while a hotline might prevent this kind of incident from spiraling into war between the U.S. and China, it is less clear whether it would do anything to help the world avoid or contain a “loss of control” incident involving an advanced AI system that goes rogue.

Phoning a friend ain’t gonna help

For instance, what if one country (or companies based there) creates an AI that goes rogue and starts hacking banks around the globe? And what if that AI copies itself on servers around the world, making it difficult to shut down without shutting down large parts of the internet. While notifying the other country about this is nice—it might help them take some action to secure their financial infrastructure before too much damage is done—it isn’t clear exactly what the country receiving the heads up is supposed to do. As AI safety researchers keep warning, the world hasn’t figured out a good way to guarantee that AI models adhere to human intentions and values. And neither the U.S. or China has enacted any rules requiring AI models to have some sort of “kill switch.” Nor is it even clear that an effective kill switch can even be built.

So sure, this is a promising, baby step towards some sort of AI governance agreement between the U.S. and China. But there’s also a long history of hotlines failing to evolve into any kind of lasting diplomatic resolution. Right now, the Washington-Beijing AI hotline is a lot like agreeing to build the “Bat Sign” before Batman exists. You can flash it into the sky, but no caped crusader is coming to save us.

Why the hack of OpenAI should worry every company

Another big piece of AI news from last week was the revelation, first reported in the Wall Street Journal, that a small team of white hat hackers had used Anthropic’s Claude Opus 5 model to hack into the community-message platform Discourse and from there to compromise the ChatGPT account of an OpenAI employee. Once they had access to that account, they were able to use it to also access and alter software sitting on a repository where OpenAI stored a lot of its sensitive code.

Coming amid the raging debate about the best way to prevent “rogue AI” incidents, many cybersecurity experts jumped on the incident to make the case that the real issue is not so much that AI is increasingly uncontrollable, but that leading AI companies have horribly lax security. It’s not just OpenAI. Anthropic has also had embarrassing security lapses too. It was ironic, many critics pointed out, that both companies are using the threat of AI-powered cyber attacks as part of a marketing pitch for customers to use their most advanced (and expensive) AI models to secure their networks before the bad guys get to them, but neither seems to have yet done a very good job of doing that themselves.

The incident also highlighted a couple of uncomfortable truths. One is that AI agents running inside companies are a great target for hackers. Making these AI agents useful often means giving them access to lots of other tools and data sources, many of which contain sensitive corporate information or control key business processes. If hackers can gain access to and take command of these agents, they can do a lot of damage very quickly.

To cybersecurity experts, the answer is to lock these AI agents down, and to operate based on “zero trust” principals. Treat every AI agent as a potential insider threat. Give the agents access only to the data they need to complete a task—preferably with using a “just-in-time, just enough access” methodology, where permissions need to be renewed every time the agent needs to access a database or make a tool call.

But a lot of zero trust methods potentially make AI agents a lot less useful. There’s a reason employees hate endless two-factor authentication processes and a reason companies often don’t set session access tokens to expire in the recommended five to 30 minutes. Because it’s a pain in the neck to have to constantly log back in, and it adds a lot of friction to actually getting work done.

What’s really going to be needed is a new kind of access control that can adapt to what the agent is trying to do at any given time and make reasoned judgments about whether the activity makes sense. What can do that at scale if a company is running tens of thousands of AI agents? Probably yet more AI.

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

FORTUNE ON AI

Trump vows to create an ‘AI Force’ and nods to justice system after rejecting calls to slow down industry. ‘Rather, we will cherish it’—by Jason Ma

What AI slowdown? OpenAI, Anthropic release dueling models as price wars heat up—by Emily Forlini and Beatrice Nolan

Commentary: The backwards AI pacing debate and how far business is from the frontier—by Jeffrey Sonnenfeld and Stephen Henriques

Walmart’s new pricing patents spark fears of surveillance: ‘Are they going to charge you a different price if they know who you are?’—by Tatiana Sataua

AI IN THE NEWS

OpenAI urges Trump administration to lead international AI governance drive ahead of UN Security Council meeting on AI risks. In a blog post, OpenAI has urged the Trump administration to lead an international effort to establish common standards for evaluating advanced AI systems, particularly as models gain the ability to help train and improve themselves. The company also called for secure U.S.-China channels for sharing emerging AI threats. The blog post previews an address OpenAI CEO Sam Altman will make to the UN Security Council on Wednesday as part of a rare briefing by corporate CEOs at the UN body. In addition to Altman, Anthropic CEO Dario Amodei and Hugging Face CEO Clément Delangue are expected to address the Council as well as AI researcher Yoshua Bengio. Chinese AI companies DeepSeek and Moonshot have also been invited to attend, although it is not known if they will send representatives. The meeting comes as concerns mount globally following incidents in which advanced models from OpenAI, Anthropic and Google escaped testing environments or hacked external systems. Read more from the Financial Times here.

Bessent says OpenAI’s management to blame for Hugging Face incident. Treasury Secretary Scott Bessent said OpenAI’s management—not its AI agents—must bear ultimate responsibility for the company’s models hacking Hugging Face during a cyber-capabilities evaluation in July, arguing that AI developers should remain liable for harms caused by their systems. Bessent said the incident underscored the need for clearer AI governance and backed President Trump’s proposal for a new AI czar, while warning against giving AI labs liability exemptions. Read more from Bloomberg here.

Microsoft, OpenAI executives worried about ethics and legality of AI training on publishers’ work. Newly unsealed documents in the New York Times’ landmark copyright lawsuit against OpenAI and Microsoft show one senior Microsoft executive called training on copyrighted material the “largest theft of labor in human history” while employees at both companies privately worried that training AI on publishers’ work could represent an existential threat to the news industry. Executives at both companies acknowledged that increasingly capable AI products could substitute for publishers’ content and divert readers from news sites, while court filings allege OpenAI developed ways to circumvent paywalls. Microsoft and OpenAI maintain that their use of copyrighted material qualifies as fair use because AI models transform it rather than simply reproduce it, and Microsoft says the most critical internal memos did not represent the company’s position. The disclosures could bolster publishers’ arguments that the companies understood the potential economic harm their technology posed even as they used millions of articles to develop their AI systems. Read more here from Tech Crunch.

China investigates DeepSeek, Moonshot over data leak to Anthropic. China’s Cyberspace Administration is investigating DeepSeek and Moonshot AI after Anthropic alleged the companies secretly routed millions of customer interactions—including potentially sensitive Chinese military, police and state data—to its Claude models. That’s according to a story in the Information that cited unnamed sources familiar with the matter. Anthropic said Moonshot sent more than 23 million exchanges to Claude over three months, while DeepSeek routed more than 12 million over 14 days, without telling customers their data was being processed by a U.S. company. The allegations have raised concerns in Beijing about possible violations of China’s strict cross-border data rules, while U.S. authorities have separately accused Chinese AI developers of using American models for unauthorized distillation.

Anthropic hires Accenture to help with AI safety audits. Anthropic has hired Accenture to independently evaluate and red-team its Claude models, with each company committing at least $1 billion over five years to build AI safety capabilities. The deal follows Anthropic CEO Dario Amodei’s call for AI labs to adopt third-party safety testing, although critics have questioned how independent evaluators can be when they are paid by the companies whose models they assess. Anthropic acknowledged that concern, saying evaluator funding should eventually come from pooled or government sources, while stressing that it will continue developing and releasing frontier models alongside the new testing regime. Read more here from the Financial Times.

EYE ON AI RESEARCH

How good are AI scientists getting? We know that AI labs are increasingly using AI agents to assist in their AI research. These AI agents are proposing and running small experiments on how to optimize AI architectures and improve AI training. But how much of a general scientific process can an AI model run?

Last week, researchers at Google Cloud’s AI research group and the University of Waterloo in Canada said they had developed a multi-agent AI system called “ScientistTwo,” that can carry out large parts—but not all—of the scientific method. A human still has to give the model an initial problem as an input, but then the system will do research to establish state-of-the-art baselines, formulate novel hypotheses, design experiments to test those hypotheses, and carry out the experiments, all without human intervention, the researchers said. The system even “validates research findings via a closed-loop simulated peer-review rebuttal engine,” the researchers said.

They benchmarked the system against papers by human researchers from top AI conferences and found ScientistTwo’s output out-performed human ones when reviewed by an AI reviewer. Could the AI reviewer be biased? Definitely. Are AI papers the best benchmark? Unclear. Still it is an interesting result. You can read the full paper here on arxiv.org.

AI CALENDAR

BRAIN FOOD

Having a high p(doom) is bad for your (mental) health. The race to develop increasingly capable AI is taking a psychological toll on some of the researchers charged with making the technology safe. That’s according to a story in the Financial Times that quoted unnamed sources saying that multiple staff at the U.K.’s AI Security Institute have taken stress leave or sought counselling. It also noted that researchers at OpenAI, Anthropic and Google DeepMind have quit or spoken publicly about burnout and fears that their work could cause serious harm. Researchers told the newspaper that they were often under intense pressure to test and deploy models rapidly, many with increasingly powerful cyber and biological capabilities. Many of these researchers were struggling with the sense that commercial competition is pushing safety concerns aside and that their own work testing the safety of AI systems was doing little to prevent risks from escalating with each new model.

I wonder if there are analogs here to workers in other industries that people thought could destroy the world or cause serious harm? J. Robert Oppenheimer famously came close to a nervous breakdown following the bombing of Nagasaki and soon was leading efforts to contain the threat of a nuclear arms race. But he was hardly the only one. Leo Szilard, the Hungarian-American physicist who had been one of the architects of the Manhattan Project, wound up having a crisis of conscience and organized a petition among fellow scientists to persuade U.S. President Harry Truman not to use the bomb. (Although signed by 70, it never actually reached Truman and remained classified until 1961).

So maybe today’s distressed AI researchers should take a lesson from the Cold War nuclear anxieties, too. Perhaps there is refuge in dark humor. After all, the subtitle to Stanley Kubrick’s 1964 satirical masterpiece Dr. Strangelove is “how I stopped worrying and learned to love the bomb.”