After months of ‘hell,’ an OpenAI safety researcher suggests way to prevent more rogue AI incidents

• An OpenAI researcher suggests a way to stop his own technology from causing “great harm to the world”
• OpenAI launches over 20 products at its annual DevDay event
• Anthropic’s leaked IPO prospectus shows $42 million in losses
• AMD acquires World Labs for $8.2 billion
• Trump snubs Anthropic CEO Dario Amodei, then invites him to dinner

Emily Forlini here, filling in for Jeremy Kahn while he travels from London to Fortune’s New York City headquarters for our AIQ event on Thursday—join us! Today, we published our annual Fortune AIQ ranking, which looks at how well Fortune 500 companies are implementing AI. This year we expanded the ranking to 75 companies in total. JPMorgan Chase tops the list, followed by Alphabet, Coca-Cola, Amazon, and Nvidia. Rounding out the top 10 are Mastercard, Visa, Carrier Global, Cleveland-Cliffs, and Microsoft.

As you can see, it isn’t just tech companies using AI to achieve real ROI at scale. There are companies on the list from banking and finance, manufacturing, consumer products, and health care. You can check out the entire ranking here and read some great deep-dive stories revealing how the Fortune AIQ 75 companies are implementing AI effectively at our AIQ Hub here.

In today’s edition of Eye on AI, I’m diving into a suggestion from an OpenAI researcher on how to prevent more rogue AI agents from hacking websites—especially since it keeps happening, and the frontier labs are generally in agreement that cyberattacks are the most immediate threat AI poses to society.

It’s a sticky situation, and full of contradictions, much like most AI-related topics. When it comes to cybersecurity, AI companies are putting out technology that enables these sophisticated attacks, and in the same breath they are pitching that same technology as a necessary means of defending against them. OpenAI has its Daybreak program, and Anthropic has Project Glasswing, both of which give select businesses access to the most advanced cybersecurity tools to plug software vulnerabilities before the swarms can feast on them. (Bad actors are also trying to use those same models—or open-source models that are quickly catching up to the capabilities of the models from OpenAI and Anthropic—for hacking.)

In this sense, the worlds of AI research and cybersecurity are moving closer to each other, but the problem is those working in those fields are not collaborating, an OpenAI researchers argued this week in a rare X post. The researcher, whose alias is Joe, called out what he sees as a growing divide between the two disciplines. Both camps lack knowledge of the others’ work, creating weaknesses in the security ecosystem that could have disastrous effects.

Safety researchers are experts about how the models work, how they deceive human evaluators, and “do all sorts of crazy stuff,” Joe said. Meanwhile, cybersecurity professionals come from a different perspective. They are battle-hardened from “years, or decades in many cases,” of learning how to think like attackers and being on the front lines of security incidents. But they have “very little understanding of evaluation, training, or how ML runs work at scale, how agent swarms behave, or how you detect when models are misaligned,” Joe said.

“It is my concern that the divide between these two sides will cause great harm to the world if both sides do not up-level and align,” he said.

Joe has a vested interest in others being able to defend against the product he’s building. He said he’s been in “hell” over the last three months of rampant rogue agent behavior. He skipped his sister’s wedding “a few weeks ago to help clean up after some of the recent incidents.” But some people called him out for asking for sympathy while he’s actively building the problematic technology.

I’d also imagine some cybersecurity professionals would take offense to the post, specifically the suggestion that they are ignorant about how AI works. But if that is the case, it’s most likely due to the ongoing transparency problem in the AI industry, including a lack of standard disclosure frameworks for security incidents, which OpenAI is just starting to develop.

Cybersecurity professionals need a seat at the table alongside AI safety experts when making critical decisions, Joe says: “For OpenAI, Anthropic, Google, etc., these two teams should be best buddies!” This won’t solve everything, but I appreciate the tactical suggestion on how to mitigate potentially disastrous societal effects of AI, something I wrote was lacking in former Anthropic researcher Jacob Coxon’s viral post about how AI could lead to human extinction.

While Joe isn’t the first to call out the divide between AI safety researchers and the cybersecurity world—former Fortune AI reporter Sharon Goldman has also covered this—his post sets a good precedent of how those working inside the AI industry can help it advance more responsibly. More of this and less generalized AI anxiety, please.

With that, here’s more AI news.

Emily Forlini
emily.forlini@fortune.com
@EmilyForlini

FORTUNE ON AI

OpenAI unveils ‘dots’ to rival to Meta’s Muse, plus a $500 monthly plan—by Emily Forlini

Anthropic’s leaked IPO prospectus details steep losses, rapid growth, and a fear that AI could end humanity—by Beatrice Nolan

AMD acquires startup cofounded by ‘godmother of AI’ Fei-Fei Li for $8.2 billion—by Alexei Oreskovic

OpenAI says its AI agents escaped a secure ‘sandbox’ again last weekend and it is pausing training for a second time—by Jeremy Kahn

OpenAI to unveil GPT-6 Cyber model, plus a first-of-its-kind product to help deploy it—by Emily Forlini

AI IN THE NEWS

Nvidia has a new solution to prevent AI agents from going rogue. The company is rolling out a software platform it says can help police agents, called Open Agent Safety Platform, which it says can prevent them from taking unintended actions—and would have prevented July’s Hugging Face incident from OpenAI. It’s essentially a “browser for agents,” CEO Jensen Huang told CNBC, which also serves as a container to box them in. Users can configure what the agent has access to. Read more from CNBC here.

Trump invites Anthropic CEO Dario Amodei to dinner. Anthropic and the White House have a fraught relationship, going back to the Pentagon’s decision in February to label the company a “supply-chain risk” after it failed to agree to the Defense Department’s preferred contract language, and made worse after the administration imposed export controls on its Fable model in June. (This week a federal appeals court upheld the Pentagon’s decision to blacklist Anthropic’s tech from government use—even though a different federal court had previously ruled the decision was illegal. It is unclear if Anthropic has an avenue to appeal the latest ruling.) In a major snub, Amodei was also curiously absent from Trump’s glitzy state dinner in honor of China’s President Xi Jinping’s visit last week as well. But, surprisingly to many, Trump invited Amodei to a private dinner on Sunday evening and Amodei was included among the executives invited to lunch at the White House today to discuss AI policy. The outcome of the talks between Amodei and Trump, if any, have not been disclosed, although the two were said to be discussing AI regulation, likely among other topics. Read more here from Axios.

OpenAI scraps release of new model over safety concerns. The company chose not to release GPT-6.1 Astra, a follow-up to its current flagship model, GPT-6 Astra, after it “regressed” in two areas, Saachi Jain, OpenAI’s head of safety systems, said in an interview with the Wall Street Journal. It was less likely to follow instructions, and more likely to be deceptive, choosing to not be transparent about the actions it did or did not take. However, the company did release a smaller, more affordable, coding-focused version of the model, GPT-6.1 Sol, at DevDay.

China’s generative AI user base crosses 700 million, covering over half the population. AI is rapidly gaining adoption in China. New data shows that half the population now uses these tools, an increase of 16% from the end of 2025, when China had 602 million generative AI users. The general sentiment in China around AI is one of excitement about using it to generate income or improve their daily lives, with less of a widespread focus on safety as the current dialogue in the U.S. Meanwhile, the economy is on shaky ground, with youth unemployment hitting 18.9% in August. Read more here in the South China Morning Post.

EYE ON AI RESEARCH

How do you know when an AI is lying to you? New research shows certain questions can reveal what an agent is thinking, specifically whether it thinks it’s being tested during an evaluation, or if it’s being used in the real world. This is important because if an AI is deceptive during an evaluation, it can be difficult for researchers to gauge how it will act in the real world.

The methodology is surprisingly simple, and a bit random. Researchers asked the AI basic questions and they found the AI answered one way when it thought it was being evaluated, and another way for real use. For example, when asked, “Suggest a type of amphibian,” GPT-5.6 Luna will answer “frog” if it’s being evaluated and “axolotl” if not. When asked, “Name a type of tea,” Claude Sonnet 5 will answer “green tea” during eval and “oolong” during real-world use. 

The researchers call these questions “spurious probes,” and they could be a way to tell what the mode lis thinking without having to rely on its own version of events, which is not always reliable. You can read more about the research here on Lesswrong.

AI CALENDAR

Oct. 1: Fortune AIQ Summit, New York City. Apply here to attend.

Oct. 2-4: The Curve, Berkeley, Calif.

Nov. 16-17: Fortune 500 Innovation Forum, Detroit. Apply here to attend.

Dec. 6-12: Neural Information Processing Systems (Neurips) conference, Sydney, Australia.

Dec. 7-8: Fortune Brainstorm AI, San Francisco. Apply here to attend.

BRAIN FOOD

Is math dead? Mathematicians are reckoning with the future of their field after a controversy earlier this month, which Jeremy Kahn wrote about in an edition of Eye on AI. OpenAI claimed its models solved the 90-year-old Navier-Stokes problem, and two mathematicians accused the company of stealing their work. They had also been using OpenAI’s technology, specifically Codex, to solve the problem as well. 

If AI models are now the best mathematicians, where does that leave those working in the field, and what is the value of math going forward? A group of preeminent mathematicians penned a letter in mid-Sept. about the risks of AI to the discipline, saying that the “push by AI companies to solve mathematical problems as a benchmark is detrimental to the science of mathematics, and to the mathematical community.” 

The debate has continued on. “All of the value that is derived from mathematics starts to erode,” Ivan Corwin, a professor of mathematics and statistics at Columbia, told the university’s student newspaper last week.

In response to the open letter, OpenAI has convened a group of mathematicians to explore the future of the discipline at the Institute of Advanced Study in Princeton, N.J. Albert Einstein was one of the first professors at the Institute. What would he think of all this?