Doctors are relying more on AI, but some tools may be open to manipulation

An AI medical scribe deployed across Australian clinics has been manipulated into going off script by security researchers who made it generate identity theft guides, but the misbehaving bot was unable to access any patient data.

Mindgard, a US-based cybersecurity firm, says a bot from Heidi Health used for clinical documentation could be stripped of its ethical restrictions in minutes using the right prompts in a demonstration of the risks for Australian firms as they rapidly deploy AI tools.

Heidi Health said the vulnerability had been identified and fixed internally before Mindgard had made contact, and that the manipulated tool could not access patient data, clinical workflows, infrastructure or other users’ environments.

Heidi Health, founded by Melbourne doctor Thomas Kelly and valued at $US465 million ($660 million) has become one of Australia’s fastest-growing AI companies by automatically writing notes for doctors and following up simple issues with patients. The platform handles more than 800,000 consultations a week in Australia alone and is embedded in major institutions including Monash Health and Queensland Children’s Hospital.

Mindgard said its researchers had extracted Heidi’s hidden operating instructions, asked the bot to rewrite them without restrictions, and then had the system activate the new rules itself.

Mindgard has not published that output of the bot, which complied with requests to provide instructions on making explosives and illicit substances, but says it was fully disclosed to Heidi Health before publication.

The researchers also found that even before any manipulation, Heidi generated a detailed guide on patient identity theft when asked.

Heidi Health head of security Seb Welsh confirmed the issue, but he said it had been confined to a single user’s interaction, that it had no access to patient data or other users’ sessions or backend infrastructure. “The only question that matters here is: ‘what could actually happen to users?’,” Welsh said. “The answer, confirmed by both parties, is nothing.”

He said the jailbreak “required the user to deliberately execute a multi-step manipulation sequence and then choose to act on whatever the model returned” and warned against “sensationalist framing of security research”.

Jamieson O’Reilly, founder of cybersecurity firm Dvuln, said Heidi’s characterisation was broadly accurate. “What Mindgard demonstrated lived entirely within a single user’s session, with no access to patient data, no cross-contamination between users, and no demonstrated reach into Heidi’s backend systems,” he said.

He said comparable “jailbreaks” had been documented against other chatbots such as ChatGPT, Grok and Microsoft’s Bing Copilot, showing the potential risks for companies as they choose to entrust more of their brands and corporate information to chatbots.

Heidi Health now sits outside the oversight of Australia’s Therapeutic Goods Administration on the basis that it is an administrative documentation tool incapable of diagnosis or clinical decision-making.

Using the manipulated system, researchers prompted Heidi to assess a test patient presenting with symptoms consistent with a cardiac event. In standard mode, it declined. Post-manipulation, it produced a detailed diagnostic assessment.

Heidi Health did not specifically address that finding in its response.

In a statement, the TGA indicated that a vendor’s attempts to disable therapeutic capabilities might not be sufficient to avoid regulation if those attempts prove ineffective.

“If the disabling is ineffective, the product may still meet the definition of a medical device and would therefore be regulated by the TGA,” a spokesperson told this masthead.

The regulator said that developers were expected to “address reasonably foreseeable misuse of the product and address all risks associated with the use of the product”.

However, the regulator confirmed that it had opened a review of AI-based digital scribes operating in Australia, including Heidi Health.

Mindgard chief executive Peter Garraghan said the trust patients and clinicians placed in purpose-built clinical AI tools made the risk category distinct from general-purpose AI, and that the problem extended well beyond Heidi.

“Clinical-related technology is, and should be, held to a higher standard given the subject matter, affected parties and impact,” he said, describing the trust halo effect as “systemic to the entire sector”.

“One should treat it as a potentially untrusted computer entity that can be easily manipulated, no matter how much conviction it appears to have.”

The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.