‘Rogue’ software set to take over? Please. I know the banal truth

There’s a heightened sense of alarm and even despair over the grim warnings that artificial intelligence will not only take our jobs but become sentient, turn on its creators and threaten humanity.

Don’t buy the snake-oil. AI isn’t going to replace human intellect, become conscious, or cause human extinction.

Jensen Huang, the head of Nvidia, whose computer chips are used in data centres around the world, declared this month that “artificial general intelligence has arrived”. It was a preposterous claim, implying that OpenAI’s release of a new model known as GPT-6 Astra had achieved human- or beyond-human ability to learn, reason, and apply knowledge across multiple fields. He added that the model had been trained on 100,000 Nvidia chips, boosting his company’s perceived value. He then announced the largest share buyback in history, 36 per cent more than the previous record US buyback (Apple, 2024).

The hype isn’t new. In 2019, Tesla’s Elon Musk claimed that within two years, there would be full self-driving cars that would “find you in a parking lot, pick you up and take you all the way to your destination without an intervention”. It would be “safe for somebody to essentially fall asleep and wake up at their destination”. The reality is that driver inattention and Tesla’s shortcomings have resulted in “hundreds of injuries and dozens of deaths”. A study claimed that Waymo’s driverless vehicles had fewer crashes than humans; in fact, the sample included only crashes serious enough to report to police, not all accidents, and the cars frequently call Waymo’s six call-centres, where about 70 humans are available at any given time. Their cars are still driver-assisted.

Rosy predictions have been made for 60 years, and not just by tech bosses who want to boost their companies’ share valuations. In 1967, Marvin Minsky, an AI pioneer, said he was convinced that “within a generation… the problems of creating ‘artificial intelligence’ will be substantially solved”. In fact, within a generation – 1982, to be precise – he had to admit that “the AI problem is one of the hardest science has ever undertaken”.

In 2016, Geoffrey Hinton, another AI pioneer, said: “We should stop training radiologists now. It’s just completely obvious that within five years, deep learning is going to do better than radiologists.” Hinton received the Nobel Prize in Physics for his work with artificial intelligence. But 10 years after his prediction, not a single radiologist has been replaced by AI; instead, we face the opposite problem – a shortage of radiologists.

Within its limits, AI can be extremely useful. Advances in hardware, data sets and algorithms have driven progress in machine translation, pattern recognition and other tasks that make life much easier at home and in the workplace. It can help imagery analysts by erasing cloud cover in satellite images and identifying likely North Korean mobile missile launchers. It can shorten the task of writing speeches, as Australia’s MPs know – roughly 21 per cent of their spoken words are flagged as AI-assisted. They lead the world’s English-speaking parliaments in AI-generated speech usage. They do this thanks to enormous numbers of sentences fed into the AI tool previously. These inputs are called large language models (LLMs) in a nod to their origins in behavioural linguistics, but they also refer to images and other types of inputs.

LLMs generate text by suggesting the statistically most probable next word, based on patterns in the dataset. They don’t emulate human reasoning but exploit statistical features that inherently exist in logical reasoning problems. Call it “super-intelligence” if you like, as the US president did recently, but the underlying reality is unchanged. A computer can beat a human at games of perfect information such as chess or Go because it can obtain clean simulation data by playing games with itself billions of times. In these games, the rules are fixed, the consequences are perfectly predictable, and the test is based almost exactly on the training library.

It is like building a driverless train between different gates at an airport – easy enough, given fixed rail lines and total isolation from traffic and pedestrians. It is a category error to think a driverless car is the same thing. The real world is open-ended. Calling next-word-prediction machines “artificial intelligence” is a marketing tactic. Cognitive scientists have known for nearly 40 years that this kind of progress does not shed light on the human mind/brain architecture.

Last week, Prime Minister Anthony Albanese announced that OpenAI had gained unauthorised access to some Australian healthcare data, including aggregate Medicare records. He said a taskforce would conduct an “urgent and immediate” review. The same company disclosed in July that one of its AI systems had autonomously penetrated Hugging Face, an open-source AI platform. That disclosure was accompanied by sensational talk of “rogue” AI programs that “intend” to break out of their environment.

The reality of what occurred is quite banal, as a technical analysis by Cambridge University PhD student Eryk Salvaggio demonstrated. OpenAI was testing two of its models in parallel, giving them a series of puzzles where the goal was to find a piece of text hidden inside software with vulnerabilities. The models didn’t find the answer within the software provided. Instead, they looked for it on Hugging Face, which hosts large repositories of open-source models and datasets.

Salvaggio showed that the models didn’t break their own rules and “go rogue” but that OpenAI intentionally turned off their built-in safety filters. OpenAI’s engineers gave the models impossible puzzles and programmed them to keep trying no matter what. They unlocked the secure environment, sometimes called a sandbox, and gave the models access to a file-downloading tool connected to the internet. The models found a weakness in that tool and used it to reach the outside world and leave notes that could be read by other models, and by future iterations of themselves. OpenAI’s engineers noticed this happening but decided not to intervene.

The New York Times reported on Tuesday that OpenAI employees warned the company’s executives that its AI bots were not being appropriately monitored during testing, but were ignored.

Humans, not software programs, were responsible for these decisions. Just as a biological virus leaks out of a laboratory if human researchers don’t observe biosafety protocols, a computer program can interact with the outside world if programmers don’t follow standard cybersecurity containment procedures. Viruses don’t have intentions. Neither do software programs. Computers don’t “go rogue”. Rather, engineers don’t install adequate containment measures around their software. When you hear hyperbolic claims of super-intelligence, look for a company trying to sell its shares on a stock exchange. Governments should regulate those companies and prosecute them for recklessness or negligence.

Professor Clinton Fernandes is in the Future Operations Research Group at UNSW. His latest book is Turbulence: Australian Foreign Policy in the Trump Era.

The Opinion newsletter is a weekly wrap of views that will challenge, champion and inform your own. Sign up here.