I’ve been hearing a lot about the rise of AI in the past few years and, apart from a few little tries of ChatGPT, haven’t thought much about it. But in the past few months, three colleagues have told me they’ve used AI as part of their daily tasks. In one instance, even though I’m no expert on AI or this colleague’s area of work, I guess I could see how an advanced chatbot might help.
In the other two instances, I was flummoxed. Both of the other colleagues are capable, bright people. Both seemed to be outsourcing basic reasoning and human creativity to AI. Am I misunderstanding how incredible this technology has become since I played with it (about a year ago), or has this AI craze gone to a ridiculous new extreme?
people have started using tools such as ChatGPT to replicate highly complex human processes. Credit: John Shakespeare
I think it’s easy to be confused about how far AI technology has come in the last little while. So I’m going to hedge on the answer to your ultimate question, and say it’s probably yes and no.
Yes, chatbots using large language model (LLM) AI technology have progressed so rapidly that if you haven’t tried working with one for 12 months, you may well be surprised by what they can do today.
To give a very personal example, I did something I’ve never done before after receiving your email. I gave two well-known LLMs the basic parameters for writing this column – and then asked them to respond to a typical (but not real) Work Therapy question.
I wish I could say what they came up with was robotic nonsense – two years ago it might have been. In October 2025, however, they both produced something genuinely interesting and potentially very helpful, replete with a few lovely turns of phrase.
It’s simply a fact that LLMs still produce wildly inaccurate, sometimes farcically silly, responses to questions or task requests.
One had a couple of sentences that seemed a touch off (I’d go so far as to say grammatically wrong as the AI tried to force a word I’d used in the prompt into the response), but the other was harder to fault.
But, no, LLM-based chatbots are not so advanced that they have become modern-day Delphic Oracles: sources of god-like wisdom and unimpeachable advice.
This may sound blatantly obvious to people like you – those who have remained vaguely impressed but fundamentally sceptical of this new technology – or those who understand the technology inside out.
But to many, including, as you’ve said, “bright and capable” professionals, it’s not obvious at all. And so people have started using tools such as ChatGPT to replicate highly complex human processes or, yes, even as a source of objective truth.
Suddenly, we hear people unironically telling us “ChatGPT told me …” or earnestly discussing what gender they assign to the chatbot, as if it’s basically a digital person.
There’s a whole philosophical and ethical argument to be made about why this is dangerous. I don’t have the space or the erudition to go down that path. So I’ll just concentrate on two practical problems with this kind of unrestrained faith, namely LLMs’ current propensity for sycophancy and hallucination.
It’s simply a fact that LLMs still produce wildly inaccurate, sometimes farcically silly, responses to questions or task requests. Experts refer to this as “hallucination”.
To go back to my experiment asking chatbots to write a Work Therapy-esque column, when I asked them to help me understand where they had got the data to inform their choices of style and tone, their answers were confused. Then, after more specific questioning, they began presenting blatantly incorrect information as fact.
AI sycophancy is a phenomenon distinct, but not entirely separate, from hallucination. It, too, can lead to incorrect assertions or dubious advice.
During a recent study on LLMs described in Nature, researchers “tested how 11 widely used large language models responded to more than 11,500 queries seeking advice, including many describing wrongdoing or harm”.
The article said that “AI Chatbots – including ChatGPT and Gemini – often cheer users on, give them overly flattering feedback and adjust responses to echo their views, sometimes at the expense of accuracy”.
In that same article, a data science PhD student is quoted as saying: “Sycophancy essentially means that the model trusts the user to say correct things”. What you’ve observed is that this mistaken trust is being reciprocated by some users. And I don’t think your anecdotal evidence is any kind of exception or anomaly.
People are starting to use LLMs not as handy, but limited tools. They’re going way beyond time-saving requests such as summarising a long, boring email or tidying up a pre-written speech.
(I have reservations about handing either of those tasks over to AI, but I also recognise that the horse has well and truly bolted on these practices and that I’m now part of a rapidly diminishing minority holding on to such a sentiment.)
They are, instead, treating this technology as if it has advanced to such a degree that there is nothing it cannot do – that there is, as you put it, no “reasoning” exercise or “creative” endeavour humans can’t “outsource” to it.
This may be true in the future – and is almost certain in the distant future – but it is absolutely not true now. You may slightly underestimate what AI is capable of today – or at least be indifferent to its utility. I think that’s preferable to massively overestimating what it can do.
Most Viewed in Business
Disclaimer : This story is auto aggregated by a computer programme and has not been created or edited by DOWNTHENEWS. Publisher: www.smh.com.au
