At a glance
- In this post we will examine the idea of AI hallucinations and ask common AI tools to give their views. The conclusion below is entirely my own view, but everything else was written (or even hallucinated) by AI.
- I asked ChatGPT-o3’s to give its view on Hallucination, which it did and it even referred to ChatGPT-4.
- I asked Claude and Microsoft Co-pilot to give a view on Hallucination and how they agree and disagree with ChatGPT-o3’s answer.
- My point of view is shared in the Conclusion at the end of the post.
Introduction
There is a lot of time spent talking about AI hallucinations and for many different reasons. I’ve had some odd conversations with ChatGPT when using it for code generation, which I’ll share in another post. But, those odd conversations didn’t result in any harm. We were coding some WordPress customisations, not determining the correct drug prescription for a patient.
Here are ten concrete examples in which an AI model’s hallucinations—confident but unfounded outputs—could put people at real‑world risk:
- Misdiagnosing medical conditions.
- A symptom‑checker chatbot confabulates a rare disease based on weak textual correlations, leading a frightened user to delay seeking qualified care for an actual heart attack.
- Prescribing unsafe drug regimens.
- An AI clinical‑decision support tool hallucinates an interaction‑free combination of medications; the suggested cocktail produces a fatal serotonin syndrome when followed.
- Inventing non‑existent legal precedents.
- A lawyer relies on an LLM to draft a motion; the model fabricates citations and quotes, causing sanctions for professional misconduct and jeopardising the client’s case.
- Falsifying aircraft maintenance steps.
- A predictive maintenance assistant generates an imagined service bulletin for a jet engine. A mechanic, trusting the document, skips a required inspection, raising the risk of in‑flight failure.
- Generating erroneous industrial control commands.
- In a chemical plant, an autonomous agent hallucinates a valve‑opening sequence it never learned from validated manuals, leading to a runaway exothermic reaction.
- Producing fake investment data.
- A portfolio‑management bot invents quarterly earnings figures for a startup; an investor commits funds based on the fabricated metrics and suffers heavy losses.
- Creating phantom cybersecurity alerts.
- A security‑operations assistant hallucinates indicators of compromise, triggering an unnecessary shutdown of hospital IT systems during surgery schedules.
- Misguiding emergency response routing.
- A disaster‑relief planning tool fabricates bridge capacities and road statuses, sending rescue convoys onto impassable routes and delaying aid delivery.
- Confusing autonomous‑vehicle perception.
- A multimodal model hallucinates a clear road in foggy conditions and overrides sensor fusion vetoes, causing the vehicle to accelerate into an unseen obstacle.
- Fabricating scientific protocols.
- A laboratory automation agent generates a reagents list that includes an impossible concentration of a toxic compound; the resulting experiment releases hazardous fumes and injures staff.
In each case, the danger stems from the model’s persuasive linguistic fluency masking its lack of real-world knowledge or calibrated uncertainty. When humans assume that eloquence equals accuracy, hallucinations are not just embarrassing, they can lead to tangible harm.
ChatGPT-o3 “A Thesis on AI Hallucination”
Prompt: ‘Can you give me a 400 word thesis on why AI is accused of hallucinating and why it does this or is it a myth?’
Large‑scale language models (LLMs) such as GPT‑4 are frequently accused of “hallucinating”—producing fluent statements that are factually untrue or unsupported. The charge arises because users perceive the system as a knowledgeable agent, yet it sometimes confabulates details with confident prose. Understanding why this happens requires viewing the model through two complementary lenses: its statistical architecture and the socio‑cognitive framing humans bring to it.
Technically, an LLM is an autoregressive probability machine trained to predict the next token given billions of text fragments. It has no memory of external reality, only a compressed map of patterns in its corpus. When a prompt leads it into a low‑density region of that map—an edge case, outdated topic, or novel combination—the model must still output the most probable continuation. Lacking explicit uncertainty calibration or a mechanism to say “I don’t know,” it will interpolate, stitching together semantically plausible but unverified snippets. The wider and noisier the training data, the more latent correlations exist for the model to mistakenly treat as ground truth. Fine‑tuning, retrieval augmentation, and tool‑use modules can reduce the incidence, but the core incentive—maximize likelihood over synthetic sentences—remains.
Socially, hallucination is amplified by anthropomorphism. People credit chatbots with intentional reasoning, so errors feel like deception rather than statistical drift. Human conversation normally includes repair strategies (“let me check”) that signal epistemic humility; LLMs simulate the style of expertise without possessing the metacognition that would throttle speculation. Evaluation compounds the problem: casual users rarely fact‑check triumphs, but every conspicuous failure spreads virally, reinforcing the narrative that the system “lies.”
Is hallucination a myth? Only in the sense that the term smuggles a psychiatric metaphor into engineering discourse. The model is not perceiving false stimuli; it is producing text under objective mathematical rules that make certain mistakes inevitable. Calling those mistakes hallucinations usefully warns practitioners that outputs can be unreliable, yet it risks over‑psychologizing the phenomenon and obscuring the responsibility to design safer interfaces.
In sum, LLM hallucination is neither deliberate mendacity nor mere myth. It is an emergent artifact of token‑level optimization interacting with human expectations. Mitigation will come not from exhorting the model to “tell the truth,” but from grounding its generations in verifiable data, exposing calibrated uncertainty, and teaching users to treat eloquence as suggestion rather than authority. Ultimately, the accusation indicts our design goals as much as the algorithm: we built an oracle, but delivered a parrot instead.
Written by: ChatGPT o3 on the 25th July 2025.
Do the AIs agree with themselves on this?
Claude’s view
Prompt: ‘Can you give a view on the following article? It is discussing AI Hallucination. What do you agree with and what do you disagree with?’
This is a thoughtful and nuanced analysis of AI hallucination that I find largely accurate. The author demonstrates a solid understanding of both the technical mechanisms and social dynamics at play.
I strongly agree with several key points:
The technical explanation is spot-on. LLMs are indeed probability machines that must generate something even when venturing into low-confidence territory. The author correctly identifies that lacking explicit uncertainty mechanisms, models will “interpolate” plausible-sounding content rather than admitting ignorance. This captures the core technical challenge well.
The social dimension is equally important and well-articulated. The anthropomorphism point is crucial—when people interpret statistical outputs through the lens of human intentionality, routine prediction errors feel like deliberate deception. The observation about humans having “repair strategies” that LLMs lack is particularly insightful.
The distinction between “deliberate mendacity” and inevitable mathematical artifacts is valuable. Framing these as emergent properties of the optimization process rather than character flaws helps focus attention on the right solutions.
However, I’d offer some nuances:
- The piece slightly understates recent progress in uncertainty calibration. While perfect solutions don’t exist, techniques like constitutional AI, RLHF, and improved refusal training have made meaningful improvements in getting models to express uncertainty appropriately.
- The “oracle vs. parrot” conclusion, while rhetorically effective, may be overly binary. Modern LLMs demonstrate genuine reasoning capabilities alongside their pattern-matching tendencies. The reality is more complex than pure mimicry.
- The article could better acknowledge that some level of “hallucination” might be unavoidable and even useful—creative tasks often benefit from models that can extrapolate beyond strict factual boundaries.
Overall, this is an excellent synthesis that avoids both techno-optimism and techno-pessimism while highlighting the genuine challenges we face in building more reliable AI systems.
Written by: Claude-Sonnet-4 on the 25th July 2025.
Microsoft Co-pilot’s view
Prompt: ‘Can you give a view on the following article? It is discussing AI Hallucination. What do you agree with and what do you disagree with?’
Critique of “AI Hallucination” Article
Points of agreement
- The article rightly frames hallucination through both statistical architecture and social expectations. Understanding LLMs purely as probability machines explains why they can’t “know” truth—they only estimate token likelihoods based on training density.
- Highlighting edge cases is crucial. When prompts land in low-density regions, models default to plausible interpolations without real-world grounding, so factual drift is baked into next-token maximization.
- Anthropomorphism amplifies perceived intent. Users treat confident prose as deliberate reasoning, not statistical drift, so errors feel like deception rather than algorithmic artifact.
- Calling these mistakes “hallucinations” is useful for flagging risk but risks over-psychologizing. It obscures design responsibility: we must engineer interfaces that express uncertainty and integrate verifiable data sources.
Nuances and points of partial disagreement
- The article downplays progress in uncertainty calibration. Recent research into confidence scoring, token-level entropy, and selective generation can in fact empower models to “say I don’t know.” It’s not purely a missing feature; it’s an active research frontier with promising prototypes.
- While retrieval augmentation reduces hallucinations, the piece understates how well integrated RAG systems perform in commercial deployments. In practice, tightly coupled knowledge bases and real-time search layers can cut factual errors dramatically when engineered end-to-end.
- Calling LLMs “oracles turned parrots” is rhetorically strong but overlooks emerging hybrid designs—knowledge graphs, long-term memory modules, and grounding layers—that blur the line between parroting and genuine referencing.
- The social framing focuses on user expectations but omits the role of developer tooling. Better annotation interfaces, real-time fact-check pipelines, and model-in-the-loop QA can shift responsibility back into design processes rather than solely user education.
Recommendations for safer interfaces
- Integrate provenance and source citations directly into model outputs, so users see exactly which documents or data snippets underpin each statement.
- Expose token-level confidence (e.g., shading text by probability), visually signaling uncertainty instead of hiding it behind fluent prose.
- Incorporate adversarial hallucination detectors that flag outputs exceeding a divergence threshold from known fact repositories.
- Build interactive “I’m not sure” prompts, prompting the model to ask for clarification or to invoke retrieval when confidence falls below a set bar.
- Educate users through micro-tutorials embedded in interfaces, reinforcing that eloquence isn’t proof of truth.
Beyond the article
Aside from grounding and uncertainty, there’s exciting work on:
- Chain-of-thought prompting to surface reasoning steps, offering checkpoints for factual verification.
- Self-critique modules that attempt to refute their own outputs before sending them.
- Regulatory frameworks and standards for truthfulness benchmarks (e.g., HELM, TruthfulQA) that define industry best practices.
Exploring these avenues can transform LLMs from oracle-style parrots into responsible assistants that know—and admit—what they don’t know.
Written by: Microsoft Co-pilot “Think Deeper” on the 25th July 2025.
Conclusion
This conclusion was almost entirely written by a human.
Let’s take a look at this statement from GPT-o3: “People credit chatbots with intentional reasoning, so errors feel like deception rather than statistical drift.“. It feels human, it even says “Thinking…”, but it’s not, at least not in the way that humans think. When we imbue AI with human consciousness we may include emotional considerations which sometimes results in users becoming upset or angry with the AI’s responses. This positioning of AI as friend or foe exacerbates the impact of an AI mistake, when it is actually just trying to give an answer based on probability – it gives the “most probable answer” not an emotional one.
So let’s not believe the hype just yet and make sure we prompt with as much detail in as concise and logically correct way as possible. And research the model you’re using, not all LLMs are the same and they can’t all cover every area of human life to the same accuracy – I prefer Claude over Chat GPT for coding but like Sora for complex image work for example.
If your need for AI support has potential hazard attached to it, then use an Agentic approach. Make sure your agent follows a prescribed workflow and take care to implement guardrails to reduce the risk. AIs checking the output of other AIs is one example. Either ask the AI to double check its output against a set of predetermined risks or ask a different AI to do the checks – one that is trained to check, not create.
If you don’t have access to an AI Agent with proper workflow and guiderails, but still need AI support with your work, make sure you drive the AI better. Ask the AI to check its own answer against a list of potential bad outcomes and perhaps get a second opinion from another AI model.
If your chosen AI model is hallucinating, look for a different one that is more aligned with your needs and research ways to prompt out the hallucinations. You can also ask the model itself why its hallucinating and what would help it think in a better direction. And finally, your ask of the model may just be too hard and you will need to expect and cater for some hallucination.
