In June 2023, a pair of New York lawyers submitted a court filing that cited six legal cases in support of their client’s position. The cases did not exist. ChatGPT had invented them, complete with plausible-sounding names and docket numbers. The lawyers submitted the filing believing the citations were real. The court sanctioned both lawyers and fined each $5,000.
That incident involved a law firm, but the mechanism behind it plays out in owner-managed businesses every day in less dramatic ways. An AI tool produces a confident-sounding answer about an HMRC position, a regulatory threshold, or a clause in a supplier contract. The answer sounds authoritative. The person reading it has no particular reason to doubt it. They act on it.
This post explains why AI produces that confident tone regardless of whether the underlying answer is accurate, where the risk sits for a UK service firm, and what to do differently.
What is actually happening when AI gives a confident but wrong answer?
Large language models like ChatGPT generate text by predicting the next word in a sequence, based on statistical patterns learned from vast amounts of text. They are not retrieving facts from a database or checking whether a statement is true before they produce it. NIST describes this as “approximating patterns in training data”, which can produce fluent and coherent language that is factually incorrect.
The result is what NIST and the wider research community call “confabulation” or “hallucination”: a model producing statements with complete confidence that have no basis in fact. OpenAI’s technical documentation for GPT-4 acknowledges this risk directly.
What makes the problem persistent is that training and evaluation of these models often rewards giving some answer over saying “I don’t know.” A model tuned to produce helpful, fluent responses tends towards confident guesses rather than explicit uncertainty. Villanova University researchers summarise it this way: the system is “optimised to produce a good next sentence, not to prove the sentence is true.”
A 2025 Carnegie Mellon University study put numbers on this. Google Gemini correctly identified fewer than 1 out of 20 hand-drawn images in a recognition task, yet when asked to estimate how many it had answered correctly, it predicted more than 14. The model’s self-assessment bore almost no relationship to its actual performance.
Why does this matter for an owner-managed business?
The practical risk is that AI’s authoritative tone reduces the likelihood that you or your team will check the answer. When a tool states something crisply and confidently, the natural human response is to trust it, especially under time pressure. Research confirms that well-formed, confident language is treated as credible even when the content is wrong.
For UK businesses, the risk is compounded by regulatory accountability. The ICO’s 2024 guidance on generative AI is direct: organisations must ensure that AI-generated content involving personal data is accurate, and they must have processes to correct mistakes. That duty belongs to the business, not to the tool provider.
The FCA makes the same point for regulated firms. Full responsibility for statements and decisions produced by AI, including customer-facing chatbots, rests with the firm. The NCSC recommends treating AI outputs as untrusted input, subject to the same scrutiny as user-generated content.
The Mata v. Avianca case illustrates what happens when that scrutiny breaks down. The mechanism the lawyers relied on was exactly the one your team encounters when asking an AI for a regulatory interpretation or a quick fact about a competitor. The confident tone felt like a signal of reliability. It was not.
Where will you actually run into this in a service firm?
The “confident but wrong” problem appears across three broad areas that owner-managed service firms typically use AI for: factual retrieval (asking AI what a rule, rate, or regulation says), claims about specific people or companies (competitors, clients, named individuals), and high-stakes written content (contracts, HR communications, regulatory submissions). These are the areas where fabricated specifics cause real damage.
For factual retrieval, the highest-risk queries involve legal and regulatory detail. An AI asked what UK employment law says about a specific situation will produce a confident-sounding answer that may be accurate, outdated, or entirely fabricated. NIST notes that hallucinations are more likely when models are asked about niche, ambiguous, or time-sensitive topics without being given source material to draw from.
Claims about individuals carry a different kind of risk. In 2023, a US radio host sued OpenAI after ChatGPT falsely told a journalist that the host had embezzled funds, citing a made-up legal complaint. The ICO’s guidance explicitly warns that AI outputs involving personal data can be defamatory or unfair, exposing businesses to data protection and reputational risk.
Written content for external use, including client proposals, statements of work, and HR letters, is often where fabricated specifics travel furthest before anyone checks them. If a team member copies AI-generated text into a client document without review, the accuracy problem leaves with it.
When does the confident tone become a red flag, and when can you work with it?
The risk from AI overconfidence varies significantly by use case. For tasks like drafting a proposal, generating marketing ideas, or reframing a paragraph, there is no single correct answer to retrieve, so the confident tone matters relatively little. The problem sharpens when the AI is expected to verify something factual, particularly when you cannot easily check it yourself.
Villanova’s practical boundary is useful here. AI is lower-risk for drafting and editing content you will review, summarising documents you have provided, and generating options where accuracy is not the point. The risk rises sharply for anything that “needs to be right, not just plausible”: a legal clause, an HMRC position, or a specific claim about a named individual.
One practical habit reduces the risk significantly: if accuracy matters, provide the source yourself. Paste or upload the relevant document, ask the AI to summarise from it and quote the relevant sections, then spot-check those quotes. This keeps the model working from material you can verify, rather than drawing on its training data where fabricated specifics are hard to detect.
Ask the AI to list its assumptions and flag anything that needs external verification. A 2025 Carnegie Mellon University study found that models are not reliably self-aware about their own accuracy, so treat self-reported confidence flags as hints rather than guarantees. A named human review step before anything AI-produced reaches a client or regulator is the practical backstop.
What else connects to this?
Understanding AI overconfidence sits at the intersection of two broader questions: how to prompt AI effectively, and how UK regulators expect you to govern AI use. Getting the prompting right reduces the frequency of confident errors. Getting the governance right means you are protected if an error gets through. The two work together, and neither is sufficient on its own.
On prompting: the “no source, no claim” habit is a practical form of what technical teams call retrieval-augmented generation, constraining the model’s responses to a specified set of supplied documents. You do not need to understand the underlying technology. You need the workflow: paste source, ask for summary with quotes, spot-check the quotes.
On governance: the ICO expects businesses processing personal data with AI to document their accuracy risks and controls, typically in a Data Protection Impact Assessment. For a five to fifty-person firm, this need not be complicated. A one-page note explaining what AI is used for, what the human review step is, and who signs off before AI-generated content goes to clients or regulators covers the compliance groundwork for many small service firms.
On accountability: the FCA, ICO, and NCSC all arrive at the same point from different angles. The business owns the output. Building that assumption into your processes from the start is cheaper than correcting the reputational damage if a confident wrong answer reaches a client and you have no record showing it was checked.
If you would like help working out where AI fits in your firm and what oversight to build around it, Book a conversation.
