A few months ago, a consultant I know asked ChatGPT to summarise a supplier contract for a client. The output was fluent, structured, and confident. He sent it on with a brief covering note. Three weeks later, the client called. The AI had missed a break clause that cost them significantly to exit. The contract was real. The summary was plausible. The confidence was the problem.
Every founder using AI eventually faces the same underlying question, even if they have not framed it this way yet. You can use AI as a drafting partner, where a human reviews every output before it affects anything that matters. Or you can let AI influence actual decisions, where the output triggers an action without that review step. The gap between those two positions is where most of the avoidable risk lives.
What choice are you actually facing?
The UK’s Alan Turing Institute describes large language models as “statistical pattern matching at scale” rather than reasoning. That matters because fluent output is not the same as accurate output. ChatGPT, Claude, Gemini, and Copilot are next-token predictors: they generate the most probable continuation of whatever text they have been given. They do not check facts, hold beliefs, or understand consequences. The gap between sounding right and being right is where founders run into trouble.
When you ask AI to draft an email, write a proposal, or summarise a document, the cost of an error is the time it takes to catch and correct it. When you ask AI to score a lead, pre-screen a job applicant, flag a fraud risk, or make a pricing call, the cost of an error can reach an ICO investigation, an FCA remediation programme, or a professional negligence claim. The two uses look similar on the surface. The risk profiles are very different.
When does treating AI as a drafting assistant make sense?
For many UK SME owners, the drafting assistant role is where AI delivers the most value with the least exposure. The ICO’s guidance on AI and data protection confirms that using AI to assist staff without automated individual decision-making generally sits within standard UK GDPR controls, provided you have a lawful basis for any personal data involved and a named person signing off the output.
The use cases that fit this role well are text-heavy and repetitive: drafting client emails and proposals, summarising meeting notes or contracts, writing marketing copy variations, and generating code snippets for a developer to review. Founded.ai’s 2026 SME guide reports that early adopters in this mode see productivity gains of up to 133% in certain functions. The condition that makes those gains real without the liability is that a competent person reviews every output before it reaches a client, a customer, or a consequential internal decision.
The distinction is about matching the tool’s role to the reversibility of errors. A mis-drafted proposal gets caught before it goes out. A mis-scored loan application or a mis-screened job candidate may not get caught at all, or may only surface when the complaint arrives.
When can AI influence real decisions, and what conditions must be in place?
There are SMEs for which AI-driven decision support is appropriate, but the conditions that justify it are specific. The FCA’s discussion paper DP22/4 and its ongoing Consumer Duty guidance make clear that where AI is used in credit, affordability, claims, or advice, firms must demonstrate explainability, fair treatment, and genuine human oversight, not a rubber-stamp review that follows whatever the model recommends.
The conditions that make this defensible are: strong, clean historical data you can test models against rigorously; genuine in-house or contracted expertise in data science and compliance rather than just a vendor relationship; efficiency gains large enough to justify the governance investment; and a real human-in-the-loop for edge cases and complaints, with a documented escalation path. If you cannot point to a named person who reviews AI outputs, can explain to a regulator how the AI influences outcomes, and can demonstrate how someone affected can challenge a decision, you are not in a position to meet those conditions. Keep AI in the drafting role until you are.
What does it cost when you get this wrong?
The consequences of treating a confident AI output as a considered decision are not theoretical. In 2023, a New York law firm was sanctioned after submitting six fictitious case citations that ChatGPT had generated. Anthropic’s own documentation describes hallucinations as “confidently stated incorrect or unverifiable information,” and notes that even advanced models still fabricate citations under pressure to be helpful. The fluency is genuine. Accuracy still requires your verification.
For a UK services firm, the financial exposure is concrete. The ICO can fine up to £17.5 million, or 4% of global turnover, for serious GDPR breaches involving automated decision-making. The FCA, for regulated firms, can require remediation programmes, skilled-person reviews under section 166, and restrictions on activities. Professional indemnity insurers including Hiscox report that defence costs in negligence claims regularly exceed £50,000 before settlement, and these figures assume you catch the problem quickly. Decisions made by an AI system running unchecked for several weeks compound all of these.
There is also the reputational dimension. UK SMEs depend heavily on referrals and local reputation. A single AI-generated mis-advice reaching a client, or a discriminatory screening decision surfacing publicly, can affect the pipeline for years.
What should you ask before you commit to either path?
The ICO and the FCA both hold firms, not vendors, responsible for outcomes when AI is involved. That accountability principle is the right frame for any founder deciding how to deploy these tools. Before you let AI move from drafting partner to decision influencer, several questions are worth putting to yourself honestly rather than optimistically.
Does the AI output materially affect someone’s money, job, or access to services? If yes, treat it as high-risk from the start and design under Article 22 and Consumer Duty expectations. Can you explain in plain English how the AI influences the outcome, and how someone affected can challenge it? Who, by name, is accountable for the decision, and how often do they actually review the AI’s outputs rather than confirm them?
Two questions founders commonly skip: what does your vendor actually document about training data, limitations, and failure modes; and what is the realistic worst-case exposure if the AI produces bad outputs undetected for a month? If you cannot answer those honestly, the drafting role is the right starting point. The governance can grow as the use case matures. Starting with governance gaps and hoping to catch problems later is where the costly calls tend to get made.
A Stanford and UCLA study found that people given AI-generated summaries were 19 percentage points more likely to believe false statements they had read. Fluency raises perceived credibility, and AI tools are very fluent. For decisions that matter to your clients, your staff, or your regulators, that gap between sounding right and being right is the one worth watching. Use AI well and you get the productivity. Give it the wheel before the governance is in place and you carry the consequences.
