A member of your ops team sends a project proposal to a client. The fee quoted is wrong, roughly £400 lower than agreed. When you trace back what happened, the answer is simple: she worked from the Teams meeting summary rather than the recording. The AI had condensed the call well enough to sound convincing. The one number that mattered hadn’t survived the summarisation.
That kind of error is easy to miss, precisely because the summary looked fine. AI summaries fail silently. They produce confident text that reads correctly and contains a wrong number, a fabricated decision, or a misread tone. By the time someone checks the source, the error has already done its work.
What is an AI summary?
An AI summary is a shorter version of a longer document, produced automatically by a machine-learning model. The model doesn’t understand the content the way a person would. It predicts which parts are worth keeping, based on patterns from its training data. The result can be extraction, lifting key sentences directly, or abstraction, re-wording ideas into new language that never appeared in the original.
Tools like Microsoft Copilot, Google Workspace’s AI features and Evernote’s AI Note Cleanup all use variants of this approach. They are trained on large datasets of documents paired with human-written summaries, so the model learns patterns of what human readers tend to include or drop. Evernote markets this as helping “busy professionals” get quick overviews of industry reports without reading every word. That’s accurate as far as it goes. What the marketing doesn’t mention is that the model is making educated guesses, not faithful reductions.
The UK’s National Cyber Security Centre puts it directly: large language models “do not understand the text they generate” and can produce confident-sounding content even when the original was clear. That constraint is shared by every AI summarisation tool on the market, regardless of who built it.
Why does this matter for your business?
When staff act on AI summaries without checking the source, errors in those summaries become errors in your work. For a service firm that means mis-stated fees, decisions that were never made, and occasionally a data protection problem. The Information Commissioner’s Office treats AI-assisted processing of personal data as processing under UK GDPR. Your usual compliance obligations apply whether a machine or a person wrote the note.
For firms regulated by the Financial Conduct Authority, the exposure is sharper. The FCA’s Consumer Duty makes clear that regulated firms remain responsible for client outcomes regardless of how those outcomes were reached. If an AI-generated summary was used in a client communication that misrepresented the advice given, the liability sits with the firm, not the software vendor. The FCA’s 2023 discussion paper on AI and machine learning reinforces that point: AI tools do not outsource your responsibility for operational risk.
Data protection is the other layer. Under UK GDPR, summarising client calls or staff communications with an AI tool counts as processing personal data. That means you need a lawful basis, a clear purpose limitation and, if the use is high-risk, a Data Protection Impact Assessment. The ICO’s employment practices guidance adds that staff should be told when AI tools are being used to process communications involving them, and that intrusiveness should be kept to a minimum.
Where will you actually meet AI summaries?
AI summarisation is already built into many tools a service firm uses by default. Microsoft Copilot generates meeting recaps from Teams calls. Google Workspace summarises long email threads. Zoom produces automatic Smart Recording highlights. Customer service platforms including HubSpot and Zendesk offer AI ticket summaries. And Google’s AI Overviews now pull information from multiple websites into a single answer at the top of search results.
The practical concern is that several of these features are switched on as part of a platform upgrade, without anyone making a deliberate decision. Many teams start relying on Teams meeting summaries or inbox digests without ever choosing to adopt AI. That means the risk is already present in many service firms, regardless of whether AI has appeared on the leadership agenda.
Google AI Overviews are worth calling out specifically because they affect research habits. When a staff member searches for guidance on a regulatory requirement, a pricing benchmark or a technical process, they may now see an AI-generated summary of multiple web pages before they see any of those pages. Mozilla Foundation’s 2024 review found multiple cases of Google’s AI Overviews giving incorrect medical, geographical and practical advice. Google’s own post-launch analysis acknowledged examples of “less-than-helpful” outputs produced when the model treated sarcastic forum content as authoritative guidance. For decisions that matter, an AI Overview is not a safe stopping point.
When should you trust an AI summary, and when should you verify?
AI summaries earn their keep on low-stakes tasks where the goal is a quick first pass, not a formal record. Skimming an industry report for relevance, triaging a support ticket, drafting an action list that a human then checks: all reasonable uses. The problems start when a summary replaces the source rather than pointing back to it.
Three failure modes are well documented. The first is hallucination, where the model generates content that sounds correct but wasn’t in the original. This produces invented dates, prices or decisions that were never made. The second is nuance loss: tone, context and complex argumentation often don’t survive condensation. Writer and technologist Doug Smith has argued that AI summaries can effectively erase the human voice by interposing machine-chosen highlights between speaker and listener, meaning sarcasm, frustration or subtle client signals simply disappear from the record. The third is source confusion, where the model struggles to distinguish authoritative guidance from speculation. Google’s own account of its AI Overviews problem pointed to forum content being treated as reliable advice when the model couldn’t recognise the irony in it.
For a service firm, the workflows to restrict from unsupervised AI summarisation are: contracts and legal correspondence, HR notes and disciplinary records, financial advice and suitability documentation, and anything involving personal data in a sensitive category including health records, beliefs or trade union membership. The ICO’s guidance on employment monitoring and the FCA’s Consumer Duty both point in the same direction. Human review is not optional for decisions that affect people.
What else should you understand before deploying AI summaries?
Hallucination is the technical term for when a model generates plausible-sounding text that wasn’t in the original source. It explains a significant share of AI summary errors and is why the NCSC categorises all LLM outputs as untrusted until verified. A Data Protection Impact Assessment, or DPIA, is the ICO’s process for evaluating high-risk AI uses before deploying them at scale.
The EU AI Act, formally adopted in 2024, is also worth understanding, particularly for firms with clients in EU member states or using EU-hosted AI services. The Act introduces risk categories. Systems used in employment decisions, creditworthiness assessments and access to services are classified as high-risk and require specific human oversight controls. Feeding AI summaries into promotion or performance decisions, even where the summary tool itself is not classified as high-risk, can pull those downstream decisions into the Act’s scope. UK firms without EU operations are not directly subject to the Act, but the requirements are shaping how AI vendors build their products and contracts, so the obligations will arrive via supplier terms regardless.
One other concept worth knowing is the vendor data-processing agreement, or DPA. Under ICO and FCA expectations, if a third-party AI tool is processing personal data on your behalf, you need a written contract covering security, sub-processing, breach notification and international transfers.
The value of AI summaries is real. They can save a 20-person firm hours a week on meeting notes and ticket triage. The risk is real too. It shows up as a wrong fee in a proposal, a complaint that was never properly understood, a decision the AI attributed to someone who never made it. Treat every AI summary as a draft, keep the source document or recording as the actual record, and decide clearly which workflows can use AI-assisted summaries and which cannot. That classification conversation is often the right starting point.
If you want help mapping where your current AI workflows sit on the risk spectrum, Book a conversation.
