England’s exam regulator, Ofqual, ran an algorithm in summer 2020 to standardise A-level grades when exams were cancelled. The system downgraded around 40% of predicted grades, disproportionately affecting students from state schools. Within days, the government scrapped it and reverted to teacher-assessed grades. The algorithm did exactly what it was designed to do. The problem was that no human check caught the outcome before it landed on thousands of young people.
The same pattern plays out on a smaller scale across UK businesses every week. A CRM scoring model deprioritises a promising lead because it hasn’t seen enough examples from that sector. An HR screening tool filters out a strong candidate because their CV layout doesn’t match historical patterns. A pricing tool quietly charges higher rates to customers in certain postcodes. These decisions happen in the background, and often nobody checks.
Any founder using AI tools needs to know where those decisions are and who is accountable for them. The sooner you’ve mapped that out, the better.
What does splitting decisions between people and AI actually mean?
Splitting decisions between people and AI means assigning each a role that fits its strengths. AI handles pattern recognition and scoring. Humans retain accountability and make the final call. UK GDPR Article 22 prohibits decisions based solely on automated processing that carry legal or similarly significant effects, unless specific safeguards apply. That line covers credit, employment, insurance, and access to services.
For an SME, this usually means treating AI as an adviser that drafts, scores, or surfaces options, while named people are responsible for approving or reversing the outcome. Monzo does this with fraud flags: an automated system raises the alert, but a human analyst reviews account freezes before they stand. Lloyds Banking Group states publicly that experienced colleagues make the final decisions in mortgage approvals and complex lending, with AI used to surface risk indicators. The principle scales down to any business size.
Where this gets complicated is in less obvious tools. Many project management and CRM platforms now include AI features that score and rank automatically, often switched on by default. Check which decisions in your business are being shaped by AI-generated scores, and whether those scoring systems have ever been reviewed or calibrated. The ICO’s accountability framework asks organisations to document the purposes and logic of automated decision-making. A brief internal audit is often enough to find out where you stand.
Why does this matter for your business right now?
The UK regulatory landscape is clearer than many founders realise. The ICO updated its AI and data protection risk toolkit in 2024 to stress that high-risk AI systems must include meaningful human involvement, with a genuine ability to change or reverse the outcome. The AI Regulation White Paper from 2023 instructs all UK regulators, including the ICO, FCA, and CMA, to apply cross-cutting principles that include appropriate human oversight and contestability of AI-assisted decisions.
The CMA warns that firms remain responsible under competition and consumer law for AI-generated outcomes, regardless of whether the decision was produced by an algorithm. The FCA is equally clear that senior managers cannot delegate accountability to a model. The NCSC advises businesses using generative AI to keep a human in the loop for any use case where the tool could fabricate content, leak data, or be manipulated by prompt injection. If you use any AI-enabled tool that touches customers, credit, pricing, or people, you are already in scope for these frameworks.
Where will you actually encounter this in practice?
Many founders are already running AI-influenced processes without having labelled them as such. A CRM that scores leads makes an automated recommendation about where your sales team spends time. An HR tool that shortlists CVs makes a recommendation about who gets an interview. A generative AI tool drafting customer responses makes a recommendation about what your business says.
None of these are inherently problematic. The CIPD found in 2023 that 55% of UK HR professionals were using or planning to use AI tools for recruitment or people analytics within 12 months, and only 28% had formal governance policies specifying where human review was mandatory. The challenge is knowing where the human checkpoint sits. If your sales team always follows the CRM ranking without question, the tool is effectively making the decision. If your hiring manager reads every shortlisted CV independently, the tool is advising. The difference lies in how the workflow is designed, not in which software you’re running.
When can you automate fully, and when must a person decide?
A simple grid helps here. Low-impact tasks with clear rules and strong data, such as routing customer enquiries, matching invoices, or flagging obvious errors, can reasonably be automated with monitoring in place. The ICO and NCSC both implicitly accept full automation where risks to individuals are low and humans can still intervene if needed.
For medium-impact decisions, such as marketing targeting or sales prioritisation, AI proposes and a human reviews. For high-impact decisions, particularly anything affecting a person’s money, job, or access to services, humans lead and AI advises. The FCA is explicit: in regulated financial services, boards remain responsible for AI-enabled decisions and must ensure appropriate oversight and a clear allocation of responsibility. For businesses outside financial services, the principle holds. A named person needs to own the outcome.
Starting with low-stakes tasks is the sensible approach. Research from Harvard and the NBER on GPT-4 at work shows meaningful performance gains when AI is used as a thinking partner on knowledge tasks. Build confidence in the tools by learning how they behave in low-consequence settings before applying them where the stakes are higher.
What does real human oversight actually look like?
The gap between nominal sign-off and genuine human oversight is where many AI governance failures happen. A 2023 BCG survey found that 89% of CEOs believed AI would create competitive advantage, but only 29% said their organisations had adequate governance in place. Regulators are looking for evidence of effective oversight. The ICO is explicit: meaningful human involvement means genuine influence over decisions, not the appearance of review.
Real oversight means three things. First, the person reviewing must understand why the AI reached its conclusion. The ICO and the Alan Turing Institute publish joint guidance on explaining AI decisions, and the EU AI Act requires that human oversight functions can interpret outputs and decide not to follow them. Second, the reviewer must have genuine authority to disagree, not just a formal sign-off to complete. Third, the decision, including the reason for agreeing or overriding the AI, should be documented.
For an SME, documentation can be straightforward: a mandatory field in your CRM or HR system asking whether AI was used, whether the output was checked against an alternative, and whether the human agreed or overrode the recommendation. A one-line record demonstrates meaningful involvement. A 2023 NBER working paper on GPT-4 at work found that workers with AI access improved performance on complex tasks by 40% on average, but also showed automation bias, over-trusting the model even when it was wrong. Train your team on where the tools fail, and build the habit of checking rather than assuming.
