A recruitment team adds an AI screening tool to handle the first round of applications. Nobody sets up the bias controls. There is a page for them in the admin panel, but no one reads it. Three months in, the shortlists look statistically different from the year before. The firm did not intend to discriminate. Under UK law, intent is not the point.
That gap between intent and outcome is where the current case law on AI discrimination sits. Courts in the US have already decided that a company deploying a vendor’s AI system can face discrimination claims even when it did not build the model. UK law and UK regulator guidance point in the same direction. If you are using AI to make or shape decisions about people, you are already in scope.
What is AI discrimination liability?
AI discrimination liability is the legal risk a business carries when an AI system produces outcomes that disadvantage a protected group, regardless of intent. Under UK data protection law and the Equality Act 2010, what matters is the effect on individuals, not the purpose of the system. If a neutral-looking process produces a disproportionate impact on people with a protected characteristic, the business operating it may need to justify that result or face a claim.
The ICO is clear in its AI fairness guidance that unjust discrimination through AI processing can breach the UK GDPR fairness principle, and that this runs independently of any Equality Act exposure. The two legal routes are parallel. Compliance with one does not guarantee compliance with the other. A firm can face a data-protection challenge even if no employment tribunal claim is ever brought.
Why does the vendor relationship not protect you?
The three US cases that have moved furthest through courts each involved companies using third-party systems rather than models built in-house. Courts consistently held that deploying a system which drives discriminatory outcomes is enough to ground liability. For UK firms, the ICO confirms the same position: the business using AI is responsible for its effects on individuals, regardless of who built the technology.
In May 2025, a court certified a collective discrimination claim in Mobley v Workday, treating the hiring platform’s AI as an active participant in age discrimination against candidates over forty. A court allowed a disparate-impact claim to proceed in Huskey v State Farm in September 2023, where plaintiffs alleged that an algorithmic fraud screen imposed disproportionate delays and scrutiny on Black policyholders. SafeRent agreed to pay more than two million dollars to settle housing-score discrimination claims in 2024. All three defendants were deploying systems designed, trained, and sold by others.
For UK owner-managers, the lesson is operational as much as legal. Procurement terms, configuration choices, bias testing, monitoring, and human override processes all matter. Due diligence does not end at sign-off. If you cannot describe what the system optimises for, what data trained it, and how you would detect a skewed output, you are carrying a risk you have not scoped. The contract with your vendor does not resolve that.
Where will you actually encounter this risk?
For an owner-managed service firm, the relevant risk sits in tools that get purchased quietly and used at volume. Applicant screening, lead scoring, complaint triage, customer eligibility decisions, and rota allocation all change who gets seen, who gets a response, and who gets declined. Each of those is a decision point where a disadvantaged group can ground a claim.
The ICO’s position makes this harder to sidestep than it first appears. Bias can arise from proxy variables and inferred data, not only from fields that explicitly name a protected characteristic. A scoring model that does not process ethnicity can still produce ethnically skewed outputs if its inputs, such as postcode or purchase history, correlate with ethnicity in the underlying data. Saying “we do not use age or race data” does not close the risk. If the model’s outputs correlate with a protected characteristic, the indirect discrimination argument is available to anyone who experienced a worse outcome.
When should you take this seriously, and when is the risk lower?
The risk is highest when AI influences who gets an opportunity, a price, a service, or a decision about their individual circumstances. If the tool is used for internal administrative support with no effect on individual outcomes, such as summarising documents, drafting templates, or building management dashboards, the discrimination exposure is materially lower.
The threshold question is whether the AI changes the result for a specific person. Hiring decisions, customer eligibility checks, credit and collections workflows, complaint escalation, and any form of automated triage all sit above that line. Internal productivity tools that never affect an individual outcome sit below it. When a tool sits right on the boundary, the safer assumption is that it sits above it.
The FCA’s AI governance thinking for financial-services firms reflects the same logic: it focuses its oversight expectations on systems affecting individual customers, not every AI deployment in the business. That framing is useful even for firms outside FCA regulation, because it gives you a consistent test. If a customer, applicant, or employee could plausibly argue that the AI shaped a decision about them, you need to have thought about the fairness of that process.
What are the related legal and regulatory concepts?
The Equality Act 2010 covers both direct discrimination and indirect discrimination. Indirect discrimination is where a neutral practice puts a protected group at a particular disadvantage, unless the firm can show the practice is a proportionate means of achieving a legitimate aim. AI systems are structurally prone to indirect discrimination because they optimise for patterns in historic data, and historic data frequently embeds past bias in the form of proxy correlations.
The EU AI Act classifies certain AI uses as high-risk, including systems used in employment and access to essential services. For in-scope systems, it requires technical documentation, human oversight, bias monitoring, and risk management. UK firms selling into the EU, using EU staff, or running EU-facing workflows should verify whether their deployments fall within the Act’s scope. The NCSC’s AI security guidance adds the operational layer: controlled deployment, documented supplier accountability, and maintained human review paths rather than complete reliance on model outputs.
For the owner-manager running a ten to thirty person firm, the minimum control set is small and deliberate rather than large and bureaucratic. Map the processes where AI influences individual decisions. Identify which protected groups pass through those processes. Review outputs periodically for patterns that should not be there. Keep a human override path. Put your supplier’s obligations in writing, including what they will produce if you need audit logs, bias reports, or feature explanations. That is a day’s work to design and a quarterly check to maintain, not a large compliance programme. The case law and the ICO’s guidance ask only that someone in the business has thought about it, and can show that they did.
