A recruitment consultant running a twelve-person firm in Birmingham added an AI CV-screener to their hiring process last year. The tool ranked candidates by fit score, they spent less time on initial reading, and shortlists came back quickly. Three months in, a rejected candidate asked why they’d been screened out. The answer the tool provided was statistically defensible on paper. It was also, their employment solicitor advised them later, potentially unlawful under the Equality Act 2010.
The gap between “the software decided” and “you are responsible” is narrower than many business owners expect.
What does “AI discrimination” actually mean?
AI discrimination happens when an algorithm consistently treats one group less favourably than another because of patterns in the data it learned from. If those outcomes align with a protected characteristic under the Equality Act 2010, such as age, race, sex, or disability, the result can be unlawful discrimination regardless of whether a human or a piece of software made the call.
There are two types that matter in employment law. Direct discrimination is when a tool treats someone worse because of a protected characteristic: an AI screener that downgrades applications mentioning a particular religion, for instance. Indirect discrimination is subtler: a tool that applies a rule or process which consistently puts a protected group at a disproportionate disadvantage, even if nobody designed it that way. An AI screener that filters out CVs with long employment gaps can indirectly disadvantage disabled applicants or those who took time out for caring responsibilities.
Amazon’s internal recruitment algorithm, built between 2014 and 2017, is the most widely cited example. The system learned from historical CVs, the majority of which came from men, and began downgrading applications containing words like “women’s chess club captain”. The project was abandoned after internal audits exposed the pattern. The lesson for an owner-managed business is the same: AI often reflects the skews in the data it was trained on, and once baked in, those patterns compound.
Why does this create legal exposure for owner-managed businesses?
UK law doesn’t carve out an exception for automated decisions. The Equality Act 2010 focuses on outcomes, not on how a decision was reached. Legal commentators are clear that employers remain fully liable even when software makes the call, and that AI cannot justify discriminatory criteria. If a tool you deploy in hiring, promotion, or performance management consistently disadvantages a protected group, the liability sits with your business.
The ICO reinforces this from a data-protection angle. Its guidance makes clear that demonstrating an AI system isn’t unlawfully discriminatory is a separate obligation from general UK GDPR compliance. Both have to be satisfied. The financial exposure can be significant: under UK GDPR, the ICO can issue fines of up to £17.5 million or 4% of global annual turnover for serious breaches. In 2022, it fined facial-recognition provider Clearview AI £7.5 million for unlawfully scraping biometric data on UK residents.
The Employment Rights Act 1996 adds a further strand. Section 98 sets out the test for fair dismissal, requiring both a fair reason and a reasonable procedure. If you rely heavily on AI outputs in a disciplinary process without proper human oversight, and without giving the employee any way to understand or challenge how the decision was reached, the dismissal can be procedurally unfair regardless of whether the underlying reason was valid.
A 2023 Capgemini survey found that 65% of executives globally were concerned that AI could produce discriminatory or biased outputs, particularly in HR applications. UK employment law doesn’t scale its requirements to the size of your firm.
Where will you actually meet this risk?
For an owner-managed services business, the highest-risk territory is anything that touches decisions about people: CV-screening and ranking tools, automated video-interview scoring that analyses facial or vocal cues, performance dashboards that flag underperformers without human review, and customer-scoring systems tied to service or credit eligibility. These are the contexts where protected characteristics emerge as hidden variables, because the historical data AI tools train on often reflects the same inequalities that employment law exists to correct.
The mechanisms are worth understanding. Historical recruitment data in many industries skews heavily towards groups that have predominantly held the relevant positions. A machine-learning model trained on ten years of successful hires from a firm where senior roles were held largely by men will learn to replicate that pattern. Proxies compound this: postcode, university attended, employment gaps, and certain professional accreditations all correlate with race, socioeconomic background, or disability status. A tool optimising for “cultural fit” can produce indirect discrimination without a single explicit reference to a protected characteristic.
The ICO requires organisations to understand and explain how AI systems reach high-impact decisions. Vendors unwilling to disclose their training data, testing methodology, or model logic place you in a difficult position if a hiring or scoring decision is ever challenged. One empirical review of algorithmic hiring tools found that up to 60 to 80% of applicants were being automatically filtered before any human review in large-scale deployments. That scale of pre-filtering raises systemic exclusion questions that individual human bias rarely reaches.
When is the risk genuinely lower?
Not every use of AI raises a discrimination question. Tools that operate on non-personal or aggregated data, with no connection to individual employment or service decisions, sit well outside the Equality Act’s reach. A scheduling tool, an energy-management system, or a document-summarisation tool used purely by internal staff to cut admin time pose little discrimination risk, though data-protection and intellectual property obligations still apply.
Simple automation with transparent, self-designed logic is also in a different category from opaque machine learning. A rule you write yourself, such as chasing an invoice if it is more than 30 days overdue, is easy to audit: the logic is visible, the criteria are entirely yours, and there are no proxy variables you haven’t considered. The more a tool relies on a black-box model trained on external data, the harder it is to confirm that protected characteristics aren’t influencing outputs.
There is also a positive case. The EHRC has noted that well-designed AI has the potential to standardise decision criteria and expose patterns of human bias that might otherwise go undetected. Some research suggests properly audited models can flag situations where comparable applicants are treated differently, creating an evidence trail that manual processes rarely produce. The caveat is that this requires deliberate design, ongoing monitoring, and the willingness to act on what the data shows.
What other legal frameworks apply alongside the Equality Act?
Two further frameworks apply when AI meets personal data or employment decisions. UK GDPR, applied through the Data Protection Act 2018, requires that AI processing of personal data be fair, explainable, and subject to substantive human review when it significantly affects individuals. The Employment Rights Act 1996 adds that opaque AI outputs in dismissal processes, with no transparency or right of challenge, can make a decision procedurally unfair.
On the data-protection side, the ICO has broad enforcement powers and clear guidance: human review of AI outputs must be substantive, not a procedural sign-off. A manager who rubber-stamps an AI recommendation without genuinely assessing it won’t satisfy the ICO’s standard.
There is currently no single UK AI Act. Regulators including the ICO, FCA, and CMA apply existing laws. Two Private Members’ Bills are working through Parliament, one focused on public-sector algorithmic decisions, another proposing a central AI Authority and a requirement for certain organisations to appoint an AI officer. Neither is law yet. The EU AI Act, adopted in 2024, classifies many HR and recruitment tools as high-risk, imposing documentation, testing, and transparency obligations on providers. Owner-managed businesses using EU-hosted SaaS tools may find vendors adapting their products to EU standards and passing new compliance requirements downstream.
The practical question is whether your current AI use can withstand scrutiny under the laws already in force. A consistent checklist emerges across legal and regulatory guidance: map where AI touches people decisions, ask vendors blunt questions about training data and bias testing, keep a named human reviewer accountable for every consequential output, run an equality and data-protection impact assessment before deploying high-risk tools, tell people when AI is involved and how to challenge it, and put an AI policy in writing. Used with that care, AI can genuinely speed up admin and improve consistency. Used without it, you carry liability for decisions you may not fully understand.
