The owner has had a feeling for a few weeks now. The AI-drafted client briefings that used to come back sharp are slightly off, less specific, more padded, occasionally bland in a way she cannot quite name. She is not sure whether the team has stopped editing as carefully, whether the AI tool itself has changed, whether the client base has shifted under her, or whether her own standards have crept up. She has nothing recorded that would tell her which.
This is the typical shape of AI quality drift in owner-operated services firms. Gradual, distributed, noticed first by the person who remembers what good used to look like, and impossible to act on without evidence. The proportionate fix is a small set of quality signals tracked over time, light enough for the team to maintain and specific enough to tell you what changed.
What does AI quality drift look like in practice?
In an owner-operated firm, drift rarely arrives as a single failure. It accumulates as small deviations over weeks. A legal practice notices the AI starts to omit clauses it used to flag. A consulting firm finds strategic recommendations have turned generic. The shape is gradual, distributed, and usually noticed first by someone who remembers what the last version sounded like.
The scale of the exposure is not theoretical. Stanford’s 2025 AI Index reports 78% of organisations using AI in 2024, up from 55% the year before. A BBC and European Broadcasting Union investigation in October 2025 found around 45% of AI news queries to ChatGPT, Copilot, Gemini and Perplexity produced significant errors. Mount Sinai researchers running large language models against fabricated discharge notes saw incorrect medical advice in 47% of test cases. These are the baseline conditions in which an SME’s AI workflow sits.
Why does AI output drift in the first place?
Drift has four distinct causes that tend to be misdiagnosed because the felt experience is the same. Vendor model drift comes from the AI provider updating the underlying model. Prompt drift comes from the team tweaking the wording without recording the change. Data drift comes from the inputs shifting, often because the client mix has moved. Standards drift comes from the team’s own threshold quietly loosening over time.
Each cause needs a different fix. A vendor model change calls for a structured conversation with the vendor and a decision about whether to stay on the new version. A prompt change calls for resetting to a documented baseline and tightening version control. A data change calls for a look at what is flowing into the system and possibly a retraining or retrieval update. A standards change calls for the team to recalibrate against the original quality bar. Tracking signals is what lets you tell the four apart before you act.
Which three signals are worth tracking?
Three complementary signals catch the four causes between them at a cost a small team will sustain. Per-category error sampling, once a week, by a nominated reviewer who spends thirty minutes on five to ten recent outputs. Exemplar tracking, the same prompt run on the first of every month. A quarterly external read by someone outside the daily AI workflow. Together they cost twenty to thirty minutes a week.
Take the first signal in more detail. The categories are specific to the practice. For a legal drafting team, accuracy of precedent, completeness of clause coverage, jurisdictional fit and tone. For a consulting practice, logic soundness, client-specificity, data accuracy and strategic relevance. The reviewer records a simple count per category, week by week, in a shared sheet. Over eight to twelve weeks, a trend becomes visible.
The second is exemplar tracking. Pick one or two recurring outputs the firm generates with a stable prompt. A standardised portfolio risk assessment, an industry benchmarking summary, a contract clause extraction against a public document. Run the same prompt on the first of every month and save the output with the date. Compare across months for length, structure and semantic consistency. The exemplar is the control variable, the input is held constant, so any change in output points to the vendor or the model rather than to the team.
The third is the quarterly external read. Every three months, someone who is not part of the daily AI workflow reads twenty to thirty recent outputs and answers a single question, has quality stayed the same, improved, or degraded compared to three months ago? Anchoring bias is the reason this signal matters. A team that lives inside the workflow stops noticing gradual change. The outside reader catches what daily exposure has smoothed over.
How do the signals tell you what kind of drift you are seeing?
Each cause of drift produces a distinct fingerprint across the three signals. A vendor model update shows the exemplar shifting, category errors rising across multiple types rather than concentrating, and the external reviewer confirming a general decline. A team prompt change shows the exemplar holding steady while category errors rise in the specific areas a recent prompt edit touched.
A data change shows category errors clustered in particular input types or client segments, the exemplar staying steady because it uses clean representative data, and the external reviewer showing mixed results depending on which outputs were sampled. A standards change is the trickiest, category errors stay flat or even improve, the exemplar is unchanged, but the external reviewer reports degradation. That divergence is the signal that the problem is internal, the team has unconsciously lowered the bar. Real-world incidents bear out the pattern. Stanford HAI found bespoke legal AI tools still hallucinated between 17% and 34% of the time on benchmark queries, with consistent error types over time pointing to tool choice rather than prompt change. The diagnostic logic is the same in a services firm at a smaller scale.
What do you do when drift is confirmed?
The next steps are sequenced by cause. If the signals point to a vendor model update, bring the exemplar outputs and the category data to a structured conversation with the vendor. Ask whether the model has been updated recently, request release notes, and confirm whether your account is on the latest version. NIST’s AI evaluation guidance is explicit that a vendor model change can improve one use case and degrade another.
If the signals point to a team prompt change, audit the prompt history. Look at edit logs, comments, shared documents, anything that records when the prompt last moved. Walk the team through the original design choices and find the point where the change crept in. Reset to the documented baseline and impose version control going forward, every change tested against the exemplar before it goes into production. If the signals point to a data change, audit the inputs. Has the client mix moved? Have document formats changed? Is the retrieval source still current? If the signals point to a standards change, the fix is internal, not technical, bring the team back to the original quality standard, place the old and current outputs side by side, and recalibrate. Set escalation thresholds in advance, for example category error count up by more than half in eight weeks, or external review degraded for two consecutive quarters, so drift never sits in ambiguity waiting for someone to formally call it.
If the conversation about AI quality monitoring is on your mind because you can feel something has shifted but cannot prove it yet, that is exactly the moment a small set of signals starts paying back. Book a conversation and we will work out which three signals fit your firm.
