A 24-staff insurance brokerage I spoke with has its sales team using ChatGPT for client emails. The Custom Instructions field has grown over twelve months from “be helpful and professional” to roughly 1,800 tokens of compliance language, brand voice, and FCA-relevant disclosures. Three team leads added clauses at different points. Nobody has read the result end to end. Cost per query has crept up because every call now ships 1,800 tokens before the user’s question is processed. When a senior advisor asked at a management meeting “who owns this, and what is the change history?”, the answer was silence.
That field is a system prompt. The brokerage has been writing one for a year without naming it. The interesting question is no longer “what is a system prompt”, it is “who owns ours, where is it versioned, and how much is it costing us on every call?”
What is a system prompt vs a user prompt?
A system prompt is the persistent instruction the AI reads first on every call: role, tone, constraints, disclosure rules, brand voice. A user prompt is what your team types into the chat each time. The model treats them as separate fields, gives the system prompt higher priority, and applies it across every session until you change it. The system prompt is the constitution; the user prompt is the request that operates inside it.
Custom Instructions in ChatGPT, output styles in Claude, profile-level prompts in Microsoft Copilot Studio, and the system field in the OpenAI, Anthropic, and Google Vertex APIs all describe the same architectural slot. The user prompt changes with every interaction. The system prompt persists, often invisibly, across thousands of conversations. Once you see that asymmetry, you can stop treating the prompt as a settings field and start treating it as configuration.
A short example makes it concrete. The system prompt for a brokerage chatbot might say: “You are a helpful product specialist. Never quote premiums without a manager check. Always escalate complaints to the senior team.” Every customer who opens the chat operates inside that frame. Their user prompts are what they type into the box. The frame applies; the conversation varies.
Why does this matter for your business?
System prompts have quietly become operating infrastructure, with three concurrent costs that owners typically cannot see. The first is on the bill, because every token in the prompt is billed on every call. The second is governance, because the prompt is now where regulated rules live. The third is intellectual property, because a finely engineered prompt has commercial value the law will protect only if you protect it first.
On cost, the maths is unforgiving but the fix is mundane. Stripping repetition, moving boilerplate to external configuration, and deleting clauses no one can defend commonly cuts API costs and latency by 10 to 20 per cent. The first move is to read your own prompt end to end; many owners have never done that, and the act of reading it is usually enough to find the first three things to remove.
On governance, the prompt is where operational rules now live. “Never share customer account numbers.” “Always confirm identity before discussing balances.” “Flag any regulated request for human review.” Under the EU AI Act, in enforcement since August 2026, organisations using high-risk AI must document how those systems are instructed and monitored. If a regulator asks for the version that was in production on a given date, the team running prompts in Git with version tags can answer. The team running them in a shared Google Doc cannot.
On IP, a finely engineered prompt encodes years of compliance expertise and brand voice, and trade secret law protects that only if you take reasonable steps to keep it confidential. A repository with role-based access and signed commits documents those steps. A shared Slack channel does not.
Where will you actually meet it?
You meet it under a different name on every platform you use, which is the main reason teams write system prompts for months without realising it. ChatGPT calls it Custom Instructions. Microsoft Copilot Studio splits it into a default layer and profile-level prompts. Anthropic’s Claude exposes output styles inside the Agent SDK. Google Vertex AI calls it system instructions. The architectural slot is the same; the marketing label varies.
In direct API work the surface is named explicitly. The OpenAI, Anthropic, and Google APIs all carry a system field separate from the user message, and any production code making API calls is shipping a system prompt whether or not the developer flagged it as one. That is the slot to audit when you ask your team “where does the prompt actually live, and who can edit it?”
The vendor pitches you sit through are the third place to spot it. When a demo shows a configurable assistant, the questions to ask out loud are: where is this stored, can we version it, can we roll back, who can edit it, and is there an audit log? Vendors who answer cleanly are selling production AI. Vendors who say “you can change it in settings” without surfacing version history or access control are selling a demo wrapped in a UI.
When should you formalise governance, and when can you ignore it?
You should formalise governance the moment a system prompt touches customers, regulated data, or an operational decision the firm would not want second-guessed. Customer-facing chatbots, proposal generators, anything reading contracts, anything plugged into a CRM or HR system, anything subject to FCA or ICO oversight. If your AI use ticks any of those, the prompt is a controlled artefact, and the governance question is when, not whether.
You can ignore the governance overhead when the AI use is genuinely individual and low-stakes. One person using ChatGPT to draft internal emails for their own review. A solo founder using Claude as a thinking partner. A consultant rewriting a paragraph in their own draft. There is no shared prompt, no customer exposure, no regulated data, no decision being delegated. Adding Git and approval gates to that use case is theatre.
The shape between those poles is incremental. Start with the highest-stakes use case and govern that one prompt: store it in version control, write a one-page card naming the owner and the purpose, and run a small set of representative queries before any change goes live. Once that cycle works, extend to the next use case. An SME running two or three stable prompts rarely needs a dedicated platform on day one; an owner, a repository, and a written rule that no production change ships without a review will cover it.
What related concepts should I know?
Three sit close enough to the system prompt that you will meet them in the same conversations. The first is prompt management as a tool category. Platforms such as Agenta, PromptHub, and LangSmith provide version control, A/B testing, and rollback specifically for prompts. They are useful when change frequency is high or several teams edit the same prompts. They are overkill for an SME running two or three stable prompts in a Git repo.
The second is prompt injection, the security failure mode where a malicious instruction hidden in a document or webpage tricks the AI into ignoring its system prompt. The system prompt is the first line of defence, so anything sensitive embedded inside it (credentials, role IDs, internal database names) becomes attacker-readable when an injection succeeds. OWASP now lists system prompt leakage as one of the top 10 LLM vulnerabilities. Externalise secrets, and assume the prompt itself is low-trust storage.
The third is function calling and the wider context of agentic AI. When an AI can call tools on your behalf (send emails, write to a CRM, move money), the system prompt is where the operational guardrails live: which actions are allowed, which require human approval, which are out of scope. A weak system prompt in an agentic context is no longer a quality issue; it is a control issue.
The system prompt is the highest-impact configuration surface in the AI stack you are buying or building. Treat it that way and the cost, security, and compliance questions get cheaper. Treat it as a free text field and they get more expensive every month. If you want a candid look at where your prompts sit, book a conversation.
