System Prompts — laranevans.com

A system prompt is the persistent layer of a language-model interaction that sets role, format, and behavioral constraints across an entire conversation or workload. Unlike user prompts — which represent a specific request and change turn by turn — the system prompt stays fixed across the conversation and shapes how the model interprets every user message that follows. Most modern LLM APIs separate the two with distinct roles in the message structure: system, user, assistant.

The boundary matters because the model treats these roles differently. A system message is the developer's contract with the model about what the assistant is and how it should behave; the user messages are the requests that arrive against that contract. Conflating them — putting persistent instructions in user prompts, or stuffing one-off task content into the system prompt — is one of the most common sources of inconsistent behavior in LLM applications.

What goes in a system prompt

The content that earns its keep in the system prompt is the content that applies across every user turn:

Role assignment. Who the assistant is, what it knows, what it doesn't, what it refuses to do. "You are a Python code reviewer focused on async-correctness; ignore style issues unless they impact correctness." Vague roles ("you are a helpful assistant") produce vague behavior; specific roles produce specific behavior.
Output format constraints. The shape of the model's responses, whether prose, structured JSON, a specific section template, or a particular voice. Tight format constraints often pair with structured outputs for hard guarantees rather than aspirational ones.
Behavioral guardrails. Refusal patterns, tone constraints, safety boundaries, things the assistant should never do. These belong in the system prompt because they apply to every turn, not just to the next one.
Persistent context. Anything the assistant needs to know about the current run that doesn't change with the user's specific request: the current date, the user's known preferences, the system's tool surface, the data the model is operating against.

What does not belong in the system prompt:

Single-turn task specifics. A request that applies to one user message belongs in that user message.
Examples that demonstrate behavior for one specific kind of input. These usually belong in the user message they exemplify, or in dynamic few-shot retrieval if the examples vary by query.
Anything the user might legitimately want to override. The system prompt should encode the contract; user-tunable preferences belong elsewhere in the application's state, not baked into the model's role.

How modern APIs handle the system role

What you can put in the system prompt also depends on what the API exposes. The big three commercial APIs all support a distinct system role, with model-specific variations in syntax and behavior. On Claude, the system prompt is a top-level parameter in the API call rather than a message role within the messages array, and Anthropic recommends placing XML-tagged structural elements inside it to organize the prompt into named sections. On OpenAI's GPT family, the system prompt is the first message in the messages array with role: "system". On Google's Gemini, system instructions are passed via a systemInstruction field separate from the contents array.

Practitioner observation suggests Claude models tend to hold to system constraints more reliably than GPT models against strong user-message overrides, though the picture varies by version, task, and the specific kind of override attempted. Verify on the current model before depending on the difference.

Smaller and older models often don't have a distinct system message at all. The convention in those cases is to embed system content as a preamble inside the user message, with a clear delimiter (a row of equals signs, an XML tag, a labeled section header) so the model treats the rest of the user message as the request against that preamble. The delimiter is doing protocol work that the API would otherwise do; it's a less reliable channel, but it's the only one available.

A few model families have features that change the system prompt's economics. On Claude, prompt caching can hold the system prompt in a server-side cache across requests, which substantially reduces per-call cost when the system prompt is long and reused. Across providers, the system prompt is typically counted as input tokens against context-window limits, which makes it a budgeting concern at scale.

Patterns that earn their keep

Across those API differences, a small set of patterns recur in well-built system prompts. They aren't a checklist; they're the recurring shapes.

Explicit role and scope. State what the assistant is, what it's qualified to do, and what's out of scope. The scope statement is often more load-bearing than the role statement — telling the model what it should not do narrows behavior more reliably than telling it what it should.
Behavior first, format second, edge cases third. Structure the system prompt as a small set of named sections in that order. On Claude, XML tags are the recommended structuring mechanism; on other models, markdown headers or labeled sections work. Consistency within the prompt matters more than the syntax choice across prompts.
Refusal templates over refusal rules. If the assistant should refuse certain requests, show what the refusal looks like rather than describing when to refuse. A demonstration response is more reliable than a rule, for the same reason in-context examples generally outperform descriptions of the desired behavior.
Tool surface declaration. When the assistant has access to tools, the system prompt should declare which tools are available and the rough decision rule for which to reach for. The tool definitions themselves live in the API's tool-call schema; the system prompt's job is to set the policy for using them.
Explicit negative space. Stating what the assistant should not do is often more reliable than stating what it should. "Never invent file paths; if you need a path, ask" is a tighter constraint than "be accurate about file paths."

How system prompts fail in production

Most of these patterns can be undone by the same failure modes that affect prompts more broadly, plus a few specific to the system / user boundary.

User input can override system instructions. The same dynamics that drive prompt injection in retrieved content also apply to the system / user boundary. Hostile user input can attempt to override system constraints; on some models this works more often than others. The defense lives architecturally — minimum privileges, human-in-the-loop for actions that matter — not in stronger system-prompt wording.
Influence drifts over long conversations. System-prompt influence tends to weaken as conversations grow long, because the model's attention spreads across more recent context. Long-running agentic workloads sometimes re-inject the system prompt periodically, or at key decision points, to reset the contract.
Over-constraint degrades quality. The simple-prompts-outperform-complex-ones failure mode applies as much to system prompts as to user prompts. A system prompt with twenty rules is often worse than one with three: the model spends attention parsing rules instead of doing the task, and contradictions between rules surface as inconsistent behavior.
Some guarantees are aspirational, not enforced. A system prompt saying "always respond in JSON" is reliably enforced only when paired with constrained decoding or schema validation; "never use the word 'just'" will fail at some rate. Distinguishing hard guarantees from soft preferences is part of writing a system prompt you can actually depend on.
System prompts behave differently across model versions. A prompt tuned for one model version often behaves differently on the next, even within the same family. The cross-model transfer failure mode applies — plan for re-evaluation when a new version ships.

Treating system prompts as code

The thread running through these failures is the same: a system prompt earns its reliability the way code does, through review, version control, evaluation, and disciplined change management. The most consequential reframing of system prompts in practice is to stop treating them as configuration and start treating them as code. A production system prompt is an artifact with all the lifecycle properties of code: it gets reviewed, versioned, tested against an evaluation set, deployed deliberately, and rolled back when it regresses. Prompt versioning and prompt evaluation are where that machinery lives.

The system prompt is also where most of an LLM application's product identity is encoded. What the assistant is, how it responds, what it refuses, what tone it uses — these decisions sit in the system prompt, and changing them changes the product. Treating that surface with the same discipline as any other piece of user-facing behavior is the difference between an application that improves over time and one that drifts with each ad-hoc edit.