Context Engineering: Prompt Versioning, Token Budgets, and Cost Control

Every LLM feature we inherit has the same artifact at its center: one enormous system prompt that has grown by accretion for eight months. Nobody remembers why half the instructions are there. Some contradict each other. A few are addressing model behavior that was fixed two versions ago.

The reflex is to call this a prompting problem and rewrite it. It is not. It is a context engineering problem - the discipline of deciding what information enters the model on any given call, where it comes from, how much it costs, and how you change it safely.

Treating context as a managed resource rather than a text file is the difference between a feature you can operate and one you can only apologize for.

---

The Context Window Is a Budget, Not a Bucket

Larger context windows made a bad habit affordable. When you can pass two hundred thousand tokens, it feels reasonable to pass everything and let the model sort it out. It is not, for three reasons.

Cost scales with what you send, every single call. A system prompt that is four thousand tokens longer than it needs to be, on a feature handling fifty thousand calls a day, is a line item. Teams discover this on the invoice rather than in design review.

Latency scales too. Time to first token tracks input length. On an interactive feature, a bloated context is felt by every user on every request.

Accuracy does not improve monotonically with more context. Models attend unevenly across a long input. Bury the one relevant fact among forty irrelevant ones and retrieval quality inside the window degrades. More context frequently makes output worse, not better, and the failure is silent.

The question is never "will this fit?" It is "does this earn its place?"

---

Give Every Call an Explicit Budget

We assign each call type a token budget and split it into named allocations. A support agent turn might look like: system instructions capped at 800 tokens, retrieved documents at 3,000, conversation history at 2,000, tool schemas at 1,200.

Each allocation has an owner and a strategy for what happens when it overflows. Retrieved documents get truncated by relevance rank. History gets summarized oldest-first. Tool schemas get filtered to the tools relevant to the current step rather than shipping the full catalog on every turn.

The value is not the specific numbers. It is that overflow becomes a designed behavior instead of an accident. Without a budget, the failure mode is that some component quietly grows, something else gets silently pushed out of the window, and quality degrades in a way that never shows up as an error.

We log actual token usage per allocation on every call. When the retrieval allocation starts running at 95 percent of budget, that is a signal to look at chunking before it becomes a truncation bug.

---

Prompts Are Code, So Version Them Like Code

A prompt is a program. It has inputs, behavior, and regressions. The tooling should reflect that.

Store prompts as versioned files in the repo, not as strings scattered through application code and not in a database someone edits through an admin panel. Live editing of production prompts feels agile until the day you need to know what the prompt was during an incident three weeks ago.

Template them properly. Separate the static instruction text from the runtime-injected values, with a real templating layer that fails loudly on a missing variable. String concatenation across a codebase is how you end up with a prompt that renders differently depending on which caller reached it.

Attach a version identifier to every call and log it with the output. When you are debugging a bad response from last Tuesday, the first question is which prompt version produced it. Without that, you are reconstructing from deploy timestamps.

Change one thing at a time and score it. A prompt change is a code change and belongs in a pull request with an eval run attached. We covered the mechanics of that in Evals Are the Only Thing Standing Between You and a Regression - prompt versioning without evals just means you can identify which change broke things after the fact rather than preventing it.

---

Structure Beats Volume

How you arrange context matters as much as how much of it there is.

Put stable content first. Instructions and schemas that do not change between calls should sit at the front, where prompt caching can pick them up. Getting the ordering right - static prefix, then variable content - routinely cuts input cost by a large fraction on multi-turn features, and it is a purely mechanical change.

Put the most important variable content last. Recency helps. If there is one retrieved document that matters most, it should not be buried in the middle of nine others.

Label everything. Clear delimiters and labels for each section - retrieved documents, conversation history, current request - measurably help the model tell them apart, and they make prompt injection harder because untrusted content is visibly demarcated from instructions.

Never mix untrusted content into the instruction region. Retrieved documents and user input are data. If they can be read as instructions, they will be, by someone who is trying.

---

Managing History Without Losing the Thread

Conversation history is where context budgets go to die. Three approaches, in rough order of how often we reach for them:

• Sliding window. Keep the last N turns verbatim, drop the rest. Simple, predictable, and fine for short-horizon tasks where nothing from turn two matters at turn twenty.

• Rolling summary. Keep recent turns verbatim, and maintain a running summary of everything older, regenerated periodically. Costs an extra call, preserves the thread. This is our default for anything conversational.

• Structured state extraction. Rather than summarizing prose, extract the facts that matter into a typed state object - the customer ID, the issue category, what has been tried. Cheapest to carry, most reliable, and it makes the state inspectable. Works when you can enumerate what matters up front.

Whichever you choose, the important part is that the working state of the task lives in an explicit object rather than being implicitly recoverable from the transcript. A task whose state exists only as prose in the history is a task you cannot resume after a crash.

---

Controlling Cost Without Degrading Quality

The levers, ordered by how much they usually return relative to the effort:

Prompt caching. Restructure so the stable prefix is genuinely stable and cacheable. Often the single largest win available, and it costs a refactor rather than a quality tradeoff.

Model routing by difficulty. Not every call needs the strongest model. Classify the request cheaply, route the easy majority to a smaller model, escalate the rest. The eval suite tells you where the boundary safely sits.

Trimming what nobody reads. Audit the system prompt against the eval suite by removing sections and measuring. We have cut prompts by 40 percent with no measurable quality change more than once. The instructions that survive contact with an eval are the ones that were doing work.

Tightening retrieval. Passing eight chunks when three would do is a cost multiplier on every call. Better ranking is cheaper than more context.

Capping output length. Output tokens usually cost several times what input tokens cost. An instruction to be concise, plus a hard max, is close to free.

---

Retrieval Is Part of the Context Budget

When a feature uses retrieval, the retrieval config is a context decision even though it usually lives in a different part of the codebase and is owned by a different person.

Chunk count is a budget line, not a tuning knob. Going from three chunks to eight because it fixed one bad case adds that cost to every call forever. Before widening retrieval, check whether better ranking would have surfaced the right chunk in the top three - it usually would, and reranking a shortlist is cheaper than carrying five extra chunks on every request.

Deduplicate before injecting. Overlapping chunks from adjacent sections of the same document are common with sliding-window chunking, and they consume budget to say the same thing twice. A cheap similarity check at injection time recovers real headroom.

Include the metadata that matters and drop the rest. A source title and date help the model cite correctly and reason about recency. A full JSON blob of internal document IDs, permissions, and ingestion timestamps helps nobody and is pure cost.

Consider whether retrieval was needed at all. A surprising number of calls retrieve documents that turn out to be irrelevant to a request the model could have answered directly or should have refused. A cheap upstream classification that skips retrieval for those cases cuts both cost and latency. The deeper mechanics of getting retrieval right are covered in RAG in Production.

---

Instrument Before You Optimize

You cannot manage a budget you cannot see, and the default logging in most LLM applications records the total token count and nothing else. That number tells you that something grew without telling you what.

What we log on every call, at minimum:

• Token count broken down by allocation - system, tools, retrieval, history, user input - rather than one total. This is the single most useful thing on the list and it turns a mystery into a chart.

• Prompt version identifier, so you can join cost and quality changes to specific releases.

• Cache hit rate on the stable prefix. A deploy that accidentally makes the prefix dynamic - injecting a timestamp, reordering a tool list - silently destroys caching, and the only symptom is the bill.

• Truncation events. When any allocation overflows and content is dropped, that should be a recorded event, not a silent slice. Silent truncation is one of the hardest quality bugs to diagnose after the fact, because the output looks plausible and nothing errored.

• Output token count separately from input, since they are priced differently and are driven by different things.

With that in place, a weekly look at the distribution usually surfaces one or two obvious problems: a tool catalog that grew to forty entries, a history summarizer that stopped running, a retrieval step returning far more than intended.

---

Where to Start

Instrument first. Log token counts per component on every call for a week, and you will almost certainly find that one allocation you were not thinking about dominates the bill.

Then move prompts into version control with explicit template variables, attach a version to every logged call, and set a budget per call type. None of that requires changing model behavior - it just makes the system legible enough that the next round of changes can be deliberate.

If you are working through this on a feature that is already in production, our LLM Automation Consulting practice does exactly this kind of unpicking.

A prompt nobody is willing to edit is technical debt with a monthly bill attached.