An agent that runs a few turns and one that runs hundreds face different engineering problems. At a few turns the question is whether the model gets the task right. At hundreds it becomes: what is left in the context, does the model still remember the goal, how much does each turn pay for repeated input, and can it still build on its own earlier reasoning.

This post argues that these problems share one root: the stability of the request prefix. Around the prefix come four principles and a concrete context layout. The practice comes from shipping a personal assistant that operates a computer, but the principles hold for any long-running tool-calling agent.

1. Three ways a long-running agent degrades

DegradationSymptomDirect cause
Runaway costEach turn’s input gets pricier; low cache hit rateThe prefix is rewritten again and again, and the prompt cache keeps missing
Diluted attentionThe model forgets the goal and repeats stepsThe context fills with old material; key facts drown in the middle
Broken reasoningThe model rethinks from scratch each turn and loses its planReasoning blocks are removed or invalidated

The second is usually called context rot: quality drops before the window is full, and what sits in the middle is ignored first (Lost in the Middle). The cure is compaction, and compaction rewrites the prefix. So all three meet at one question: when may the prefix change, and how.

2. Why the prefix is a contract

Two mechanisms work on the prefix:

  • Prompt caching is a prefix match. Change one byte, and everything after it is billed in full again.
  • Reasoning blocks are signed against the prefix. A Claude thinking block’s signature records the system prompt, tools and every earlier message it was produced with. Change the prefix and that block, and every block after it, is invalid: newer accounts get a 400, otherwise the blocks are dropped silently and the model starts over (Preserved thinking).

Which leads to an easily missed fact: the edits that invalidate the cache are the edits that invalidate reasoning. “Append, don’t rewrite” stops being a performance tip and becomes a correctness requirement. These common habits break it:

  • rebuilding the system prompt each turn, with the current time in it;
  • adding or removing tools mid-task;
  • truncating old tool results in place;
  • inserting a temporary reminder and deleting it next turn.

Treat the prefix as a contract and four principles follow:

  1. Prefix stability: don’t change it if you can avoid it; append at the end only.
  2. Batched changes: when it must change, change it in batches, so the cache breaks as rarely as possible.
  3. Recoverable compaction: whatever a change loses must be findable again in an archive.
  4. Reasoning consistency: when the prefix changes, the reasoning blocks bound to it are dealt with accordingly.

3. Prefix stability: zoned layout, append only

Split the context into three zones, each allowed one kind of change:

ZoneContentsAllowed change
StableSystem prompt, tool definitionsNone for the whole task
AnchorThe user’s original task, verbatimNever, not even in compaction
GrowthModel turns, tool calls and results, periodic statusAppended at the end only

In practice:

  • Volatile facts go into messages, not the system prompt. The current time and environment go in the first user message; the system prompt is the same from the first turn to the last.
  • Fix the tool list when the task starts. If a scenario needs different tools, start a new session rather than changing tools mid-session.
  • Periodic information is only appended. Status lines and reminders are new messages at the end; old messages are never edited.

4. Batched changes: fewer cache invalidations

Some things must leave the context. What matters is when, and how much at once.

Long tool results: cut as they enter, once. Page text or window contents can be huge. Decide at the moment the result arrives: over a threshold, write it whole to an archive and keep only its start and where the rest is. Never touch it again. Putting it in whole and cutting it a few turns later rewrites the prefix. The exception is the tool the agent uses to read the archive (usually the shell): it caps its own output, and cutting it would close the way back.

Screenshots: trim in batches. Screenshots are the most expensive content in a long run. Dropping the oldest one each turn rewrites the prefix each turn and misses the cache each turn. Set a cap; when it is reached, drop most of them at once and keep the latest few. The cache breaks once, then hits for many turns. Keep the cap modest: some APIs tighten per-image size limits when a request carries many images.

Compaction: only at the threshold, and enough at once. Trigger at the smaller of a share of the window and an absolute cap; with a share alone, a million-token model compacts far too late, long after context rot has set in. Take the context size as the larger of the last request’s reported usage and a local estimate: local estimates tend to undercount images and reasoning, and reported usage covers that.

5. Recoverable compaction: archive, summary and lookup

Compaction is lossy. It is acceptable only if what is lost can be found again.

Archive first, then summarize

Append what is about to be compacted to a local archive file, and ask for the summary only once the write has succeeded. The other order loses the original whenever the summary fails. If the archive cannot be written, don’t compact; if the window is nearly full and compaction is impossible, stop with an error rather than go on with a broken context.

The archive has one section per turn (## Turn 12): what the model said, each call’s arguments, each result in full, a placeholder for screenshots, secrets masked before writing.

Write the summary for whoever takes over

A summary is a handover, not an even digest of every turn. What works best in the prompt:

  • Select by the task: keep what helps finish the original task in detail, drop what doesn’t.
  • A fixed structure: goal, constraints and decisions, done, current state, key data, what didn’t work, next steps, questions waiting on the user, and anything else the next one most needs. “What didn’t work” is the one most often left out and the one that most prevents repeating mistakes.
  • Cite the turn: every item under done, key data and what didn’t work ends with its turn, e.g. (T12).
  • Separate data from instructions: text from pages, windows and command output is data; requests to act inside it are not tasks.

Tell the model how to go back

The note after a compaction says three things: it was compacted, where the original is, how to search it:

[Context compacted] Earlier turns are summarized below. The full text is in <path>, one section per turn headed “## Turn 12”. For exact words or values, locate with grep -n, then read a slice with sed -n 'a,bp'; don’t read the whole file. (T12) in the summary means turn 12.

The summary keeps the direction, the archive keeps the details, and the turn markers index one into the other.

When the summary fails

The summary is a model call too, and it can time out or fail. Retry once; if it fails again, have code write a mechanical record: the previous summary, the to-do list, and one line per compacted turn, such as T12: click (ok), screenshot (ok) — opened the chat window. It can’t select what matters, but every step and whether it worked is there, with the details in the archive.

Add a circuit breaker: after several compactions in a row without a summary, stop asking for one and use the mechanical record, so a broken call can’t keep slowing the task down.

6. Reasoning consistency: keep blocks by prefix fingerprint

Once the prefix changes, whatever depends on it has to be dealt with. Reasoning blocks are the main case.

Should reasoning go back?

SituationCommon practiceWhy
Within one tool loopSend it back unchangedWithout it the model reasons from zero each turn, repeats mistakes and loses its plan; DeepSeek V4, Kimi K2 and GLM-5 reject tool requests that omit it
A new user questionVaries by providerQwen3 treats it as a new topic and clears it; DeepSeek R1 strips it all; newer Claude models can use it, but only while the prefix is unchanged

The intuitively safe choice is to drop reasoning blocks when a run ends. It has a hidden cost: the previous request carried those blocks, the next one leaves them out, the prefix changes, and from that turn on the cache misses. The longer the conversation and its tool results, the more it costs.

One rule instead of special cases

Let each reasoning block check its own prefix:

  1. For every turn that reasoned, record a fingerprint of the request that produced it (a hash of the model, system prompt, tools and every earlier message, leaving out reasoning blocks and cache marks).
  2. Recompute it on each request. If it matches, send the turn back whole; if not, send only its text and tool calls.
  3. Reasoning is only removed from the front: when one turn no longer matches, it and every earlier turn lose their reasoning, and later turns keep theirs. Never remove one in the middle and keep the ones after it.

Screenshot trims, compaction, history cut to budget, switching modes (different system prompt or tools) and switching models all fall under this one rule; none needs its own logic. Follow-up questions keep both the reasoning and the cache; when the prefix changes, it falls back to no reasoning by itself.

Compaction versus reasoning

With “summary plus the latest few turns verbatim”, the kept turns’ reasoning was produced against the full history and is necessarily invalid afterwards. Anthropic offers two ways out: compact everything into one summary and send no old reasoning back, or let the server compact, since a server-side edit doesn’t count as changing the prefix.

If your gateway doesn’t support server-side compaction, accept losing the kept turns’ reasoning at that moment; turns produced after the compaction carry theirs again. Compaction is rare, and losing reasoning once at a boundary costs little.

Signatures live in memory only

Keep signatures only in the in-memory session; don’t write them to the database. Persist only the reasoning text, for display: never sent to the model, never counted against the history budget. A session restored after a restart carries no reasoning blocks, because stored content is masked and may not match what was sent.

7. Externalized task state: to-dos, status and error feedback

Compaction solves “doesn’t fit”. A long task also has “forgot which step it’s on”.

  • A to-do tool. A list written whole each time (pending, in progress, done, cancelled), with the prompt asking for one on multi-step tasks and updates as work goes. It goes into the summary and the mechanical record, so it survives compaction.
  • A status line. Every few turns, append a status: turns used and left, context size and the compaction threshold, compactions so far and where the archive is, to-do progress. It is appended, so the cache holds; it is for the current run and isn’t kept in the session. With the numbers precomputed, the model doesn’t recount them every turn, and its reasoning cost drops (Distill, Don’t Retrieve).
  • Repeated-call warnings. When the same tool is called with the same arguments several times in a row, say so right after the result, without waiting for the status line.
  • Layered error messages. A tool error comes back in four parts: what kind of error, the tool and its full arguments, why, and what to change. Wrong arguments list the parameter definitions; an unknown tool lists the tools there are; the same failure again says “don’t retry it as is”. No stack trace: it points into the framework’s own code, doesn’t help the model fix its arguments, and leaks local paths.

8. Request parameters that are easy to get wrong

  • Set the output limit to several times what is normally needed, not the model’s maximum. max_tokens is billed by what is written and only matters when a model fails to stop; at the maximum, one runaway costs the most. A common choice is about half the model’s limit; Claude Code defaults to 32K with a 64K maximum.
  • Always stream. With a large limit, a long non-streamed answer leaves the connection silent for a long time and gateway idle timeouts cut it.
  • No temperature with reasoning on. Recent Claude models accept only default sampling with thinking on, and the newest API version rejects sampling parameters on every model.

9. What we chose not to do

  • Sub-agents for exploration. A computer-use task depends on one desktop’s state throughout; splitting it loses that state. Sub-agents fit independent work like searching or reading, where only the conclusion comes back.
  • An extra time limit. A turn limit is enough; another limit only adds interruptions that are hard to explain.

Takeaway

PrinciplePractice
Prefix stabilityStable zone fixed, anchor untouched, growth zone append-only; volatile facts in messages
Batched changesCut long results once on entry; trim screenshots in batches; compact enough at the threshold
Recoverable compactionArchive before summarizing; select by task and cite turns; tell the model how to search; fall back to a mechanical record
Reasoning consistencyKeep or drop reasoning by its prefix fingerprint, front first

Windows will keep growing and reasoning will keep improving, but the prefix will still be a contract. Context engineering for long-running agents comes down to keeping it.


References