#204 — Pi: How compaction works

Compaction and RAG solve different problems. RAG retrieves external documents or data the model never saw before, pulling relevant chunks into context on demand. Compaction compresses context the model already generated — your own conversation history — into a smaller summary so the session can continue. You'll often see both in production agents: RAG for knowledge, compaction for memory management.

The core mechanism is similar across coding agents — Claude Code, Codex, and Pi all use an LLM call to summarize older conversation history once the context window nears its limit. The implementation details differ: Pi's default retention window is a 20,000 token budget (roughly 5-20 turns), uses a distinct 'context summarization assistant' system prompt, and stores summaries as portable plain text, while other agents may tune retention size, trigger thresholds, or summary structure differently.

In Pi, you can force compaction at any point using the /compact command rather than waiting for the automatic trigger, which only checks after a turn ends. This is useful before starting a new complex task, since you get a clean handoff summary at a moment you choose rather than mid-task.

That spike is almost certainly caused by compaction breaking your prompt cache. Prompt caching only discounts requests that share an exact token-for-token prefix with a previous request — once compaction replaces old history with a summary, the prefix changes and the first post-compaction request must be fully recomputed at full price. Costs return to normal on subsequent requests once a new cached prefix is established.

Context rot refers to the measurable decline in LLM output quality as context size grows, even well before the model hits its hard token limit. Chroma's context rot research found accuracy degradation on long-context tasks scales with input length across major models — meaning your coding agent can feel noticeably 'dumber' in hour three of a session purely from context bloat, not a model downgrade.

Yes — Pi is extensible, so you can ask it to create a custom extension with your own compaction prompt instead of the default 'goal, progress, key decisions' structure. This matters for founders building domain-specific agents: a legal-review agent or a biotech research agent may need different fields preserved (e.g., citations, compliance flags) than a general coding assistant.

Pi's default token budget for retained recent messages is 20,000 tokens, which typically covers 5 to 20 turns depending on message density, and this is configurable rather than fixed. Lowering the budget triggers compaction more frequently (more caching resets, but tighter context), while raising it delays compaction (fewer resets, but more context rot risk before the summary kicks in).

It depends on whether you need continuity. Starting fresh restores peak model performance instantly but discards prior decisions, open TODOs, and constraints you've established — risky mid-build. Compaction preserves a summarized handoff so work continues uninterrupted, at the cost of some fidelity and a one-time cache-reset fee; most founders should default to compaction for active builds and reserve fresh starts for genuinely new tasks.

Plain text keeps the summary model-agnostic and portable — since the compaction result isn't tied to any particular model's expected input schema, you can switch the underlying LLM mid-session (e.g., from a frontier model to a cheaper one) and the new model can still parse and use the handoff summary without a translation step.

Not necessarily, and it's a deliberate design choice in Pi. Because the compaction request is standalone and doesn't depend on the existing conversation's model context, it can be routed to a smaller, cheaper model purely for the summarization task without degrading the main session's reasoning quality — the expensive model stays reserved for actual coding work.

A turn spans one full user message through the final assistant response, including everything in between. It starts with [system][tools][user], may include one or more assistant tool-call and tool-result cycles as the agent executes actions and feeds results back to the model, and ends once the assistant produces its final output. Each new turn appends another full cycle to the conversation history, which is why token counts grow non-linearly in agentic coding sessions compared to simple chatbot Q&A.

AGENTS.md is a convention for a project-level instructions file that coding agents like Pi load automatically at the start of a session, similar to a README aimed at the AI rather than humans. It gets included in every request's context alongside the system prompt and tool definitions, so a bloated AGENTS.md file effectively taxes your available context budget before you've written a single message — keeping it concise directly delays when compaction triggers.

Prompt caching lets providers skip recomputing tokens that exactly match the prefix of a previous request, charging a fraction of the normal rate for the cached portion. In an active coding session where history mostly just grows by appending, this means each new request only pays full price for the newly added tokens — until something like compaction changes the prefix and forces a full recompute on the next request.

This happens when a single turn — not just the cumulative history — pushes past the context limit before the agent's normal end-of-turn compaction check runs, for example during a turn with many tool calls or a large file load. Pi handles this by triggering compaction mid-turn rather than only checking after a turn completes, preventing the session from dying outright on an overflow.

Tool results and loaded file contents inside the compacted range are folded into the summary rather than preserved verbatim — only the retained recent messages keep their original tool outputs untouched. This means if you need an agent to recall an exact file diff or command output from far earlier in a session, that specificity may be lost to summarization unless it's still within the retention window.