#189 — How to effectively context engineer for AI agents

Prompt engineering is about how you write instructions. Context engineering is the broader discipline of deciding what information is in your model's context window at any given moment — system prompts, tools, examples, message history, external data, and everything else the model sees. Anthropic's applied AI team defines it as 'the art and science of curating what will go into the limited context window from that constantly evolving universe of possible information.' The shift matters because agents running in a loop generate increasing amounts of data across turns — context engineering is how you decide what survives each turn.

Yes — measurably. Context rot is the documented phenomenon where a model's ability to accurately recall and reason over information decreases as total token count increases, regardless of the model or provider. It's not a hard cliff but a performance gradient. The root cause is architectural: the transformer model creates n² pairwise relationships for n tokens, so as context grows, attention gets spread thinner. For production agents, this means a bloated context doesn't just waste money — it silently degrades output quality in ways that are hard to detect without systematic evals.

As short as it needs to be — and no shorter. Anthropic's guidance is to target the minimal set of instructions that fully defines expected behavior, organized into clearly labeled sections (Role, Tool guidance, Output format, etc.) using Markdown headers or XML tags. Two failure modes to avoid: (1) brittle if-else hardcoded logic for every edge case, which fragments easily in production; and (2) vague platitudes like 'be helpful' that falsely assume shared context. The practical approach: start with the smallest prompt that passes your test suite on your best model, then add instructions only to fix observed failure modes — not hypothetical ones.

As few as possible. The most common failure mode in production agents is bloated tool sets with overlapping functionality that create ambiguous decision points. Anthropic's test: if a human engineer can't immediately say which tool should be used in a given situation, an AI agent can't either. Tools should be self-contained, robust to error, and have minimal functional overlap — the same design principles you'd apply to a well-structured codebase. When in doubt, cut tools rather than add them, especially for long-horizon tasks where tool call history compounds in the context window.

Just-in-time retrieval means your agent maintains lightweight references — file paths, stored queries, URLs — and dynamically loads data at runtime rather than pre-processing everything upfront. Claude Code uses this approach for complex data analysis: instead of loading full database objects into context, the model writes targeted queries and uses Bash commands like head and tail to sample large files without bloating context. Use JIT when your data is dynamic or large-scale. Use embedding-based RAG when latency matters and data is relatively static (legal documents, finance reports). For most production agents, a hybrid strategy — dropping high-signal stable context (like a CLAUDE.md) upfront while letting the agent explore dynamically — outperforms either approach alone.

Three techniques work in practice: Compaction summarizes the conversation as it nears the context limit and reinitiates with a compressed version — preserving architectural decisions, unresolved bugs, and key state while discarding raw tool outputs. Structured note-taking lets the agent write progress to a persistent file (e.g., NOTES.md) and pull it back into context later. Sub-agent architectures delegate deep work to specialist agents with clean context windows; each may use tens of thousands of tokens exploring, but returns only a 1,000–2,000 token summary to the lead agent. Anthropic's Pokémon-playing Claude maintained coherent strategy across 1,234+ game steps using only self-directed note-taking — no special memory prompting required.

Compaction is the practice of summarizing a context window nearing its limit and restarting with the compressed version. The art is in deciding what to keep. Start by maximizing recall — your initial compaction prompt should capture every potentially relevant piece of information. Then iterate to improve precision by eliminating content that's genuinely superfluous. The easiest wins: clear old tool call results from message history once a tool call is deep in the conversation — there's rarely a reason to keep the raw output. Anthropic has shipped tool result clearing as a native feature on the Claude Developer Platform, making this the lightest-touch compaction lever available.

Yes — Anthropic's applied AI team still considers few-shot examples one of the highest-leverage prompt engineering techniques available. The mistake most teams make is using them to enumerate edge cases: a laundry list of 'if X, then Y' rules stuffed into the prompt. Don't. Instead, curate a small set of diverse, canonical examples that portray the full range of desired behavior. For LLMs, examples are the pictures worth a thousand words — they communicate expected outputs more efficiently than any rule set, and they're far less brittle. Fewer, better examples consistently outperform comprehensive rule lists.

A sub-agent architecture uses specialized agents with clean context windows to handle focused subtasks, reporting condensed summaries back to a coordinating lead agent. Each sub-agent might consume 10,000+ tokens exploring a problem space but returns only 1,000–2,000 tokens to the lead agent — keeping the orchestrator's context lean and strategic. Anthropic's internal multi-agent research system showed substantial performance improvements over single-agent systems on complex research tasks using this exact pattern. Use sub-agents when tasks involve parallel exploration, deep technical work across isolated domains, or any workflow where one agent maintaining full state across the entire project would result in compounding context pollution.

More than most teams realize. To an agent navigating a file system, a file named test_utils.py in a /tests/ folder implies a fundamentally different role than the same file sitting in /src/core_logic/. Folder hierarchies, naming conventions, and timestamps are all context signals — they help agents understand how and when to use information without you having to explicitly state it. This is called progressive disclosure: agents build understanding layer-by-layer through exploration, assembling relevant context incrementally rather than being handed a pre-loaded dump. Good repo hygiene and consistent naming conventions aren't just engineering best practices — they're a lever for making your agents smarter.

Not reliably. Anthropic's engineering team is explicit on this: context windows of all sizes are subject to context pollution and information relevance degradation, especially for tasks where strongest agent performance is required. Larger windows buy more headroom, but they don't fix the fundamental attention budget problem — they just push it further out. Teams that wait for bigger context windows instead of implementing context engineering practices will find themselves facing the same coherence and precision issues at higher token counts. Engineering context quality is a more reliable path to performance gains than simply expanding the window.

MCP (Model Context Protocol) is an open standard developed by Anthropic that lets AI agents connect to external tools and data sources in a standardized, composable way — without requiring custom integrations for every data source. In context engineering terms, MCP is the infrastructure layer that enables just-in-time context retrieval at scale: rather than pre-loading everything or writing bespoke connectors, agents can call MCP servers to pull precisely what they need, when they need it. For teams building multi-tool or multi-agent systems, adopting MCP early reduces integration debt and keeps your tool layer token-efficient.

Every LLM has a finite capacity to form meaningful relationships between tokens in its context window. Anthropic describes this as an 'attention budget' — every token you add depletes some of that budget, whether or not the token is useful. This is why irrelevant tokens aren't neutral: they're an active tax on the tokens that do matter. For builders, this means the quality of what you put in the context window matters more than the quantity. A 5,000-token context with 90% signal will consistently outperform a 50,000-token context of mixed relevance — on accuracy, on coherence, and on cost.

Progressive disclosure is the agent design pattern where an agent builds its understanding of a task incrementally through exploration — rather than being front-loaded with exhaustive context at initialization. Agents navigate their environment autonomously, each interaction surfacing new context that informs the next decision: file sizes signal complexity, naming conventions hint at purpose, timestamps serve as proxies for relevance. Claude Code implements this natively, using tools like glob and grep to navigate codebases dynamically rather than ingesting entire syntax trees upfront. The result is a focused, high-signal context window that scales to large and dynamic environments without ballooning token usage.

A CLAUDE.md file is a persistent, plain-text context anchor dropped into your project root that gets naively loaded into the agent's context at initialization — before any dynamic exploration begins. Think of it as the one document a new engineer on your team would read on day one: project architecture, key decisions, environment setup, conventions, and anything the agent should never have to rediscover. Anthropic's Claude Code uses this as the 'up-front retrieval' layer in its hybrid context strategy, pairing stable CLAUDE.md context with dynamic JIT retrieval for everything else. Keep it focused and factual — narrative prose and excessive edge-case caveats waste attention budget on tokens that rarely fire.

These are related but distinct mechanisms. Compaction is a reactive operation: when a context window nears its limit, you summarize the conversation and restart with the compressed version — used to extend task continuity in a single session. Summarization is a technique used within compaction or as a standalone step to distill verbose outputs (tool results, research dumps) before they enter or persist in context. Memory (or structured note-taking) is a proactive, persistent pattern: the agent writes state to an external file (NOTES.md, a database, Anthropic's memory tool) that survives context resets entirely and gets pulled back in as needed. Compaction handles context overflow; memory handles cross-session and long-horizon persistence. Most production agents need both.

The diagnostic is straightforward: test the same task with a clean, minimal context window. If performance improves dramatically on a short, focused interaction versus a long agent loop, you have a context problem, not a model problem. Other signals of context engineering failure: the agent frequently re-asks for information it was already given; reasoning quality degrades over a multi-turn session; the model ignores specific instructions that appeared early in a long prompt; outputs become more generic the longer the session runs. These are all attention budget symptoms. Model capability problems look different — they manifest as task failures even on the first turn with clean, well-curated context.

Anthropic's applied AI team recommends this sequencing: Start with the smallest system prompt that produces correct behavior on your best model. Add instructions only to fix observed failures — not anticipated ones. Then design your tool set with zero overlap, test that a human can unambiguously route any task to the right tool, and cut everything that fails that test. Only once those two layers are clean should you invest in memory, compaction, or sub-agent architectures — which are complexity multipliers that amplify good foundations and entrench bad ones. The trap most teams fall into is building infrastructure for scale before their core context is clean.