#188 — Why your AI agent is broken and you don't know it yet
July 5, 2026·12 min read
Contents
Note: If you're new to evals, start with our primer on evals.
Why it matters: Most founders building on LLMs test manually, ship, wait for user complaints, then scramble. That reactive loop is a slow death: fix one failure, create another, and your team can never tell the difference between a real regression and noise. The name for the fix is evaluations (evals). And their value is easy to miss — costs are visible upfront, benefits compound later.
The Vocabulary You Need
Before you build anything, align your team on these terms:
- Task — A single test with defined inputs and success criteria (also called a problem or test case)
- Trial — One attempt at a task. You run multiple trials per task because model outputs vary between runs
- Grader — Logic that scores some aspect of the agent's output. A task can have multiple graders
- Transcript — The complete record of a trial: all tool calls, reasoning, intermediate results, and outputs. For the Anthropic API, this is the full messages array
- Outcome — The final state in the environment, not just what the agent said. A flight-booking agent saying "your flight is booked" is not the outcome — the reservation in the SQL database is
- Evaluation harness — The infrastructure that runs evals end-to-end: provides tools, runs tasks concurrently, records steps, grades outputs, and aggregates results
- Agent harness (scaffold) — The system that enables a model to act as an agent. When you evaluate "an agent," you're evaluating the harness and model together
- Evaluation suite — A collection of tasks designed to measure specific capabilities or behaviors
The Anatomy of an Eval
An eval is conceptually simple: give an agent an input, apply grading logic to its output, measure success.
Single-turn evals are straightforward — prompt, response, grade. But agents are multi-step and stateful, which means mistakes compound across turns. A coding agent that misreads a spec on turn 2 may write broken tests on turn 8 with no obvious link. That's what makes agent evals genuinely hard — and why most founders skip them until it's too late.
One important edge case: frontier models can find creative solutions that technically "fail" a rigid eval but actually produce better outcomes. Anthropic's Opus 4.5 once solved a flight booking problem by finding a legitimate policy loophole — the eval marked it a failure. Your graders need to accommodate valid paths your eval designers didn't anticipate.
The Two Eval Types That Matter
| Type | Question it answers | Starting pass rate | Purpose |
|---|---|---|---|
| Capability / Quality evals | "What can this agent do well?" | Low — intentionally | Gives your team a hill to climb |
| Regression evals | "Does it still do what it used to?" | Near 100% | Protects against backsliding |
Once a capability eval reaches saturation (near 100% pass rate), it graduates into a regression suite — tasks that once measured "can we do this at all?" now measure "can we still do this reliably?"
The Three Grader Types
Code-based graders
Fast, cheap, objective, and reproducible. Best for pass/fail correctness checks.
- String match (exact, regex, fuzzy)
- Binary tests (fail-to-pass, pass-to-pass)
- Static analysis (lint, type checking, security scans)
- Outcome verification (did the record actually appear in the DB?)
- Tool call verification (did it use the right tools with the right parameters?)
- Transcript analysis (how many turns, how many tokens?)
Watch out for: Too brittle for open-ended tasks — they penalize valid solutions that don't match a narrow expected pattern.
Model-based graders (LLM-as-judge)
Flexible and scalable. Use rubric-based scoring, natural language assertions, pairwise comparison, reference-based evaluation, or multi-judge consensus.
Critical rule: LLM graders must be calibrated against human experts. They're non-deterministic and will drift. Give the LLM a way out — instruct it to return "Unknown" when it doesn't have enough information to grade confidently. Grade each dimension of a task with an isolated LLM call rather than asking one judge to grade everything at once.
Watch out for: False confidence if calibration lapses.
Human graders
The gold standard. Use for SME review, crowdsourced judgment, spot-check sampling, A/B tests, and inter-annotator agreement. Primary role in practice: calibrating your LLM graders and handling tasks too subjective for code or models.
Watch out for: Expensive and slow. Reserve for calibration runs, not as your primary grading mechanism.
For tasks with multiple components, build in partial credit. A support agent that correctly identifies a problem and verifies the customer but fails to process the refund is meaningfully better than one that crashes immediately — your scoring should reflect that continuum.
Eval Design by Agent Type
Coding Agents
Software is naturally measurable: does it run, do the tests pass? Use deterministic graders as your foundation.
The key gotcha: if a task asks the agent to write a script but doesn't specify a filepath, and your tests assume a specific path, the agent fails through no fault of its own. Ambiguity in task specs becomes noise in your metrics. A 0% pass rate across many trials is almost always a broken task, not a broken agent.
Useful grader stack for coding: unit tests for correctness, LLM rubric for code quality, static analysis for security/lint, state checks for environment outcomes, tool call verification.
Conversational Agents
Success is multidimensional and the quality of the interaction itself is part of what you're measuring. Unlike coding agents, you often need a second LLM to simulate the user.
Grade across all three axes simultaneously: was the ticket resolved? (state check), did it finish in under 10 turns? (transcript constraint), was the tone appropriate? (LLM rubric).
Research Agents
The hardest to evaluate because "comprehensive" and "well-sourced" are context-dependent. Combine:
- Groundedness checks — Are claims supported by retrieved sources?
- Coverage checks — Does the answer include the key facts it must?
- Source quality checks — Are the sources authoritative, not just first-retrieved?
- LLM rubric — Coherence and completeness for open-ended synthesis
LLM-based rubrics for research agents should be calibrated against expert human judgment more frequently than for other agent types, because the subjective surface area is much larger.
Computer Use Agents
These interact through screenshots, mouse clicks, and keyboard — not APIs. The challenge is balancing token efficiency with latency. DOM-based interactions are fast but token-heavy; screenshot-based interactions are slower but more efficient at scale. Build evals specifically to verify your agent is selecting the right interaction method for each context.
Evaluate by checking real environment state: file system, application configs, database contents, UI element properties — not just whether a confirmation page appeared.
The Metrics That Tell Opposite Stories
Two metrics every founder building customer-facing agents needs to understand:
pass@k — Did the agent get at least one correct answer in k attempts? As k increases, this score rises. A score of 50% pass@1 means the agent succeeds at half of tasks on the first try. Use this for tools where any valid solution counts.
pass^k — Did the agent succeed on all k trials? As k increases, this score falls. If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%. Use this for customer-facing agents where users expect consistent, reliable behavior every time.
At k=1 they're identical. By k=10, they tell completely opposite stories: pass@k approaches 100% while pass^k approaches 0%. Which metric you optimize for depends entirely on your product requirements.
The 8-Step Playbook
Step 0: Start before you think you're ready
You don't need 500 test cases. Twenty to fifty tasks from real failures is a great start. Early in development, each change has a large, noticeable impact — small sample sizes are enough. Evals get harder to build the longer you wait. Early on, product requirements naturally translate into test cases. Wait too long and you're reverse-engineering success criteria from a live system.
Step 1: Start with what you already test manually
Begin with the manual checks you run before each release. If you're in production, go to your bug tracker and support queue. Convert user-reported failures into test cases. Prioritize by user impact to invest effort where it counts.
Step 2: Write unambiguous tasks with reference solutions
A good task is one where two domain experts, working independently, would reach the same pass/fail verdict. Ambiguity in task specs becomes noise in metrics. The same applies to rubrics for LLM graders — vague criteria produce inconsistent judgments.
For every task, create a reference solution: a known-working output that passes all graders. This proves the task is solvable and verifies your graders are configured correctly.
Step 3: Build balanced problem sets
Test both the cases where a behavior should occur and where it shouldn't. One-sided evals create one-sided optimization. If you only test whether the agent searches when it should, you may end up with an agent that searches for almost everything. Build coverage in both directions and actively avoid class-imbalanced test sets.
Step 4: Build a robust, isolated eval harness
Each trial must start from a clean environment. Shared state between runs — leftover files, cached data, resource exhaustion — causes correlated failures that reflect infrastructure flakiness, not agent performance.
One specific failure mode to watch: if your agent can examine git history from previous trials, it gains an unfair advantage on subsequent tasks. Make sure trials are genuinely isolated.
Step 5: Design graders thoughtfully
Grade what the agent produced, not the path it took. Checking that your agent called tools in a specific sequence is brittle — agents regularly find valid approaches that eval designers didn't anticipate.
Make your graders resistant to bypasses. Tasks and graders should be designed so that passing genuinely requires solving the problem, not exploiting a loophole. This is harder than it sounds: Opus 4.5 initially scored 42% on CORE-Bench until a researcher discovered that rigid grading penalized "96.12" when the expected answer was "96.124991…" After fixing bugs and using a less constrained scaffold, the score jumped to 95%.
Step 6: Read the transcripts
You will not know if your graders are working unless you read them. When a task fails, the transcript tells you whether the agent made a genuine mistake or whether your grader rejected a valid solution. Failures should seem fair: it must be clear what the agent got wrong and why. When scores don't climb, you need confidence it's the agent's performance — not a broken eval.
Step 7: Watch for eval saturation
An eval at 100% pass rate tracks regressions but gives no signal for improvement. As an eval approaches saturation, large capability improvements appear as small score increases — deceptive and demoralizing. When this happens, build harder tasks. Startups have gotten burned by this: Qodo was initially unimpressed with Opus 4.5 because their one-shot coding evals didn't capture gains on longer, more complex tasks — they had to build a new agentic eval framework to see the full picture.
Step 8: Treat evals like unit tests — they're a living artifact
Establish clear ownership. At Anthropic, dedicated evals teams own the core infrastructure while domain experts and product teams contribute most tasks and run the evaluations themselves.
Practice eval-driven development: build evals to define planned capabilities before the agent can fulfill them, then iterate until the agent performs well. This also makes capability bets visible — capability evals that start at a low pass rate tell you exactly which bets paid off when a new model drops.
Beyond Automated Evals: The Full Stack
Automated evals are your first line of defense, but they're one layer of a complete picture. Think of this like the Swiss Cheese Model from safety engineering — no single layer catches everything.
| Method | Best for | The catch |
|---|---|---|
| Automated evals | Pre-launch, CI/CD, every model upgrade | Can create false confidence if tasks don't match real usage |
| Production monitoring | Detecting distribution drift post-launch | Reactive — problems reach users first |
| A/B testing | Validating significant changes with real traffic | Slow; needs sufficient traffic to reach significance |
| User feedback | Surfacing problems you didn't anticipate | Sparse, self-selected, skews toward severe issues |
| Manual transcript review | Building intuition for failure modes | Doesn't scale, inconsistent coverage |
| Systematic human studies | Calibrating LLM graders, subjective tasks | Expensive and slow; use sparingly |
The recommended operating cadence: Run automated evals on every agent change and model upgrade. Monitor production constantly post-launch. Sample transcripts weekly. Run A/B tests for significant changes once you have traffic. Reserve systematic human studies for calibrating LLM graders.
The Competitive Moat Nobody Talks About
When a new model drops — and they drop constantly — teams without evals spend weeks validating manually. Teams with evals run the suite, find the regressions, tune their prompts, and upgrade in days.
That's a compounding advantage. The teams stuck in reactive loops can never move as fast because every change is a risk they can't quantify. The teams with evals have something rare in early-stage startups: a clear, shared definition of what "better" means — and the tooling to prove it.
The Starting Point
You don't need a framework. You don't need a team. You need:
- 20 tasks sourced from your bug tracker and support queue
- Unambiguous success criteria two people would agree on
- Balanced coverage — both the should-do and should-not-do cases
- A reference solution for each task that proves it's solvable
- A grader — start with code-based where you can, LLM rubric where you must
- A transcript reader — someone on the team who reads failures before every release
That's it. Start there. Everything else scales from this foundation.
Frequently asked questions
How many test cases do I actually need to start running evals on my AI agent?
You don't need hundreds. 20–50 tasks drawn from real user failures is enough to start — and that's exactly what Anthropic recommends for early-stage teams. In early development, each change has a large, noticeable impact, which means small sample sets produce statistically meaningful signal. The bigger risk is waiting: the longer you delay, the harder it becomes to reverse-engineer success criteria from a live system.
What's the difference between a capability eval and a regression eval?
Capability evals measure what your agent can do — they should start with a low pass rate and give your team a clear hill to climb. Regression evals measure whether it still does what it used to — they should sit near 100% at all times. Once a capability eval reaches saturation (near 100%), it graduates into the regression suite. Teams like Descript and Bolt each run both suites in parallel: one for benchmarking quality, one for catching drift.
Can I use an LLM to grade my AI agent's outputs instead of writing tests manually?
Yes — and most production teams do. Model-based graders (LLM-as-judge) are flexible, scalable, and handle open-ended outputs that code-based tests can't. The critical rule: always calibrate your LLM grader against human expert judgment before trusting it. Vague rubrics produce inconsistent scores. Build structured, dimension-specific rubrics, give the judge a way out (instruct it to return 'Unknown' when uncertain), and calibrate regularly — especially for research or conversational agents where subjective surface area is large.
What is pass@k vs pass^k and why does it matter for AI agent evals?
pass@k asks: did the agent succeed at least once in k tries? As k increases, this rises — it's the right metric for tools where any valid solution counts. pass^k asks: did the agent succeed every single time across k tries? As k increases, this falls — it's the right metric for customer-facing agents where users expect reliability. At k=1 they're identical. By k=10 they tell opposite stories: if your agent has a 75% per-trial success rate, pass^3 is only (0.75)³ ≈ 42%. Choose your metric based on what your product actually requires.
Why is my AI agent's eval score low even though the agent is clearly working correctly?
This is more common than you'd think — and almost always a broken eval, not a broken agent. Anthropic found that Claude Opus 4.5 initially scored 42% on CORE-Bench because rigid grading penalized '96.12' when it expected '96.124991…', tasks were ambiguous, and the scaffold constrained the model. After fixing grader bugs and clarifying task specs, the score jumped to 95%. A 0% pass rate across many trials is nearly always a signal to audit your task specification and graders, not your agent.
Should AI agent evals check the steps the agent took, or just the final output?
Grade the outcome, not the path. Checking that an agent called tools in a specific sequence is brittle — agents regularly find valid approaches eval designers didn't anticipate. Anthropic's Claude Opus 4.5 once solved a flight-booking task on τ2-bench by discovering a legitimate policy loophole. The eval marked it a failure, but the agent had found a genuinely better solution for the user. Build graders around what the agent produced and the final state of the environment, not the exact route it took.
How do I evaluate a conversational AI agent differently from a coding agent?
Coding agents have natural binary graders — does the code run, do the tests pass? Conversational agents require multidimensional grading across three axes simultaneously: was the goal completed (state check), did it finish within a turn budget (transcript constraint), and was the tone appropriate (LLM rubric)? You also typically need a second LLM to simulate the user. Unlike coding evals, many conversational tasks have multiple valid resolutions, so model-based graders are the foundation rather than an add-on.
How do I prevent my AI agent from cheating on its own evals?
Isolate every trial by starting from a clean environment. Leftover files, cached data, or access to git history from previous runs can give your agent an unfair advantage — Anthropic observed Claude gaining measurable performance boosts on certain internal evals by reading prior trial history. Each trial must be genuinely isolated. Beyond that, design tasks and graders so that passing genuinely requires solving the problem: if an agent can hit 100% by exploiting a loophole rather than doing the work, your eval is measuring the wrong thing.
What eval framework should I use to get started with AI agent testing?
The framework matters far less than the quality of your test cases. Harbor is strong for containerized, cloud-scale agentic evals and ships popular benchmarks like Terminal-Bench 2.0 out of the box. Braintrust combines offline evals with production monitoring and experiment tracking. LangSmith integrates tightly with LangChain; Langfuse is the self-hosted open-source alternative for teams with data residency requirements. Arize Phoenix adds open-source LLM tracing and debugging. Many teams start with simple evaluation scripts and add infrastructure only when scale demands it.
How do I know when my eval suite is no longer useful?
When it hits 100% pass rate consistently, it stops giving you improvement signal — this is called eval saturation. Qodo, the code review startup, was initially unimpressed with Claude Opus 4.5 because their one-shot coding evals were already saturated and couldn't detect gains on longer, more complex tasks. They had to rebuild their eval framework around agentic, multi-step tasks to see the real picture. As a rule: when the hardest tasks in your suite feel easy for the model, it's time to build harder tasks — not to celebrate a solved problem.
When in the startup lifecycle should I build evals — pre-launch or after I have users?
The earlier, the better — but there's a cost to waiting either way. Pre-launch evals force your team to write down what 'good' actually means, resolving spec ambiguities that two engineers would otherwise interpret differently. Post-launch, your bug tracker and support queue give you the highest-signal source of test cases: real failures from real users. Claude Code started with fast iteration based on employee feedback, then added evals incrementally — first for narrow behaviors like concision, then for complex behaviors like over-engineering. Both paths work; the key is treating evals as a core artifact, not a future project.
How should I evaluate an AI agent that controls a computer or browser?
Verify real environment state — not just whether a confirmation screen appeared. Benchmarks like WebArena and OSWorld check actual outcomes: was the order placed in the database, was the file saved, did the application config change? Beyond outcome verification, computer use agents require a deliberate balance between DOM-based interactions (fast but token-heavy) and screenshot-based interactions (slower but token-efficient). Anthropic built dedicated evals for Claude for Chrome specifically to verify the agent was selecting the right interaction method for each context — this alone improved both speed and accuracy meaningfully.
What is an AI agent eval harness and do I need to build one?
An eval harness is the infrastructure that runs evaluations end-to-end: it provides tools and instructions to the agent, runs tasks concurrently, records every step in a transcript, grades outputs, and aggregates results. You don't necessarily need to build one from scratch. Harbor, Braintrust, and LangSmith all provide harness infrastructure. The critical distinction is between the eval harness (the test runner) and the agent harness (the scaffold that lets your model act as an agent). When you evaluate your agent, you're evaluating both together — which means harness bugs can masquerade as agent failures.
What does 'outcome' mean in AI agent evaluation and why does it matter?
The outcome is the final state of the environment — not what the agent said it did. A flight-booking agent saying 'Your flight has been booked' is not the outcome. The outcome is whether a reservation exists in the SQL database. This distinction is critical because agents can produce convincing-sounding outputs without actually completing a task. Always verify state in the environment (database records, file system changes, API confirmations) rather than grading the agent's self-reported success. This is the most common hidden failure mode in production AI agents.
How do I write AI agent test cases that two engineers would agree on?
A good task passes what Anthropic calls the 'two domain experts' test: two subject-matter experts, working independently, should reach the same pass/fail verdict without discussion. If they'd disagree, the task needs refinement. In practice: define exactly what the grader will check, create a reference solution (a known-working output that passes all graders), and verify that everything the grader measures is explicitly clear from the task description. When auditing Terminal-Bench, Anthropic found tasks that asked agents to write a script without specifying a filepath — but the grader assumed a specific path. The agent failed through no fault of its own.
What is eval-driven development for AI agents?
Eval-driven development means writing evals to define planned capabilities before your agent can fulfill them, then iterating until the agent performs well — the AI equivalent of test-driven development (TDD). At Anthropic, teams often build features that work 'well enough' today but are explicit bets on what future models will enable. Capability evals that start at a low pass rate make those bets visible and measurable. When a new model drops, running the suite immediately reveals which bets paid off — without weeks of manual validation.
How do evals help when upgrading to a new AI model like Claude Opus 4.5 or GPT-5?
Teams without evals face weeks of manual validation every time a new model ships. Teams with evals run the suite, find regressions, tune prompts, and upgrade in days. This is a compounding competitive advantage: the gap widens with every model release. Practically, you run your regression suite first (it should stay near 100%), then your capability suite (it may improve significantly). Bolt built their eval system in 3 months and now uses static analysis, browser agents, and LLM judges to validate every model upgrade systematically — rather than waiting for production complaints.
How do production monitoring and automated evals work together for AI agents?
They're complementary, not interchangeable. Automated evals are your pre-launch and CI/CD defense — they run on every agent change and model upgrade, catching regressions before users see them. Production monitoring is your post-launch ground truth — it reveals real user behavior, distribution drift, and failure modes your evals didn't anticipate. The recommended cadence: run automated evals on every commit, monitor production constantly, sample transcripts weekly, run A/B tests for significant changes once you have traffic, and reserve systematic human review for calibrating LLM graders.
What is partial credit grading in AI agent evals and when should I use it?
Partial credit grading scores agents on a continuum rather than binary pass/fail when a task has multiple components. A support agent that correctly identifies a problem and verifies the customer identity but fails to process the refund is meaningfully better than one that crashes on the first turn — your eval should reflect that difference. Use partial credit whenever a task has multiple independently graded components (e.g. identity verification + problem diagnosis + resolution + tone) and when you need to measure direction of improvement, not just whether an agent crosses a threshold.
What is a transcript in AI agent evaluation and why should I read them?
A transcript (also called a trace or trajectory) is the complete record of a single trial: every tool call, piece of reasoning, intermediate result, and response the agent produced. For the Anthropic API, this is the full messages array at the end of a run. Reading transcripts is how you verify your eval is measuring what actually matters. When scores plateau, transcripts tell you whether the problem is the agent or the grader. Anthropic invests in dedicated tooling for transcript review and treats it as a non-negotiable part of the eval process — not an optional deep-dive.
What does a balanced AI agent eval problem set mean and why does it matter?
A balanced problem set tests both when a behavior should occur and when it shouldn't. One-sided evals create one-sided optimization: if you only test whether your agent searches when it should, you may end up with an agent that searches for almost everything. Anthropic learned this building search evals for Claude.ai — preventing over-triggering (searching when it shouldn't) while preserving appropriate search behavior required building evals covering both directions and many rounds of refinement. Actively avoid class-imbalanced test sets where one outcome dominates.
How should product managers and non-engineers contribute to AI agent evals?
Product managers, customer success managers, and salespeople are the best-positioned people to define what 'good' looks like — not engineers. At Anthropic, domain experts and product teams contribute most eval tasks, while dedicated evals teams own the core infrastructure. With current model capabilities, a product manager can use Claude Code to contribute an eval task as a pull request. Anthropic actively encourages this. The process of writing a concrete eval task is also one of the best ways to stress-test whether product requirements are specific enough to start building against.
Keep reading
#189 — How to effectively context engineer for AI agents
How you feed information to your AI agents matters more than the prompts you write. This single insight is reshaping how the best AI-native teams build.
#190 — Agent skills are context management, not magic
Agent skills are not a new capability. Their value comes from routing and progressive disclosure, not from smarter prompts.
#191 — Canada's AI strategy: The founder's playbook
Ottawa recently dropped its national AI playbook, and if you're building an AI-native company, there's real money and market signal buried in the policy-speak.