#195 — Migrating production AI agents from one frontier model to another

Not without checking the eval harness first. In one head-to-head test between Claude Opus 4.8 and GPT-5.6, roughly a third of the raw failures traced back to test infrastructure biased toward the incumbent model, not actual quality gaps . One dataset had a silent default score threshold of 1.0, so a build scoring 0.98 was marked as a failure purely because nobody set the threshold explicitly . Always triage the actual traces, not just the pass rate, before trusting a benchmark.

Some models fill in every optional tool parameter with invented values instead of omitting unused ones, and a placeholder like offset: 0 is indistinguishable from a real one to your code . In production traces, this caused 52-64% of file reads from GPT-5.6 to come back empty while the tool still reported success: true, meaning the agent never learned it was working with blank data . The fix is a schema-level change, not a prompting fix: making every optional parameter required-but-nullable (anyOf: [T, null]) and stripping nulls before the tool executes .

Because it's often a structural behavior baked into how a model emits function calls, not a comprehension issue. Testing showed that tool-description hints telling the model to 'omit unused parameters,' per-field hints, and even the provider's official strict mode all produced identical behavior — the model kept sending every parameter regardless . The only fix that worked was transforming the tool schema itself at the API boundary.

Usually because of prompt caching, not the model's actual pricing. In one real migration, a new model looked 50% more expensive than the old one purely due to cache misconfiguration — once the caching setup was corrected, the same model ended up cheaper than the incumbent . If you're cost-comparing two models and one has a cold or misconfigured cache, you're comparing your setup, not the models.

Anthropic's caching can be org-wide and shared across any conversation or workspace with a single entry, reaching 92-96% hit rates with minimal setup . OpenAI's newer models dropped partial-prefix matching entirely, requiring explicit cache breakpoints plus a mandatory cache key, with each key limited to roughly 15 requests per minute before traffic spills to a separate, cold cache node . This means OpenAI-style caching requires deliberate key-scoping design that Anthropic's architecture doesn't.

Scope it per-workspace or per-customer, not per-conversation or globally. A per-conversation key produced a 0% first-call cache hit rate in testing, while a single global key overloaded the throughput cap and spilled traffic to unrelated cold caches . Workspace-scoped keys, combined with layered cache breakpoints (shared static layer, workspace layer, session layer), pushed first-call hit rates from near 0% to 83.7% .

It depends on the provider's cache architecture, and for some models it's structurally impossible. Because certain caching systems tie cache identity to a specific key rather than the whole organization, cross-tenant sharing of a static prompt prefix isn't achievable — teams instead budget for one small cold-write cost (around $0.18) per idle window, per tenant, as a bounded but unavoidable expense .

This typically happens when a model API replays prior-turn reasoning as a server-side reference rather than sending the full content back to you, and that server-side state can expire or go missing during a live conversation . The fix is requesting self-contained, encrypted reasoning content instead of relying on server-side pointers, which eliminates dependency on state you don't control .

If your setup relies on server-side reasoning or state replay, the effective prompt reaching the model can shift upstream of you even when the text you're sending is byte-for-byte identical and append-only . This is a known trap that looks like model nondeterminism but is actually a side effect of server-managed state — switching to self-contained reasoning replay removes this variable.

Fix the eval harness first, then audit tool-call schemas, then rebuild the caching architecture, then check reasoning or state replay behavior last . This sequence matters because each earlier step tends to mask or distort the problems in the steps after it — for example, a biased eval harness can hide real cost differences caused by cache misconfiguration.

Not automatically — model upgrades often trade one weakness for another. In one comparison, the newer model wrote dramatically leaner code (a 2,508-character stylesheet with 45 variables versus the older model's 17,957 characters and 174 mostly-unused variables) but showed a tendency to default to clean, generic layouts unless actively steered toward a specific brand's design system . Faster and cheaper doesn't guarantee stylistically superior output without additional steering work.

In one production comparison across 10-11 matched builds, the newer model averaged $2.22 per completed build versus $3.06 for the incumbent, a 27% cost reduction, while cutting wall-clock time from 8 minutes to 3 minutes 42 seconds, more than 2x faster . Output token usage nearly halved too, dropping from 33.0K to 17.1K tokens per build, which is often the bigger lever since output tokens typically cost more than input tokens .

It standardizes the API surface but doesn't eliminate provider-specific behavior underneath it. Even using a universal SDK, one team still had to discover through failed evals that tool-argument filling, prompt caching, and reasoning replay were all provider-specific behaviors their stack had quietly specialized around . A unified SDK reduces boilerplate, not the underlying architectural differences between model providers.

Effective visual evals use binary, specific checks rather than vague scoring — for example, ten yes/no questions like 'the hero is a full-bleed photographic scene' or 'primary CTAs are rounded rectangles, not pills' scored against a reference design, combined with content checks, tool-trajectory checks, and file assertions . Every failed case should be triaged against its full execution trace, not just the numeric score, since silent defaults (like an inherited minimum score threshold) can produce false failures .

One real-world comparison used 10-11 completed builds per model as an initial signal before rolling out a default model change, which is small enough to be directional rather than statistically definitive . Teams treating small-sample results as a green light should pair them with production monitoring after rollout, since the true failure modes (like tool schema handling or cache misconfiguration) often only surface at higher volume .

It refers to models storing their internal reasoning from a previous turn on the provider's servers and passing back only a reference ID for future turns, instead of returning the full reasoning content to your application . This creates a dependency on infrastructure you don't control — if that stored reference expires or isn't found, your agent throws mid-conversation errors, which is why requesting self-contained encrypted reasoning blobs is the more resilient production pattern .