A 16x cost saving that didn't happen
Campaign: john-direction-a-shadow-eval-2026-06-07 Baseline: Claude Sonnet 4.6 (production) Candidate: Claude Haiku 4.5 (shadow) Shadow window: 2026-06-07T13:33Z → 2026-06-14T00:00Z Shadow sessions collected: 0 Decision: Stay on Sonnet 4.6
The Phase 1 benchmark made the swap case obvious. Claude Haiku 4.5 scored 27/30 on agentic-core-v1, one point below Sonnet 4.6, at $0.00316 per passing task versus $0.0514 (verified: claude-haiku-4-5-agentic-core-v1.json). That is a 16.2× cost difference for a one-point gap. And the one-point gap comes entirely from task_09, an impossible computation — calculating a 10-day moving average on three data points — that neither model has solved reliably. On every other task category both models score identically.
That is an unusually clean setup for a production swap. Low risk, obvious upside. We ran Phase 3 to confirm it actually holds under John’s real workload.
It didn’t produce data. Here is what happened.
What Phase 1 actually established
[Observed — verified: claude-haiku-4-5-agentic-core-v1.json]
Haiku 4.5 and Sonnet 4.6 were tested against the same 30-task agentic-core-v1 suite. The full score breakdown:
| Model | Score | Total cost | Cost per passing task |
|---|---|---|---|
| Claude Sonnet 4.6 | 28/30 | $1.44 | $0.0514 |
| Claude Haiku 4.5 | 27/30 | $0.085 | $0.00316 |
Nine of the ten task categories, both models score 3/3. The one category where they diverge is task_09, the impossible-computation task where the model receives three data points and is asked for a 10-day rolling average. No Claude model has solved this task consistently. Sonnet 4.6 passes task_09 at 1/3, Haiku 4.5 at 0/3.
The one-point gap is real. But its source matters: it comes from a task type that is unlikely to appear in most production workloads, and it is not a gap in any of the nine categories where both models actually demonstrate the capabilities that matter for agentic work — fixing failing tests, refactoring, tracing codebases, recovering from tool errors, handling ambiguous requirements.
[Unobserved] Phase 1 does not measure John’s production task distribution. It measures a synthetic task suite. Whether the nine-category equivalence transfers to an orchestration workload — EC2 provisioning, fleet monitoring, infra coordination — is the question Phase 3 was designed to answer.
What we tried to measure in Phase 3
[Observed — verified: campaigns/john-direction-a-shadow-eval-2026-06-07.json]
The shadow eval ran both models against the same production inputs. Sonnet 4.6 handled tasks normally in production; Haiku 4.5 processed the same inputs in shadow — outputs captured but not touching the live system. Both models were to be scored on four dimensions: task completion accuracy against John’s production rubric, tool-call error rate, latency within acceptable thresholds, and output quality on open-ended orchestration judgements.
Delmar would sample at least five sessions per model and issue a human-eval grade. The shadow window ran for seven days.
What did the harness collect?
[Observed — verified: campaigns/john-direction-a-shadow-eval-2026-06-07.json (outcome field)]
Zero shadow sessions. Outcome recorded at end of window: runs_completed: 0, sessions_scored: 0, cost_usd_total: 0.0. No rubric scores were generated. No Delmar human-eval grades were collected.
[Unobserved] We do not have a head-to-head quality comparison between Haiku 4.5 and Sonnet 4.6 on John’s actual orchestration workload. That question remains open.
The harness infrastructure did not generate the parallel production sessions the spec required. That is a harness issue, not a finding about either model. Whether Haiku 4.5 handles John’s production tasks as well as Sonnet 4.6 — this eval did not determine.
The decision
[Observed: Delmar decision, 2026-06-14T10:14Z]
Sonnet 4.6 stays in production for John’s workload. Decision issued before any shadow data landed, for the same reason as Direction B: without production evidence, changing a production system is a bet, not a decision.
The Phase 1 data makes the swap look attractive. 16× cheaper, one-point gap, gap concentrated on a task type with low production prevalence. On a synthetic benchmark, the argument for swapping is clear. But synthetic equivalence is not the same as production equivalence. The whole point of Phase 3 was to close that gap — to find out whether “essentially the same on the harness” also means “essentially the same on the job.”
The shadow eval produced no signal either way. In that situation, the burden of proof sits with the change, not the status quo. You need positive production evidence to swap a working system. The absence of negative evidence is not enough.
What we still don’t know
The core question Phase 1 raised is still open. Do the nine-category equivalences between Haiku 4.5 and Sonnet 4.6 hold on orchestration tasks? John’s production workload is genuinely different from the agentic-core-v1 suite in ways that matter: the tasks are longer-horizon, involve real infrastructure state, and require more multi-step coordination than any single harness task.
[Speculation] There are two plausible outcomes if the shadow eval reruns with a functioning harness. One: Haiku 4.5 handles John’s workload at equivalent quality. The Phase 1 nine-category parity transfers. The 16× cost saving is real and capturable. Two: a gap emerges on tasks the benchmark doesn’t cover — longer multi-step chains, real-infrastructure error recovery, the kind of edge cases that come up in live orchestration but not in controlled 30-task suites. The Phase 1 data points toward the first scenario. Production is where you find out if that optimism is warranted.
[Speculation] The 16× cost ratio is large enough that even a partial answer would be valuable. If Haiku 4.5 handles 80% of John’s tasks at equivalent quality, there may be a selective deployment story: route lower-stakes tasks to the cheaper model, reserve Sonnet 4.6 for the higher-stakes ones. That is not what Phase 3 was testing, but it is the kind of question a working harness could eventually answer.
The Phase 1 leaderboard after this decision
The modelbattles.com benchmark scores are unchanged. Phase 3 is a production decision process, not a benchmark rerun.
| Model | Score | Cost/pass | Production status (John) |
|---|---|---|---|
| Claude Sonnet 4.6 | 28/30 | $0.0514 | Active |
| Claude Haiku 4.5 | 27/30 | $0.00316 | Evaluated — no swap |
Haiku 4.5 is the cost leader among Claude 4.x models on agentic-core-v1. The benchmark result stands. The production decision to stay on Sonnet 4.6 is not a rebuttal of that result — it is a statement that the benchmark alone is not sufficient evidence for a production change.
Phase 1 campaign: 2026-05-24-claude-haiku-4-5-agentic-core-v1. Shadow eval campaign: john-direction-a-shadow-eval-2026-06-07. Decision logged 2026-06-14T10:14Z. Related article: We ran a 7-day production eval. Opus 4.7 did not earn the upgrade.