Sonnet 5 matches Opus 4.8 on frontier-eval-v1. The cost difference is $136.

July 2, 2026 · campaign-reports

Campaign: 2026-07-02-claude-sonnet-5-frontier-eval-v1
Model: global.anthropic.claude-sonnet-5 (AWS Bedrock, eu-west-1)
Harness: frontier-eval-v1 (7 tasks × 5 runs = 35 runs)
Date: 2026-07-02

Why Sonnet 5?

When Opus 4.8 ran frontier-eval-v1 in June, it scored 25/35 and cost $166.26 (verified: verification/cost_breakdown.csv). That established the frontier-eval-v1 baseline — 71.4% on the harness’s hardest seven tasks, with two task failures that appear to be structural rather than fluky.

The next logical question: is Opus 4.8’s performance tier-locked, or can the mid-tier Anthropic model replicate it?

Sonnet 5 is the answer to that question. It sits below Opus in the Anthropic model family, costs significantly less per token, and has never run frontier-eval-v1 before this campaign. If it matches Opus 4.8, the cost case for running Opus on agentic workloads collapses.

Pre-campaign predictions were filed at campaigns/claude-sonnet-5-frontier-eval-v1.predictions.md before the run.

What frontier-eval-v1 asks models to do

frontier-eval-v1 has seven tasks, each run five times. The tasks are longer-horizon than agentic-core-v1: multi-file migrations, autonomous debug campaigns, dependency upgrades, chart analysis, and pipeline work with self-monitoring. A run passes when the model produces a correct output before hitting the turn limit. A run fails as wrong_answer when the model submits an incorrect output.

What a pass requires varies by task. task_01 needs all 13 tests green after migrating an 18-file Python 2.7 codebase to Python 3.11. task_04 needs all 8 bugs fixed across 6 files the model hasn’t seen before. task_05 needs a working Express 4 to Express 5 upgrade on a 25-file Node.js scaffold. These are not simple tool-call tasks — they require the model to sustain a coherent plan across dozens of steps.

On Opus 4.8, task_03 and task_05 failed. On Sonnet 5, task_03 and task_05 failed. Everything else passed.

The score

[Observed — verification/pass_rate_by_task.csv, verification/cost_breakdown.csv, verification/tool_calls_by_task.csv, verification/latency_distribution.csv]

Same pass rate. $136.96 less.

What Sonnet 5 did on the passing tasks

[Observed — evidence/long_tail_turn_count.md, verification/tool_calls_by_task.csv]

task_01: Python 2 to 3 migration

Five passes. 208 seconds average, 42 tool calls average, $1.07/run. Methodical approach — the model works through the 18-file codebase sequentially, patches imports and syntax, and validates against all 13 tests before finishing.

The Opus 4.8 comparison is instructive. Opus averaged 59.4 tool calls and 186.6 seconds per run on this task; Sonnet 5 averages 42 calls and 208 seconds. Fewer tool calls, slightly more clock time, same outcome. Whether that reflects a different exploration strategy or just run variance is not distinguishable from turn counts alone.

21 of 35 runs across the campaign appear in the long-tail turn count evidence bundle (verified: evidence/long_tail_turn_count.md). task_01 and task_07 in particular run 40+ tool calls per session.

task_04: Autonomous debug campaign

Five passes. 163 seconds average, 35 tool calls average, $0.60/run. Eight bugs across six files. Opus 4.8 averaged 45.6 calls on this task; Sonnet 5 uses 35.

task_02, task_06, task_07

task_02 (large codebase comprehension): 5/5, 12 tool calls average, $0.19/run. The cheapest scoreable task — read-heavy, low output.

task_06 (cross-file utility extraction): 5/5, 24 tool calls average, $0.41/run.

task_07 (pipeline with self-monitoring): 5/5, 24 tool calls average, $0.78/run.

The two failures

[Observed — verification/pass_rate_by_task.csv]

task_03: chart analysis pack

0/5, wrong_answer across all five runs. Total cost: $0.49. Average cost per run: $0.10. The model exits quickly and confidently incorrect.

Sonnet 5 shows the same failure mode as Opus 4.8 on this task. Opus 4.8’s task_03 result was an infrastructure failure — the harness wasn’t delivering image payloads correctly, and runs terminated at $0.006 average. That bug was fixed in TASK-636 (PR #175, 2026-06-13). Sonnet 5 is running on the fixed harness. The model receives the task and fails it on content, not on infrastructure.

[Speculation] Both Anthropic models — Sonnet 5 and Opus 4.8, post infrastructure fix — appear to fail chart analysis on this harness. Whether this is a chart-interpretation ceiling in the current Anthropic family, a harness vision pipeline issue, or something specific to these five chart types is not distinguishable from wrong_answer outcomes alone. Follow-up test with a non-Anthropic model would help triangulate.

[Unobserved] Whether Sonnet 5 passes chart analysis tasks under different conditions. We have 5 data points: 5 failures with identical failure mode.

task_05: dependency major-version upgrade

0/5, wrong_answer across all five runs. Total cost: $13.59. Average: $2.72/run.

This is the most expensive task in the campaign by a significant margin. Average latency: 473 seconds. Average tool calls: 81 per run. Average input tokens across five runs: 6.09M total. The model works hard — 81 tool calls at 473 seconds is a model doing real work, not giving up early — and still can’t close.

Opus 4.8 also failed this task, at $15.70/run before the original campaign stopped at the budget ceiling. Sonnet 5 fails faster and cheaper: $2.72/run versus $15.70/run for the same wrong_answer result. That’s cold comfort, but it matters for debugging economics. If you’re testing whether a dependency upgrade task is solvable, failing at $2.72 beats failing at $15.70.

[Observed — evidence/tool_call_redundancy.md] The tool call redundancy forensics flagged 3 of 35 runs in this campaign showing back-to-back identical tool calls — 1 run in task_01, 2 runs in task_04. None occurred in task_05. Whatever is driving task_05’s failures, redundant re-reads aren’t the mechanism; the high input token load there (6.09M across five runs) comes from something else. What that something else is isn’t answerable from this data.

The pattern we looked for and didn’t find

[Unobserved — evidence/diagnosis_then_regression.md]

On longer tasks, some models commit to a plan, hit a wall mid-task, reverse their earlier diagnosis, and retry from scratch. This tends to produce runs that spiral rather than converge.

The forensics scanner ran against all 35 transcripts for this pattern. Zero instances found. Zero. The model commits and follows through.

This is consistent with Opus 4.8’s result on the same harness — also zero. Whether this is a feature of frontier-eval-v1 task design (tasks where reversals are expensive and the model learned not to) or a behavioral property of current Anthropic models is not answerable from two campaigns on the same harness.

Predictions scoring

Predictions were filed at campaigns/claude-sonnet-5-frontier-eval-v1.predictions.md before the campaign ran.

The miss on task_03 is the most interesting result from a predictions standpoint. The prediction assumed Sonnet 5 might show better multimodal chart performance than Opus 4.8 — that the Opus task_03 failure was partly infrastructure, and a fixed harness would reveal meaningful capability. That was wrong. Sonnet 5 fails chart analysis on the fixed harness just as cleanly as Opus 4.8 does post-fix. The chart analysis failure may not be a harness issue at all.

The cost picture

[Observed — verification/cost_breakdown.csv]

Total campaign input tokens: 12,281,438 (verified: verification/cost_breakdown.csv). That is roughly 12% more than Opus 4.8’s estimated ~11M — Sonnet 5 generates more input load, possibly from its exploration patterns, but the per-token cost difference dominates the total.

Opus 4.8 at $166.26. Sonnet 5 at $29.30. The delta is $136.96 (82% lower).

What the leaderboard looks like now

frontier-eval-v1 now has two Anthropic data points:

Two models, same score, 5.7× cost difference. For builders running Anthropic models on agentic workloads, this is the benchmark case for paying the Opus premium: on the seven frontier-eval-v1 tasks, that case does not exist.

The more important data gap is the third column. One data point from a non-Anthropic model would tell us whether 71.4% is a ceiling (the tasks are that hard for any model at current capability) or a coincidence (both Anthropic models happen to share specific failure modes on tasks where other models might differ).

What we still don’t know

Three open questions from this campaign:

On task_03: Is the chart analysis failure specific to this harness’s vision pipeline, to these five chart types, or to the Anthropic model family at current capability? We have 10 Anthropic failures across two campaigns on the fixed harness. We need a non-Anthropic model on the same task.

On task_05: The model runs 81 tool calls at 473 seconds and still fails. Whether a different scaffolding approach (smaller context, different file layout, explicit dependency mapping) would change the outcome is not answerable from wrong_answer outcomes alone. The question is whether the task is solvable at all on this harness, not whether Sonnet 5 specifically can’t do it.

On the 5.7× cost delta: This campaign shows Sonnet 5 at $29.30 vs. Opus 4.8 at $166.26 on these specific seven tasks. Different task distributions will produce different ratios. The ratio narrows on tasks that are input-token-heavy (both models hit similar input loads); it widens on tasks where Opus does more iterative work. Any workload with significant task_05-style deep-context iteration will see a different cost profile than this campaign shows.

Related reading

• Claude Opus 4.8 on frontier-eval-v1 — the model Sonnet 5 tied at 71.4%
• Mistral Medium 3.5 on frontier-eval-v1 — another data point on the same harness