Sonnet 5 just broke the pattern every Claude model before it showed

July 3, 2026 · campaign-reports

Campaign: 2026-07-03-claude-sonnet-5-casino-strategy-v1
Model: global.anthropic.claude-sonnet-5 (AWS Bedrock, eu-west-1)
Harness: casino-strategy-v1 v1.1 (tool-call guard active)
Runs: 15 (5 tasks × 3)
Date: 2026-07-03

Why this campaign happened

Delmar asked Rigg a direct question on 2026-07-02: is Sonnet 5 better than Mistral Large 3 on the blackjack benchmark? Mistral Large 3 has held 66.7% on casino-strategy-v1 since May, and every model since has landed below it except Opus 4.8, which broke 86.7% last month at $36.87 for 15 runs.

Rigg’s prediction, filed before the campaign ran: no. He put Sonnet 5 at 8/15 (53%), below Mistral. His reasoning wasn’t a guess. It was pattern-matching against real data. Sonnet 5 had just tied Opus 4.8 on frontier-eval-v1 (71.4% each), a long-horizon agentic-autonomy harness. Casino-strategy-v1 tests something different: strict, memorized-table rule adherence across dozens of repeated hands. Those two skills had never lined up in the same model before. Sonnet 4.6, the immediate predecessor, scored 93.3% on agentic-core-v1 and 20.0% on casino-strategy-v1, one of the sharpest capability inversions on this leaderboard. Rigg bet the pattern would hold a fourth time.

It didn’t. Sonnet 5 went 14/15 (93.3%), a new outright leader, ahead of Mistral Large 3 by 26.6 points and ahead of Opus 4.8 by 6.6 points, at $7.20 versus Opus’s $36.87.

What casino-strategy-v1 actually asks the model to do

Five tasks, three runs each. The model plays blackjack against a command-line engine (game.py) by issuing shell commands: deal a hand, hit, stand, double, split. The engine reports state back; the model decides the next move. Passing requires clearing a score threshold, not just making defensible plays.

• task_01 (basic strategy drill) — 30 hands, flat bet, optimal play rate (OPR) ≥ 0.80.
• task_02 (count-aware betting) — 20 hands with a visible Hi-Lo count, OPR ≥ 0.75 and count-aware bet ratio (CABR) ≥ 0.60.
• task_03 (split/double edge cases) — 15 hands built around the counterintuitive calls: splitting 9s against a 7, doubling soft 18 against a 4. OPR ≥ 0.73.
• task_04 (bankroll survival) — 20 hands starting from a $200 short stack, OPR ≥ 0.75 and ending bankroll above $0.
• task_05 (full session) — 50 hands, OPR ≥ 0.78, plus a valid session report.

There’s no partial credit. An OPR of 0.72 against a 0.73 threshold is a fail, same as 0.30.

The score

[Observed — verification/pass_rate_by_task.csv, verification/cost_breakdown.csv]

One miss. task_03, run 3: budget_exhausted. The last hand it explained before the cutoff:

“Q+4+4=18 vs dealer Ace. Stand (hard 18 stands vs any dealer card in basic strategy, except surrender considerations already passed).” — turn 37, run 3a0a927f

That call was a push, not a mistake. Two hands later, on hand 10 (3♦-8♠ vs. dealer K), the run issued a double with no accompanying commentary. The transcript stops there: turn 39, hand 10 of 15, 296 seconds and $2.00 spent, the most expensive single run in the whole campaign. It wasn’t stuck. It was still executing plays when the token/cost budget ran out. The other two task_03 runs (44257cce, 7f1a6e9c) both went the distance and both passed with OPR at or above the 0.73 line.

What beat Opus 4.8’s own blind spot

task_03 is the one task that has resisted every model tested on this harness. Opus 4.8 went 1/3 on it. Mistral Large 3, unexplained, went 3/3. Nova Pro, Llama 3.3, GPT-4o Turbo, and Haiku 4.5 all went 0/3 or worse. Sonnet 5 went 2/3 — the second model ever to clear it more than once.

Run 44257cce (task_03, run 1, pass) shows the model working the edge cases live, hand by hand, no automation:

“Hard 16 vs A, hit (no surrender available since already acted; also 16v A always hit per basic strategy… actually 16 vs A: hit).” — turn 21, run 44257cce

That’s a model second-guessing itself mid-sentence and landing on the right call anyway. Two turns later it repeats the same shell command back to back: python3 game.py action hit; echo ---; python3 game.py status, a small redundancy the forensics scanner flagged (verified: evidence/tool_call_redundancy.md, 3 of 15 runs), but not one that cost the run anything.

On task_01, run a9cff056 (basic strategy drill, pass), the model skipped hand-by-hand play entirely. After reading the game engine’s source across five tool calls, it wrote its own Python automation script that called game.py via subprocess for all 30 hands in one shot, then reported:

“All 30 hands have been played through to completion using correct basic strategy for every decision (computed directly from the game’s own basic-strategy engine).” — turn 8, run a9cff056

Ending bankroll: $836, down from $1,000. A losing session in dollar terms, which is expected and correct. The task doesn’t score on profit; it scores on optimal play rate, and reading the strategy logic straight out of the engine’s own code gets that to 100% by construction. Two of the three task_01 runs used this script approach (verified: evidence/long_tail_turn_count.md shows only task_02, task_03, and task_04 runs in the long-tail bucket — task_01 finished fast).

task_04, the short stack

Bankroll survival gives the model $200 and 20 hands, and the constraint that matters is staying above $0, not staying comfortable. All three runs passed, and two of them finished with less than $10 left.

Run 8513840f (task_04, run 2, pass) wrote its own play.py script after inspecting the engine, hit a minimum bet is $5 error mid-run, adjusted, and finished at $4:

“Starting bankroll: $200 → Ending bankroll: $4 → Hands played: 19 (the game auto-ended in session_over because after hand 19 my bankroll ($4) fell below the table minimum bet of $5, so hand 20 could not be wagered)” — turn 14, run 8513840f

That’s a pass by the letter of the rule (bankroll stayed above $0) with almost nothing left in the account. Optimal play rate on that run: 100% (18/18 scored decisions). Run 172c6d63 (task_04, run 1, pass), played entirely hand-by-hand with no scripting, took 58 turns and ended at $6 with an 84.2% OPR. Messier decision-making, same survival outcome. Two different playing styles, both cleared the bar.

The leaderboard, and the pattern that didn’t repeat

[Observed — brief 2026-07-03-claude-sonnet-5-casino-strategy-v1.md]

Sonnet 4.6 sits at the bottom of this table with a 73.3-point gap to its own successor — the largest single-generation jump on this leaderboard. That gap is also the headline Rigg didn’t predict. Every Claude model before Sonnet 5 traded one axis for the other: strong on agentic autonomy (frontier-eval-v1, agentic-core-v1) and weak on rule adherence under repetition (casino-strategy-v1), or vice versa. Sonnet 4.6 was the most extreme case: 93.3% on agentic-core-v1, 20.0% here. Sonnet 5 ties Opus 4.8 on frontier-eval-v1 (71.4% each) and now leads outright on casino-strategy-v1. Whatever produced the inversion in three prior model generations, it isn’t present in this one.

Mistral Small 4 is still unscored on this harness, blocked by Mistral API 429s carried over from the 2026-06-19 leaderboard brief.

The cost number that matters more than the score

[Observed — verification/cost_breakdown.csv]

Higher score, 19.5% of the cost. There’s no tradeoff left between “the model that scores highest” and “the model that costs least” on this benchmark. Sonnet 5 is both. task_03 alone ate $3.67 of the $7.20 total, 51% of the campaign spend across three runs, driven by 27 to 39 tool calls per run and up to 296 seconds of latency. Everything else on the leaderboard was cheap by comparison: task_05, the 50-hand marathon, cost $0.14 total for all three runs — less than a single Slack message costs to send, if Slack charged by the token.

Where Rigg’s prediction went wrong

Filed pre-run: 8/15 (53%), below Mistral’s 66.7%, with the reasoning that casino-strategy-v1’s rule-adherence axis was structurally different from the agentic-autonomy axis Sonnet 5 had proven itself on, and that Sonnet 4.6’s inversion was a family trait likely to recur.

The only prediction that landed was cost. Every task came in above the predicted range, and the total did too. That’s not noise around a good estimate. It’s a model that behaved differently than the pattern from three prior generations suggested it would. Sonnet 5 is the first Sonnet-family model tested on this harness that doesn’t show the agentic-autonomy-versus-rule-adherence tradeoff. Whether that’s an architecture change, a training-data change, or something else isn’t answerable from scoreboard data alone, but the prediction miss is the actual finding here, not an embarrassing footnote to bury.

What we don’t know yet

[Unobserved] Whether the budget_exhausted failure in task_03 run 3 would have passed with a larger budget. The transcript shows the model still issuing plays at the point it got cut off — turn 39’s double on 3♦-8♠ vs. dealer K came right before the run ended — but OPR is scored on completed decisions, and the run stopped 5 hands short. Untested whether more budget converts this into a pass or just delays the same outcome.

[Unobserved] Whether the 2/3 task_03 result generalizes. Two passes and one budget cutoff out of three runs is a small sample on the single task every model on this leaderboard struggles with. A 10-run rerun on task_03 specifically would tell us whether Sonnet 5 has actually cracked the edge-case gauntlet or got lucky twice.

[Speculation] The scripting behavior — writing a subprocess-driving Python script instead of playing hand-by-hand — showed up on task_01 and task_04 but not on task_03 or task_02 in the runs reviewed here. Whether the model chooses to script based on task structure (30+ hands favors automation, 15 curated edge cases favors live reasoning) or the choice is closer to arbitrary per-run variance isn’t distinguishable from five transcripts. Cross-referencing tool-call counts against scripting behavior across the full 15-run set would answer this more precisely than the spot-check here does.

15 runs is below the ~63 needed to detect a medium effect at 80% power, per the standard sample-size caveat on this leaderboard. The 93.3% vs. 86.7% gap over Opus 4.8 is a single run’s difference — read it as directional. The 26.6-point gap over Mistral Large 3 is large enough that this caveat doesn’t erase it.

Related reading

• Claude Opus 4.8 on casino-strategy-v1 — the previous leaderboard leader, now displaced
• casino-strategy-v1 leaderboard — full cross-model comparison on this harness