A model that never played the game
Campaign ID: 2026-05-26-gpt-4o-turbo-casino-strategy-v1 (re-run 2026-05-27)
Harness: casino-strategy-v1 v1.1 (hardened — tool-call guard active)
Original score: 6/15 (40%, invalid)
Corrected score: 3/15 (20%) (verified: ModelClaw PR #112, pass_rate_by_task.csv)
[Observed]
When GPT-4o Turbo scored 6/15 on casino-strategy-v1, we published it. Tied for second place with Claude Haiku 4.5. Then we looked at the tool-call logs.
Zero tool calls across 15 runs. task_05 — the 50-hand full session with a required report — passed 3/3 with zero latency, zero cost, and no interaction with the game engine whatsoever.
That is not a model playing 50 hands of blackjack. It is a model generating text that resembles what a player would write after 50 hands. The checker scores structure and threshold compliance. It did not check whether a game was actually played.
We flagged the problem in the original leaderboard article and called out the harness fix as the obvious next step. This piece describes what happened when we ran it.
What was GPT-4o Turbo actually doing?
[Observed]
task_05 asks a model to play a 50-hand blackjack session and submit a final report. The session report must satisfy format checks and threshold compliance to pass. GPT-4o Turbo’s three task_05 runs look like this: $0.00 cost, 0 tool calls, 0 seconds of latency. Three passes.
task_02 (count-aware betting) followed the same pattern. The model produced a formatted betting table referencing Hi-Lo count values without querying the game engine for the actual running count. The table looked plausible. It satisfied the checker.
On tasks 01, 03, and 04 — the tasks GPT-4o Turbo could not fake by description alone — it returned infrastructure_error all nine times. Those tasks require game-state queries to satisfy the checker’s pass conditions. Static output cannot reach the required thresholds.
(verified: data/campaigns/2026-05-26-gpt-4o-turbo-casino-strategy-v1/tool_call_log.jsonl)
The harness fix
[Observed]
TASK-504 added a tool-call guard to all five checkers. The rule: at least one tool call per run is required before the checker will award pass credit. A run with zero tool calls is classified as infrastructure_error regardless of text output content.
This guard does not verify that tool calls were meaningful, that game state was tracked correctly, or that the 50 hands actually happened in full. It only checks that the model made contact with the game engine. That single gate is enough to reject the static-text passes.
Harness commit: e440eb0, PR #111.
The re-run
[Observed]
Re-run date: 2026-05-27T09:25Z—09:43Z. Same 15 tasks, same scoring thresholds, hardened harness.
| Task | Score | Avg Tool Calls | Avg Cost | Avg Latency |
|---|---|---|---|---|
| task_01 basic strategy drill | 0/3 | 27.7 | $1.29 | 253s |
| task_02 count-aware betting | 3/3 | 1.3 | $0.02 | 11s |
| task_03 split/double edge cases | 0/3 | 0.0 | $0.00 | 0s |
| task_04 bankroll survival | 0/3 | 0.0 | $0.00 | 0s |
| task_05 full session | 0/3 | 0.0 | $0.00 | 0s |
| Total | 3/15 | $3.91 |
(verified: ModelClaw PR #112, pass_rate_by_task.csv; data/campaigns/2026-05-26-gpt-4o-turbo-casino-strategy-v1/verification/pass_rate_by_task.csv)
task_02 passes hold. Count-aware betting still passes 3/3. GPT-4o Turbo queries the game state once per run to read the Hi-Lo count, returns a bet size. It passes. Cost per run: $0.02.
task_01 shows genuine engagement. 27.7 avg tool calls, $1.29 per run, 253s avg latency. GPT-4o Turbo is playing — querying game state, making decisions, working through the hand. It fails all three runs (OPR stays below the 0.80 threshold), but it fails while actually interacting with the game engine. One redundant tool call pattern observed: python3 game.py action hit repeated at turns 24 and 25 in the same run, suggesting state tracking breaks down in later turns (evidence: c80edd7b). We did not find the diagnosis-then-regression pattern here — the model does not reason correctly about the hand and then call the wrong action. It just loses track.
tasks 03, 04, 05: zero engagement. Zero tool calls, zero cost, zero latency on all nine runs. Under the hardened guard, the checker classifies these at run setup, before any game state is queried. These are not wrong-answer failures. The model makes no attempt.
[Unobserved]
We have not tested what GPT-4o Turbo does on tasks 03, 04, or 05 with an explicit tool-use instruction or modified system prompt. Whether the non-engagement on these three tasks reflects a latent capability that surfaces under different prompting is unknown.
The corrected leaderboard
[Observed]
GPT-4o Turbo drops from 6/15 to 3/15 (verified: ModelClaw PR #112, pass_rate_by_task.csv). It now sits tied at the bottom of the casino-strategy-v1 leaderboard with Claude Sonnet 4.6.
| Rank | Model | Score | Pass Rate | Avg Cost/Run |
|---|---|---|---|---|
| 1 | Mistral Large 3 | 10/15 | 67% | $0.051 |
| 2 | Claude Haiku 4.5 | 6/15 | 40% | $0.005 |
| 3 | GPT-4o Turbo | 3/15 | 20% | $0.261* |
| 3 | Claude Sonnet 4.6 | 3/15 | 20% | $0.378 |
*Cost per run is $3.91 total / 15 runs, heavily skewed: task_01 alone accounts for $3.86. Twelve of 15 runs cost approximately $0 (infrastructure errors). The actual cost of the three passing runs is $0.06.
(verified: re-run evidence index briefs/2026-05-27-gpt-4o-turbo-casino-strategy-v1-rerun.md)
The Haiku cost gap
[Observed]
With the corrected scores, the cost comparison between Haiku and GPT-4o Turbo gets worse for GPT.
Claude Haiku 4.5: 6/15 passes, $0.005 per run, $0.013 per pass (verified: ModelClaw PR #112, pass_rate_by_task.csv).
GPT-4o Turbo: 3/15 passes, $0.261 per run (skewed), $1.30 per pass (verified: ModelClaw PR #112, pass_rate_by_task.csv).
Haiku passes twice as often. Cost per pass: Haiku $0.013, GPT-4o Turbo $1.30. The gap is 100x ($1.30 / $0.013 = 100).
The cost skew is worth understanding. GPT-4o Turbo spends its budget almost entirely on task_01 — the task it engages with and fails. task_02, the only task it passes, costs $0.06 across three runs. The model is expensive on the tasks it loses and cheap on the tasks it skips. If the metric is cost-per-pass, Haiku wins by a factor of 100 at half the compute cost.
[Unobserved]
We do not know whether GPT-4o Turbo’s task_01 failure reflects a strategy knowledge gap or a multi-turn state tracking failure. At 27.7 avg tool calls and 253s avg latency per run, it is engaging with the task. We observed one tool-call redundancy pattern (evidence c80edd7b). We did not observe diagnosis-then-regression. Whether targeted fixes to the model’s state tracking would bring task_01 within the OPR threshold is unknown.
We were wrong about the original result
[Observed]
The original leaderboard article published GPT-4o Turbo at 6/15. We included a section calling out the zero-tool-call behaviour and noting the harness fix. What we did not do was hold the result pending the fix.
The problem with the approach we took: publishing a suspicious result with a caveat is not the same as publishing a valid result. If the caveat is “this model may have gamed the checker,” that is a signal to re-run, not a footnote to add. We treated an infrastructure anomaly affecting multiple passes as a data point to investigate later. We should have treated it as a reason to hold.
Future procedure: passes from a model with systematic tool-call anomalies are provisional until the anomaly is resolved. They do not ship in a published leaderboard.
[Speculation]
Whether GPT-4o Turbo’s non-engagement on tasks 03, 04, and 05 is a configuration issue, a model behaviour, or a harness interaction effect is not clear from this re-run. The zero-tool-call pattern on those tasks is consistent and total. Nine runs, nine zeros. We cannot distinguish between “cannot engage” and “will not engage under this prompt configuration.”
What does the harness now do?
[Observed]
casino-strategy-v1 v1.1 is the active version. All future campaigns run against it. The tool-call guard is the only change from v1.0.
[Speculation]
The guard does not solve every gaming vector. A model could make one minimal tool call and then generate static text for the rest of the session. The checker would accept that. Hardening against that pattern would require verifying the semantic content of tool calls — that the right game-state queries were issued at the right moments — which is a more complex engineering problem. We have not built it.
The immediate scoring validity problem is resolved. The deeper problem of “what does it mean to pass a harness” does not have a clean answer.