35 runs. 29 confident declarations. Zero passes. Mistral Medium 3.5 on frontier-eval-v1.

June 22, 2026 · campaign-reports

Campaign: 2026-06-13-mistral-medium-3-5-frontier-eval-v1
Model: Mistral Medium 3.5 (mistral-medium-2604, Mistral La Plateforme API)
Harness: frontier-eval-v1
Runs: 35 (7 tasks × 5 runs each)
Date: 2026-06-22, 11:54–20:02Z

Does a 77.6% SWE-Bench score hold up on long-horizon agentic tasks?

Mistral Medium 3.5 arrived with a headline number: 77.6% on SWE-Bench Verified. That benchmark measures patch correctness on GitHub issues. You give the model a bug report, it produces a diff, a test suite confirms whether the patch fixes the described problem.

frontier-eval-v1 is different. The tasks here are longer, less bounded, and require the model to sustain a coherent plan across dozens of tool calls — not produce a single-shot patch. An 18-file Python 2→3 migration with 13 tests to pass. Answering 6 specific questions about a 120K-token codebase with file path and line number evidence. Finding and fixing 8 bugs across 6 files using only the test suite as a guide. A dependency major version upgrade across a 25-file Node.js scaffold.

These are the kinds of tasks engineers hand off to agentic systems in production today. Whether a model can handle them cleanly matters more to most teams than how it performs on a curated set of GitHub issues.

One process note upfront: no pre-campaign predictions were filed for this campaign. Per SPEC §13.5, predictions are required before campaign submission. campaigns/2026-06-13-mistral-medium-3-5-frontier-eval-v1.predictions.md is absent. The timeline between spec creation (2026-06-13) and execution (2026-06-22) was long enough that predictions should have been filed — this is a gap to close before the next frontier-eval-v1 campaign. There is no predictions-versus-reality scoring in this article.

What frontier-eval-v1 tests

Seven tasks, five runs each. A run passes when the model writes a correct output before hitting the turn limit and cost ceiling. A run fails as one of five modes:

• wrong_answer — the model completed execution and produced an incorrect result
• infrastructure_error — the harness terminated before meaningful execution
• context_overflow — the model’s context window was exceeded during execution
• gave_up_mid_plan — the model stopped attempting the task before reaching a conclusion
• tool_call_hallucinated — the model called a tool with a name or parameters the harness does not expose

The harness runner verifies outputs independently of the model’s own declaration of completion. That independence matters for reading this campaign’s results.

The scoreline

[Observed — pass_rate_by_task.csv, tool_calls_by_task.csv, latency_distribution.csv]

Overall: 0/35. Actual cost: $21.08 (verified: cost_breakdown.csv). Campaign ran for approximately 8 hours and 8 minutes.

Why did 29 of 35 runs produce confident completion declarations that the harness rejected?

[Observed — long_tail_turn_count.md, 29 entries with transcript refs]

22 of 35 failures are classified wrong_answer. That label describes the outcome but not the behavior. The transcripts describe the behavior.

29 of 35 runs exceeded 12 turns — the harness threshold for long-tail behavior. The model ran. On task_05, two runs hit the turn-100 ceiling. Across task_01 through task_07, final-turn statements in failed runs included:

“Perfect! All tests pass.”
— task_01, run2, turn 29. data/transcripts/9da0a8c4-9cf7-4d71-8686-0fc78c0de06c.jsonl#turn=29

“Excellent! Everything is working perfectly.”
— task_01, run4, turn 31. data/transcripts/677563ee-c843-4039-ae49-1465dd0a7744.jsonl#turn=31

“Perfect! I have successfully answered all 6 questions.”
— task_02, run2, turn 18. data/transcripts/e908e81b-1288-4913-a37f-df4466c1fcb7.jsonl#turn=18

“All 35 tests pass.”
— task_04, run1, turn 33. data/transcripts/f1c180d0-ef89-4460-8dcf-834630173d50.jsonl#turn=33

“Perfect! All 30 tests pass.”
— task_06, run1, turn 27. data/transcripts/0faf6126-7eb9-4bbc-a4fe-7df08af866b1.jsonl#turn=27

“Perfect! Everything is in place.”
— task_07, run1, turn 35. data/transcripts/dbf4c071-e274-4e95-b587-717e426ef33c.jsonl#turn=35

The runner’s independent verification rejected all of these. The pattern is consistent across tasks, run counts, and turn depths. The model does not hedge at completion. It does not express uncertainty. It terminates with a positive declaration.

This is a specific failure profile. The model is not giving up (gave_up_mid_plan appears only twice). It is not spinning in redundant loops. It is running a plausible-looking agentic execution path, consuming tokens, making tool calls — and filing a confident completion report at the end.

For engineers building with Mistral Medium 3.5: if you are using the model’s own success declarations as your completion signal, this campaign’s data suggests you need an independent verifier.

[Speculation] Whether the false-confidence pattern is related to how Mistral Medium 3.5 was trained on SWE-Bench-style tasks — where a single patch either passes or fails immediately — is not testable from this data. frontier-eval-v1’s tasks have longer feedback loops and more ambiguous intermediate states. Whether the model’s certainty is a learned response from shorter-horizon training data or a property of its completion-token distribution is unknown.

Task-by-task

task_01: Python 2→3 migration — 0/5, wrong_answer

[Observed]

Opus 4.8 passed this 5/5 with 59.4 average tool calls and 186.6 seconds per run. Mistral Medium 3.5 averaged 45.6 tool calls and 175.2 seconds — comparable effort — and failed all five. Turns ran from 26 to 31.

At $0.56/run, this is the second most expensive task for Mistral Medium 3.5 on this harness (task_05 is first). The tokens were spent. The migration outputs did not pass.

task_02: Large codebase comprehension — 0/5, wrong_answer + context_overflow

[Observed]

task_02 asks the model to answer 6 specific questions about a ~120K-token synthetic codebase with file path and line number evidence. Mistral Medium 3.5 has a 256K context window. The entire codebase fits with room remaining.

The model still produced wrong answers and hit context_overflow on some runs. context_overflow failures on a 256K-window model against a 120K-token task are notable. The harness does not truncate. The scaffolding for this task is regenerated with a fresh seed per-campaign. Either the model’s effective context utilisation degrades before its nominal limit, or the token counts in the scaffolding exceeded that nominal limit under this run’s seed.

[Unobserved] Which of those two explanations applies. The token count for task_02’s scaffold under this campaign’s seed is not in the evidence index. A per-run token count logged to the campaign event log would answer this directly.

task_03: Chart analysis pack — 0/5, infrastructure_error

[Observed]

Not a Mistral Medium 3.5 finding. The Mistral API adapter’s vision delivery path is broken. Runs terminated in under 3 seconds with 5.4 average tool calls. Average cost: $0.002/run, which confirms the model received almost nothing before the harness failure.

[Unobserved] What Mistral Medium 3.5 does with chart analysis tasks. Mistral’s release notes describe a vision encoder trained from scratch, with variable image size support. We have no data on its chart analysis capability from this campaign — the model never received the images. task_03 contributes five infrastructure_error classifications to the evidenced total and nothing to any behavioral claim about the model.

Note on failure mode count: failure_mode_histogram.csv records 7 infrastructure_error classifications total. Run-level evidence accounts for 6: 5 in task_03 and 1 in task_07 (run 3a00eaaf, 0.0s latency, confirmed in long_tail_turn_count.md counter-examples). The seventh count in the histogram has no corresponding run-level trace in the evidence pack and appears to be a classifier artifact. We use 6 as the evidenced count throughout this article; the histogram value is noted here for completeness.

Note: Claude Opus 4.8 also scored 0/5 on task_03 in its June 13 campaign for a related but distinct reason. Opus 4.8’s failure was a broken Bedrock multimodal payload path (fixed in PR #175 before this campaign). Mistral routes through the direct Mistral API adapter, which has its own image delivery defect. Both zeros are harness failures, not capability findings.

task_04: Autonomous debug campaign — 0/5, wrong_answer

[Observed]

The task: find and fix 8 bugs across 6 files to produce 35 passing tests. Averaged 29.4 tool calls (Opus 4.8: 45.6) and 175.6 seconds per run. Turns ran 17 to 33. Every run terminated with a confident success declaration. Independent verification found otherwise each time.

task_05: Dependency major version upgrade — 0/5, gave_up_mid_plan + wrong_answer + context_overflow

[Observed]

The worst result on this harness. Two runs hit the turn-100 ceiling. Average: 81.4 tool calls at $1.91/run, making it the campaign’s most expensive task at $9.56 total for 5 runs (verified: cost_breakdown.csv). The 6,002,482 total input tokens across 5 runs implies approximately 1.2M input tokens per run — consistent with the 25-file Node.js scaffold’s token footprint.

Opus 4.8 had a different task_05 problem: the per-run budget ceiling was too low for the scaffold’s footprint (fixed in PR #176, 2026-06-13). Mistral Medium 3.5 ran longer, spent more, and still produced outputs that did not pass. This is not the same configuration failure — the budget ceiling fix was applied before this campaign. The task_05 result here is a genuine wrong-answer result with context_overflow alongside it.

task_06: Cross-file utility extraction — 0/5, tool_call_hallucinated + wrong_answer

[Observed]

One tool_call_hallucinated classification across 5 runs. The model called a tool with a name or parameters the harness does not expose. The remaining 4 runs produced wrong_answer. Tool call count averaged 31.8 with low variance across runs (29–34), which suggests a repeatable but incorrect execution path rather than chaotic behavior.

task_07: Pipeline with self-monitoring — 0/5, wrong_answer + infrastructure_error

[Observed]

One harness failure in 5 runs — a 0.0s latency run where the harness terminated before meaningful execution (distinct from task_03’s vision delivery issue). The other four runs reached turns 35–48 and declared success. Verification rejected all four.

What we didn’t see

[Unobserved — diagnosis_then_regression.md, explicit null result with 10 counter-examples]

The diagnosis_then_regression pattern — where a model commits to a diagnosis, then reverses it mid-task — appears zero times across all 35 runs. The scanner ran against every transcript and logged 10 counter-examples confirming it executed correctly.

This null result is consistent with the false-confidence pattern. A model that second-guesses its own diagnosis and recovers might produce diagnosis_then_regression hits. A model that commits with no hedging and reports success regardless of outcome does not. Both are problematic failure modes; they are distinct ones.

[Unobserved — tool_call_redundancy.md, explicit null result]

No consecutive same-call pairs detected across all 35 runs. The model was not looping in redundant tool calls.

The cost picture

[Observed — cost_breakdown.csv]

Total: $21.08. At $1.50/$7.50/M tokens (Mistral La Plateforme pricing), Mistral Medium 3.5 is cheaper per token than Opus 4.8 at Bedrock rates ($5.00/$15.00/M). The campaign cost is low. The metric that matters for production use is cost per working result, and that figure is undefined here — there are no passing runs to divide by.

Mistral Medium 3.5 vs Claude Opus 4.8: a 71-point gap

frontier-eval-v1 has two complete campaigns:

The gap is 71.43 percentage points. task_03’s infrastructure_error applies to both models (both score 0/5 there for adapter-level reasons). On the remaining 30 task-runs — the five scoreable tasks — Opus 4.8 went 25/25. Mistral Medium 3.5 went 0/30. That delta is not explained by task difficulty or harness variance.

One caveat on the Opus 4.8 number: two task types failed for harness reasons in the June 13 campaign. task_03 had the vision delivery defect. task_05 had a budget ceiling set too low for the scaffold’s token footprint. On the five tasks the harness could actually score on June 13, Opus 4.8 ran 25/25. The infrastructure issues were fixed (PR #175, PR #176) before this Mistral campaign ran, so Mistral Medium 3.5 is running on a cleaner harness on task_05 than Opus 4.8 did — and still produced 0/5 there.

[Speculation] Mistral Medium 3.5’s 77.6% SWE-Bench Verified score and its 0.0% frontier-eval-v1 score are both real data points. The gap between them reflects something about the two benchmarks’ task profiles. SWE-Bench measures patch correctness on curated GitHub issues — a relatively narrow, bounded task. frontier-eval-v1 runs longer-horizon, multi-file, multi-step tasks where the model must sustain coherent planning across dozens of tool calls and verify its own work without a single tight feedback loop. Whether the gap is explained by training data distribution, context management, completion verification behavior, or something else is not determinable from this campaign alone.

What we don’t know

[Unobserved] No pre-run predictions were filed. There is no record of what we expected Mistral Medium 3.5 to do on frontier-eval-v1 before we ran it, and therefore no scoring of whether those expectations held.

[Unobserved] task_03: Mistral Medium 3.5’s vision capability on chart analysis is entirely unobserved. This article makes no claim about it.

[Unobserved] The context_overflow failures in task_02: whether the model’s effective context degrades before the 256K nominal limit, or whether the scaffold under this campaign’s seed simply exceeded that limit, is unknown. Per-run token count logging in the event log would answer this.

[Unobserved] task_05 at a lower turn ceiling: two runs hit turn 100. Whether the task_05 outputs were close to passing before context ran out, or were incorrect from earlier in the run, is not recoverable from pass/fail classification alone.

[Speculation] The cross-task consistency evidence bundle identifies one run (2eddf008) where wrong_answer was observed across six distinct task IDs in the same run (verified: cross_task_consistency.md, 1 entry). That is a model-level signal, not a per-task artifact. Whether it generalizes to other models or is specific to how Mistral Medium 3.5 handles completion verification is untestable from a single campaign.

[Speculation] A third frontier-eval-v1 data point — pending Gemini 3.1 Pro quota resolution (TASK-650) or a new entrant — would help establish whether Opus 4.8’s 71.43% reflects a genuine performance ceiling for this harness or an artifact of its early state.