1/35. Google's fastest GA model ran frontier-eval-v1 clean — and nearly scored zero.

June 24, 2026 · campaign-reports

Campaign: 2026-06-23-gemini-3.5-flash-frontier-eval-v1
Model: Gemini 3.5 Flash (gemini-3.5-flash, version 3.5-flash-05-2026, GA May 2026)
Provider: Google Generative AI (direct API)
Harness: frontier-eval-v1
Runs: 35 (7 tasks × 5 runs each)
Date: 2026-06-23

Why this campaign existed

[Situation]

The previous attempt to run a Google model through frontier-eval-v1 ended before it produced any useful data. Gemini 3.1 Pro Preview (TASK-650) hit Google’s 250 request/day daily cap for preview-tier models. Twenty-eight of 35 runs returned RESOURCE_EXHAUSTED. The data was invalid, the campaign was void, and the leaderboard still had no Google row.

Gemini 3.5 Flash is GA — standard service tier, no preview caps. The question was whether it would actually clear 35 runs without the quota wall coming down.

It did. Zero RESOURCE_EXHAUSTED errors across all 35 runs. The GA quota thesis holds.

The second question: could a Flash-tier model pass any frontier-eval-v1 tasks? Flash had scored 36.7% on agentic-core-v1 (verified: google-gemini-3-5-flash-agentic-core-v1-2026.mdx). frontier-eval-v1 is harder — longer horizons, deeper context, more sustained planning required. The gap between the two benchmarks tends to expose models that run competently on short tasks but fall apart across 80 turns.

The answer: 1/35, a 2.86% pass rate.

Pre-run predictions (filed 2026-06-23T20:00Z per SPEC §13.5): Rigg predicted 3–8/35 (8–23%). The actual result landed below the floor of that range. Prediction on long-horizon agentic tasks (task_01, task_04, task_05, task_06, task_07) was accurate: all predicted 0/5, all scored 0/5. The miss was on task_03 (chart analysis, vision), which was predicted 1–2/5 and scored 0/5.

What the tasks actually ask

[Task]

frontier-eval-v1 runs seven tasks at five repetitions each. A run passes when the model produces a correct output verified by the harness before hitting the turn limit. The harness verifies independently — the model’s own declaration of success counts for nothing.

The tasks cover sustained agentic work:

• task_01: Migrate an 18-file Python 2.7 codebase to Python 3.11. Thirteen tests must pass. The model cannot see the tests before it starts.
• task_02: Answer 6 specific questions about a ~120K-token synthetic codebase with file path and line number evidence. Pass threshold: 0.55 (at least 3.3/6 correct with evidence).
• task_03: Analyze a pack of rendered chart images. Extract specific numeric values. Grader tolerance is tight — approximate readings fail.
• task_04: 28 failing tests, no modifying test files. Find and fix the bugs. The model never sees a diff of what changed.
• task_05: Major version upgrade across a 25-file Node.js scaffold (Express 4 to 5), 22 Jest tests passing required.
• task_06: Extract shared utilities from a cross-file codebase without breaking any consumers.
• task_07: Build a self-monitoring pipeline that reports its own execution anomalies.

These are not one-shot tasks. task_05 averaged 21.8 tool calls per run. task_01 averaged 7.2. A model needs to track what it has already done, update its working state, and course-correct without forgetting what the previous turns established.

What Flash actually did

[Action]

The results by task (verified: verification/pass_rate_by_task.csv, verification/tool_calls_by_task.csv, verification/cost_breakdown.csv, verification/latency_distribution.csv):

Overall: 1/35 (2.86%). Total cost: $5.84 (verified: verification/cost_breakdown.csv).

The one pass

[Observed — evidence/long_tail_turn_count.md]

Run 4c0d70f5 on task_02 (run3) is the campaign’s only pass. The reason it worked is structural: task_02 is a comprehension task, not a modification task. Flash reads the codebase, answers questions, stops. It does not need to maintain a working state across 90 turns of file writes and test executions. The 1M context window gave Flash room to hold ~120K tokens in view. Pass threshold was 0.55 — Flash needed 3.3 of 6 questions right with file path and line number evidence.

This same run shows 5 consecutive identical fs_read calls on src/serialise/__init__.py (turns 15–19) and 90 total tool calls — more than any other run in the campaign (verified: evidence/tool_call_redundancy.md). Flash re-read the same file five times without tracking that it had already done so. It passed not because it was efficient but because the task does not care how many times you re-read a file. The comprehension question is graded on answer accuracy, not on tool discipline.

[Unobserved] Whether task_02 performance is repeatable. One pass from five attempts is too thin to call a capability. A targeted mini-eval with 10 runs on task_02 would tell us whether this is a real signal or a single lucky run through a 0.55 threshold.

The 34 failures and what they look like

[Observed — evidence/cross_task_consistency.md]

34 failures. All 34 are classified wrong_answer. No context_overflow. No gave_up_mid_plan. No tool_call_hallucinated. Flash ran to completion, reached a conclusion, and delivered it as correct. The harness disagreed.

The wrong_answer classification is cross-task: every task type in the suite (verified: evidence/cross_task_consistency.md, 1 evidence entry covering all 7 task IDs). This is not a single task where Flash has a blind spot. It is the same failure mode across the full benchmark.

The task_04 numbers tell part of the story. Flash dispatched 3.2 tool calls on average — the fewest of any task. Task_04 has 28 failing tests; no modifying test files. Flash diagnosed the problem quickly, pushed a fix, and stopped. All five runs failed. The diagnoses were wrong and Flash did not know it.

Thinking was enabled (thinking=true, API-confirmed) and billed at $9/M thinking tokens. Task_02 averaged $0.495/run and task_05 averaged $0.497/run — those are the two tasks with the most expensive runs, both high-context and multi-step. The thinking budget shows up in the cost, not in the pass rate.

On task_05, Flash had 82 turns in one run. The transcript shows identical grep commands against node_modules/express/lib/ at turns 57, 62, and 79 (verified: evidence/tool_call_redundancy.md). Thinking activated. Flash reconsidered each step. It did not fix what it found three turns ago. The issue is not single-turn reasoning — it is state management across turns: Flash does not carry forward what earlier turns established.

The null result on diagnosis-then-regression

[Unobserved — evidence/diagnosis_then_regression.md, 10 counter-examples, 0 occurrences]

Across all 35 runs, the runner found zero instances of the diagnosis_then_regression pattern — a model stating a diagnosis, then reversing it with “actually, let me reconsider” before trying again. Flash does not second-guess itself. It commits and pursues. This is partially why the failure mode is wrong_answer rather than gave_up_mid_plan. Flash never recognizes the plan is failing because it never pauses to ask.

What the results mean

[Result]

[Observed — leaderboard context]

Flash scores one place above Mistral Medium 3.5 by exactly one pass. But the failure mode is different. Mistral failed through disorientation — context overflows, tool call hallucinations, runs that derailed before finishing. Flash failed through false confidence — ran all the way, declared success, was wrong. Two very different things for a builder to work with.

Flash at $0.17/run average is the cheapest frontier-eval-v1 campaign to date. Opus 4.8 cost $4.75/run. For tasks where Flash can actually pass, the cost efficiency is real. The problem is knowing in advance which tasks those are. On frontier-eval-v1, the answer is: comprehension tasks where the model reads and answers, context window holds, state tracking is minimal. Everything else, Flash fails confidently.

We predicted 3–8 passes. We got 1. The miss was task_03 vision. Flash has multi-modal support. Task_03 asks for precise numeric extraction from rendered chart images. The grader tolerance requires exact values, not approximations. Flash read the charts and delivered numbers close enough to look right — but the harness grader does not give partial credit. That precision gap was not anticipated. It is worth a targeted mini-eval to determine whether prompt engineering could close it (open question: see below).

What we don’t know yet

[Speculation]

The Flash vs Gemini 3.1 Pro comparison on frontier-eval-v1 remains open. TASK-650 is blocked pending Vertex AI quota escalation. agentic-core-v1 data shows a 24-point gap (Flash 36.7%, Pro 76.7%). On frontier-eval-v1, the harder benchmark, that gap will almost certainly be larger — but we cannot quantify it without clean Pro data.

Task_02 repeatability is unknown. One pass from five runs at a 0.55 threshold could be a genuine capability or variance in the grader. Ten more runs would answer it.

The tool_call_redundancy pattern appeared in 6 of 35 runs (verified: evidence/tool_call_redundancy.md), including the passing run. A system prompt instructing the model not to re-read files it has already retrieved might reduce tool call count — but whether that translates to pass rate improvement is untested. State management is the underlying problem; a reminder at prompt time may not fix a context management gap.

Data pack: data/campaigns/2026-06-23-gemini-3.5-flash-frontier-eval-v1/
Evidence index: briefs/2026-06-23-gemini-3.5-flash-frontier-eval-v1.evidence-index.md