Gemma 4 12B on agentic-core-v1: 91.7% on the tasks it actually ran

June 11, 2026 · campaign-reports

Campaign: 2026-06-10-gemma-4-12b-agentic-core-v1
Model: Gemma 4 12B IT (gemma-4-12b-it-Q4_K_M.gguf, llama.cpp, Q4_K_M)
Harness: openclaw@2026.4.22
Runs: 30 (10 tasks × 3 runs each)
Hardware: g6.4xlarge (A10G 24 GiB, eu-central-1a)
Campaign date: 2026-06-11

The question for this campaign was simple: does the 12B version of Gemma 4 carry meaningful agentic capability, and how does it hold up against its 31B sibling on the same task suite? At Q4_K_M quantization, the 12B weighs roughly 7.5 GB of VRAM. The 31B needs roughly 20 GB. If the 12B comes close in capability, that VRAM difference is significant for anyone running local inference on constrained hardware.

Gemma 4 31B scored 23/30 (76.7%) in its campaign. The 12B came in at 22/30 (73.3%) raw. But the raw number is misleading in a way that’s worth being precise about.

What agentic-core-v1 tests

[Observed]

The suite runs 10 tasks, 3 times each. Each task has a deterministic pass/fail checker. The 15-turn budget is fixed per run. Failure modes are logged at run time: wrong_answer when the checker rejects the output, gave_up_mid_plan when the turn budget expires, and infrastructure_error when the harness fails to complete the run at all.

The 10 tasks cover: fixing a failing test, refactoring duplicated code, investigating a log file, tracing through a codebase, writing a minimal fix under a line constraint, handling an ambiguous requirement with two-artifact output, multi-step sequential planning, recovering from a tool error involving multibyte character byte-counting, knowing when to stop on an underspecified problem, and SQL investigation using native tool calls.

What happened in 30 runs

[Observed]

22 of 30 runs passed. The failure breakdown by mode: 6 × infrastructure_error, 2 × wrong_answer, 0 × gave_up_mid_plan (verified: verification/failure_mode_histogram.csv).

(verified: verification/pass_rate_by_task.csv, verification/latency_distribution.csv, verification/tool_calls_by_task.csv)

Six of the eight failures are infrastructure_error on tasks 09 and 10. Those runs show 0 tokens dispatched, 0 tool calls, and timestamps collapsed to a single moment at the tail end of the campaign (2026-06-11T10:10:02Z). The SSM port-forward tunnel or the EC2 instance dropped right at the end. Tasks 09 and 10 never ran.

Excluding those six non-runs, the model went 22/24 (91.7%) on the tasks it actually ran (verified: verification/pass_rate_by_task.csv). That is higher than the 31B sibling’s effective score on the same eight tasks.

The EC2 drop: what it was and what it wasn’t

[Observed]

Tasks 09 and 10 were the final two tasks in execution order. All six runs (three per task) show infrastructure_error with 0 tokens dispatched. The run timestamps for all six are clustered at 2026-06-11T10:10:02Z — simultaneous with the campaign marking done. The EC2 instance or the SSM port-forward tunnel became unavailable in the campaign’s final minutes.

This is not a model result. We cannot say whether Gemma 4 12B would have passed task_09 (know when to stop, the hardest task in the suite, one that no model below Claude Sonnet 4.6 has reliably passed) or task_10 (SQL investigation). The 31B scored 3/3 on task_10, so there is a real open comparison sitting here.

The fix for next time is operational: keepalive settings on the SSM tunnel for the full campaign duration.

12B vs 31B: where they split

[Observed]

Both are Gemma 4 IT at Q4_K_M, tested on the same harness and task suite. The comparison is direct.

(verified: verification/pass_rate_by_task.csv for 12B; Gemma 4 31B campaign verification/pass_rate_by_task.csv)

The 12B beats the 31B by 2 runs on task_02 (refactoring) and by 1 run on task_03 (investigation). The 31B returns the favor on task_05 (3/3 vs 2/3), where the 12B had a degenerate run.

[Speculation]

The refactoring result is the most striking inversion. The 12B went 3/3 on a task where the 31B went 1/3. The 12B also averaged 2.3 tool calls on task_03 (log investigation) versus the 31B’s 1.3 — more conservative about committing early, which helped. Whether this reflects a tighter instruction-following fine-tune at the smaller scale, a different post-training emphasis, or just variance in a 3-run sample is not resolved by this campaign alone. The result is 3/3 versus 1/3, which is not noise — but it would need a longer run at matched context lengths to be confident the pattern holds.

Chat template leakage on task_05 run 3

[Observed]

task_05 asks the model to fix a bug in src/price.py with a diff no longer than 10 lines. Runs 1 and 2 passed cleanly. Run 3 (e71740e9, transcript: data/transcripts/e71740e9-6664-419a-9eae-85fa72ff825d.jsonl) did not.

In run 3, the model produced this in a single assistant text turn:

<|channel>fs_read{file: src/price.py}
<|channel>fs_read{file: tests/test_price.py}
<|channel>shell{command: python3 -m unittest tests/test_price.py}
<|channel>shell{command: python3 -m unittest tests/test_price.py}
<|channel>shell{command: python3 -m unittest tests/test_price.py}
... [~200 more identical lines]

All of that was a single text turn. No actual tool calls were dispatched. The output hit the 4096-token ceiling at 146.8 seconds. No file was written, no test was run.

The <|channel> token is a Gemma chat template delimiter. The model generated its internal tool-call representation directly into the text stream instead of using the structured function-call interface. This is not a hallucination and not a misunderstanding of the task. The model knows what to do (the right tools, the right sequence) but emitted the representation as raw template syntax rather than JSON tool_calls.

The runner classified it wrong_answer because the output produced no work product. The underlying issue is the leakage.

[Speculation]

Why run 3 and not runs 1 or 2? The most plausible trigger is context pressure. Runs 1 and 2 had 4–8 tool calls and shorter conversation histories. Run 3 may have started from a different context state that pushed the model into a regime where template tokens bleed through. The smoke test confirmed clean JSON function calling at low context depths; the leakage is a degenerate mode under higher context load. Frequency in this campaign: 1 in 30 runs.

The Gemma 3 tool_code text-block pattern (where the model emits tool calls as text blocks consistently, every run) is a related failure mode but a different regime. That was consistent output format. This was a degenerate edge case triggered by context state.

For builders: validate that tool_calls is non-null before treating an assistant turn as a function call result. A text-only turn during a tool-use loop is an error signal, not a response. The symptom is easy to detect: <|channel> in content with tool_calls null.

What the pattern detectors found

[Unobserved]

Three detectors ran against all 30 transcripts.

diagnosis_then_regression: 0/30 runs. No case of a stated diagnosis being reversed in a later turn.

long_tail_turn_count: 0/30 runs. No run exceeded 13 of the 15-turn budget on tasks that actually ran.

tool_call_redundancy: 1/30 runs. task_05 run 1 (31a6df12) showed two consecutive identical tool calls — shell{command: 'python3 -m unittest tests/test_price.py'} repeated at turns 4 and 5 (ref: data/transcripts/31a6df12-5ec7-4e82-a1a4-d9cc59a53b9e.jsonl#turn=4). The run passed. The redundancy appears to be a test re-run rather than a degenerate loop — the model ran the test suite twice before committing its fix. Every other run was clean on this pattern.

Where it sits on the leaderboard

[Observed]

(verified: verification/cost_breakdown.csv, verification/pass_rate_by_task.csv)

One run behind its 31B sibling on raw score. At Q4_K_M, the 12B runs on an A10G (24 GiB). The 31B needs an L4 or equivalent. On a g6.4xlarge (A10G, the hardware used in this campaign), the 31B would require careful GPU sizing. The 12B fits without constraint.

The gap to the API tier is real. At 73.3% raw (or 91.7% on tasks that ran), the 12B is 20 points below the top cluster. For deployment decisions that need 95%+ task reliability, a local 12B model is not the answer. For workloads where VRAM budget, cost, or data locality are binding constraints, 91.7% on eight of ten task types is something to measure against.

What we don’t know

[Speculation]

Three open questions from this campaign:

task_09 and task_10 results: We have no result on task_09 (know when to stop) or task_10 (SQL investigation) because the EC2 connection dropped before those tasks ran. task_09 is the hardest task in the suite — no model below Claude Sonnet 4.6 has gone better than 1/3 on it. Whether Gemma 4 12B would match or exceed the 31B (0/3 model failures) on task_09 is unknown. On task_10, the 31B scored 3/3; we have no comparable number for the 12B.

Leakage frequency under longer contexts: The chat template leakage appeared once in 30 runs. Whether that rate changes meaningfully in production workloads with longer conversation histories — multi-turn sessions, large system prompts, mid-conversation tool loops — is untested. A targeted follow-up with higher context loads would give a cleaner frequency estimate.

No predictions filed: No predictions file was committed before this campaign. The adversarial predictions workflow (per the campaign spec) requires a predictions.md committed before the run; it was not present. Same process gap as the 31B campaign. There is no formal pre-run versus post-run scoring of what was expected versus what happened.

The case for running the 12B

[Observed]

At Q4_K_M quantization, Gemma 4 12B weighs roughly 7.5 GB of VRAM. It passed 8 of the 10 task types it actually ran, at 91.7%. It beat its 31B sibling on code refactoring (3/3 vs 1/3) and log investigation (2/3 vs 1/3). API cost: $0.00.

The failure modes are bounded: one context-triggered leakage event in 30 runs, one task where the model short-cut the investigation, and an infrastructure drop at the end of the campaign that had nothing to do with the model.

The production concern is the chat template leakage. It fires at low frequency, but when it fires the output is 4096 tokens of unusable text and 146 wasted seconds. Any wrapper that sends Gemma 4 12B into a tool-use loop should validate tool_calls is non-null before parsing the assistant turn. Text-only turns during function calling are an error signal, not a soft fallback.

Beyond that, the 12B is a 7.5 GB model that handles the agentic core well enough to be worth measuring. The gap to frontier API models is real and measured. So is the cost difference.