Nova 2 Lite: 21/30 on agentic-core-v1. One point ahead of Nova Pro, half the cost, different failures.
Campaign: 2026-06-20-nova-2-lite-agentic-core-v1
Model: Amazon Nova 2 Lite (us.amazon.nova-2-lite-v1:0, AWS Bedrock Converse API, us-east-1, cross-region INFERENCE_PROFILE)
Harness: agentic-core-v1 (10 tasks × 3 runs = 30 total)
Campaign date: 2026-06-20
Amazon’s v2 lightweight model runs the same 30-task harness we put Nova Pro through in May 2026. The headline: one more pass, roughly half the cost per pass, and a failure profile that looks nothing like its predecessor.
Nova 2 Lite scored 21/30 (70%) at $0.0035/pass ($0.074 total). Nova Pro scored 20/30 at $0.0068/pass. The gap on score is small. The gap on cost is real. And under those near-identical totals, the tasks where each model fails are different enough that comparing them at the summary level misses the actual story.
The nine failures were all wrong_answer: the task checker rejected the output. No gave_up_mid_plan, no tool_call_hallucinated, no sessions that ended without producing anything. The model attempted every run and produced output. Some of that output was wrong.
What the harness measures
[Observed — harness spec]
agentic-core-v1 runs 10 software engineering tasks, 3 independent runs each, 30 total. The work covers what a deployed agent would actually encounter: fix a failing test, refactor duplicated code, investigate a log file, trace execution through a codebase, make a targeted minimal code fix, handle an ambiguous requirement with a parallel deliverable, execute a sequential multi-step plan, recover from an injected tool error, recognize when a requested computation is impossible, and run a SQL investigation.
Two tasks are structurally different from the others. task_09 presents three rows of data and asks for a ten-day moving average: three data points cannot support a ten-day window, and the correct response is to refuse and say why, rather than produce a number. task_08 deliberately injects a file-not-found error on the first tool call; the model needs to read the error, find the correct path, and produce verified output before the run ends.
A pass requires correct task completion. Failure modes tracked by the harness: wrong_answer (output rejected by the checker), gave_up_mid_plan (hit the turn limit before producing a final answer), tool_call_hallucinated, tool_call_malformed.
What happened
[Observed — pass_rate_by_task.csv, cost_breakdown.csv]
| Task | Score | Fail mode |
|---|---|---|
| task_01_fix_failing_test | 3/3 | — |
| task_02_refactor_duplicated_code | 2/3 | wrong_answer (1 run) |
| task_03_investigate_log | 3/3 | — |
| task_04_trace_through_codebase | 1/3 | wrong_answer (2 runs) |
| task_05_minimal_fix | 0/3 | wrong_answer (all runs) |
| task_06_handle_ambiguous_requirement | 3/3 | — |
| task_07_multi_step_plan | 3/3 | — |
| task_08_recover_from_tool_error | 3/3 | — |
| task_09_know_when_to_stop | 0/3 | wrong_answer (trap — expected) |
| task_10_sql_investigation | 3/3 | — |
Total: 21/30. $0.0741 total. $0.0035/pass.
Six tasks clean. Four tasks with failures, all classified wrong_answer. The failure mode histogram from the evidence pack reads: passed=21, wrong_answer=9. That is the complete distribution. (verified: failure_mode_histogram.csv)
The failure mode swap
[Observed — pass_rate_by_task.csv, cross_task_consistency.md]
On paper, 21/30 vs 20/30 is one pass. Look at which tasks account for those totals and the picture changes.
Nova Pro’s two consistent failure points were task_02 (refactoring contract) and task_04 (trace through codebase). Both scored 0/3. Nova Pro read the refactor task and optimized for making tests pass, which meant rewriting the test API instead of preserving it. Three runs, same wrong strategy. On task_04, the model produced a trace every run and the checker rejected it every time.
Nova 2 Lite improved on both. task_02 went from 0/3 to 2/3. task_04 went from 0/3 to 1/3. Neither is a full fix, but the model is no longer failing those tasks uniformly.
The cost of those gains was task_05.
Nova Pro scored 3/3 on task_05, the minimal fix task. Nova 2 Lite scored 0/3. The task asks for a specific bug fix with a hard constraint: the diff must be 10 lines or fewer. At an average of 5.0 tool calls per run, Nova 2 Lite was not giving up early; it was working through the problem and producing output that the checker rejected every time. [Speculation] The most likely failure modes are a patch that fixes the symptom but not the root cause, or a correct fix that exceeds the 10-line diff constraint. Which one isn’t visible from the campaign data alone.
task_06 (handle ambiguous requirement) also moved, from 2/3 to 3/3. This task requires both writing the implementation and leaving assumption notes in a separate file. Nova Pro passed 2 of 3 runs on the parallel deliverable discipline; Nova 2 Lite passed all three.
Put together: +2 on task_02, +1 on task_04, +1 on task_06, -3 on task_05. Net: +1. The models arrived at similar totals via different paths.
Why did task_05 regress?
[Observed — tool_calls_by_task.csv, failure_mode_histogram.csv]
This is the outstanding question from the campaign.
Nova Pro handled task_05 cleanly across all three runs: fix a bug, keep the diff small, verify the test passes. It was one of Nova Pro’s most reliable tasks. Nova 2 Lite attempted it seriously (5 tool calls per run average) and failed all three.
[Observed] The failures are all wrong_answer, meaning the output file was present but the checker rejected it. The model did not time out or give up.
[Speculation] The constrained code fix format, which evaluates both correctness and diff size, may be sensitive to how the model constructs its output. A model that has been tuned toward verbose multi-step tool use (which the v2 agentic improvements suggest) might naturally produce larger diffs. That’s speculative. What is not speculative is that Nova 2 Lite consistently failed a task that Nova Pro consistently passed, and the failure mode is wrong_answer rather than anything the harness can pinpoint further.
Where does 21/30 land on the leaderboard?
[Observed — leaderboard data, cost_breakdown.csv]
| Model | Score | Pass rate | Cost/pass |
|---|---|---|---|
| Claude Sonnet 4.6 | 28/30 | 93.3% | $0.0514 |
| Devstral 2 123B | 27/30 | 90.0% | $0.0019 |
| Mistral Large 3 | 27/30 | 90.0% | $0.0022 |
| GPT-5.5 | 27/30 | 90.0% | $0.0700 |
| GPT-OSS 20B | 25/30 | 83.3% | $0.000481 |
| Gemini 3.1 Pro | 23/30 | 76.7% | $0.0369 |
| Palmyra X5 | 23/30 | 76.7% | $0.0022 |
| GPT-OSS 120B | 23/30 | 76.7% | $0.0013 |
| Nova 2 Lite | 21/30 | 70.0% | $0.0035 |
| Llama 3.3 70B | 20/30 | 66.7% | $0.0045 |
| Nova Pro (v1) | 20/30 | 66.7% | $0.0068 |
Nova 2 Lite at 21/30 finishes four points below GPT-OSS 20B, which scored 25/30 (83.3%). The cost contrast is still worth noting: GPT-OSS 20B runs at $0.000481/pass against $0.0035/pass for Nova 2 Lite. Higher score at lower cost means it remains the stronger choice if budget is the deciding factor at this end of the leaderboard.
The vertical comparison (Nova 2 Lite vs Nova Pro) is the more relevant one for AWS users. One extra pass, $0.0033 cheaper per pass. If you are running Nova Pro for agentic workloads, there is no case for staying on it. The v2 upgrade is strictly better on both dimensions.
The gap between 21/30 and the 23/30 cluster (Gemini 3.1 Pro, Palmyra X5, GPT-OSS 120B) is two tasks. Recovering task_05 (0/3 to 2/3 would be enough) or stabilizing task_04 (1/3 to 2/3) would close most of it. Both are tractable failure modes, not fundamental capability gaps.
The cost profile
[Observed — cost_breakdown.csv]
Total campaign cost: $0.0741. Average per run: $0.0025. Most expensive task: task_03 (log investigation) at $0.0126 for three runs: the model reads through a substantial log file on each pass, which drives input token counts up (37,616 total input tokens for task_03 alone). Cheapest task: task_08 (error recovery) at $0.0037 for three runs.
[Observed] The campaign completed in under 5 minutes wall-clock time for 30 runs. All tasks completed under 7 seconds average. task_04 was the slowest at 4.92 seconds average (reading multiple files for the codebase trace). task_08 was fastest at 2.03 seconds average.
Amazon Nova models are fast. That held in v2.
What we still don’t know
[Speculation — not tested in this campaign]
task_05’s 0/3 failure is the unresolved question. The model worked the problem (5 tool calls per run average) and got it wrong every time. [Speculation] Inspecting the rejected output from each run would distinguish between a wrong-logic failure (patching the symptom) and a constraint violation (correct fix but diff too long). The campaign data shows the outcome, not the mechanism.
task_09 (the impossible-computation trap) is failed by every non-reasoning model in this dataset with one or two single-run exceptions. [Speculation] Whether an extended-thinking configuration changes that outcome for Nova 2 Lite is untested. Nova 2 Lite ran in standard mode.
task_04 passed 1 of 3 runs. [Speculation] Whether that single pass is stable capability (the model can do it but inconsistently) or a single-run statistical anomaly is unclear from three runs. The variance is worth watching in any follow-on campaign.