AgentBench / results
← All models

Nemotron Nano 9B v2 Q4_0

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_0 · 5.34 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_0

Q4_0 · 5.34 GB of weights · vulkan · context 16384 · GPU layers 99

Medium · 5–under 20B · dense · 8.9B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

36.4 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking offDetails
Everyday coding tasksTask pass rate
Pass rate
6/16 · 38%
Answers completed
16/16 · 100%
Correct / completed
6/16 · 38%
6 full credit10 incorrect
16/16 scored8.9s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
18.2%
Full-credit rate
2/11 · 18%
Answers completed
11/11 · 100%
Full credit / completed
2/11 · 18%
2 full credit9 incorrect
11/11 scored19.7s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
30.8%
Full-credit rate
8/34 · 24%
Answers completed
34/34 · 100%
Full credit / completed
8/34 · 24%
8 full credit9 partial credit17 incorrect
34/34 scored2.9s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
6.7%
Full-credit rate
0/11 · 0%
Answers completed
11/11 · 100%
Full credit / completed
0/11 · 0%
4 partial credit7 incorrect
11/11 scored2.7s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
19/23 · 83%
Answers completed
23/23 · 100%
Correct / completed
19/23 · 83%
19 full credit4 incorrect
23/23 scored4.9s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
14/16 · 88%
Answers completed
16/16 · 100%
Correct / completed
14/16 · 88%
14 full credit2 incorrect
16/16 scored1.3s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
2/7 · 29%
Answers completed
7/7 · 100%
Correct / completed
2/7 · 29%
2 full credit5 incorrect
7/7 scored1.8s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
35.9%
Full-credit rate
0/8 · 0%
Answers completed
8/8 · 100%
Full credit / completed
0/8 · 0%
8 partial credit
8/8 scored9.1s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
0/12 · 0%
Answers completed
12/12 · 100%
Correct / completed
0/12 · 0%
12 incorrect
12/12 scored2.5s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Nemotron Nano 9B v2 Q4_0 is useful for passage-based answers and tightly formatted responses with thinking off, and for handling challenged source information with thinking on. Larger practical tasks and difficult math and reasoning remain weak even with thinking enabled. Choose it for those narrower uses rather than unattended project work.

Evidence for this judgment
Reading without guessing · Thinking off
Tasks passed14 of 16 (87.5%)
Test coverage16 of 16 tasks scored
Tasks timed16 tasks
Typical time per task (median)1.3 seconds
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking off
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking on
Tasks passed11 of 12 (91.7%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score36.4/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 0 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking on
Average score18.2/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes2 full credit · 0 partial · 9 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T05:41:21.802642Z
Review time
2026-10-22T07:26:27.186628+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
3f164af6f4bbce09c3bf89ab62c5b6a69cc1895dfe3803faa302ff5c8f8be35c
Original analysis input hash
dea9524cc37b366f2b78e8876030e88d1aa8ff3e00ac905148d0f35e76d4e471
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-7b1c8ec2ba3763e6ad2152742368081e4ea50d917671549d493137b8418fd9c6
Summary revision
summary-148cab43c95fb5bfdbcd7e58589f16e37f77289b4031a43159b5acf995fbf17f

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (21)
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e68876bc12d05f8a88f0641fed81693f1d880d3adbd845463a4bda947df49c9f
    Summary revision
    summary-0a760c923943535a417bfd6436bba212e00236376597410c126a109426ab4efe
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    249036d883fbb655fe6b2fb8b3a55ab1e881597eae2553ae1a743bd506e5a569
    Summary revision
    summary-1f559e6e61574cdaedb1bfeca60652226d0aad835e0dd8f4905c47046c4462bd
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e7d9dba17dd6a307739e328488920b80fd53c632b0c6b47f5c62f3f4d56c85e8
    Summary revision
    summary-2835f0342d3ffe636f8dfefa92c4bdc4ae9a9abb7338d89e84d6e258fbcf7490
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8700730c508bd094d3ecde3d5f4b4dc3f367d7e08ede162a645b2e3d04022fd4
    Summary revision
    summary-3ad7bf2f0fb47c3a0382c27561cbf352c1905e7042707ea7fb40d343be84b7fd
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    34ae7ee4f4f5f1413d8b9f64a98e1b52021e235f8cd719e6fda5979fe74a34da
    Summary revision
    summary-528b703b1f49652699ff4d1b8409b26714e0fc793f7937fb13b4d3e728be46c4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    40cff63928cb41e8324f80eeb36a51e3d1e4bd5f8a42d83b46aac688a35a94ad
    Summary revision
    summary-5dd3885a19ba3b252e026ab782b5fd6036b6f1b05da8a15bc0ff690f4c17c695
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5ed16fdf8bc03ea4d1327f0ef299edf67e2eb66812c876e30460cf72bcb43f94
    Summary revision
    summary-74d71ea48b332e26ef669dae330da7500e6913f11606f5ccb64380381a210540
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d4ce86aee48990ebb831c3d8e437e9cb84ef79d7e593556e2c9d1d2a7f443a91
    Summary revision
    summary-8945077d0a6340971e8987c2e76fe9d0d9d69d3054d02ffb18f3e73e20ae3d6e
    Input summary revisions
    summary-1f559e6e61574cdaedb1bfeca60652226d0aad835e0dd8f4905c47046c4462bd, summary-5dd3885a19ba3b252e026ab782b5fd6036b6f1b05da8a15bc0ff690f4c17c695, summary-ac591ebfaa290de8795fb04a05f75d1f813d51440836a78ea1ea79f1b905e22d, summary-d6a2a3b0a41ded0b180abcb1a9b50097f3fa9ef42a8f054d835d68c6cd9e1217, summary-ece10cb714ec928fe4c76723dfacf0c0af369d750c3d2e682c3378cfcac4a717
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7940dfaac6ea2f8ea8791f90fa065aa389f46f373ebc9f05446377b3d8d7dcb6
    Summary revision
    summary-89795532f44f1bd31ceb8ed03a6f6bdb329ef9cb4e4d797eea47d7ab73e99681
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8e94523e690fdaf48da98bf6d9f6c5cbe6fd34953f03b47be3dc41d9ff76517a
    Summary revision
    summary-90328f39a444e9f7646b28aa1c13959c33682c42f51b371c7f0f42942b5fc676
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    40b9c2c124181fefe9e2decc30b14caa7368c42d608603d64c4b37e9c3066360
    Summary revision
    summary-9fa63ff585fbcb3166ff9f28df32bbfdbcd7bc772594a35b27c8ace2126bde2f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    28bbbba49b231a17d25864f56e0ec55fbb7a362994c8f4e02d3423d62c34a2ff
    Summary revision
    summary-ac591ebfaa290de8795fb04a05f75d1f813d51440836a78ea1ea79f1b905e22d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6ddfa3b169b4196474b5f68d0a851b5f7649099b03646548a29134f4e367bae4
    Summary revision
    summary-b7f7bb0ff0680c1828f52873420f49071a8e12032e0942ddfc9a229e537b685f
    Input summary revisions
    summary-2835f0342d3ffe636f8dfefa92c4bdc4ae9a9abb7338d89e84d6e258fbcf7490, summary-3ad7bf2f0fb47c3a0382c27561cbf352c1905e7042707ea7fb40d343be84b7fd, summary-528b703b1f49652699ff4d1b8409b26714e0fc793f7937fb13b4d3e728be46c4, summary-8945077d0a6340971e8987c2e76fe9d0d9d69d3054d02ffb18f3e73e20ae3d6e, summary-89795532f44f1bd31ceb8ed03a6f6bdb329ef9cb4e4d797eea47d7ab73e99681, summary-9fa63ff585fbcb3166ff9f28df32bbfdbcd7bc772594a35b27c8ace2126bde2f, summary-b9c342edba36d75b68afe9978e3da706381df6e8e8ace78e389efd9a4726596d, summary-e2d7996674114340fa2b641548097f65b3746a9fd0b0dcb43ff1871dd1e66903, summary-ed2226966f986db0cd91765489e7f9ca8084d552232240fae70b8319f3ead7ee
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    662ceff2451a2e753cb9afae6f5c1f145b1690e0bdc1ec404733f7270d9538c5
    Summary revision
    summary-b9c342edba36d75b68afe9978e3da706381df6e8e8ace78e389efd9a4726596d
    Input summary revisions
    summary-0a760c923943535a417bfd6436bba212e00236376597410c126a109426ab4efe, summary-74d71ea48b332e26ef669dae330da7500e6913f11606f5ccb64380381a210540, summary-90328f39a444e9f7646b28aa1c13959c33682c42f51b371c7f0f42942b5fc676, summary-c15492a4dc1e21971f0941692b5ddff70cbefaa0e436e1a3948884b187e42d01, summary-fd05efd9984a4349a42aff990e40bd0fcf2381f9840ab8773068e80905c50ceb, summary-ffdbc6edfe52478e4a6e18a0b5d92473706e8e723825d2ec5ead6169dac3aa39
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a73af94245ac73d3927183739f0a161e3b0523139193e632a99e6f6b55bb8ca0
    Summary revision
    summary-c15492a4dc1e21971f0941692b5ddff70cbefaa0e436e1a3948884b187e42d01
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1f0d55e721dafd13abfda6c2b5bd75f03d8ee8f787c0a299a44c676bb259c418
    Summary revision
    summary-d6a2a3b0a41ded0b180abcb1a9b50097f3fa9ef42a8f054d835d68c6cd9e1217
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d7e247680e71e85edf2d2eb3004e11b6a3d471274cac4c6c66bf4044a2069a26
    Summary revision
    summary-e2d7996674114340fa2b641548097f65b3746a9fd0b0dcb43ff1871dd1e66903
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6054fa75c57e4bef7c61ae6596a62ed6f3e8c72fe0562c9484c9e093f7e5a575
    Summary revision
    summary-ece10cb714ec928fe4c76723dfacf0c0af369d750c3d2e682c3378cfcac4a717
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e94cd302e345d4d30d1d71ffcb52bb6c18d2d6dcd9675af94d986ea0eb895530
    Summary revision
    summary-ed2226966f986db0cd91765489e7f9ca8084d552232240fae70b8319f3ead7ee
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    186f59ea4eaa7ee472543d29d00af53b094a6e0fe0c1d6f9ece823e86c37adb9
    Summary revision
    summary-fd05efd9984a4349a42aff990e40bd0fcf2381f9840ab8773068e80905c50ceb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8fed57fd74a6807c4bf123d254ca6951cdd6ea300aad7a2bb7b1230f3415a65e
    Summary revision
    summary-ffdbc6edfe52478e4a6e18a0b5d92473706e8e723825d2ec5ead6169dac3aa39
    Input summary revisions
    Direct evidence analysis

Good at

Finds answers—or recognizes missing answers—in passages, and follows detailed wording and formatting rules well with thinking off. These are useful applications, though neither was error-free.

Evidence for this judgment
Reading without guessing · Thinking off
Tasks passed14 of 16 (87.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking off
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Handles unsupported challenges and genuine source updates well with thinking enabled. This makes it useful for checking source-backed answers, despite its weaknesses on more demanding work.

Evidence for this judgment
Resisting incorrect corrections · Thinking on
Tasks passed11 of 12 (91.7%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Struggles badly with difficult math and reasoning in either mode. Most larger practical tasks also remained unsuccessful with thinking on, so app-building and file investigations need close supervision.

Evidence for this judgment
Published hard problems · Thinking off
Average score6.7/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes0 full credit · 4 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking on
Average score18.2/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes2 full credit · 0 partial · 9 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score36.4/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 0 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Small-project coding is only moderately successful even with thinking on. Treat file edits and bug fixes as work to verify, not tasks to hand off unchecked.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed10 of 16 (62.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Should you enable thinking?

Leave thinking off for routine passage lookup and strict formatting. It made both slower on these tests, with a narrow accuracy loss for passage answers but a substantial drop in instruction compliance.

Evidence for this judgment
Reading without guessing · Thinking off versus on
Thinking off → on87.5% · Exact-match accuracy → 81.3% · Exact-match accuracy
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking on-6.3 score points
Average time change with thinking on7.0 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking off versus on
Thinking off → on82.6% · Strict prompt compliance → 52.2% · Strict prompt compliance
Paired test coverage23 of 23 tasks compared in both modes
Tasks timed in both modes23 pairs
Score change with thinking on-30.4 score points
Average time change with thinking on7.3 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Enable thinking for challenged source answers: without it, none met all source-handling requirements. For small-project coding it improves success, but the substantial extra wait makes it a selective option rather than an everyday default.

Evidence for this judgment
Resisting incorrect corrections · Thinking off versus on
Thinking off → on0.0% · All-branches pass rate → 91.7% · All-branches pass rate
Paired test coverage12 of 12 tasks compared in both modes
Tasks timed in both modes12 pairs
Score change with thinking on+91.7 score points
Average time change with thinking on7.6 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off
Tasks passed0 of 12 (0.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking off versus on
Thinking off → on37.5% · Task pass rate → 62.5% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking on+25.0 score points
Average time change with thinking on42.0 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking off, passage exact-match accuracy was near the top of the medium dense comparison but not a unique lead: three peers scored one more answer. In the same-machine timing group, most alternatives finished sooner.

Evidence for this judgment

Reading without guessing
Thinking off. Medium · 5–under 20B · dense. Tied at rank 4 of 10 in this recorded comparison.

Speed: Reading without guessing
Thinking off. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Reading without guessingThinking off · Medium · 5–under 20B · dense

Answer from a passage—or say that the passage does not contain the answer.

Exact-match accuracy · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 12B11.9B total parameters93.8%
    1 task more passed
  2. =1 Gemma 4 12B QAT Q4_011.9B total parameters93.8%
    1 task more passed
  3. =1 Ornith 1.5 9B Q4_K_M9.2B total parameters93.8%
    1 task more passed
  4. =4 Ministral 3 8B Instruct Q4_K_M8.5B total parameters87.5%
    Same score
  5. =4 Nemotron Nano 9B v2 Q4_08.9B total parameters87.5%
    This model
  6. =6 Ministral 3 14B13.5B total parameters75.0%
    2 tasks fewer passed
  7. =6 Qwen2.5 Coder 7B Instruct Q4_K_M7.6B total parameters75.0%
    2 tasks fewer passed
  8. =6 Qwen3.5 9B9.0B total parameters75.0%
    2 tasks fewer passed
  9. #9 Gemma 4 E4B7.5B total parameters25.0%
    10 tasks fewer passed
  10. #10 Granite 4.2 8B Q4_08.8B total parameters12.5%
    12 tasks fewer passed

Other mode, this model only: thinking on · 81.3% · Exact-match accuracy.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Speed and task success: Reading without guessingThinking off · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
LFM2.5 1.2B Q4_00.6 seconds16 of 16 tasks timed316.8 tokens/s0 of 1616 of 16 scored
Qwen3.5 0.8B Q4_00.6 seconds16 of 16 tasks timed243.6 tokens/s10 of 1616 of 16 scored
Phi-4 Mini Instruct 3.8B Q4_K_M0.7 seconds16 of 16 tasks timed108.6 tokens/s6 of 1616 of 16 scored
Ling 3.0 Tiny Q4_00.7 seconds16 of 16 tasks timed133.7 tokens/s10 of 1616 of 16 scored
Granite 4.0 Micro 3.4B Q4_00.7 seconds16 of 16 tasks timed113.9 tokens/s10 of 1616 of 16 scored
SmolLM3 3B Q4_K_M0.8 seconds16 of 16 tasks timed128.8 tokens/s1 of 1616 of 16 scored
Mellum 2.1 12B-A2.5B Thinking0.8 seconds16 of 16 tasks timed139.5 tokens/s13 of 1616 of 16 scored
Qwen2.5 Coder 7B Instruct Q4_K_M0.9 seconds16 of 16 tasks timed66.2 tokens/s12 of 1616 of 16 scored
Ministral 3 8B Instruct Q4_K_M1.0 seconds16 of 16 tasks timed55.5 tokens/s14 of 1616 of 16 scored
Ornith 1.5 9B Q4_K_M1.1 seconds16 of 16 tasks timed54.0 tokens/s15 of 1616 of 16 scored
Nemotron Nano 9B v2 Q4_0This model1.3 seconds16 of 16 tasks timed53.5 tokens/s14 of 1616 of 16 scored
Gemma 4 12B QAT Q4_01.4 seconds16 of 16 tasks timed37.3 tokens/s15 of 1616 of 16 scored
Granite 4.2 8B Q4_04.4 seconds16 of 16 tasks timed57.0 tokens/s2 of 1616 of 16 scored
Phi-4 Mini Reasoning 3.8B Q4_K_M4.9 seconds16 of 16 tasks timed103.5 tokens/s0 of 1616 of 16 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

With thinking on, small-coding accuracy was in the lower part of the medium dense cohort, and most alternatives in the matched same-machine timing comparison finished sooner.

Evidence for this judgment

Everyday coding tasks
Thinking on. Medium · 5–under 20B · dense. Rank 8 of 9 in this recorded comparison.

Speed: Everyday coding tasks
Thinking on. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Everyday coding tasksThinking on · Medium · 5–under 20B · dense

Find information in a small project, edit files and fix bugs.

Task pass rate · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 E4B7.5B total parameters100.0%
    6 tasks more passed
  2. =1 Granite 4.2 8B Q4_08.8B total parameters100.0%
    6 tasks more passed
  3. =1 Ornith 1.5 9B Q4_K_M9.2B total parameters100.0%
    6 tasks more passed
  4. =1 Qwen3.5 9B9.0B total parameters100.0%
    6 tasks more passed
  5. =5 Gemma 4 12B11.9B total parameters93.8%
    5 tasks more passed
  6. =5 Gemma 4 12B QAT Q4_011.9B total parameters93.8%
    5 tasks more passed
  7. #7 Ministral 3 8B Reasoning 2512 Q4_K_M8.5B total parameters75.0%
    2 tasks more passed
  8. #8 Nemotron Nano 9B v2 Q4_08.9B total parameters62.5%
    This model
  9. #9 Ministral 3 14B Reasoning 2512 Q4_K_M13.5B total parameters50.0%
    2 tasks fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking off · 37.5% · Task pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Speed and task success: Everyday coding tasksThinking on · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
Ling 3.0 Tiny Q4_08.4 seconds16 of 16 tasks timed122.0 tokens/s15 of 1616 of 16 scored
LFM2.5 1.2B Thinking Q4_K_M11.0 seconds16 of 16 tasks timed293.3 tokens/s2 of 1616 of 16 scored
Ornith 1.5 9B Q4_K_M13.3 seconds16 of 16 tasks timed51.0 tokens/s16 of 1616 of 16 scored
Mellum 2.1 12B-A2.5B Thinking15.2 seconds16 of 16 tasks timed137.3 tokens/s16 of 1616 of 16 scored
Ministral 3 8B Reasoning 2512 Q4_K_M19.0 seconds16 of 16 tasks timed53.6 tokens/s12 of 1616 of 16 scored
Ministral 3 14B Reasoning 2512 Q4_K_M32.7 seconds16 of 16 tasks timed34.3 tokens/s8 of 1616 of 16 scored
Gemma 4 12B QAT Q4_033.6 seconds16 of 16 tasks timed35.3 tokens/s15 of 1616 of 16 scored
Nemotron Nano 9B v2 Q4_0This model34.5 seconds16 of 16 tasks timed53.3 tokens/s10 of 1616 of 16 scored
Granite 4.2 8B Q4_036.7 seconds16 of 16 tasks timed49.9 tokens/s16 of 1616 of 16 scored
SmolLM3 3B Q4_K_M37.7 seconds16 of 16 tasks timed109.3 tokens/s1 of 1616 of 16 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Worth knowing

Publication redactions and withheld or source-link-only material limit readers’ ability to inspect some supporting evidence.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Thinking offMean matched task credit37.5%
Thinking onMean matched task credit62.5%

Thinking on took 42.0 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on37.5% → 62.5%
Mean paired credit change+25.0 percentage points
Question outcomes4 improved · 12 unchanged · 0 worse
Mean added task time+42.0s
Mean task time, off → on12.2s → 54.2s
Median task time, off → on8.9s → 34.5s
Median of paired time changes+26.8s
Total added task time+672.1s
Ratio of summed times, on / off4.4×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking off · temperature 0.6

48/48 repeats scored · 14 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 3 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 303 · requested off · verified
  • Seed 202 · requested off · verified
  • Seed 101 · requested off · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

6 full credit10 incorrect

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-bbc4edfaee8436367e8067fd
Matching candidate set: paired-bbc4edfaee8436367e8067fd

Qwen3.5 9B

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-ec911e6561843ecc14d3482a, fact-bef3eb6c6baa2d44e3aa0649
Matching candidate set: fact-bef3eb6c6baa2d44e3aa0649, fact-eae080b73ed21e846eaa5d1d, fact-490482dd62af02e41d2fc20e, fact-6808af8c64a1e7875a14e7a6, fact-5c36b6f6bfeaa5ee466506f6, fact-1eaacebf5d4775d7901754f8, fact-2e2fe4760014b5b180de8620, fact-6d911759dd69ca545ee8be5c, fact-f0e7d2286cd806072751d966, fact-9f9b21c2355c6bdb5b2dd393, fact-3f910f804bc145423c5f4359, fact-9b097653ded9dc087919acf0, fact-e2c619b31f8a88c1472ca60e, fact-e57043e6128b693616991807, fact-bb58072f1c77808045477c91, fact-81703cf46648d0b2de523fad, fact-200d3a80040bda993709f78f, fact-eacdd70113b5633f3f223c12

Tested weights

Q4_0 · 5.34 GB

bartowski · recorded source

nvidia_NVIDIA-Nemotron-Nano-9B-v2-Q4_0.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.