AgentBench / results
← All models

Gemma 4 12B QAT Q4_0

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

QAT-Q4_0 · 6.98 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · QAT-Q4_0

QAT-Q4_0 · 6.98 GB of weights · vulkan · context 16384 · GPU layers 99

Medium · 5–under 20B · dense · 11.9B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

71.5 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking offDetails
Everyday coding tasksTask pass rate
Pass rate
13/16 · 81%
Answers completed
14/16 · 88%
Correct / completed
13/14 · 93%
13 full credit1 incorrect2 failed to finish
16/16 scored13.3s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
72.7%
Full-credit rate
8/11 · 73%
Answers completed
10/11 · 91%
Full credit / completed
8/10 · 80%
8 full credit2 incorrect1 failed to finish
11/11 scored39.1s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
80.9%
Full-credit rate
27/34 · 79%
Answers completed
34/34 · 100%
Full credit / completed
27/34 · 79%
27 full credit1 partial credit6 incorrect
34/34 scored31.3s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
56.1%
Full-credit rate
6/11 · 55%
Answers completed
11/11 · 100%
Full credit / completed
6/11 · 55%
6 full credit1 partial credit4 incorrect
11/11 scored32.6s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
20/23 · 87%
Answers completed
23/23 · 100%
Correct / completed
20/23 · 87%
20 full credit3 incorrect
23/23 scored9.4s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
15/16 · 94%
Answers completed
16/16 · 100%
Correct / completed
15/16 · 94%
15 full credit1 incorrect
16/16 scored1.4s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
5/7 · 71%
Answers completed
7/7 · 100%
Correct / completed
5/7 · 71%
5 full credit2 incorrect
7/7 scored2.0s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
0.0%
Full-credit rate
0/8 · 0%
Answers completed
8/8 · 100%
Full credit / completed
0/8 · 0%
8 incorrect
8/8 scored12.6s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
12/12 · 100%
Answers completed
12/12 · 100%
Correct / completed
12/12 · 100%
12 full credit
12/12 scored2.0s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this model for tool-assisted project work and logic puzzles, not research-paper question answering. With thinking off, it completed most larger practical tasks and performed well on puzzles. Larger-task completion took substantial time in these runs. Research-paper answers remained very poor even with thinking enabled.

Evidence for this judgment

Logic and number puzzles
Thinking off. Medium · 5–under 20B · dense. Rank 1 of 10 in this recorded comparison.

Larger practical tasks · Thinking off
Average score72.7/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes8 full credit · 0 partial · 3 no credit
Typical time per task (median)39.1 seconds
Tasks timed11 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off
Average score80.9/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes27 full credit · 1 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking off
Average score0.0/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking on
Average score10.9/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 3 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T04:01:19.959016Z
Review time
2026-10-22T07:26:27.144200+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
da5f0279df834acf66d76dfee3a0e46f60144eb181e22abaaaa5b9de67700190
Original analysis input hash
d7a3b1189787333e4f05d724b1bb4c1218d3fd5981562cce240d8e559580360e
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-9cf088025601478974945e433adb2cec26043b9a5b15bdc9a1c06d51598bea31
Summary revision
summary-f1c15763d657ea10d83d8c2a811cc88f7a3d0d1bea7640c1763e2e1e533e396a

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (21)
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1a0f88f1e446b661d2fc72a873918d936c3d10880bf4f01f66415217958e8c2b
    Summary revision
    summary-0e6e8bf2a6ca8a4f8989642ce67bc5234c0887e4fed436b578680e39f01b8a04
    Input summary revisions
    summary-0fd13b1645d8a70282419c521e2f627ea775ccab9858c2b5a15509bfa331f0b9, summary-3c8b98fc0387d5720e6e9ad7a960c4afffdc9dd0da0d6368ee13af9ceb190884, summary-6cfeeb279f1361d02608c2a00b6bea318851855b521cc99642971fbb2e218b25, summary-c14abf2bcad25cc8ce36527c5fd2cd73cd0787366d450a01f03996d3cabd4a8f, summary-ca67ce77c0b6f296af029e8eae8fb374da31963aeb693c7f9e7f2c5804f109e4, summary-cd6d2087bb3962fd35db7e9115f88d4ab952e37786ea6d1a74a2479603ce25a3
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    ab11784e69453553256385a8354c1429b0e9edcc0f2152097369c1c0fa1ef8a5
    Summary revision
    summary-0fd13b1645d8a70282419c521e2f627ea775ccab9858c2b5a15509bfa331f0b9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    70db95410d04cdfa6dbce3ff307c4e14489b186cddcf26b6e9b5da05b59a98a5
    Summary revision
    summary-2d46d5ca948d08fbf0050cb03e0412c7fd53b4b063c414000c9ae4a5a1fd2dc7
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7d4cf25ca2183e8a566336beebb197de43ee38d5747cfb3d60c58fb74efbd11f
    Summary revision
    summary-3c8b98fc0387d5720e6e9ad7a960c4afffdc9dd0da0d6368ee13af9ceb190884
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a68f3365b2e021cff65a4d11848727731879e7beefa71ce9c9e2cb1fc9e5f4ae
    Summary revision
    summary-4085bf652f5dcc6fd35a9dd3a341ce0d2ce814057b1eb13820541de4d1e56c24
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    46d89fcf3aa4f53640767b6d9b1e397837be11bb36ff3fd41b2e107b40bcece0
    Summary revision
    summary-4f0e2becd114b69d0e8564469d12d2b1e329a06f8a8983baeabeeb400c7228d8
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    aa758389549250664f04eee6e0658f6f82609023d6e7b6f471b91089d616aa32
    Summary revision
    summary-59c1c4ccf15b03aab3f8850f4f4f511213ad7002aac2e3c921a69a59fee65243
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8f2f957dff03c32d6ffa684c0e96dd262ed68acfa27ac357c924935661bfcd4e
    Summary revision
    summary-63074cd6477461d474ef097457272ba9ce4b4e27f6eec8822b5141b5d8a7db48
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f3f868db5e986fd99ebff88f5fa1ed6b4fb619eea9473e074a65416c32dc86c5
    Summary revision
    summary-691a2b4cf8adfdbda5f1e289ed183cd47dda16425faef20599570a6b00eb2319
    Input summary revisions
    summary-0e6e8bf2a6ca8a4f8989642ce67bc5234c0887e4fed436b578680e39f01b8a04, summary-2d46d5ca948d08fbf0050cb03e0412c7fd53b4b063c414000c9ae4a5a1fd2dc7, summary-4085bf652f5dcc6fd35a9dd3a341ce0d2ce814057b1eb13820541de4d1e56c24, summary-63074cd6477461d474ef097457272ba9ce4b4e27f6eec8822b5141b5d8a7db48, summary-69528f5d57959d7f531f01ce9328e85992b26463c29a3c6bad45b7b9640eb104, summary-9a02952a9779d7bd49ee9a6b1282ccbc22488e7fd5c89b663229834c2e3d372e, summary-b0067f441099323cbd75f29a9844d019e2f81e25782541515d5ef348fe5d726c, summary-eb2689a35fc7321039a62e8e1e1a308535667f7ff3c7de74d4db2bbfed04fac6, summary-fb69dd269e298f0efc2ebeb5604df338a4608a76bfd29a45e905a0d42eab6280
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d342fd7b302fdda279b55fe43f98a83497c962cbd4dca3208ddf7f86539c2d8a
    Summary revision
    summary-69528f5d57959d7f531f01ce9328e85992b26463c29a3c6bad45b7b9640eb104
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    12d64a6be6857bdd0ecbff6bf36fa0634922d3807aa37fb484f576fc9994ee31
    Summary revision
    summary-6cfeeb279f1361d02608c2a00b6bea318851855b521cc99642971fbb2e218b25
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    51854939ab85e910f091886abf36536b2af0bed8e3fb266278f217df15e4dca7
    Summary revision
    summary-857b9a38a4d4bc457f70ce5a73f03969944fd5795ec4566adcef22017460d08d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    563e4b3addbf36d6ef0b16d4905eb2f62bd4e47713795e31e1a8d4cf399b2681
    Summary revision
    summary-9a02952a9779d7bd49ee9a6b1282ccbc22488e7fd5c89b663229834c2e3d372e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    cdf4fc6d692dfc77babcc5d39c8037eae834e669376f0f8732aea1d14556cdd3
    Summary revision
    summary-a5c000097ad6fd322e6c7e9cb94384b562cbaa6ea629701d3a817e31728362c0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    adb1bc4b9160ceb6fff06063551fb3c8b10049edb05c1f4133ac7ecd350d7b15
    Summary revision
    summary-b0067f441099323cbd75f29a9844d019e2f81e25782541515d5ef348fe5d726c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c4dcb8c20cb673317204df7312c0db4bbe7c2e1f253052a799d1d8bd2441b93a
    Summary revision
    summary-c14abf2bcad25cc8ce36527c5fd2cd73cd0787366d450a01f03996d3cabd4a8f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9ecba8b6c31904e8dfd9e1cb9f7e45f0fd0951194624f9fc5660ff814ec49b95
    Summary revision
    summary-ca67ce77c0b6f296af029e8eae8fb374da31963aeb693c7f9e7f2c5804f109e4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    01ac696fc562d3b4cb6c8e9c50fc644bcb696b55d9c68e2757ee3533b14cf40b
    Summary revision
    summary-cd6d2087bb3962fd35db7e9115f88d4ab952e37786ea6d1a74a2479603ce25a3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    38d91cdd77ba2beaea9e5f26aa00dc80c961e9204de7ff947c8074b59bcde3e1
    Summary revision
    summary-eb2689a35fc7321039a62e8e1e1a308535667f7ff3c7de74d4db2bbfed04fac6
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7c19e48720cb4852a62718998a6f75860e7fe12ead46846daec7333b84d6018d
    Summary revision
    summary-fb69dd269e298f0efc2ebeb5604df338a4608a76bfd29a45e905a0d42eab6280
    Input summary revisions
    summary-4f0e2becd114b69d0e8564469d12d2b1e329a06f8a8983baeabeeb400c7228d8, summary-59c1c4ccf15b03aab3f8850f4f4f511213ad7002aac2e3c921a69a59fee65243, summary-857b9a38a4d4bc457f70ce5a73f03969944fd5795ec4566adcef22017460d08d, summary-a5c000097ad6fd322e6c7e9cb94384b562cbaa6ea629701d3a817e31728362c0, summary-fb843ebf3209e7066ab336a0fdbb89505fb002212640cd3551a51482e0f09dec
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1b66cc602e0d5c110e128e86d95d11240965d2dbd7b891f1eee1a3511d1266ab
    Summary revision
    summary-fb843ebf3209e7066ab336a0fdbb89505fb002212640cd3551a51482e0f09dec
    Input summary revisions
    Direct evidence analysis

Good at

Handles longer tasks with tools well: most larger practical tasks succeeded in either mode. That makes it worth considering for building small apps and investigating files, not just isolated code edits.

Evidence for this judgment
Larger practical tasks · Thinking off
Average score72.7/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes8 full credit · 0 partial · 3 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score81.8/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes9 full credit · 0 partial · 2 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Solves many logic puzzles without thinking enabled. It also passed every source-challenge test in that mode, keeping supported answers under pressure while accepting genuine source updates—a useful distinction from simply agreeing with corrections.

Evidence for this judgment
Logic and number puzzles · Thinking off
Average score80.9/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes27 full credit · 1 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Research-paper question answering is a major weakness in either mode, with extremely low answer-matching scores. Calculations from passages were also uneven, so check document-based answers carefully.

Evidence for this judgment
Reading research papers · Thinking off
Average score0.0/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking on
Average score10.9/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 3 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Thinking off
Tasks passed5 of 7 (71.4%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Thinking on
Tasks passed6 of 7 (85.7%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Can get stuck repeating a failed tool call instead of changing approach. In the examples checked, this exhausted tool access, although a successful final answer was still possible despite those mistakes.

Evidence for this judgment

Example 1: Larger practical tasks
Thinking on. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 2: Larger practical tasks
Thinking off. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Should you enable thinking?

Default to thinking off for routine coding and puzzles: coding improved modestly with thinking, but average task time rose substantially; puzzle scores slipped slightly. Enable it selectively for difficult math and reasoning, where the average score gain was meaningful.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on81.3% · Task pass rate → 93.8% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking on+12.5 score points
Average time change with thinking on36.0 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off versus on
Thinking off → on80.9/100 · Mean puzzle credit → 79.4/100 · Mean puzzle credit
Paired test coverage34 of 34 tasks compared in both modes
Tasks timed in both modes34 pairs
Score change with thinking on-1.5 score points
Average time change with thinking on43.0 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking off versus on
Thinking off → on56.1/100 · Mean answer credit → 73.0/100 · Mean answer credit
Paired test coverage11 of 11 tasks compared in both modes
Tasks timed in both modes11 pairs
Score change with thinking on+16.9 score points
Average time change with thinking on41.2 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking off, its larger-practical completion rate tied two other models for the best result in the dense-parameter comparison. Most alternatives in the matched timing cohort finished sooner, making this a success-over-responsiveness choice.

Evidence for this judgment

Larger practical tasks
Thinking off. Medium · 5–under 20B · dense. Tied at rank 1 of 10 in this recorded comparison.

Speed: Larger practical tasks
Thinking off. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Larger practical tasksThinking off · Medium · 5–under 20B · dense

Build small apps, organize information and investigate files or web pages.

Mean task credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 12B11.9B total parameters72.7/100
    Same score
  2. =1 Gemma 4 12B QAT Q4_011.9B total parameters72.7/100
    This model
  3. =1 Ministral 3 14B13.5B total parameters72.7/100
    Same score
  4. #4 Ornith 1.5 9B Q4_K_M9.2B total parameters54.5/100
    18.2 points lower
  5. =5 Ministral 3 8B Instruct Q4_K_M8.5B total parameters45.5/100
    27.3 points lower
  6. =5 Qwen3.5 9B9.0B total parameters45.5/100
    27.3 points lower
  7. =7 Gemma 4 E4B7.5B total parameters36.4/100
    36.4 points lower
  8. =7 Granite 4.2 8B Q4_08.8B total parameters36.4/100
    36.4 points lower
  9. #9 Nemotron Nano 9B v2 Q4_08.9B total parameters18.2/100
    54.5 points lower
  10. #10 Qwen2.5 Coder 7B Instruct Q4_K_M7.6B total parameters0.0/100
    72.7 points lower

Other mode, this model only: thinking on · 81.8/100 · Mean task credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Speed and task success: Larger practical tasksThinking off · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedAverage score
LFM2.5 1.2B Q4_01.5 seconds11 of 11 tasks timed311.1 tokens/s0.0/10011 of 11 scored
Phi-4 Mini Instruct 3.8B Q4_K_M4.8 seconds11 of 11 tasks timed100.1 tokens/s0.0/10011 of 11 scored
Qwen2.5 Coder 7B Instruct Q4_K_M7.0 seconds11 of 11 tasks timed61.6 tokens/s0.0/10011 of 11 scored
SmolLM3 3B Q4_K_M8.0 seconds11 of 11 tasks timed112.3 tokens/s18.2/10011 of 11 scored
Granite 4.0 Micro 3.4B Q4_013.4 seconds11 of 11 tasks timed93.1 tokens/s18.2/10011 of 11 scored
Ling 3.0 Tiny Q4_019.6 seconds11 of 11 tasks timed128.5 tokens/s27.3/10011 of 11 scored
Nemotron Nano 9B v2 Q4_019.7 seconds11 of 11 tasks timed53.1 tokens/s18.2/10011 of 11 scored
Qwen3.5 0.8B Q4_029.3 seconds11 of 11 tasks timed213.0 tokens/s9.1/10011 of 11 scored
Ministral 3 8B Instruct Q4_K_M29.7 seconds11 of 11 tasks timed49.5 tokens/s45.5/10011 of 11 scored
Ornith 1.5 9B Q4_K_M32.2 seconds11 of 11 tasks timed53.8 tokens/s54.5/10011 of 11 scored
Gemma 4 12B QAT Q4_0This model39.1 seconds11 of 11 tasks timed35.3 tokens/s72.7/10011 of 11 scored
Mellum 2.1 12B-A2.5B Thinking57.8 seconds11 of 11 tasks timed132.6 tokens/s27.3/10011 of 11 scored
Granite 4.2 8B Q4_073.6 seconds11 of 11 tasks timed43.9 tokens/s36.4/10011 of 11 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

With thinking on, its published hard-problem credit narrowly exceeded the nearest alternative, Ornith 1.5 9B Q4_K_M, but median task time was substantially longer.

Evidence for this judgment

Published hard problems
Thinking on. Medium · 5–under 20B · dense. Rank 1 of 9 in this recorded comparison.

Speed: Published hard problems
Thinking on. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Published hard problemsThinking on · Medium · 5–under 20B · dense

Answer selected difficult math and reasoning questions from LiveBench.

Mean answer credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Gemma 4 12B QAT Q4_011.9B total parameters73.0/100
    This model
  2. #2 Ornith 1.5 9B Q4_K_M9.2B total parameters72.3/100
    0.8 points lower
  3. #3 Gemma 4 12B11.9B total parameters63.9/100
    9.1 points lower
  4. #4 Gemma 4 E4B7.5B total parameters46.7/100
    26.4 points lower
  5. =5 Ministral 3 14B Reasoning 2512 Q4_K_M13.5B total parameters37.6/100
    35.5 points lower
  6. =5 Qwen3.5 9B9.0B total parameters37.6/100
    35.5 points lower
  7. #7 Granite 4.2 8B Q4_08.8B total parameters20.8/100
    52.3 points lower
  8. #8 Nemotron Nano 9B v2 Q4_08.9B total parameters18.2/100
    54.9 points lower
  9. #9 Ministral 3 8B Reasoning 2512 Q4_K_M8.5B total parameters11.2/100
    61.8 points lower

Other mode, this model only: thinking off · 56.1/100 · Mean answer credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Speed and task success: Published hard problemsThinking on · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedAverage score
LFM2.5 1.2B Thinking Q4_K_M13.2 seconds11 of 11 tasks timed293.2 tokens/s12.9/10011 of 11 scored
Ling 3.0 Tiny Q4_016.9 seconds11 of 11 tasks timed132.4 tokens/s10.9/10011 of 11 scored
SmolLM3 3B Q4_K_M18.3 seconds11 of 11 tasks timed109.4 tokens/s35.8/10011 of 11 scored
Mellum 2.1 12B-A2.5B Thinking30.2 seconds11 of 11 tasks timed139.1 tokens/s22.3/10011 of 11 scored
Granite 4.2 8B Q4_039.2 seconds11 of 11 tasks timed54.2 tokens/s20.8/10011 of 11 scored
Ornith 1.5 9B Q4_K_M45.8 seconds11 of 11 tasks timed54.0 tokens/s72.3/10011 of 11 scored
Ministral 3 8B Reasoning 2512 Q4_K_M59.7 seconds11 of 11 tasks timed51.8 tokens/s11.2/10011 of 11 scored
Nemotron Nano 9B v2 Q4_078.0 seconds11 of 11 tasks timed53.5 tokens/s18.2/10011 of 11 scored
Gemma 4 12B QAT Q4_0This model84.3 seconds11 of 11 tasks timed35.8 tokens/s73.0/10011 of 11 scored
Ministral 3 14B Reasoning 2512 Q4_K_M93.5 seconds11 of 11 tasks timed34.7 tokens/s37.6/10011 of 11 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Worth knowing

Publication redactions, withheld artifacts and source-linked paper text limit reader inspection of selected examples; they do not establish that the model lacked those inputs.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Thinking offMean matched task credit81.3%
Thinking onMean matched task credit93.8%

Thinking on took 36.0 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on81.3% → 93.8%
Mean paired credit change+12.5 percentage points
Question outcomes2 improved · 14 unchanged · 0 worse
Mean added task time+36.0s
Mean task time, off → on17.0s → 53.0s
Median task time, off → on13.3s → 33.6s
Median of paired time changes+20.4s
Total added task time+575.9s
Ratio of summed times, on / off3.1×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking off · temperature 0.6

48/48 repeats scored · 36 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 9 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested off · verified
  • Seed 101 · requested off · verified
  • Seed 303 · requested off · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

13 full credit1 incorrect2 failed to finish

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-daa1f114cbc0392b7d200e66
Matching candidate set: paired-daa1f114cbc0392b7d200e66

Qwen3.5 9B UD-Q5_K_XL

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-9b097653ded9dc087919acf0, fact-3f910f804bc145423c5f4359
Matching candidate set: fact-3f910f804bc145423c5f4359, fact-9f9b21c2355c6bdb5b2dd393, fact-e2c619b31f8a88c1472ca60e, fact-e57043e6128b693616991807, fact-1eaacebf5d4775d7901754f8, fact-5c36b6f6bfeaa5ee466506f6, fact-bb58072f1c77808045477c91, fact-bef3eb6c6baa2d44e3aa0649, fact-ec911e6561843ecc14d3482a, fact-eae080b73ed21e846eaa5d1d, fact-81703cf46648d0b2de523fad, fact-490482dd62af02e41d2fc20e, fact-6808af8c64a1e7875a14e7a6, fact-2e2fe4760014b5b180de8620, fact-6d911759dd69ca545ee8be5c, fact-200d3a80040bda993709f78f, fact-f0e7d2286cd806072751d966, fact-eacdd70113b5633f3f223c12

Tested weights

QAT-Q4_0 · 6.98 GB

google · recorded source

gemma-4-12b-it-qat-q4_0.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.