AgentBench / results
← All models

Gemma 4 26B-A4B QAT Q4_K_XL

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

QAT-UD-Q4_K_XL · 14.25 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · QAT-UD-Q4_K_XL

QAT-UD-Q4_K_XL · 14.25 GB of weights · vulkan · context 16384 · GPU layers 99

Large · 20B and above · MoE · 25.2B total · Active parameter count not recorded

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

79.4 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking offDetails
Everyday coding tasksTask pass rate
Pass rate
16/16 · 100%
Answers completed
16/16 · 100%
Correct / completed
16/16 · 100%
16 full credit
16/16 scored15.8s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
81.8%
Full-credit rate
9/11 · 82%
Answers completed
11/11 · 100%
Full credit / completed
9/11 · 82%
9 full credit2 incorrect
11/11 scored57.5s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
80.9%
Full-credit rate
27/34 · 79%
Answers completed
34/34 · 100%
Full credit / completed
27/34 · 79%
27 full credit1 partial credit6 incorrect
34/34 scored67.5s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
64.9%
Full-credit rate
7/11 · 64%
Answers completed
11/11 · 100%
Full credit / completed
7/11 · 64%
7 full credit2 partial credit2 incorrect
11/11 scored75.7s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
19/23 · 83%
Answers completed
23/23 · 100%
Correct / completed
19/23 · 83%
19 full credit4 incorrect
23/23 scored19.9s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
14/16 · 88%
Answers completed
16/16 · 100%
Correct / completed
14/16 · 88%
14 full credit2 incorrect
16/16 scored1.6s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
5/7 · 71%
Answers completed
7/7 · 100%
Correct / completed
5/7 · 71%
5 full credit2 incorrect
7/7 scored1.9s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
45.4%
Full-credit rate
0/8 · 0%
Answers completed
8/8 · 100%
Full credit / completed
0/8 · 0%
8 partial credit
8/8 scored11.5s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
12/12 · 100%
Answers completed
12/12 · 100%
Correct / completed
12/12 · 100%
12 full credit
12/12 scored3.5s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

This Gemma configuration is a good candidate for small project edits and passage-based answers, but longer tool-driven investigations need supervision. It passed the small coding tests without thinking and answered or abstained correctly throughout the reading tests with thinking enabled. Passage calculations were uneven, and file-search examples showed repeated calls without useful progress. These bounded successes make it worth trying for focused work, not trusting to finish a larger workflow unattended.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Thinking on
Tasks passed6 of 7 (85.7%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Example 3: Larger practical tasks
Thinking off. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 5: Larger practical tasks
Thinking off. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.0 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v2
Generation time
2026-10-10T05:38:11.642984Z
Review time
2026-10-22T05:55:43.655129+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
908c896076f4bffb619e94b43db45139ea9488daae8dec81cfe73e72dcc13a68
Original analysis input hash
510b9df158bab291870dacdb533ca003218023b5268a77129bdf8543e16d5fe3
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-7d7d1cfc40cd60be5ca3eb6ed30829f23a6924f1274c3db49829cc3a8c086c11
Summary revision
summary-c27c42d6d567f374c240f641876f987e7275d6c86f328bdccffc7a2cdaf68c49

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (15)
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a090974fa26a4575c36d55b0da7f152c9202a708203ec79d0be4c6b04f24cfe1
    Summary revision
    summary-062415ed225f8df05f70d71a5e88ff272fe44906c03a335e1cf5e7b72853803a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b319cc27dca28fca5fbaa1114234678eb3ba35a52c9221b066486002c5bb596f
    Summary revision
    summary-08175a42127683e90222af9e87ab44984c31727e2182352435b07a240f98ef14
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e46640fbb3fec40854ba5155cd5144e89db11eb5429c3f0d904d10ac37777801
    Summary revision
    summary-0f5260c33e7a70d349d3f1584a69920987846e14ed7c540baf3ad9ad6210e72e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    22591573b94b1eec1d337422a2d4d5b805fd4fb86f4c6489b37c5adda170c833
    Summary revision
    summary-241f86f347cd21d9265ea4699600d1a248d28ba8297002a64a627b0fda94b918
    Input summary revisions
    summary-5521776f62386eac78e2e3cf06b94783fe8e98f22da907e782963374faf518aa, summary-9d66b1d04a8504c5b91d6afc6b955066a63d7855850d22eeeec626130b69e88c, summary-c6096e1ee9601f50abbf4729a91fc66fee089b452ed5c971d2560e9bd313a995, summary-e35f7a56bc7d2e831df565d9c63ab1cbc5ab305720cf68f96bfb1ff6ec698ba8
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a72b91d7e3cd672340c217f5753c6313886b95b49dc6b5c32715d071559b4be6
    Summary revision
    summary-36cb465f526fdd0ea9401f0a5476b4671a201a4b8c3a39c7e03303484c1b25da
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d72bc4651f64b79a1f76897f9722735b762cc0754c34da0d2bdde0a399723d51
    Summary revision
    summary-4dcde385295398fb65c3b159ebfd0b65c46ae510f41326fc4756fbfe2e69ac02
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    be7c0aeca74809297ae9734f1a000ed41b718648d2c328c4583ce988a9b25821
    Summary revision
    summary-5521776f62386eac78e2e3cf06b94783fe8e98f22da907e782963374faf518aa
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4809c5f9c5bd27a776bd8ab3d1ccdf6843c57adbab28cdd4b2a6a80936626da5
    Summary revision
    summary-61fb3b63f8fa0efbef9ee2af96926a1031f7d669053608364d51df11c75820be
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-reader-v1
    Analysis input hash
    510b9df158bab291870dacdb533ca003218023b5268a77129bdf8543e16d5fe3
    Summary revision
    summary-8d717d6f252001892c6994bb00bc2f71b03d06976f120442043bef0fc9ed0784
    Input summary revisions
    summary-ee5d826ebdfe8992593b51edfdface91c0032f6667b998f34bff5d66c533178b
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f3c4299696b8192e15c25109db63715cfd803ba9d3410e92c69370d39d9b67c6
    Summary revision
    summary-9d66b1d04a8504c5b91d6afc6b955066a63d7855850d22eeeec626130b69e88c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6b7876a5508f98fcb218c4d6269b28027d02b5399f8dd64647fbf3689e04e3ac
    Summary revision
    summary-adffe1a100113da7e3babe2500c3e16f7337940440bedd96915e8e7911757b88
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    555ec2ff00d815ab48c5456a27c7cc97b5af07347ebe0534d5a965fd3ed59d28
    Summary revision
    summary-c6096e1ee9601f50abbf4729a91fc66fee089b452ed5c971d2560e9bd313a995
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e8fb7af9ccb2d07a67a73a3446cbb7da5b0e8ca180df6c57589da97329384eae
    Summary revision
    summary-e35f7a56bc7d2e831df565d9c63ab1cbc5ab305720cf68f96bfb1ff6ec698ba8
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fd1395d0558ec329ac21af09e1ea626d8af4a4aa038dc662033b9d15f5446a8e
    Summary revision
    summary-ee5d826ebdfe8992593b51edfdface91c0032f6667b998f34bff5d66c533178b
    Input summary revisions
    summary-062415ed225f8df05f70d71a5e88ff272fe44906c03a335e1cf5e7b72853803a, summary-08175a42127683e90222af9e87ab44984c31727e2182352435b07a240f98ef14, summary-0f5260c33e7a70d349d3f1584a69920987846e14ed7c540baf3ad9ad6210e72e, summary-241f86f347cd21d9265ea4699600d1a248d28ba8297002a64a627b0fda94b918, summary-36cb465f526fdd0ea9401f0a5476b4671a201a4b8c3a39c7e03303484c1b25da, summary-4dcde385295398fb65c3b159ebfd0b65c46ae510f41326fc4756fbfe2e69ac02, summary-61fb3b63f8fa0efbef9ee2af96926a1031f7d669053608364d51df11c75820be, summary-adffe1a100113da7e3babe2500c3e16f7337940440bedd96915e8e7911757b88, summary-f955172dcbb496d2460754384c2b7b8efd5429f83fb42d0050833300e4db979c
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4228311fa189dc0c6d8737191d62fb5011e2f75d477921309572e405400ed347
    Summary revision
    summary-f955172dcbb496d2460754384c2b7b8efd5429f83fb42d0050833300e4db979c
    Input summary revisions
    Direct evidence analysis

Good at

Handles bounded project lookups, edits and bug fixes well: every baseline small-coding task passed without thinking. Repeated coding trials are now scored: all 48 repetitions of 48 repetitions thinking-off repeats earned full credit, and 47 repetitions of 48 repetitions thinking-on repeats earned full credit.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking off · sampled repetitions
Repetitions scored48 repetitions
Repetitions planned48 repetitions
Repetitions earning full credit48 repetitions
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking on · sampled repetitions
Repetitions scored48 repetitions
Repetitions planned48 repetitions
Repetitions earning full credit47 repetitions
Explore these test results →

View this model’s answers, code and tool activity →

Can recognize when a passage does not support an answer. With thinking enabled, the examples checked correctly declined unsupported questions rather than guessing.

Evidence for this judgment

Example 2: Reading without guessing
Thinking on. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 6: Reading without guessing
Thinking on. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Struggles with

Passage calculations need checking even with thinking enabled. Research-paper answers without thinking also had modest overlap with reference answers, so verify extracted information; unavailable paper inputs limit closer diagnosis.

Evidence for this judgment
Calculating from a passage · Thinking on
Tasks passed6 of 7 (85.7%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking off
Average score45.4/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 8 partial · 0 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Source limitation
Some selected evidence has recorded publication redactions.

Example 1: Reading research papers
Thinking on. Final result: 14.5/100 · Answer token F1. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Can get stuck repeating unchanged file searches despite feedback, or stop at a shortlist without inspecting contents. Supervise longer investigations rather than assuming a plausible result means the work is complete.

Evidence for this judgment

Example 3: Larger practical tasks
Thinking off. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 5: Larger practical tasks
Thinking off. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Should you enable thinking?

Leave thinking off by default for routine coding and larger practical workflows. Matched tests took longer without improving coding scores, while larger-task scores fell.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on100.0% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking onNo change
Average time change with thinking on60.6 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking off versus on
Thinking off → on81.8/100 · Mean task credit → 72.7/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Tasks timed in both modes11 pairs
Score change with thinking on-9.1 score points
Average time change with thinking on194.8 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Enable thinking selectively for strict wording and formatting requirements, or answering from passages without guessing. Matched accuracy improved at the cost of longer waits; use it when avoiding those errors matters more.

Evidence for this judgment
Following exact instructions · Thinking off versus on
Thinking off → on82.6% · Strict prompt compliance → 95.7% · Strict prompt compliance
Paired test coverage23 of 23 tasks compared in both modes
Tasks timed in both modes23 pairs
Score change with thinking on+13.0 score points
Average time change with thinking on80.6 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking off versus on
Thinking off → on87.5% · Exact-match accuracy → 100.0% · Exact-match accuracy
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking on+12.5 score points
Average time change with thinking on28.0 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

It outscored GLM-4.7-Flash REAP 23B-A3B and Qwen3.6 29B REAP Opus distill on larger practical tasks with thinking off. With thinking on, passage-calculation scores tied the distill model and were lower than GLM Flash, a specific weakness despite stronger practical-task results.

Evidence for this judgment

Larger practical tasks
Thinking off. Large · 20B and above · MoE (total parameters). Rank 2 of 7 in this recorded comparison.

Calculating from a passage
Thinking on. Large · 20B and above · MoE (total parameters). Tied at rank 5 of 7 in this recorded comparison.

Larger practical tasks · Thinking off
Average score81.8/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes9 full credit · 0 partial · 2 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Thinking on
Tasks passed6 of 7 (85.7%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasksThinking off · Large · 20B and above · MoE (total parameters)

Build small apps, organize information and investigate files or web pages.

Mean task credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Gemma 4 26B-A4B25.2B total parameters90.9/100
    9.1 points higher
  2. #2 Gemma 4 26B-A4B QAT Q4_K_XL25.2B total parameters81.8/100
    This model
  3. #3 Qwen3.8 Flash-Next (IQ3_XXS)176.9B total parameters63.6/100
    18.2 points lower
  4. =4 Qwen3.6 28B-REAP20-A3B28.2B total parameters54.5/100
    27.3 points lower
  5. =4 Qwen3.6 35B-A3B34.7B total parameters54.5/100
    27.3 points lower
  6. #6 Qwen3.6 29B REAP Opus distill28.9B total parameters36.4/100
    45.5 points lower
  7. #7 GLM-4.7-Flash REAP 23B-A3B23.0B total parameters27.3/100
    54.5 points lower

Other mode, this model only: thinking on · 72.7/100 · Mean task credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-10 assessment. Task-specific results.

Calculating from a passageThinking on · Large · 20B and above · MoE (total parameters)

Read a passage and count, compare or calculate using its facts.

Numeric exact-match accuracy · 0–100% · higher is better
7 of 7 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 GLM-4.7-Flash REAP 23B-A3B23.0B total parameters100.0%
    1 task more passed
  2. =1 Qwen3.6 28B-REAP20-A3B28.2B total parameters100.0%
    1 task more passed
  3. =1 Qwen3.6 35B-A3B34.7B total parameters100.0%
    1 task more passed
  4. =1 Qwen3.8 Flash-Next (IQ3_XXS)176.9B total parameters100.0%
    1 task more passed
  5. =5 Gemma 4 26B-A4B QAT Q4_K_XL25.2B total parameters85.7%
    This model
  6. =5 Qwen3.6 29B REAP Opus distill28.9B total parameters85.7%
    Same score
  7. #7 Gemma 4 26B-A4B25.2B total parameters71.4%
    1 task fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking off · 71.4% · Numeric exact-match accuracy.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-10 assessment. Task-specific results.

Worth knowing

Redactions and unavailable paper inputs, artifacts and execution details limit inspection of some examples. Model-authored project checks were not independently executed for this summary, and the coding rename example lacks detailed check output.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

Example 4: Everyday coding tasks
Thinking off. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 1: Reading research papers
Thinking on. Final result: 14.5/100 · Answer token F1. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 3: Larger practical tasks
Thinking off. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Both modes earned full credit on every matched question. These matched results leave no measured quality headroom; unmatched questions are not covered by this observation.

Thinking offMean matched task credit100.0%
Thinking onMean matched task credit100.0%

Thinking on took 60.6 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on100.0% → 100.0%
Mean paired credit change0.0 percentage points
Question outcomes0 improved · 16 unchanged · 0 worse
Mean added task time+60.6s
Mean task time, off → on23.5s → 84.2s
Median task time, off → on15.8s → 39.5s
Median of paired time changes+27.3s
Total added task time+970.2s
Ratio of summed times, on / off3.6×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking off · temperature 0.6

48/48 repeats scored · 48 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 16 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested off · verified
  • Seed 101 · requested off · verified
  • Seed 303 · requested off · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

16 full credit

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-71431bc2ad83f5a0e3bf977f
Matching candidate set: paired-71431bc2ad83f5a0e3bf977f

GLM-4.7-Flash REAP 23B-A3B

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-b02c9416cc8ad504bf20376d, fact-47ad91563613ac587b7cf8d6
Matching candidate set: fact-47ad91563613ac587b7cf8d6, fact-13b07e4fd98f9401f28d5b10, fact-16816c9c04532a86f188323f, fact-9c25818e1688f1a4f03239a9, fact-5ae8ac173573895133028987, fact-fd3890f4fbd1383d2af80d72, fact-bfc465187e0c4b64d3e0faee, fact-84b68ee522c794052dbb13dd, fact-543885d1abe7569641164958, fact-d3eb63c2e0bc62fa106a3ff4, fact-3ca95ea92b5d2c9ac3e39bf6, fact-3bed0f728335765f1f64dc04, fact-f7565dbc26202571f4b7c51e, fact-2495f2a50aa1e8662cf535ad

Tested weights

QAT-UD-Q4_K_XL · 14.25 GB

unsloth · recorded source

gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.