AgentBench / results
← All models

Qwen3.6 29B REAP Opus distill

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_K_M · 17.71 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_K_M

Q4_K_M · 17.71 GB of weights · rocm · context 32768 · GPU layers 99

Large · 20B and above · MoE · 28.9B total · Active parameter count not recorded

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

67.0 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking onDetails
Everyday coding tasksTask pass rate
Pass rate
16/16 · 100%
Answers completed
16/16 · 100%
Correct / completed
16/16 · 100%
16 full credit
16/16 scored13.8s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
72.7%
Full-credit rate
8/11 · 73%
Answers completed
11/11 · 100%
Full credit / completed
8/11 · 73%
8 full credit3 incorrect
11/11 scored47.6s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
45.0%
Full-credit rate
14/34 · 41%
Answers completed
34/34 · 100%
Full credit / completed
14/34 · 41%
14 full credit7 partial credit13 incorrect
34/34 scored83.3s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
44.8%
Full-credit rate
4/11 · 36%
Answers completed
11/11 · 100%
Full credit / completed
4/11 · 36%
4 full credit2 partial credit5 incorrect
11/11 scored99.4s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
19/23 · 83%
Answers completed
23/23 · 100%
Correct / completed
19/23 · 83%
19 full credit4 incorrect
23/23 scored74.6s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
10/16 · 63%
Answers completed
16/16 · 100%
Correct / completed
10/16 · 63%
10 full credit6 incorrect
16/16 scored2.9s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
6/7 · 86%
Answers completed
7/7 · 100%
Correct / completed
6/7 · 86%
6 full credit1 incorrect
7/7 scored3.5s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
34.7%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit5 partial credit2 incorrect
8/8 scored11.2s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
9/12 · 75%
Answers completed
12/12 · 100%
Correct / completed
9/12 · 75%
9 full credit3 incorrect
12/12 scored12.9s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this model for small project edits and bug fixes: thinking off succeeded throughout the baseline test, with a median task time of 12.5 seconds on this test machine. With thinking enabled, it also completed most larger practical tasks—not just simple edits. Research-paper answers remained weak even with thinking enabled. It is a useful coding assistant, but not a good choice for research work without careful checking.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Typical time per task (median)12.5 seconds
Tasks timed16 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score72.7/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes8 full credit · 0 partial · 3 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking on
Average score34.7/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes1 full credit · 5 partial · 2 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T06:00:28.364327Z
Review time
2026-10-22T05:55:43.650019+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
2a7244387c8b29e4c0e88e11cedb930fdbb9748509ca957166f00509d69c9463
Original analysis input hash
6d98ec12e195b35bc3ea5856d3d1a94c241c0b182a0d3de7b7f878cc124591cd
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-0ea64f367177bf7d6ae2438874324dd980f7c4b2695ff1bc94ea435fb8e25e27
Summary revision
summary-7f6cce7c3fa7eb953a0c249c5137381566a05b8e2c8383b12c60272f15142ffc

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (22)
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c3d2c2708ab312437dec0c714e5a8a070eb1bb1ca782c6291cdff1e424c0395f
    Summary revision
    summary-01208b0f98f45145798e6cf01a12cefc0a773e561017d2dd9328d52ba1df18af
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4cddba3d34ceb245ab8ea93a2c462884a93e273eb1fedd71fb8dd8b79bc25c4f
    Summary revision
    summary-0c9388fd5edf87ecc759779e812aedd52e7fb305835292dbc9980cff33771078
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    bf00aca78a3628f48ed4edfd307c844bdd7af88e9ec9ff57d010ffc239b32b08
    Summary revision
    summary-0d6065c94b8cdab4eb9f7584d9f1dc0405b7d02cdbc4d90e276fa71f25d6efb1
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5c8b53e279f75a9c79ac6cc0c30137591c4557643d1ca3033434bd0c40c4db3a
    Summary revision
    summary-0d9707c4dffcdf0e2aa44fb8602fa266d20cffa40d4ea80b81a07f6f0071f771
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d710ee9949af095cd8b392a425dbbe727b88e503435052d4eebf4903736b5af2
    Summary revision
    summary-0f0631f1df2b2e00b434ef6e0b5059e34263990283147dcafd220eecee63e9aa
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fcfa8e160e09c35d7627a818f97e74286f29b5cb471d5071ca775815fbe6116c
    Summary revision
    summary-0f2cb15cd04ceb885e89421d572c87d7ccbedf9e75de931aef9661766be3c18d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b72be3f9c035b8840d8ff1444c72d6baa30f0d0de6494d9da3ce58d01ba88577
    Summary revision
    summary-22abb49bee9653651f952c84972f23b4729509e4a16ecde4e970ead83deda741
    Input summary revisions
    summary-0f2cb15cd04ceb885e89421d572c87d7ccbedf9e75de931aef9661766be3c18d, summary-37b541934b7546a891df9f4ec2e3d3ef7b203781c823d334799ac1a34903b4a8, summary-3e230874ccb47f13ddfb707e91bcf76c171e054f878d1b384742c485f7f8b650, summary-445a4920c7cbc0bb69472e39456bfef12cc1b32901dacb1839a6b46fa4ada87f, summary-81a2ff028bf79a3b75f5a3f288bf7b7db1c6b986e80ac73429ab54a21b51a548, summary-853e011de9ceade6a6ed59048c7a412ccbe68a46902a9b3bce4e6ea8d53273be, summary-940de33476c67cb092eb3bd8f15b6a443e6f3c22e4c7407836ab7ae65edaf4cc, summary-afc2d18ac81ae1bba7cb5cf054ccd646347f4d7077a2287ff47b5f7f61e785b6, summary-ccb22df29886062626c83aac9b7774598f3a23bbdfed5022f1017b4f056b1f9e
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    040f6e8f5388ceeb09bb66bcde2ad65567443c56eab4713188cfa9b5fe63ae98
    Summary revision
    summary-37b541934b7546a891df9f4ec2e3d3ef7b203781c823d334799ac1a34903b4a8
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8467aa04b17ab875eed22dfdab69cfc007c222b74ef231cd25e87fad45501c0a
    Summary revision
    summary-3991b621228b88d1efca30e160ef4a5a2fa8bcfb3171d5f9569b402697f8a429
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    306221b9765b208366b24887538019c6df890f1f1a4c597837a2ef45373dcd53
    Summary revision
    summary-3e230874ccb47f13ddfb707e91bcf76c171e054f878d1b384742c485f7f8b650
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    0af8b6ff73c1eeca2fe0e17970434f4d92d689b1d1cbb7885ff1a7d750324ae0
    Summary revision
    summary-445a4920c7cbc0bb69472e39456bfef12cc1b32901dacb1839a6b46fa4ada87f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2d9b716f2b42543fc74e88f0caf125e821aa904c19f47bfbb11c16a68489344f
    Summary revision
    summary-4940d2c44ec46b408a830cf8e17816166be22d196b6dbc48b5e126c640698cef
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    041b043f4d644f2b5a2b5c04fedf4f105a3b207a59a5dd3ebe7acce4fd33fd3c
    Summary revision
    summary-512962bb672cf99f4f3e791d3d9c9aee4793bf854a9143bade7fb96e6369cf86
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6f2727edd0b788a8efd518866352aed5e0973d892f579492fbb58051597d9274
    Summary revision
    summary-57e3b9edc44bf010f92429bbf4609d82fe7a46d3a969ca78fb3c315f98d34cb3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b2267ad3a2e507ac06b39d4d883a7346084c2d8ba12cb9392dd596f8e974f663
    Summary revision
    summary-5fe869295310790bc6f9a02d39bd89122045964d72cd3b09de41ae04b84bf3aa
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    0612f0307900399b2ccfe9da8d69638dbdfd3e843e34bd5881e72c5f66a70f05
    Summary revision
    summary-770ac00327f999033460b4f02a84eed5712ac48968fa8551f3fca179088a8246
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5c008e7dbba7fbd34484ffefe35e55c27e6ec004d9fc1e39a93b2eab97d86c03
    Summary revision
    summary-81a2ff028bf79a3b75f5a3f288bf7b7db1c6b986e80ac73429ab54a21b51a548
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4ab34ec4cc2813bdef8d5304d569ddcae3aa0484cf901663fc381ae3c9c36115
    Summary revision
    summary-853e011de9ceade6a6ed59048c7a412ccbe68a46902a9b3bce4e6ea8d53273be
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    463dffd57226f20e3e7801e988699b64b3300a53bd361abdabe19102fa140017
    Summary revision
    summary-8a2b6e52deb49421013f1d0ec394d7103e2ea03fd718c381bf19fd071d63b904
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b141e42847b6ed00ccbff6957c4fb6387a72226878638cfbbbef6008318225ee
    Summary revision
    summary-940de33476c67cb092eb3bd8f15b6a443e6f3c22e4c7407836ab7ae65edaf4cc
    Input summary revisions
    summary-01208b0f98f45145798e6cf01a12cefc0a773e561017d2dd9328d52ba1df18af, summary-0c9388fd5edf87ecc759779e812aedd52e7fb305835292dbc9980cff33771078, summary-0d6065c94b8cdab4eb9f7584d9f1dc0405b7d02cdbc4d90e276fa71f25d6efb1, summary-0d9707c4dffcdf0e2aa44fb8602fa266d20cffa40d4ea80b81a07f6f0071f771, summary-0f0631f1df2b2e00b434ef6e0b5059e34263990283147dcafd220eecee63e9aa, summary-3991b621228b88d1efca30e160ef4a5a2fa8bcfb3171d5f9569b402697f8a429, summary-4940d2c44ec46b408a830cf8e17816166be22d196b6dbc48b5e126c640698cef, summary-512962bb672cf99f4f3e791d3d9c9aee4793bf854a9143bade7fb96e6369cf86, summary-57e3b9edc44bf010f92429bbf4609d82fe7a46d3a969ca78fb3c315f98d34cb3, summary-5fe869295310790bc6f9a02d39bd89122045964d72cd3b09de41ae04b84bf3aa, summary-770ac00327f999033460b4f02a84eed5712ac48968fa8551f3fca179088a8246, summary-8a2b6e52deb49421013f1d0ec394d7103e2ea03fd718c381bf19fd071d63b904
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    43be4fca64cbf6a809aae4fca5dec293b13921c10884cea63cf4e0effd26f633
    Summary revision
    summary-afc2d18ac81ae1bba7cb5cf054ccd646347f4d7077a2287ff47b5f7f61e785b6
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    742c5da1816ea470470750db2e42a88db282b2aa846e6b96fa02246b2fb6792e
    Summary revision
    summary-ccb22df29886062626c83aac9b7774598f3a23bbdfed5022f1017b4f056b1f9e
    Input summary revisions
    Direct evidence analysis

Good at

Small project edits and bug fixes are a useful application. Thinking off passed every baseline task and nearly all sampled reruns, though the reruns were not flawless.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking off · sampled repetitions
Repetitions earning full credit47 repetitions
Repetitions scored48 repetitions
Repetitions planned48 repetitions
Explore these test results →

View this model’s answers, code and tool activity →

With thinking off, it successfully kept source-backed answers under unsupported challenges and updated them when the source changed throughout this test.

Evidence for this judgment
Resisting incorrect corrections · Thinking off
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Research-paper answering is a major weakness: thinking off produced no answer-matching success, and thinking on still scored poorly. Exact passage answers were also very weak with thinking off; these problems go beyond occasional citation mistakes.

Evidence for this judgment
Reading research papers · Thinking off
Average score0.0/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking on
Average score34.7/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes1 full credit · 5 partial · 2 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking off
Tasks passed2 of 16 (12.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Difficult reasoning needs checking. Even with thinking on, both generated puzzles and selected hard math/reasoning questions produced middling average scores, rather than consistently strong solutions.

Evidence for this judgment
Logic and number puzzles · Thinking on
Average score45.0/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes14 full credit · 7 partial · 13 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking on
Average score44.8/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 2 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Should you enable thinking?

Leave thinking off for everyday edits: enabling it added average task time without improving baseline success. It also made source-correction handling worse and slower, so thinking is not a universal upgrade.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on100.0% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Average time change with thinking on7.9 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off versus on
Thinking off → on100.0% · All-branches pass rate → 75.0% · All-branches pass rate
Paired test coverage12 of 12 tasks compared in both modes
Tasks timed in both modes12 pairs
Score change with thinking on-25.0 score points
Average time change with thinking on12.6 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Enable thinking for larger practical tasks and finding answers in passages. Both improved; larger tasks took slightly less average time, while passage-answering time barely changed. Better passage results still leave meaningful room for error.

Evidence for this judgment
Larger practical tasks · Thinking off versus on
Thinking off → on36.4/100 · Mean task credit → 72.7/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Tasks timed in both modes11 pairs
Average time change with thinking on0.9 seconds faster
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking off versus on
Thinking off → on12.5% · Exact-match accuracy → 62.5% · Exact-match accuracy
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Average time change with thinking on0.1 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking on, larger-task results substantially exceeded GLM-4.7-Flash REAP 23B-A3B and tied Gemma 4 26B-A4B QAT Q4_K_XL. It is competitive here among the other tested large MoE models, not a clear leader.

Evidence for this judgment

Larger practical tasks
Thinking on. Large · 20B and above · MoE (total parameters). Tied at rank 2 of 7 in this recorded comparison.

Larger practical tasks · Thinking on
Average score72.7/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes8 full credit · 0 partial · 3 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasksThinking on · Large · 20B and above · MoE (total parameters)

Build small apps, organize information and investigate files or web pages.

Mean task credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.6 35B-A3B34.7B total parameters90.9/100
    18.2 points higher
  2. =2 Gemma 4 26B-A4B25.2B total parameters72.7/100
    Same score
  3. =2 Gemma 4 26B-A4B QAT Q4_K_XL25.2B total parameters72.7/100
    Same score
  4. =2 Qwen3.6 29B REAP Opus distill28.9B total parameters72.7/100
    This model
  5. #5 Qwen3.8 Flash-Next (IQ3_XXS)176.9B total parameters63.6/100
    9.1 points lower
  6. #6 Qwen3.6 28B-REAP20-A3B28.2B total parameters54.5/100
    18.2 points lower
  7. #7 GLM-4.7-Flash REAP 23B-A3B23.0B total parameters45.5/100
    27.3 points lower

Other mode, this model only: thinking off · 36.4/100 · Mean task credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

With thinking off, small-edit results tied several leading peers in the tested large MoE cohort. Source-correction results also compared favorably with this cohort.

Evidence for this judgment

Everyday coding tasks
Thinking off. Large · 20B and above · MoE (total parameters). Tied at rank 1 of 7 in this recorded comparison.

Resisting incorrect corrections
Thinking off. Large · 20B and above · MoE (total parameters). Tied at rank 1 of 7 in this recorded comparison.

Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasksThinking off · Large · 20B and above · MoE (total parameters)

Find information in a small project, edit files and fix bugs.

Task pass rate · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 26B-A4B25.2B total parameters100.0%
    Same score
  2. =1 Gemma 4 26B-A4B QAT Q4_K_XL25.2B total parameters100.0%
    Same score
  3. =1 Qwen3.6 28B-REAP20-A3B28.2B total parameters100.0%
    Same score
  4. =1 Qwen3.6 29B REAP Opus distill28.9B total parameters100.0%
    This model
  5. =1 Qwen3.6 35B-A3B34.7B total parameters100.0%
    Same score
  6. =1 Qwen3.8 Flash-Next (IQ3_XXS)176.9B total parameters100.0%
    Same score
  7. #7 GLM-4.7-Flash REAP 23B-A3B23.0B total parameters93.8%
    1 task fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking on · 100.0% · Task pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Resisting incorrect correctionsThinking off · Large · 20B and above · MoE (total parameters)

Keep a source-backed answer when challenged, but change it when the source is updated.

All-branches pass rate · 0–100% · higher is better
12 of 12 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 26B-A4B25.2B total parameters100.0%
    Same score
  2. =1 Gemma 4 26B-A4B QAT Q4_K_XL25.2B total parameters100.0%
    Same score
  3. =1 Qwen3.6 29B REAP Opus distill28.9B total parameters100.0%
    This model
  4. #4 Qwen3.6 28B-REAP20-A3B28.2B total parameters91.7%
    1 task fewer passed
  5. =5 Qwen3.6 35B-A3B34.7B total parameters83.3%
    2 tasks fewer passed
  6. =5 Qwen3.8 Flash-Next (IQ3_XXS)176.9B total parameters83.3%
    2 tasks fewer passed
  7. #7 GLM-4.7-Flash REAP 23B-A3B23.0B total parameters8.3%
    11 tasks fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking on · 75.0% · All-branches pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Worth knowing

Publication redactions restrict readers’ inspection of some examples, artifacts and paper text.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Both modes earned full credit on every matched question. These matched results leave no measured quality headroom; unmatched questions are not covered by this observation.

Thinking offMean matched task credit100.0%
Thinking onMean matched task credit100.0%

Thinking on took 7.9 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on100.0% → 100.0%
Mean paired credit change0.0 percentage points
Question outcomes0 improved · 16 unchanged · 0 worse
Mean added task time+7.9s
Mean task time, off → on14.2s → 22.1s
Median task time, off → on12.5s → 13.8s
Median of paired time changes+5.3s
Total added task time+125.9s
Ratio of summed times, on / off1.6×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking on · temperature 0.6

48/48 repeats scored · 43 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 13 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested on · verified
  • Seed 303 · requested on · verified
  • Seed 101 · requested on · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

16 full credit

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-6f9231ccf2741146ac531254
Matching candidate set: paired-6f9231ccf2741146ac531254

Qwen3.6 28B-REAP20-A3B

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-715b4dac385ebc58266864e4, fact-eab8e02b2ca6f6125554960e
Matching candidate set: fact-eab8e02b2ca6f6125554960e, fact-483531924d0e765052461c57, fact-e8904ea75031d6ca51c8c950, fact-c07386361967c222d8028664, fact-513cdcd418215fb18de91eaf, fact-323b97d3fdf7355dc2770c66, fact-293c9f59094b11858da40443, fact-f173b9794ef7b0017910a9bd, fact-c42b6bb6242e9fbbcb40000c, fact-85df5b9585f4e6e43ea084c4, fact-e33976d73a46b28357f03760, fact-7aafa6cbb04481beb867a2e4, fact-24e468344d64cb03badda736, fact-ea0d7c942a224d49dc767879, fact-bc60b4f12cd6927a928b8266

Tested weights

Q4_K_M · 17.71 GB

barozp · recorded source

Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP-Q4_K_M.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.