AgentBench / results
← All models

Qwen3.5 4B

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_K_M · 2.71 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_K_M

Q4_K_M · 2.71 GB of weights · rocm · context 32768 · GPU layers 99

Small · under 5B · dense · 4.2B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

57.1 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking offDetails
Everyday coding tasksTask pass rate
Pass rate
15/16 · 94%
Answers completed
16/16 · 100%
Correct / completed
15/16 · 94%
15 full credit1 incorrect
16/16 scored4.8s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
45.5%
Full-credit rate
5/11 · 45%
Answers completed
10/11 · 91%
Full credit / completed
5/10 · 50%
5 full credit5 incorrect1 failed to finish
11/11 scored15.9s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
58.9%
Full-credit rate
20/34 · 59%
Answers completed
34/34 · 100%
Full credit / completed
20/34 · 59%
20 full credit2 partial credit12 incorrect
34/34 scored34.4s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
55.8%
Full-credit rate
6/11 · 55%
Answers completed
11/11 · 100%
Full credit / completed
6/11 · 55%
6 full credit2 partial credit3 incorrect
11/11 scored26.2s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
19/23 · 83%
Answers completed
23/23 · 100%
Correct / completed
19/23 · 83%
19 full credit4 incorrect
23/23 scored3.5s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
10/16 · 63%
Answers completed
16/16 · 100%
Correct / completed
10/16 · 63%
10 full credit6 incorrect
16/16 scored0.7s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
5/7 · 71%
Answers completed
7/7 · 100%
Correct / completed
5/7 · 71%
5 full credit2 incorrect
7/7 scored1.0s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
43.9%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit7 partial credit
8/8 scored3.4s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
0/12 · 0%
Answers completed
12/12 · 100%
Correct / completed
0/12 · 0%
12 incorrect
12/12 scored1.1s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this Q4_K_M configuration for small-project coding and, with thinking enabled, answering questions from passages. It is a useful focused assistant, not a substitute for supervising complex work: larger practical tasks had mixed outcomes, and keeping or updating source-backed answers under challenge was a severe weakness with thinking off.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Tasks timed16 tasks
Typical time per task (median)8.0 seconds
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking on
Tasks passed15 of 16 (93.8%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking off
Average score45.5/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes5 full credit · 0 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off
Tasks passed0 of 12 (0.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T05:49:55.667110Z
Review time
2026-10-22T05:55:43.629552+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
7d9263d865e3166f231e9809155c7d72a6faa710bcc2b2dc9562c788a7d2a821
Original analysis input hash
fe473def922d2181bdd1fee2b2a91c17560f67fbb4d374282baf1076be355854
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-b2ecb29ed835ad2af575f31ee2d34fa1e22979785a8228705f538d5eb10b1043
Summary revision
summary-136647dac0c3ab58c0e9e72719c78616c6dd9ab23bbd7324a465337974ad318f

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (35)
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c673903b8755db3fa7b1845f26a39e5d9bc83914df7e35b19866257e14ab52f8
    Summary revision
    summary-00b44580616d82de66a06c61b3546a8ddc45fb349f532ca36dcbf8a11f7f5536
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    faeab0db75f80aacbb76ed49259810e510d95a638aecd759d217e0fe3244e3a5
    Summary revision
    summary-04f72d5351adabdf551592f08c06ebdb94aebd8e6d30dcc6ef41b61fe660644d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a099cb4a99ff8ddc966af544160a99ee17618e203891d420ecd3c3da042858b9
    Summary revision
    summary-1ba1ed0727258890741e40dc577d058cc005281d6ce7ebc36b42111bd75cc300
    Input summary revisions
    summary-00b44580616d82de66a06c61b3546a8ddc45fb349f532ca36dcbf8a11f7f5536, summary-573955cb290c492bd1258c22da63d76111405143ad0e1f21c888558ad33e348c, summary-5b26de682d8fcb6eadd10465ff33808e3d3ccd1bb8253eedb4185ad2f6bebc61, summary-92a986d83457f4fb4b99cca88105b810f8562da5376ca3bb23f042f806131d4e, summary-a986afc60e0b95561b3ca60d355d58aa533ab69ea00eec7ac6d3f613a301518b, summary-bce71904f317da6ec19e3a15b262854692374d476be7b9f7eb5b307c6944bcad, summary-bfc6ef5c4e45a54152bdd7c3304379a13526d665a8f18e390d5735ccf0aa911e, summary-c4f9fc09d0a93f09d1219180009a3605db0f9982dab1a2702db64d48296e68a6, summary-c97b7f7c0ed7770b4262a83960c5a5c7d5b30a8b63951b24840ad85ae13b1009
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e48c7b8eeafd64c05c090b9b9e0375c13397ab4c7ca71637a4fa9d292b9a00db
    Summary revision
    summary-1fe94be4122e88231b96088ca86f9230c5dccd9267ad6dcdacecb7d8e39ce99e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5d05321c1f15039f2b8f7428545b85f2bd115c61a56b23dfe7d2d13c91d57e43
    Summary revision
    summary-238ab8fdff16a67933284feb12ba4b176b59db377e9bb1f6d709e313d81c4e0a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3041c30cdf806856d6bf0076dc79f00f3287438b9e15ff308a8e16163242c20c
    Summary revision
    summary-23ca1eaf15954a3dbd11d401f81bbd801c0bdd98a1734a4b68bc99780890e642
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7ceff27eea9b6ad493e5d38de68440a8587504fed4c6710064d847d28c98d0a9
    Summary revision
    summary-2bb56e9f4a3a0b00f7871cdcc90326199e58199974d88b657e65557511bd9444
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    90d0e6b19a276c8307243e149f42485d140caf8c3dab95e35f94b8713048586b
    Summary revision
    summary-33b8f779dd36f4aaa5bcda34b18919fceceebc6405b9cc77fa14e390ed8ab79e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5703394f13530ac4853de2bc767bfb39995074643d2d54b665c3c230bb67da5b
    Summary revision
    summary-4be12005b96ba5a1bcbd5b1aeacfe7b70be06b591e6f6c7863a4055bc72ec8cc
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2303a06a45bc5f4c9ea2026a3671523442a3371d3d0f956ff9cb90a5293e1f39
    Summary revision
    summary-535fdee30dce43ca10e56f23add9a9664ab8c5cb8d0d6b1547717368fbb8ee27
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4d9c683961c404931037f83cb15345ae29e5051742086b9e09a8f719e68db3b6
    Summary revision
    summary-54b47afa2bded5100fe7672ec29f060404e96faa67ed58c6e370b2264175dbe4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c62741b4ef002e0eba1eb50f17dc093da7d4c5cecc0ab85f84bda24a46e1d0d6
    Summary revision
    summary-573955cb290c492bd1258c22da63d76111405143ad0e1f21c888558ad33e348c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    619ab65a49ab0b0b5d2a9d70a1e300ef8112f611db0d47baeaacb10b29e3d772
    Summary revision
    summary-5b26de682d8fcb6eadd10465ff33808e3d3ccd1bb8253eedb4185ad2f6bebc61
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    90622bb44f7923c76bcf405753694615b2150a7e7d813ec3d6a9234251797694
    Summary revision
    summary-5f8376672b6ee946900bc13a9cdeed8350c92b60e7e1526c2c67df040664b862
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    905c8ca2ec68e9d286f2fcaf07906acbf77357db95527174cc008e70d0f483ec
    Summary revision
    summary-643671b5f92eb581e890becb21da295d4d2c25981364ba07e5af6a2136de6e69
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2a2e77244b7c2e2c302e2649d0e48665c8fce05394a0b3b8aab5fe6189cb8617
    Summary revision
    summary-66e746501ac63e1ff883cf68bd065a13c22f9bb76254b46cd43408d3397df499
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    006f087f6c2350ad827fbf923f106e26f7b4b73a4d815e7c6af2b450688da641
    Summary revision
    summary-7455522a158d13f8266b027865edd4d5d49eacf85da42d7410fb31eedc508b8f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f253ea51c9117af4ce600107317ecd71b735bedac2fc4c8d7f7a6986187cf221
    Summary revision
    summary-92a986d83457f4fb4b99cca88105b810f8562da5376ca3bb23f042f806131d4e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6f708ba1df199364a2896f2e69b09877e3f79f5f80614ac1abd1409dd12cf4fc
    Summary revision
    summary-95c01fe47e9d95b576e1588fb5cea0b01c5be4172a92d50e170b050c29f28dcc
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    0869cc316e1ec5ba8f136443762d445cb79fe1987a70c162188cf0677b621bc3
    Summary revision
    summary-9bb1407e488ede703b3b8c1435d0eba79ce18004df10b3c542a60a5786c0d488
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e69f8b5396706a02f2a2d196c2ca75285701a09c44a3362d1b0ef0d59cc49355
    Summary revision
    summary-9be5a808d53476afe8f5e15040035998290f212305e048a4f5717c159f1f44e2
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1f993a9867322ee7c3f252f04d598446e20c5067e4b0110c6a6fb733072f9589
    Summary revision
    summary-a986afc60e0b95561b3ca60d355d58aa533ab69ea00eec7ac6d3f613a301518b
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    707c07135ba29a4ccf804e2b1ab64a3aa617ea06666b106fd794ee2d78621d29
    Summary revision
    summary-a9b634b87502691551496528d1ec67d295a409af52b7cc41879ed09f218ff0d3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5888894c0e028e9c3b7f797a03d88ef474ecaaf3df43be44e14ad3277ec3a306
    Summary revision
    summary-b41083593c4251c08edcb181733bb715e6628475f6df0103254f74f4a7e23d13
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    61a237aaea11038bab5d0db7ff3dd3d1a17d59e9163218eaed3c1abb8b44100f
    Summary revision
    summary-bce71904f317da6ec19e3a15b262854692374d476be7b9f7eb5b307c6944bcad
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fd40b5fed039d732072cf457ccc8e6fa8718992a97b27eb2cf49cac27fdc3ed0
    Summary revision
    summary-bfc6ef5c4e45a54152bdd7c3304379a13526d665a8f18e390d5735ccf0aa911e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2a8890db345a25048b815a32a18936b8ffa2daa089ff3db4dff8f5fba46b408c
    Summary revision
    summary-c4b5aee0751fce2456206156777df1d11d01c318bb385345a93f109256a95302
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5d5b78d492549df704730f5129b6fb68fc9234d6fe7c2b91885ffc3b684e5d4b
    Summary revision
    summary-c4f9fc09d0a93f09d1219180009a3605db0f9982dab1a2702db64d48296e68a6
    Input summary revisions
    summary-04f72d5351adabdf551592f08c06ebdb94aebd8e6d30dcc6ef41b61fe660644d, summary-1fe94be4122e88231b96088ca86f9230c5dccd9267ad6dcdacecb7d8e39ce99e, summary-238ab8fdff16a67933284feb12ba4b176b59db377e9bb1f6d709e313d81c4e0a, summary-23ca1eaf15954a3dbd11d401f81bbd801c0bdd98a1734a4b68bc99780890e642, summary-2bb56e9f4a3a0b00f7871cdcc90326199e58199974d88b657e65557511bd9444, summary-33b8f779dd36f4aaa5bcda34b18919fceceebc6405b9cc77fa14e390ed8ab79e, summary-4be12005b96ba5a1bcbd5b1aeacfe7b70be06b591e6f6c7863a4055bc72ec8cc, summary-535fdee30dce43ca10e56f23add9a9664ab8c5cb8d0d6b1547717368fbb8ee27, summary-54b47afa2bded5100fe7672ec29f060404e96faa67ed58c6e370b2264175dbe4, summary-5f8376672b6ee946900bc13a9cdeed8350c92b60e7e1526c2c67df040664b862, summary-643671b5f92eb581e890becb21da295d4d2c25981364ba07e5af6a2136de6e69, summary-66e746501ac63e1ff883cf68bd065a13c22f9bb76254b46cd43408d3397df499, summary-7455522a158d13f8266b027865edd4d5d49eacf85da42d7410fb31eedc508b8f, summary-95c01fe47e9d95b576e1588fb5cea0b01c5be4172a92d50e170b050c29f28dcc, summary-9bb1407e488ede703b3b8c1435d0eba79ce18004df10b3c542a60a5786c0d488, summary-9be5a808d53476afe8f5e15040035998290f212305e048a4f5717c159f1f44e2, summary-a9b634b87502691551496528d1ec67d295a409af52b7cc41879ed09f218ff0d3, summary-b41083593c4251c08edcb181733bb715e6628475f6df0103254f74f4a7e23d13, summary-c4b5aee0751fce2456206156777df1d11d01c318bb385345a93f109256a95302, summary-c9ecb21b06df131481d33fd9cf807530f2025f15298b6417b81198d5fa31ef24, summary-d8aa8b8b6a7329467bc12aa44b719650021234478ecc984c226dbd9345167fff, summary-e179af7cd70e02cf2ff9d78143b83ff2289717009756ddde6ca00191a1ec6d29, summary-e637ff4be6e868a5d736e522a217b0fe039119b9d791b446bff58f7a56f0ae55, summary-e9421215a87a41f3cd2b1ff13b16d2d3574d9ac195e69f78f6e96d80f4653bbe, summary-ec58fd9ba9b8417ea57ab2d103843a7132c129610c96b3a821c1ec356064ab2f
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2ef4b64b62875171014ed7821a07d8ac9d2bdac574032a425ac3e78e7ca5c47e
    Summary revision
    summary-c97b7f7c0ed7770b4262a83960c5a5c7d5b30a8b63951b24840ad85ae13b1009
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9e9e971587c82b4ff994b4f8d442e8cc11ded4f7ee1b8037728ae67ea7f9e02a
    Summary revision
    summary-c9ecb21b06df131481d33fd9cf807530f2025f15298b6417b81198d5fa31ef24
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9a5bb51b3dcc70f26c2bbaf2a7f1a440e9e43fc6efbafbe290c4466fff036bd8
    Summary revision
    summary-d8aa8b8b6a7329467bc12aa44b719650021234478ecc984c226dbd9345167fff
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8986a0b038f8e0d8d2b9ef6c416e1dddcb41560b36282bb5c0240790bbd927b0
    Summary revision
    summary-e179af7cd70e02cf2ff9d78143b83ff2289717009756ddde6ca00191a1ec6d29
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    edbf65e82c287b2451849ae466f4c84e17a8ec9039f0e0e61234b2b06a2e7734
    Summary revision
    summary-e637ff4be6e868a5d736e522a217b0fe039119b9d791b446bff58f7a56f0ae55
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    db37935b372af042b4876ed6bb43246301941c7c8267965cc26ad39d5328ad86
    Summary revision
    summary-e9421215a87a41f3cd2b1ff13b16d2d3574d9ac195e69f78f6e96d80f4653bbe
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9d9e9af46d514ecbf6bdb362b06ababbc1b1e71b9dcb737f505ce6a15878ebc7
    Summary revision
    summary-ec58fd9ba9b8417ea57ab2d103843a7132c129610c96b3a821c1ec356064ab2f
    Input summary revisions
    Direct evidence analysis

Good at

Small-project coding is a practical use: it completed every test with thinking on. Bounded file edits and bug fixes are sensible assignments, without assuming the same success on larger projects.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

With thinking on, it answered passage questions accurately and usually obeyed strict wording and formatting rules—useful for extracting information and producing constrained responses.

Evidence for this judgment
Reading without guessing · Thinking on
Tasks passed15 of 16 (93.8%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking on
Tasks passed21 of 23 (91.3%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Without thinking, it failed the complete source-challenge checks across the test set. Do not rely on this mode to preserve a supported answer under pressure while still updating it when evidence changes.

Evidence for this judgment
Resisting incorrect corrections · Thinking off
Tasks passed0 of 12 (0.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical work needs supervision: without thinking, it completed 5 of 11 tasks. Difficult math/reasoning and research-paper answer matching were also uneven, so small-edit success should not set expectations for these jobs.

Evidence for this judgment
Larger practical tasks · Thinking off
Average score45.5/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes5 full credit · 0 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking off
Average score55.8/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes6 full credit · 2 partial · 3 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking off
Average score43.9/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes1 full credit · 7 partial · 0 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Should you enable thinking?

For quick small edits, thinking off is reasonable: enabling it fixed a narrow baseline shortfall but added time. Enable it for passage questions and especially for keeping or updating source-backed answers under challenge.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on93.8% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Score change with thinking on+6.3 score points
Average time change with thinking on3.0 seconds slower
Tasks timed in both modes16 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking off versus on
Thinking off → on62.5% · Exact-match accuracy → 93.8% · Exact-match accuracy
Paired test coverage16 of 16 tasks compared in both modes
Score change with thinking on+31.3 score points
Average time change with thinking on25.1 seconds slower
Tasks timed in both modes16 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off versus on
Thinking off → on0.0% · All-branches pass rate → 100.0% · All-branches pass rate
Paired test coverage12 of 12 tasks compared in both modes
Score change with thinking on+100.0 score points
Average time change with thinking on36.5 seconds slower
Tasks timed in both modes12 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Do not enable thinking automatically for harder work: it added time without improving larger practical-task scores, barely changed research-paper answer matching, and reduced difficult math/reasoning scores.

Evidence for this judgment
Larger practical tasks · Thinking off versus on
Thinking off → on45.5/100 · Mean task credit → 45.5/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Score change with thinking onNo change
Average time change with thinking on23.8 seconds slower
Tasks timed in both modes11 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking off versus on
Thinking off → on43.9/100 · Answer token F1 → 45.3/100 · Answer token F1
Paired test coverage8 of 8 tasks compared in both modes
Score change with thinking on+1.4 score points
Average time change with thinking on28.3 seconds slower
Tasks timed in both modes8 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking off versus on
Thinking off → on55.8/100 · Mean answer credit → 28.5/100 · Mean answer credit
Paired test coverage11 of 11 tasks compared in both modes
Score change with thinking on-27.3 score points
Average time change with thinking on10.5 seconds slower
Tasks timed in both modes11 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking on, small coding and passage answering outscored the other tested small dense models. Larger practical-task results also led that group, despite mixed absolute results: standing out for size does not make complex work dependable.

Evidence for this judgment

Everyday coding tasks
Thinking on. Small · under 5B · dense. Rank 1 of 5 in this recorded comparison.

Reading without guessing
Thinking on. Small · under 5B · dense. Rank 1 of 5 in this recorded comparison.

Larger practical tasks
Thinking on. Small · under 5B · dense. Rank 1 of 5 in this recorded comparison.

Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking on
Tasks passed15 of 16 (93.8%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score45.5/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes5 full credit · 0 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasksThinking on · Small · under 5B · dense

Find information in a small project, edit files and fix bugs.

Task pass rate · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters100.0%
    This model
  2. =2 Gemma 4 E2B Q4_04.6B total parameters81.3%
    3 tasks fewer passed
  3. =2 Qwen3.5 2B Q4_K_M1.9B total parameters81.3%
    3 tasks fewer passed
  4. #4 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters12.5%
    14 tasks fewer passed
  5. #5 SmolLM3 3B Q4_K_M3.1B total parameters6.3%
    15 tasks fewer passed

Other mode, this model only: thinking off · 93.8% · Task pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Reading without guessingThinking on · Small · under 5B · dense

Answer from a passage—or say that the passage does not contain the answer.

Exact-match accuracy · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters93.8%
    This model
  2. #2 Gemma 4 E2B Q4_04.6B total parameters75.0%
    3 tasks fewer passed
  3. #3 Qwen3.5 2B Q4_K_M1.9B total parameters68.8%
    4 tasks fewer passed
  4. #4 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters12.5%
    13 tasks fewer passed
  5. #5 SmolLM3 3B Q4_K_M3.1B total parameters6.3%
    14 tasks fewer passed

Other mode, this model only: thinking off · 62.5% · Exact-match accuracy.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Larger practical tasksThinking on · Small · under 5B · dense

Build small apps, organize information and investigate files or web pages.

Mean task credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters45.5/100
    This model
  2. #2 Gemma 4 E2B Q4_04.6B total parameters27.3/100
    18.2 points lower
  3. #3 Qwen3.5 2B Q4_K_M1.9B total parameters18.2/100
    27.3 points lower
  4. #4 SmolLM3 3B Q4_K_M3.1B total parameters9.1/100
    36.4 points lower
  5. #5 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters0.0/100
    45.5 points lower

Other mode, this model only: thinking off · 45.5/100 · Mean task credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

With thinking off, difficult math/reasoning scores clearly exceeded the other tested small dense models. Its puzzle-score lead over the nearest peer was negligible, however; these are different-sized advantages, not evidence of broad reasoning superiority.

Evidence for this judgment

Published hard problems
Thinking off. Small · under 5B · dense. Rank 1 of 9 in this recorded comparison.

Logic and number puzzles
Thinking off. Small · under 5B · dense. Rank 1 of 9 in this recorded comparison.

Published hard problems · Thinking off
Average score55.8/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes6 full credit · 2 partial · 3 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off
Average score58.9/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes20 full credit · 2 partial · 12 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problemsThinking off · Small · under 5B · dense

Answer selected difficult math and reasoning questions from LiveBench.

Mean answer credit · 0–100 points · higher is better
11 of 11 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters55.8/100
    This model
  2. #2 Gemma 4 E2B Q4_04.6B total parameters20.3/100
    35.5 points lower
  3. #3 Qwen3.5 2B Q4_K_M1.9B total parameters19.4/100
    36.4 points lower
  4. #4 SmolLM3 3B Q4_K_M3.1B total parameters14.9/100
    40.9 points lower
  5. #5 Granite 4.0 Micro 3.4B Q4_03.4B total parameters11.7/100
    44.1 points lower
  6. #6 Phi-4 Mini Reasoning 3.8B Q4_K_M3.8B total parameters10.5/100
    45.2 points lower
  7. #7 Phi-4 Mini Instruct 3.8B Q4_K_M3.8B total parameters5.2/100
    50.6 points lower
  8. #8 LFM2.5 1.2B Q4_01.2B total parameters3.5/100
    52.3 points lower
  9. #9 Qwen3.5 0.8B Q4_00.8B total parameters3.3/100
    52.5 points lower

Other mode, this model only: thinking on · 28.5/100 · Mean answer credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Logic and number puzzlesThinking off · Small · under 5B · dense

Solve newly generated puzzles using arithmetic, deduction and planning.

Mean puzzle credit · 0–100 points · higher is better
34 of 34 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters58.9/100
    This model
  2. #2 Gemma 4 E2B Q4_04.6B total parameters58.9/100
    0.0 points lower
  3. #3 LFM2.5 1.2B Q4_01.2B total parameters38.3/100
    20.5 points lower
  4. #4 Qwen3.5 2B Q4_K_M1.9B total parameters35.4/100
    23.5 points lower
  5. #5 Granite 4.0 Micro 3.4B Q4_03.4B total parameters28.8/100
    30.1 points lower
  6. #6 Phi-4 Mini Reasoning 3.8B Q4_K_M3.8B total parameters26.5/100
    32.4 points lower
  7. #7 Phi-4 Mini Instruct 3.8B Q4_K_M3.8B total parameters26.0/100
    32.9 points lower
  8. #8 SmolLM3 3B Q4_K_M3.1B total parameters15.2/100
    43.7 points lower
  9. #9 Qwen3.5 0.8B Q4_00.8B total parameters5.9/100
    52.9 points lower

Other mode, this model only: thinking on · 61.8/100 · Mean puzzle credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Worth knowing

Publication redactions, withheld artifacts and source-link-only paper exports limit readers’ inspection of some answers and grading details.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

Evidence for this judgment

Source limitation
One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

What changed in these results

One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Thinking offMean matched task credit93.8%
Thinking onMean matched task credit100.0%

Thinking on took 3.0 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on93.8% → 100.0%
Mean paired credit change+6.3 percentage points
Question outcomes1 improved · 15 unchanged · 0 worse
Mean added task time+3.0s
Mean task time, off → on5.7s → 8.8s
Median task time, off → on4.8s → 8.0s
Median of paired time changes+2.7s
Total added task time+48.5s
Ratio of summed times, on / off1.5×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking off · temperature 0.6

48/48 repeats scored · 46 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 14 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 303 · requested off · verified
  • Seed 101 · requested off · verified
  • Seed 202 · requested off · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

15 full credit1 incorrect

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-e67140a291b2bf2f88b997b1
Matching candidate set: paired-e67140a291b2bf2f88b997b1

Gemma 4 E2B Q4_0

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-f3caf8beeec8b9313dcb4eaf, fact-c316deeab93502c3e19d46ba
Matching candidate set: fact-c316deeab93502c3e19d46ba, fact-b158bb60a71073ed7cff8c86, fact-8cd25fbbd82bdae0d5893fac, fact-9e831223924ab330a13b6b40, fact-4bb7924a4c33187258857975, fact-f61609b3038ba8e6778d4e5a, fact-775ed46998b6dd1af2acd16c, fact-4c13720a63efcc49c96b4505

Tested weights

Q4_K_M · 2.71 GB

lmstudio-community · recorded source

Qwen3.5-4B-Q4_K_M.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.