AgentBench / results
← All models

Mellum 2.1 12B-A2.5B Thinking

3 tested weight formats · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_K_M · 8.07 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_K_M

Q4_K_M · 8.07 GB of weights · vulkan · context 16384 · GPU layers 99

Medium · 5–under 20B · MoE · 12.1B total · Active parameter count not recorded

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 2 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

43.9 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking offDetails
Everyday coding tasksTask pass rate
Pass rate
14/16 · 88%
Answers completed
15/16 · 94%
Correct / completed
14/15 · 93%
14 full credit1 incorrect1 failed to finish
16/16 scored7.5s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
27.3%
Full-credit rate
3/11 · 27%
Answers completed
8/11 · 73%
Full credit / completed
3/8 · 38%
3 full credit5 incorrect3 failed to finish
11/11 scored57.8s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
41.2%
Full-credit rate
14/34 · 41%
Answers completed
34/34 · 100%
Full credit / completed
14/34 · 41%
14 full credit4 partial credit16 incorrect
34/34 scored31.5s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
30.0%
Full-credit rate
3/11 · 27%
Answers completed
11/11 · 100%
Full credit / completed
3/11 · 27%
3 full credit2 partial credit6 incorrect
11/11 scored30.2s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
14/23 · 61%
Answers completed
23/23 · 100%
Correct / completed
14/23 · 61%
14 full credit9 incorrect
23/23 scored3.6s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
13/16 · 81%
Answers completed
16/16 · 100%
Correct / completed
13/16 · 81%
13 full credit3 incorrect
16/16 scored0.8s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
2/7 · 29%
Answers completed
7/7 · 100%
Correct / completed
2/7 · 29%
2 full credit5 incorrect
7/7 scored1.2s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
38.8%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit5 partial credit2 incorrect
8/8 scored4.2s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
0/12 · 0%
Answers completed
12/12 · 100%
Correct / completed
0/12 · 0%
12 incorrect
12/12 scored1.6s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this Q4_K_M configuration with thinking enabled for small project edits and source-based questions, not unattended app-building. It completed every assessed small-coding test and handled challenges to source-backed answers well, but larger practical tasks remained weak. Keep human checks on more ambitious work.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Tasks timed16 tasks
Typical time per task (median)15.2 seconds
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T03:57:49.477273Z
Review time
2026-10-22T07:26:27.176952+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
3d72e75851c3a2b39c078b841e18dce250f6ce31d32cee71ee271711d7f5f57a
Original analysis input hash
7b61ce70664a5a06adecf98f4973383d11215d22417fae71de3327756884c140
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-fd48ef55048777795f53a8d939e48b32e38f5248cb7dbec58d23909716a61ae4
Summary revision
summary-9204c0ec08ada69ac031e3c01064cbdb75d22942f1c8f7b568507ac9e5debdbd

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (54)
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    73a34fb3ff151e5ce053f1c4cba4cac167a66d2b011f35e04d25ea16dadfc66f
    Summary revision
    summary-04ae2a2c2f05d5b4d9cd8dad7ae24ee1a97ac153ee870a60bf4cd65a2c36e602
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    74fb8fc7dd27a3567d2bbcb982d0be04164db4634d971641e2c34eae260e92bb
    Summary revision
    summary-0500b0406543ba65534dcefe7cf9f446f7db071f09bf2a2a49253dc9ece06281
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c3189dd771c44a25324ffa010599f79f36c2e02eec3c0411fd85207115617a4f
    Summary revision
    summary-0e14879bbde178e13c8f48bd7d3804bfe81488604d08c49d414d66582c40d950
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c9f5d8c8db8ad8fa2108965b077e667f8cc39f3835d2277fd12927efcac13aa8
    Summary revision
    summary-14cc5960cf66614f0fff13b0ac691c85c0bdc29b39f5c87102995f924896d157
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a5b3c593ffd9af05f0bacc789fd03a49fb1e7ef6d9cda6994651a3f45599e1a0
    Summary revision
    summary-15346dc6d0961fc11b58bf97d1b7050f83926eac7b13ec748d2d30023db26681
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1160c3c26f3a5ff557c54602a7dbde47a79402b54f96e76e5fa9b0411fb711f4
    Summary revision
    summary-158315b5fd7adcd653babff3e8982c5762d9a333fb0f6ad21ab66b6145e14ea4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4b4828792693dfed07bc36c4aa86977a442f47e4ecadb66af7645789248dfdb6
    Summary revision
    summary-2015a0f1fc437c5e74641e81d10cb87ba8bc80250cba6b2c52b98308c73e4162
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d002187ddafcc0efabdc45ed36d392c6454a129c5226e80922696238d0a0ca03
    Summary revision
    summary-217966952d542079883e0629cbc5e15d5f7417bd7b2493daf8cc2d9ac4d74ec3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fe8338096be903f5f92b1414f02dfd88115059e116d59ccbe2c56e02f49e48a1
    Summary revision
    summary-28308ed703380935707236c8d7e68dd41c5ac51970119a3210aabec01d8b492a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    ffab4de5d356bd5abec1646a0a707184fd48aab3eb6925b8fc12dd486f48e404
    Summary revision
    summary-2839ca8b2fd6f27ae92818ad67fbf51f806d3d1c8d7284925dba822d9a53813d
    Input summary revisions
    summary-2015a0f1fc437c5e74641e81d10cb87ba8bc80250cba6b2c52b98308c73e4162, summary-466d082032d5e7b4016c869e3ed66ae5adde349cd1e2debe94867eb22ce24cb7, summary-569202fd8f38ee5828e894a6ab03a8196b093f8d3133c884196f9d0d53abe2d0, summary-69f908a7007f97bab3a45cda0742c5ab3cec5f63dd73b2da10f70facde212aba, summary-6b2c9b66fb8ea77dc575521e03f4e6ddcc03fdf6ea0fdb858dd9158854007f35, summary-75c4b34486d5d9a95910378a1e29743df0e4fa4520c9b544ab29548071de5210, summary-783c628647b3e58124323fc635e8427317815c620ac2bbd7860d3a05ada9c688, summary-91fb25351d6b270b7a35829231e3a51cd5b40f1f3662fb5e04583af07484d858, summary-b4b93085444ffa41c233221ce2292eee6b97214e131b53a559541c7ebf7a6ed4
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3b1bea027afa40cc95d792270af2d58eb4de06e2240ef6ffc52ecc3ec38f42b8
    Summary revision
    summary-2975f53400f338a8e5a6a9a472d6bc36c8368c9cd67a5696dc9d69b4b91d688c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b7d51208175a299da9ddcc45dc16d432f70369ba05fd8e91d3ef240e5939e3c6
    Summary revision
    summary-2e0148708b9acccfbaf052b0e6d23f4c626b822766d3dbf69b8a8aa1a0df2afa
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    ef0eb0f083b6b07e5e33d267de28682728731b8619a0e669e4761709127805e0
    Summary revision
    summary-35c22b6fffff3295bfb4c7e8e08d90a128e12d2e8da44fc8db59420565b5af3d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    da22c2710b355c2127c74f700c312ec3fa13f2927d657e7dafeeaa442ab72689
    Summary revision
    summary-377c1fcf05c4b64790e6cbcd3addba47b933dc4483ef2b2a0c8e6d8de146453d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    184aeedc83ba6286db103d3f50a5a48d3f0d79c3b2c647057e1956b5afa6ed1a
    Summary revision
    summary-3bd0fe7b355199b8434a0e364b648996e54f55ff5db1f35fab77f4af891d8e57
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    822a6068306d2b65d671ab0c61d2ffd25b1e8ddae311b2604b29457a081c9700
    Summary revision
    summary-3c7e43da4b334cc403ffaf10355d8eda7082466f90e4b2a9f6c4212035f617fe
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b638fb27f4a42812d47dd109485f1d572a994fb4c930f889e97d2749ce8301ea
    Summary revision
    summary-466d082032d5e7b4016c869e3ed66ae5adde349cd1e2debe94867eb22ce24cb7
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    89e60742dc1b6d9601a9196e7c695117cd527db70b98a6fec25c35140e5d8e8f
    Summary revision
    summary-467364032d3fc28a6fe4f6b06f81df3d287734e25eb880d66cafe1831273c0c9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9d007677712793b6032eaa4747c81fe6499f7ed40687dcf2058034d47a456e23
    Summary revision
    summary-4b1b983e3024fbf8a23605ff5b6a760f5d9e4b3d0973b94d4c8b0aaecd2cd7bb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    abbe7c69ed5025c07234466bd6d95348cf4eb975d7278a5679eeae5c607d134e
    Summary revision
    summary-4b95f24a73507e65f2d31a80766915d45c2be4d8be402c8290595f41fcc57e2c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    38cf9b60ba056887a86b281ea3eb2683f7e1dba499ec4631947bb348e7564bdf
    Summary revision
    summary-4d4b2086e2b547becf08119b0c1cf21ef45d127e2c594bc800dd06e203e1d235
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7330b98851b94802c3b161cd244f9872d7e0aed8861b4493e745c0e8043d7808
    Summary revision
    summary-4f469f206330f62e760bc63a44dfdba433ff87b637f276962cf38373fed484b4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e93601490803a75f9002749fa5630f12c3e5ec7ebe1e2bdd4edc4ff72381d00b
    Summary revision
    summary-5047a77285dc3a95ad095325e7d88150509672d988f0b4610ef0d7a3fb3f36b3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    42af2ee19f848b1a441d083b4979bbebd5f1a00cd47bd5dd4bbcb03b3a8ff2b7
    Summary revision
    summary-569202fd8f38ee5828e894a6ab03a8196b093f8d3133c884196f9d0d53abe2d0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    60a1ce26bf1e585e2d7c71edaf09348ff17959ea03dc6c1c6f49fa6894bcb75e
    Summary revision
    summary-585ebe5418a15f9e0f5e60eb9d04df5cf2ca23ff15ee44748c7a32ecd669cc85
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1d921e113a2afec203669287983b21dcc49f3b867e2b3cbc9609aa875676b3a4
    Summary revision
    summary-69f908a7007f97bab3a45cda0742c5ab3cec5f63dd73b2da10f70facde212aba
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5487292293c972c08058f3f6a835633c8ae226860e0e790fb3db0c6887c61f10
    Summary revision
    summary-6b2c9b66fb8ea77dc575521e03f4e6ddcc03fdf6ea0fdb858dd9158854007f35
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f71801619ed642425e9098d9df19470705d1905e4208cc4b95d7b1ca12a8831a
    Summary revision
    summary-75c4b34486d5d9a95910378a1e29743df0e4fa4520c9b544ab29548071de5210
    Input summary revisions
    summary-0e14879bbde178e13c8f48bd7d3804bfe81488604d08c49d414d66582c40d950, summary-35c22b6fffff3295bfb4c7e8e08d90a128e12d2e8da44fc8db59420565b5af3d, summary-3bd0fe7b355199b8434a0e364b648996e54f55ff5db1f35fab77f4af891d8e57, summary-4b1b983e3024fbf8a23605ff5b6a760f5d9e4b3d0973b94d4c8b0aaecd2cd7bb, summary-4d4b2086e2b547becf08119b0c1cf21ef45d127e2c594bc800dd06e203e1d235, summary-85ae6488a7e99149cdf549013a2422cf51e6cb941c1ea69e0abcff128c31835a, summary-8c7a3a4ce7ffcb1186547c21ff3b74983e9003be2ffe9b76fe1f334ac2547ddf, summary-a521fef62ba1032cdefa033b43dacfa7a08acf85e193e5598962af69ad5bfa0d, summary-c4890a6b4ac2da55bb370cf6dbea8c192bc0d6f1b727e338b88f0ba791d5c914, summary-d67a322f2113719f933889336fdf8f18297f8cee7901dcbcfff7e043137055f5, summary-dc70f912ad78eda2067047445a28bd6574884f88b76bce9a17b3d68803f0405e, summary-ee7980749b96c7f2c13af45f43718071e2cb04b7bbea6a049b35beef7f486dcd, summary-fac545fb974be33b7ce856743f412ccb0c102ffd35d0aa77f178a31509e9be18
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    860fbcfef7024d44c07452b063d5c9606bdf4b6fcad8fd988f5608cf9279ceea
    Summary revision
    summary-783c628647b3e58124323fc635e8427317815c620ac2bbd7860d3a05ada9c688
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3f987eaf7c8f7c051f346c33ef0df1d4bff2f6b61163299bc743dd3d63554337
    Summary revision
    summary-7ef7cadcacdd36c6990edeceb9fbea8e03ac9a6608a73163156fc75b5eb56687
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b2ce3a168384787bb642e347ee19484e8611b6cff2b86c2867162675533bbfc9
    Summary revision
    summary-85ae6488a7e99149cdf549013a2422cf51e6cb941c1ea69e0abcff128c31835a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    0ae0781b120c36a238a5b4718b37d21222c053f9c79c028e9118033f85a4dcad
    Summary revision
    summary-8c7a3a4ce7ffcb1186547c21ff3b74983e9003be2ffe9b76fe1f334ac2547ddf
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3178fe776103330261a79cbe8df4274313539c1f2c8408c9893e96c65f55e430
    Summary revision
    summary-91fb25351d6b270b7a35829231e3a51cd5b40f1f3662fb5e04583af07484d858
    Input summary revisions
    summary-04ae2a2c2f05d5b4d9cd8dad7ae24ee1a97ac153ee870a60bf4cd65a2c36e602, summary-0500b0406543ba65534dcefe7cf9f446f7db071f09bf2a2a49253dc9ece06281, summary-14cc5960cf66614f0fff13b0ac691c85c0bdc29b39f5c87102995f924896d157, summary-15346dc6d0961fc11b58bf97d1b7050f83926eac7b13ec748d2d30023db26681, summary-158315b5fd7adcd653babff3e8982c5762d9a333fb0f6ad21ab66b6145e14ea4, summary-217966952d542079883e0629cbc5e15d5f7417bd7b2493daf8cc2d9ac4d74ec3, summary-28308ed703380935707236c8d7e68dd41c5ac51970119a3210aabec01d8b492a, summary-2975f53400f338a8e5a6a9a472d6bc36c8368c9cd67a5696dc9d69b4b91d688c, summary-2e0148708b9acccfbaf052b0e6d23f4c626b822766d3dbf69b8a8aa1a0df2afa, summary-377c1fcf05c4b64790e6cbcd3addba47b933dc4483ef2b2a0c8e6d8de146453d, summary-3c7e43da4b334cc403ffaf10355d8eda7082466f90e4b2a9f6c4212035f617fe, summary-467364032d3fc28a6fe4f6b06f81df3d287734e25eb880d66cafe1831273c0c9, summary-4b95f24a73507e65f2d31a80766915d45c2be4d8be402c8290595f41fcc57e2c, summary-4f469f206330f62e760bc63a44dfdba433ff87b637f276962cf38373fed484b4, summary-5047a77285dc3a95ad095325e7d88150509672d988f0b4610ef0d7a3fb3f36b3, summary-585ebe5418a15f9e0f5e60eb9d04df5cf2ca23ff15ee44748c7a32ecd669cc85, summary-7ef7cadcacdd36c6990edeceb9fbea8e03ac9a6608a73163156fc75b5eb56687, summary-9310406ad172dcee3c6a602579649757d281a554889848bb9a8e4935674771c9, summary-bafe9b98764badcb00287ca1a8b03149b098ec89e1c589cdb480a540a22a9a0c, summary-c5918c9a8fdfbc357ca3a4ce936f7ebcafdf49211815a1e696e19b77df3a56ef, summary-cc2631ac3e5b3937309fa10ee4af74de2568b7a60c339896153c809603bd1d2e, summary-cebfe32952fa5e836478db710ffe3e3fe5895c45314dc8df33282fb75ebd35cc, summary-d8eda607cf2af91d75e16eedc15b83568e79884021ae72da5571ffb157c77b59, summary-d91681b2cbcbb4748c1f7dba9f9b9b0e54456038b360f33f8261cdde21975a27, summary-dd3bbd54659d6fe2de99bdeecc50a7a05df298f6abe52cc2afd5198c5c504d0f, summary-e2579f013e3b8fbbe5ee15a587db2d436119b4ab2ebcbe8614f41ae5575e504d, summary-e5ce7694b13f7f38308159927908e9b729f28848a25bdfabe47ddaf007a16764, summary-ef8a4692fdf8320e26b56da393e138f364ade49faf499240257c71317adfcbb2, summary-f1352be98cab97367d5a67f7b6f84bb590a98d847f6652fae1dbf9fa56c4fd89, summary-f709fe4f5230fc5cb61fa13ad2098a6e5eeae579ff942ee7e62e78f7a82d7ab9, summary-fce7b31f05f1210bc5ffd3d852ef3647236d09919cbe80d75ca89cc4f0ac2af5
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    60bc5b19ce80f4a2e460274a4c636568a44d8ef84f8295f648685d94d1a98cfc
    Summary revision
    summary-9310406ad172dcee3c6a602579649757d281a554889848bb9a8e4935674771c9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7417bec3947a93b27ed1a44ede2be8f6daf16ea55bd5aeba8026ed2a68611103
    Summary revision
    summary-a521fef62ba1032cdefa033b43dacfa7a08acf85e193e5598962af69ad5bfa0d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    67eb1610e1c5ae90423ea15c24f5fefdf517d47f6c383850aee81e28c3b081ba
    Summary revision
    summary-b4b93085444ffa41c233221ce2292eee6b97214e131b53a559541c7ebf7a6ed4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    616bc42619b084914925f8c05277c2ff31f0f4e3272e346a53da4b3e4b0ebfce
    Summary revision
    summary-bafe9b98764badcb00287ca1a8b03149b098ec89e1c589cdb480a540a22a9a0c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    eb8f7cbd25ac02ea4baaea983d324069fc7b66c84912852f55b64e3979fd04a3
    Summary revision
    summary-c4890a6b4ac2da55bb370cf6dbea8c192bc0d6f1b727e338b88f0ba791d5c914
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e012ffbb415971f64255056dbf24d4e7b013634d114a1354564732daabc023b5
    Summary revision
    summary-c5918c9a8fdfbc357ca3a4ce936f7ebcafdf49211815a1e696e19b77df3a56ef
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6c7e7085b29a8dac80b64d3e9b03f57e75f7a47465b07f89c4c3390bf7b18716
    Summary revision
    summary-cc2631ac3e5b3937309fa10ee4af74de2568b7a60c339896153c809603bd1d2e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    934b2ed243c4c40afa6fc8b0e8f975c21a4615eacc484c1108defd0464c74da9
    Summary revision
    summary-cebfe32952fa5e836478db710ffe3e3fe5895c45314dc8df33282fb75ebd35cc
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b704bc6568a7001f92b204f3b3f9cf5018b407c985f5ac679440c329defa09f5
    Summary revision
    summary-d67a322f2113719f933889336fdf8f18297f8cee7901dcbcfff7e043137055f5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c0366e83ea0dfc09f1e589fdf7b07f65a6ccedded3b0aa7d7f5dc6a3d9ce8162
    Summary revision
    summary-d8eda607cf2af91d75e16eedc15b83568e79884021ae72da5571ffb157c77b59
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    772a92e36a1ac00052b50ed1ea115bcbe9614cfd7b90f0d8ea91724f83534e5e
    Summary revision
    summary-d91681b2cbcbb4748c1f7dba9f9b9b0e54456038b360f33f8261cdde21975a27
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6ffd71aa8279c05a94ee3ded0f428f065657786a91b34024c09a5c122a97ac01
    Summary revision
    summary-dc70f912ad78eda2067047445a28bd6574884f88b76bce9a17b3d68803f0405e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    5c59c9b0c0e554eb697edadff4c4cbae1efcd24fc4a87533b693e40d935e472f
    Summary revision
    summary-dd3bbd54659d6fe2de99bdeecc50a7a05df298f6abe52cc2afd5198c5c504d0f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    24ad0e64cc7848a4923777effb46932b52e9804c0a635da9e02116b50f9478f6
    Summary revision
    summary-e2579f013e3b8fbbe5ee15a587db2d436119b4ab2ebcbe8614f41ae5575e504d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c228ba044ab7b4552b7ac19d2c6eda3b5305b57c884eb16d2ed9cac8d093ee7c
    Summary revision
    summary-e5ce7694b13f7f38308159927908e9b729f28848a25bdfabe47ddaf007a16764
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    28b943c7ae994bdbd9f3f2ce7d46b975c13e96220a9cf8fb273b517f862b8194
    Summary revision
    summary-ee7980749b96c7f2c13af45f43718071e2cb04b7bbea6a049b35beef7f486dcd
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8742a8041990996f8c02a5a1d25b19b24335917ce267496e8f05a6c08ac82d81
    Summary revision
    summary-ef8a4692fdf8320e26b56da393e138f364ade49faf499240257c71317adfcbb2
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    86ff41e19b2d670f4bb10586021d4f0e75542cf976cb380c17d807d2c5b148c7
    Summary revision
    summary-f1352be98cab97367d5a67f7b6f84bb590a98d847f6652fae1dbf9fa56c4fd89
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    1375b74b083d789c35c7c52a588cb9170f274e263b083d7bd07a9454e77d06cc
    Summary revision
    summary-f709fe4f5230fc5cb61fa13ad2098a6e5eeae579ff942ee7e62e78f7a82d7ab9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8eeaef5731aef882272cd0e8a562773e05a6b175ffec45f434a6b70008d6bb75
    Summary revision
    summary-fac545fb974be33b7ce856743f412ccb0c102ffd35d0aa77f178a31509e9be18
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f74d294c3ab53fcda01befde1d6bb64bf794903164d594cd1c4be836a8b7cea5
    Summary revision
    summary-fce7b31f05f1210bc5ffd3d852ef3647236d09919cbe80d75ca89cc4f0ac2af5
    Input summary revisions
    Direct evidence analysis

Good at

Handles bounded project work well with thinking enabled, completing every small-coding test. This supports using it to find information in project files, make small edits and fix bugs—not assuming broader autonomous coding ability.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Does well at calculations from passages and at keeping or updating source-backed answers appropriately with thinking enabled. That makes it useful for document questions where unsupported challenges should not override the source.

Evidence for this judgment
Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Thinking on
Tasks passed7 of 7 (100.0%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Struggles with longer practical work such as building apps and investigating files or web pages. Overall task scores were poor in either mode, so small-edit success should not justify handing it an unattended project.

Evidence for this judgment
Larger practical tasks · Thinking off
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Difficult math and reasoning questions expose substantial limits, as do arithmetic, deduction and planning puzzles. Neither mode produced strong overall results on these tests; check demanding solutions independently.

Evidence for this judgment
Published hard problems · Thinking off
Average score30.0/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 2 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking on
Average score22.3/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes2 full credit · 3 partial · 6 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off
Average score41.2/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes14 full credit · 4 partial · 16 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking on
Average score38.3/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes13 full credit · 2 partial · 19 no credit
Explore these test results →

View this model’s answers, code and tool activity →

With thinking off, it failed every complete test of preserving source-backed answers under challenge while updating them when evidence changed. Avoid that mode when handling corrections accurately matters.

Evidence for this judgment
Resisting incorrect corrections · Thinking off
Tasks passed0 of 12 (0.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Should you enable thinking?

Enable thinking for small coding and handling challenges to source-backed answers. Coding success improved, though typical completion time rose despite lower average time. Handling corrections improved dramatically, at a modest additional wait in these runs.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on87.5% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Score change with thinking on+12.5 score points
Median time with thinking off7.5 seconds
Median time with thinking on15.2 seconds
Average time change with thinking on5.2 seconds faster
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off versus on
Thinking off → on0.0% · All-branches pass rate → 100.0% · All-branches pass rate
Paired test coverage12 of 12 tasks compared in both modes
Tasks timed in both modes12 pairs
Score change with thinking on+100.0 score points
Average time change with thinking on3.3 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Do not expect thinking to rescue longer work or puzzles. Larger practical-task scores were unchanged while average time increased; puzzle scores slipped slightly despite a small reduction in average time.

Evidence for this judgment
Larger practical tasks · Thinking off versus on
Thinking off → on27.3/100 · Mean task credit → 27.3/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Tasks timed in both modes11 pairs
Score change with thinking onNo change
Average time change with thinking on21.7 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off versus on
Thinking off → on41.2/100 · Mean puzzle credit → 38.3/100 · Mean puzzle credit
Paired test coverage34 of 34 tasks compared in both modes
Tasks timed in both modes34 pairs
Score change with thinking on-2.9 score points
Average time change with thinking on2.0 seconds faster
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking off, puzzle scores fell between the other tested medium-sized MoE models. Its absolute result remained weak.

Evidence for this judgment

Logic and number puzzles
Thinking off. Medium · 5–under 20B · MoE (total parameters). Rank 2 of 3 in this recorded comparison.

Logic and number puzzles · Thinking off
Average score41.2/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes14 full credit · 4 partial · 16 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzlesThinking off · Medium · 5–under 20B · MoE (total parameters)

Solve newly generated puzzles using arithmetic, deduction and planning.

Mean puzzle credit · 0–100 points · higher is better
34 of 34 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Gemma 4 19B-A4B REAP18.4B total parameters61.8/100
    20.6 points higher
  2. #2 Mellum 2.1 12B-A2.5B Thinking Q4_K_M12.1B total parameters41.2/100
    This model
  3. #3 Ling 3.0 Tiny Q4_07.9B total parameters29.6/100
    11.6 points lower

Other mode, this model only: thinking on · 38.3/100 · Mean puzzle credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

MoE models use only part of their parameters for each token. This group uses total size; active counts are not included in this saved comparison.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

With thinking on, small coding tied the best pass rate; Ornith 1.5 9B Q4_K_M reached the same result with a shorter median task time.

Evidence for this judgment

Speed: Everyday coding tasks
Thinking on. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Speed and task success: Everyday coding tasksThinking on · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
Ling 3.0 Tiny Q4_08.4 seconds16 of 16 tasks timed122.0 tokens/s15 of 1616 of 16 scored
LFM2.5 1.2B Thinking Q4_K_M11.0 seconds16 of 16 tasks timed293.3 tokens/s2 of 1616 of 16 scored
Ornith 1.5 9B Q4_K_M13.3 seconds16 of 16 tasks timed51.0 tokens/s16 of 1616 of 16 scored
Mellum 2.1 12B-A2.5B Thinking Q4_K_MThis model15.2 seconds16 of 16 tasks timed137.3 tokens/s16 of 1616 of 16 scored
Ministral 3 8B Reasoning 2512 Q4_K_M19.0 seconds16 of 16 tasks timed53.6 tokens/s12 of 1616 of 16 scored
Ministral 3 14B Reasoning 2512 Q4_K_M32.7 seconds16 of 16 tasks timed34.3 tokens/s8 of 1616 of 16 scored
Gemma 4 12B QAT Q4_033.6 seconds16 of 16 tasks timed35.3 tokens/s15 of 1616 of 16 scored
Nemotron Nano 9B v2 Q4_034.5 seconds16 of 16 tasks timed53.3 tokens/s10 of 1616 of 16 scored
Granite 4.2 8B Q4_036.7 seconds16 of 16 tasks timed49.9 tokens/s16 of 1616 of 16 scored
SmolLM3 3B Q4_K_M37.7 seconds16 of 16 tasks timed109.3 tokens/s1 of 1616 of 16 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Worth knowing

Publication redactions, shortened excerpts and withheld artifacts limit reader inspection. Some papers are published as source links rather than reproduced text; this does not establish that the model lacked those inputs.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

How do the weight formats compare?

Everyday coding tasks · Task pass rate · requested / displayed reasoning off. Size is the weight download, not RAM. Different runtimes and placement can also affect results.

MXFP4_MOE · 7.03 GBLocal · test machine with 32 GB RAM · vulkan · 99 GPU layers · Recorded · some results unscored or qualifiedThinking off14/16 passed16/16 scored
Q4_K_M · 8.07 GBLocal · test machine with 32 GB RAM · vulkan · 99 GPU layers · Fully scoredThinking off14/16 passed16/16 scored
Q6_K · 10.88 GBLocal · test machine with 32 GB RAM · vulkan · Unknown GPU layers · No scored resultsThinking offNot scored0/16 scored · 16 not yet recorded · setup comparability unverified

Qualified rows show the reason they are outside the rankings; they cannot establish superiority over a fully tested format. Open the setup to inspect context, runtime and controls.

Choosing a quant

Shared assessment of MXFP4_MOE, Q4_K_M · 2026-10-22 UTC. The current results table can include newer formats.

For small edits and bug fixes, MXFP4_MOE and Q4_K_M are both reasonable tested choices. Their everyday coding pass rates tied in this test, so MXFP4_MOE's smaller download is an attractive starting point. Its stronger results on larger practical tasks also favor it for tool-driven work. This advice applies only to the requested-off setups: MXFP4_MOE reasoning suppression was not verified, and no thinking-on versus thinking-off benefit or cost is established.

Evidence for this judgment

Everyday coding tasks · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatTask pass rateTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
14/16 passed16/16 scored6.9s16/16 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
14/16 passed16/16 scored7.5s16/16 timed8.07 GBWeights, not required RAMAnswers and code

Larger practical tasks · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean task creditTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored17.4s11/11 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
27.3/10011/11 scored57.8s11/11 timed8.07 GBWeights, not required RAMAnswers and code

For larger tool-driven work with reasoning requested off, prefer MXFP4_MOE: it earned substantially more task credit than Q4_K_M with much shorter median task time. The shorter time accompanied better outcomes, not merely quicker failures.

Evidence for this judgment

Larger practical tasks · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean task creditTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored17.4s11/11 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
27.3/10011/11 scored57.8s11/11 timed8.07 GBWeights, not required RAMAnswers and code

With reasoning requested off, Q4_K_M had narrow leads in exact instruction following, logic puzzles and selected hard problems. The instruction-following gap was a single question. These results offer only a modest reason to choose its larger download for those uses.

Evidence for this judgment

Following exact instructions · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatStrict prompt complianceTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
13/23 passed23/23 scored3.2s23/23 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
14/23 passed23/23 scored3.6s23/23 timed8.07 GBWeights, not required RAMAnswers and code

Logic and number puzzles · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean puzzle creditTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
38.3/10034/34 scored32.5s34/34 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
41.2/10034/34 scored31.5s34/34 timed8.07 GBWeights, not required RAMAnswers and code

Published hard problems · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean answer creditTask timeDownloadInspect
MXFP4_MOEJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
28.2/10011/11 scored31.1s11/11 timed7.03 GBWeights, not required RAMAnswers and code
Q4_K_MSelected formatJetBrainsvulkan · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
30.0/10011/11 scored30.2s11/11 timed8.07 GBWeights, not required RAMAnswers and code
Scope and limits
  • These are comparisons of tested setups registered to one checkpoint. They do not isolate bit depth from conversion, publisher, runtime-build or template differences.
  • Only complete eligible banks with matching task controls and recorded context enter the quality comparison. Execution profiles are shown per setup; timing is compared only when every recorded execution profile matches and timing coverage is complete. Other results remain coverage, not ranked evidence.
  • Download size is not RAM or VRAM requirement. No memory-fit recommendation is established by this comparison.
  • Small score differences can reflect a single question. Similar results are not proof of equivalence; timings describe this benchmark machine.
About this quant assessment
Requested model
gpt-6-astra
Returned model
Not supplied; request identity only.
Exact provider snapshot
Not supplied.
Generation route
Codex CLI · ChatGPT account
Prompt version
quant-summary-v3
Review time
2026-10-22T07:02:10.492416+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Source edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Revision
quant-summary-d62c9799e9aaa5e7383e24b56d1e9bbcd1d05160b22e267a5cdbf4015d2a13d0

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Thinking offMean matched task credit87.5%
Thinking onMean matched task credit100.0%

Thinking on took 5.2 seconds less per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on87.5% → 100.0%
Mean paired credit change+12.5 percentage points
Question outcomes2 improved · 14 unchanged · 0 worse
Mean added task time-5.2s
Mean task time, off → on21.8s → 16.6s
Median task time, off → on7.5s → 15.2s
Median of paired time changes+4.3s
Total added task time-83.2s
Ratio of summed times, on / off0.8×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Weight formats compared

All tested weight formats of this checkpoint, using the selected thinking setting. This shared comparison updates with new results; each saved model summary keeps its original date.

Everyday coding tasks

Scores and times describe this task bank. Quick failures are not a speed advantage. Different execution settings are labeled.

VersionTask resultTypical task timeWeight downloadExplore
Mellum 2.1 12B-A2.5B Thinking MXFP4_MOEThinking off87.5%16/16 scored6.9 seconds16/16 timed7.03 GBAnswers and code

Compare

Mellum 2.1 12B-A2.5B Thinking Q4_K_MThinking off87.5%16/16 scored7.5 seconds16/16 timed8.07 GBAnswers and code
Mellum 2.1 12B-A2.5B Thinking Q6_KThinking off · Different or uncertain execution settingsNo assessed score0/16 scored · Incomplete / not rankedNot measured0/16 timed10.88 GBAnswers and code

Compare

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking off · temperature 0.6

48/48 repeats scored · 42 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 12 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested off · verified
  • Seed 303 · requested off · verified
  • Seed 101 · requested off · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

14 full credit1 incorrect1 failed to finish

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-b61fc13326404bed8c38b50c
Matching candidate set: paired-b61fc13326404bed8c38b50c

Mellum 2.1 12B-A2.5B Thinking MXFP4_MOE

Another tested weight format of this checkpoint. Complete results first, then nearest download size.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-checkpoint-matching-controls
Facts: fact-f2afff5be73b3b2bf0133034, fact-18b9d27661127ac14e5e16ab
Matching candidate set: fact-18b9d27661127ac14e5e16ab, fact-48a3c8ec09a4a055e278d563

Gemma 4 19B-A4B REAP (Q4_K_S)

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-f2afff5be73b3b2bf0133034, fact-69b7b20fcb8961feae28e4a8
Matching candidate set: fact-69b7b20fcb8961feae28e4a8, fact-1ec245f5a9717680a5818e83, fact-57f1be0ca8bc0e7cb0e343b4, fact-45e16a9e4c4c3e1b61f0f139

Tested weights

Q4_K_M · 8.07 GB

JetBrains · recorded source

Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

MXFP4_MOE · 7.03 GB

JetBrains · recorded source

Mellum2.1-12B-A2.5B-Thinking-MXFP4_MOE.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Q6_K · 10.88 GB

JetBrains · recorded source

Mellum2.1-12B-A2.5B-Thinking-Q6_K.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.