AgentBench / results
← All models

Granite 4.2 8B Q4_0

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_0 · 5.06 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_0

Q4_0 · 5.06 GB of weights · vulkan · context 16384 · GPU layers 99

Medium · 5–under 20B · dense · 8.8B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 2 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

67.1 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking onDetails
Everyday coding tasksTask pass rate
Pass rate
16/16 · 100%
Answers completed
16/16 · 100%
Correct / completed
16/16 · 100%
16 full credit
16/16 scored36.7s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
27.3%
Full-credit rate
3/11 · 27%
Answers completed
8/11 · 73%
Full credit / completed
3/8 · 38%
3 full credit5 incorrect3 failed to finish
11/11 scored303.7s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
48.8%
Full-credit rate
14/34 · 41%
Answers completed
34/34 · 100%
Full credit / completed
14/34 · 41%
14 full credit6 partial credit14 incorrect
34/34 scored40.2s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
20.8%
Full-credit rate
2/11 · 18%
Answers completed
11/11 · 100%
Full credit / completed
2/11 · 18%
2 full credit2 partial credit7 incorrect
11/11 scored39.2s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
20/23 · 87%
Answers completed
23/23 · 100%
Correct / completed
20/23 · 87%
20 full credit3 incorrect
23/23 scored41.0s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
14/16 · 88%
Answers completed
16/16 · 100%
Correct / completed
14/16 · 88%
14 full credit2 incorrect
16/16 scored11.4s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
6/7 · 86%
Answers completed
7/7 · 100%
Correct / completed
6/7 · 86%
6 full credit1 incorrect
7/7 scored18.9s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
46.6%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit6 partial credit1 incorrect
8/8 scored43.9s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
12/12 · 100%
Answers completed
12/12 · 100%
Correct / completed
12/12 · 100%
12 full credit
12/12 scored11.7s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this setup with thinking enabled for small project edits and handling source-backed answers under challenge. It passed every assessed test in those areas, but larger practical work remained weak, so keep longer jobs supervised. This is a useful bounded assistant, not a compelling autonomous project worker.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Typical time per task (median)36.7 seconds
Tasks timed16 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking off
Average score36.4/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 0 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T03:32:21.918205Z
Review time
2026-10-22T07:26:27.156773+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
df01b5f7e4021a8605a33d064a2115b2269d151a8b249b278abe51a6b2ce6ae5
Original analysis input hash
6839efed5287f26b9b5865efcadd68b32d213755de3b2996d9a439eaa654d554
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-71d362f3782e1e1c8e59bc893b74cfac908ecc38f1699b39d811384e9d9ec207
Summary revision
summary-3da86d334c0a448b4049bbc48dae7f4aeef4859c05ac04b97390ac0901fa4911

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (55)
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    78be92c44bcb874db084efae612aa7812f0358febdefff350653c45f68131d38
    Summary revision
    summary-07f8cdcfcefcc25555585f696859476911e808746627764d43a938cfb708c7ef
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2838d1804045ce370ca6ac66d3a56fa72f634d430f7222d031f53dd6866b0ef9
    Summary revision
    summary-0bb8371833868baf978479b5e524413b9af85ba74fbb59676c0abe7d2cc981cb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    76752728c5d6d620d7fcc4c43d5e85eaa813b03ec881ace6d9b9f005f005002c
    Summary revision
    summary-0ec916331fe2f02278a4a2a6e390c9bb3962922346f6e3d490e9ba01604d47d9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3fdb1789f721ffe8efa98b050d5e57f76dac741b4a99ebdee85cf85fdcb73827
    Summary revision
    summary-187b9029e14517b18b47b0522e3115d6931a889802eb408c97263b61bd049686
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a976d3ba78f728b6a64f2d16c72837776aef14a7e9ccef74c66d6749fefb4ad4
    Summary revision
    summary-1a7e19a6b4be637f193729e83e6de340f8960d0501d1ca695e5aa65e5745dc31
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2d6680c5e2335c0c6c73671c24d96b1a635efdd3bce8ea3b7c79f540388ea072
    Summary revision
    summary-21eeea20fe98d3c71b04b3c04bc5dd2496cec8006c4e9a12603ddff0663bbaa5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    210bdd30070409731d6a81f5ada3cfe6c4e4c9f6358e56ebb3827d0419704c82
    Summary revision
    summary-22e5a71b40f1fec98bd023bd44a3ace286d928d97c86ccc437e2475534a5cd98
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4a4dd4668db5df20b8d835eec8d7bcf8b6561a5f51d35629b5e22acb0fad072e
    Summary revision
    summary-239c8bddbda373676c1152fc64b58a0eb694b96c2db1764567253cef3e63ed66
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c0ea5c7036fde9bef8b81fb6575953c3ea17a4d7494e8287270f4988232fc80d
    Summary revision
    summary-276761ebc92ea0d21cf1d79d648a5866467b6148413c2a58914983f48056e2c9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    dd2e8302ef764f3d07a9c0a9a0a89a462b4c843d7445c688fb2e3558fc23c4f2
    Summary revision
    summary-305b513f7fb9cd6caa2d89bba7f28a5831d2d0d864674cc9265f3b6da0743fa0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3ef51ebf05c4fac062e58eac9c68465b4a5e3bb39c03327ec3a0442ec81dc989
    Summary revision
    summary-36d4b24234e842f7dfcfd9f5ebd026663ef319cada98f1071509f413565c8016
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    30272c32b3832515494754fe3f7c0ba5d523e904b1b271ed0be585e9463514fd
    Summary revision
    summary-37e61a93a8574a9245c38f08c428c66e6b3de1673d37163a8186d7e726ff1f80
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2d66ded1120dc45012f6835a2c0fbade6f58d1d247b9a8e02f07d22291c06653
    Summary revision
    summary-3b2d8d3c7ae9306ca5e83f8a5d6d81ffa88253f21cbb9fcb1d743d8791736f58
    Input summary revisions
    summary-0bb8371833868baf978479b5e524413b9af85ba74fbb59676c0abe7d2cc981cb, summary-36d4b24234e842f7dfcfd9f5ebd026663ef319cada98f1071509f413565c8016, summary-478c2fe88b13f71bed76d6005bb4120abef04893e7d9c118202afa615f6e0d51, summary-546a46446ef8383644aa569150fb653cc3692b508026c6c150b63f138e7cfc69, summary-8d5cce0d32a2079b0f182b5263b15f9a4b7cdedeed59222224bf980bbbad619e, summary-9439520c3fd39ed04f29d81f87c4f741969931e8301413a776bd416b399c6c0e, summary-a05ec41618de266c53e8b482822e9cd0e57e80f5d1322e4f0e7bda24b6d459ae, summary-dcbb12efc62bad434bdf95ad957dd1404b17e743fc142b1f23f8333d508f407f, summary-f6748e50a501ef101e201ebdef5d0787cde30d906947d06e9927a0faf146fa95
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    85bd09bfc784701ee64c737125996057a7cde60eda099b4590149edea60fc9a8
    Summary revision
    summary-3e9b25c203d9548e85759b961a4063f59ac943f5c8b69aafc1fbd7138638907f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b2d5478b71efd72516f492cbfbb0655822f0cdb37e179dbb119c92aa2f32ec1a
    Summary revision
    summary-478c2fe88b13f71bed76d6005bb4120abef04893e7d9c118202afa615f6e0d51
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6cc7fed9528372848e508b6647ffd399bd6d85b287f73b8e97481fbc50b00be9
    Summary revision
    summary-546a46446ef8383644aa569150fb653cc3692b508026c6c150b63f138e7cfc69
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9ca7ae7ad777e79c8795db8d672eda18d18b1e8a71bf33185732265c03228b28
    Summary revision
    summary-57e04745d32128dec983b3b42ad4065a57459a51cc7c3cbb7c3fdbe35bba1994
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a88fb265c9a586fd6cce47fff7d1960ece3cdc4f2c55749f4547e1c64d9d1b07
    Summary revision
    summary-5e634c0de0be42965be4acba07226458949203c6db0781bc3ef1d917be91c051
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b35322761454f32ae79ef87088c2b1743d6ed6fdb48839bfbc9c5c4cfc95a68a
    Summary revision
    summary-65fbfa86b2c4d3a7814de54e23e960cf1f42416828d074cf4d18923922136895
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8ae2e47f2e97b4c18ddcd27bfa060e83a9d8a95a02076795b1a1cee124f55661
    Summary revision
    summary-665229269265befa273aeaddb1d2b367309f73d2991d53b30c1d80dc538d700c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    789a37ec8a340b05a276bcd1697a042ef715410fab84e1aa32de1145d73ae850
    Summary revision
    summary-69c24ca7ace0b903906f7e12cb7a221916f4489a3ee1869ec4a639e81a394edb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    55711a27fe3ab505ffd35ea6f07bd4b48601e7a766c81ce7dc30ee0bb17a74e0
    Summary revision
    summary-714a6d9f3dc22fa4b8cb6b38af25d11af6ccb1a1884a5c25e8a4bfde6d67b4d5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    bb297492feb2d7e9b9b8aadf488a236105551725c8634231f40ee77369fb0317
    Summary revision
    summary-7401dcc77b96fe6c175984ea9ce7dd32128a3fb0902ef0d8299a1b58af32cba9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    950366940d145639b889564f06b683ed68088102be5a70c3888261a04d9ec5b7
    Summary revision
    summary-8116ed727f078e5259eceaac366049d0cf8105f8082aa3c555a0df971e07e2d6
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    66b25bb0ccd8136c1d0be14395a7dcd23d30488f7804b35ff4b4a07d8d92983a
    Summary revision
    summary-8139828ed986f8660aa2c31dae99a3ec71702e748fec1e312c908dc25d0cc890
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    66076cd3ea48e21b5587eb8fa6817c5ff15edb7aaf1900575e0dcee433af3c0b
    Summary revision
    summary-837adeca5191ee3838e4dd98f6bb7a69cd8b79e7eda3f6ce6f1c381bdd0b87b2
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f7e4cc33af89cd956041e8bb44893d44a6bafa169119dcd60c26868845625366
    Summary revision
    summary-87379605eac32b3684db8cf928e74ed704acd98b99cf9f7db82fe1046b5b4c5f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    76658021879f764facec2fa84afae893add4fc4686c0b2cd6daaf5cfe7dfbdcf
    Summary revision
    summary-8c7dbb090eadf6fdc6ab0feec42add37d71602db981bd3634de443dc8795e371
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3762ca05b35391d8e83e0a0c76d67514da08ac28f088b69078762e086d595143
    Summary revision
    summary-8d5cce0d32a2079b0f182b5263b15f9a4b7cdedeed59222224bf980bbbad619e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d25979270e9bd8d0eb54a05159ee1e10f6edf666cb8f3fc681c302adf7b4b854
    Summary revision
    summary-8dc042787f0e9941cea143906edee7b4b6a14c1e103f6fdd95aec252ac8a33b0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3eafec450a1e643657f8e357b475c6fd20584294c0341d89dc0dbb8f1ecc4c38
    Summary revision
    summary-9439520c3fd39ed04f29d81f87c4f741969931e8301413a776bd416b399c6c0e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    beedbe9bd9bf6d7c3e85a5f1b9199352db1f43b611b725f2a34ef01b0b9a9043
    Summary revision
    summary-95e34df17c7558c058ebd5e584077da88a02e1a32ae9a5e2d3e3d41e6f4d9eb7
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    27cbc8d330f8d2f7ce24e1c3701af24572d0cce7828c469b792638d591f693d1
    Summary revision
    summary-98731352a76c0779f7df944cc6fbbc5b8a131821806c6f61ed38301e0d0fd4dd
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    758c06f0f090dacdd04458b00417c5e9aee1e654fde23269fcdffcac38e5e926
    Summary revision
    summary-9ba04830e0890efbeeb2348d03678ede14837bd9f1e92b69f3ae0ab2e4c6cfff
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8698c4b5a3ddf9ef5e9ea1f0ed8abe7f8c5c5bfffd15713f848c363a0e0f9028
    Summary revision
    summary-a05ec41618de266c53e8b482822e9cd0e57e80f5d1322e4f0e7bda24b6d459ae
    Input summary revisions
    summary-0ec916331fe2f02278a4a2a6e390c9bb3962922346f6e3d490e9ba01604d47d9, summary-21eeea20fe98d3c71b04b3c04bc5dd2496cec8006c4e9a12603ddff0663bbaa5, summary-239c8bddbda373676c1152fc64b58a0eb694b96c2db1764567253cef3e63ed66, summary-276761ebc92ea0d21cf1d79d648a5866467b6148413c2a58914983f48056e2c9, summary-305b513f7fb9cd6caa2d89bba7f28a5831d2d0d864674cc9265f3b6da0743fa0, summary-37e61a93a8574a9245c38f08c428c66e6b3de1673d37163a8186d7e726ff1f80, summary-3e9b25c203d9548e85759b961a4063f59ac943f5c8b69aafc1fbd7138638907f, summary-57e04745d32128dec983b3b42ad4065a57459a51cc7c3cbb7c3fdbe35bba1994, summary-5e634c0de0be42965be4acba07226458949203c6db0781bc3ef1d917be91c051, summary-65fbfa86b2c4d3a7814de54e23e960cf1f42416828d074cf4d18923922136895, summary-69c24ca7ace0b903906f7e12cb7a221916f4489a3ee1869ec4a639e81a394edb, summary-714a6d9f3dc22fa4b8cb6b38af25d11af6ccb1a1884a5c25e8a4bfde6d67b4d5, summary-7401dcc77b96fe6c175984ea9ce7dd32128a3fb0902ef0d8299a1b58af32cba9, summary-8116ed727f078e5259eceaac366049d0cf8105f8082aa3c555a0df971e07e2d6, summary-8139828ed986f8660aa2c31dae99a3ec71702e748fec1e312c908dc25d0cc890, summary-837adeca5191ee3838e4dd98f6bb7a69cd8b79e7eda3f6ce6f1c381bdd0b87b2, summary-8dc042787f0e9941cea143906edee7b4b6a14c1e103f6fdd95aec252ac8a33b0, summary-98731352a76c0779f7df944cc6fbbc5b8a131821806c6f61ed38301e0d0fd4dd, summary-9ba04830e0890efbeeb2348d03678ede14837bd9f1e92b69f3ae0ab2e4c6cfff, summary-a72000ebeec006fa3759ded8544510ad0eb91db91ee743af507658a754c3cd76, summary-b2ab52fc39b18a8886a5e7c7b45370bad367744a0caa9becfeec0ef786e2e9c3, summary-b7678515d85b37a246fb1384af28e9923cff2de28626471a81888bcd6428de9e, summary-ba23dac0297fe48d651def5d38b9595e5c77b7de5c9991c705a4b8c9d72dd7a6, summary-bb37350d7da848a742ef4a82094114d1b95610aec9392fda20797bca0397816b, summary-c39deb78fb2ab7ca2186b7594d9e0b861bcb451006dccb42bc9403930f32890d, summary-c4c4ec274058e2d530d91e9e3ef0162969c631681e2fc01c63bab93106846d17, summary-d0280e35fb20640ba296e0a2c2d739d0e3cc127ecb64bf108fad2b1ad6d157a0, summary-d6272bc803f7cfbce00ff91ad14c8aab4c771a5f460378ab72b2aab60bf22e41, summary-dbcd132da0fcf096f6c567f767c79e5982728a1c846ee2ecb8c96504eb544a48, summary-de5794377057f3b80b6505da848dc11c06ac8c372467da706fe1086142ecc68f, summary-ec268b1c36007bcb27b66e3b23110bbe17e41b7f8ed8f166d6dc02683f4ad584, summary-efa06dc965ff2c0465a8e9e5f6e6d7cbdfd263ac826760982f09f126bcb8d5e5
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    48fb1e224188679031d496a9c8b4c83a0e532c97850e033596b796fd44a88872
    Summary revision
    summary-a72000ebeec006fa3759ded8544510ad0eb91db91ee743af507658a754c3cd76
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    83c6155483d60f117e08b3b01e02c3d74006837883145feb6540ca84ffad62dc
    Summary revision
    summary-b2ab52fc39b18a8886a5e7c7b45370bad367744a0caa9becfeec0ef786e2e9c3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e14937b6793942e36bdc6afdb181f40c2a214550b0431f753ddbe6ea1be7f5fa
    Summary revision
    summary-b4586d42ebbe5d1c0f1f86e0f60900257b92b471d6eccb5da996ebbd94e7096a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3ae0e94f6f2fc0d2d1f9497d576d71c0247539330e0996691394db23b10abeb9
    Summary revision
    summary-b7678515d85b37a246fb1384af28e9923cff2de28626471a81888bcd6428de9e
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    ae1e3bd3ef7ec0854cba3130df871813113b0cf581b2bbc6ecddbdebe28de5fb
    Summary revision
    summary-ba23dac0297fe48d651def5d38b9595e5c77b7de5c9991c705a4b8c9d72dd7a6
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    63ef9972e5e6c5888bb193e1ef2a6e7427901be9d761366178f3c2cca7e52d02
    Summary revision
    summary-bb37350d7da848a742ef4a82094114d1b95610aec9392fda20797bca0397816b
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7f9c96b38585ddd872221c0457fec225873767b35da7e27d34879c887f80bedd
    Summary revision
    summary-c39deb78fb2ab7ca2186b7594d9e0b861bcb451006dccb42bc9403930f32890d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    bcbcac0fb1c9ca7f34a37d646a6897eab2e7164cd73ba10bafb4712fb6985efc
    Summary revision
    summary-c4c4ec274058e2d530d91e9e3ef0162969c631681e2fc01c63bab93106846d17
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    3a620c3db4ba903d1de26e2ab8dea9cc3363bdc8dd094ea5347a22808212c256
    Summary revision
    summary-d0280e35fb20640ba296e0a2c2d739d0e3cc127ecb64bf108fad2b1ad6d157a0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    76a4412535b130a42f29c05ef8623c0911bac7f97fdc211584b515a314d1a01f
    Summary revision
    summary-d510d50ae8826ea3d66360c5fa73fc7153cafd280d4dca193538865f3976a3b0
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d1fb88314d7ec6261b25dcb9f2060aa2b4ae45c2efbd010b0a7c66e8d166723a
    Summary revision
    summary-d6272bc803f7cfbce00ff91ad14c8aab4c771a5f460378ab72b2aab60bf22e41
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    ef6eee744dd669827e936f887e4d7c66b65d863a3d7c11e15bd753177df5e55c
    Summary revision
    summary-dbcd132da0fcf096f6c567f767c79e5982728a1c846ee2ecb8c96504eb544a48
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    e1fa4b5092aa11a538aeff6eeb93abbfc6efff0e8fcff451afa501b762475c12
    Summary revision
    summary-dcbb12efc62bad434bdf95ad957dd1404b17e743fc142b1f23f8333d508f407f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4653c09243489149aff4296cb0e12b96043d95383c54c543e0be56554b8fdd67
    Summary revision
    summary-dd7601884c47d7a22651dea6583c7a2a62ae2b7442ec53d78b908bcad38af95a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    a8e586524ed7d548822ed154469b9a2b9a0f23fe054533c534b57d972f9a038f
    Summary revision
    summary-de5794377057f3b80b6505da848dc11c06ac8c372467da706fe1086142ecc68f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    69a8f5355ab9c959ded1582511aa1e2ac68b0bf7155099e10b245696a78def9b
    Summary revision
    summary-ec268b1c36007bcb27b66e3b23110bbe17e41b7f8ed8f166d6dc02683f4ad584
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fa4ab6b839d74876f23563445ed75da737532eccdf4b50607d6b5256bb8c5094
    Summary revision
    summary-eca7b3919cf1a207d21480e4e85b8acb8676e60b3978f0e415d170c948617c82
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f65884f860e7ffda7c7041f898d844ffb71107964013d9e4ba7734920900fe91
    Summary revision
    summary-ecbd1537bd430b45cce591b0955e5580ad9f83f448f4f8b2225df33fa3d5c30d
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9ddb41e1b29c1c3465b3778ee6a87ebe08c10ac4d35733a22331c6cb96b41671
    Summary revision
    summary-efa06dc965ff2c0465a8e9e5f6e6d7cbdfd263ac826760982f09f126bcb8d5e5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    20be745c77ab6c4ce809c971f455bf3ee78ca88a17bc4a7e76769b816aa39586
    Summary revision
    summary-f6748e50a501ef101e201ebdef5d0787cde30d906947d06e9927a0faf146fa95
    Input summary revisions
    summary-07f8cdcfcefcc25555585f696859476911e808746627764d43a938cfb708c7ef, summary-187b9029e14517b18b47b0522e3115d6931a889802eb408c97263b61bd049686, summary-1a7e19a6b4be637f193729e83e6de340f8960d0501d1ca695e5aa65e5745dc31, summary-22e5a71b40f1fec98bd023bd44a3ace286d928d97c86ccc437e2475534a5cd98, summary-665229269265befa273aeaddb1d2b367309f73d2991d53b30c1d80dc538d700c, summary-87379605eac32b3684db8cf928e74ed704acd98b99cf9f7db82fe1046b5b4c5f, summary-8c7dbb090eadf6fdc6ab0feec42add37d71602db981bd3634de443dc8795e371, summary-95e34df17c7558c058ebd5e584077da88a02e1a32ae9a5e2d3e3d41e6f4d9eb7, summary-b4586d42ebbe5d1c0f1f86e0f60900257b92b471d6eccb5da996ebbd94e7096a, summary-d510d50ae8826ea3d66360c5fa73fc7153cafd280d4dca193538865f3976a3b0, summary-dd7601884c47d7a22651dea6583c7a2a62ae2b7442ec53d78b908bcad38af95a, summary-eca7b3919cf1a207d21480e4e85b8acb8676e60b3978f0e415d170c948617c82, summary-ecbd1537bd430b45cce591b0955e5580ad9f83f448f4f8b2225df33fa3d5c30d

Good at

Handles bounded programming assistance well: with thinking enabled, it passed every small-project task covering finding information, editing files and fixing bugs. That supports supervised everyday coding, not larger autonomous development.

Evidence for this judgment
Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Useful for short, source-based questions with thinking enabled: it handled every test of resisting unsupported corrections and accepting source updates, and answered most passage questions correctly.

Evidence for this judgment
Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking on
Tasks passed14 of 16 (87.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Larger practical work is a poor fit: most tasks failed their checks in either mode. Do not treat success on small edits as evidence it can independently finish a longer project.

Evidence for this judgment
Larger practical tasks · Thinking on
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking off
Average score36.4/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 0 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Difficult math and reasoning remain weak, and logic puzzles are uneven even with thinking enabled. Research-paper answers also achieved only middling answer matching; short-passage success does not make it a dependable research assistant.

Evidence for this judgment
Published hard problems · Thinking on
Average score20.8/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes2 full credit · 2 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking on
Average score48.8/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes14 full credit · 6 partial · 14 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking on
Average score46.6/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes1 full credit · 6 partial · 1 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Should you enable thinking?

Enable thinking for small coding and finding answers in passages. Coding gained modestly at a substantial average time cost; passage accuracy improved dramatically with a smaller delay. Without thinking, passage-answer accuracy was extremely poor.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on87.5% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Average time change with thinking on46.2 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Reading without guessing · Thinking off versus on
Thinking off → on12.5% · Exact-match accuracy → 87.5% · Exact-match accuracy
Paired test coverage16 of 16 tasks compared in both modes
Tasks timed in both modes16 pairs
Average time change with thinking on5.6 seconds slower
Explore these test results →

View this model’s answers, code and tool activity →

Leave thinking off for larger tool-driven jobs if you try them: enabling it substantially increased average time while slightly lowering the task score. Neither mode made these jobs a good fit.

Evidence for this judgment
Larger practical tasks · Thinking off versus on
Thinking off → on36.4/100 · Mean task credit → 27.3/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Tasks timed in both modes11 pairs
Average time change with thinking on217.7 seconds slower
Score change with thinking on-9.1 score points
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking enabled, small-coding and source-correction results matched the best other tested medium-sized dense models. Puzzle scores lagged most of that group, though gaps varied and the nearest peer was close. With thinking enabled, small coding tied the best pass rate, but its median task time was near the slow end of the eligible same-machine comparison; only one peer was slower.

Evidence for this judgment

Everyday coding tasks
Thinking on. Medium · 5–under 20B · dense. Tied at rank 1 of 9 in this recorded comparison.

Resisting incorrect corrections
Thinking on. Medium · 5–under 20B · dense. Tied at rank 1 of 9 in this recorded comparison.

Logic and number puzzles
Thinking on. Medium · 5–under 20B · dense. Rank 7 of 9 in this recorded comparison.

Everyday coding tasks · Thinking on
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking on
Average score48.8/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes14 full credit · 6 partial · 14 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Speed: Everyday coding tasks
Thinking on. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Everyday coding tasksThinking on · Medium · 5–under 20B · dense

Find information in a small project, edit files and fix bugs.

Task pass rate · 0–100% · higher is better
16 of 16 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 E4B7.5B total parameters100.0%
    Same score
  2. =1 Granite 4.2 8B Q4_08.8B total parameters100.0%
    This model
  3. =1 Ornith 1.5 9B Q4_K_M9.2B total parameters100.0%
    Same score
  4. =1 Qwen3.5 9B9.0B total parameters100.0%
    Same score
  5. =5 Gemma 4 12B11.9B total parameters93.8%
    1 task fewer passed
  6. =5 Gemma 4 12B QAT Q4_011.9B total parameters93.8%
    1 task fewer passed
  7. #7 Ministral 3 8B Reasoning 2512 Q4_K_M8.5B total parameters75.0%
    4 tasks fewer passed
  8. #8 Nemotron Nano 9B v2 Q4_08.9B total parameters62.5%
    6 tasks fewer passed
  9. #9 Ministral 3 14B Reasoning 2512 Q4_K_M13.5B total parameters50.0%
    8 tasks fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking off · 87.5% · Task pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Resisting incorrect correctionsThinking on · Medium · 5–under 20B · dense

Keep a source-backed answer when challenged, but change it when the source is updated.

All-branches pass rate · 0–100% · higher is better
12 of 12 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. =1 Gemma 4 12B11.9B total parameters100.0%
    Same score
  2. =1 Gemma 4 12B QAT Q4_011.9B total parameters100.0%
    Same score
  3. =1 Gemma 4 E4B7.5B total parameters100.0%
    Same score
  4. =1 Granite 4.2 8B Q4_08.8B total parameters100.0%
    This model
  5. =1 Ornith 1.5 9B Q4_K_M9.2B total parameters100.0%
    Same score
  6. =1 Qwen3.5 9B9.0B total parameters100.0%
    Same score
  7. =7 Ministral 3 14B Reasoning 2512 Q4_K_M13.5B total parameters91.7%
    1 task fewer passed
  8. =7 Nemotron Nano 9B v2 Q4_08.9B total parameters91.7%
    1 task fewer passed
  9. #9 Ministral 3 8B Reasoning 2512 Q4_K_M8.5B total parameters16.7%
    10 tasks fewer passed

Several models reached full marks; this test cannot separate their top scores.

Other mode, this model only: thinking off · 83.3% · All-branches pass rate.

How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Logic and number puzzlesThinking on · Medium · 5–under 20B · dense

Solve newly generated puzzles using arithmetic, deduction and planning.

Mean puzzle credit · 0–100 points · higher is better
34 of 34 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Gemma 4 E4B7.5B total parameters82.4/100
    33.5 points higher
  2. #2 Gemma 4 12B QAT Q4_011.9B total parameters79.4/100
    30.6 points higher
  3. #3 Gemma 4 12B11.9B total parameters75.0/100
    26.2 points higher
  4. #4 Ornith 1.5 9B Q4_K_M9.2B total parameters73.5/100
    24.7 points higher
  5. #5 Nemotron Nano 9B v2 Q4_08.9B total parameters50.1/100
    1.2 points higher
  6. #6 Qwen3.5 9B9.0B total parameters50.0/100
    1.2 points higher
  7. #7 Granite 4.2 8B Q4_08.8B total parameters48.8/100
    This model
  8. #8 Ministral 3 14B Reasoning 2512 Q4_K_M13.5B total parameters40.9/100
    7.9 points lower
  9. #9 Ministral 3 8B Reasoning 2512 Q4_K_M8.5B total parameters37.8/100
    11.1 points lower

Other mode, this model only: thinking off · 29.6/100 · Mean puzzle credit.

How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-12 assessment. Task-specific results.

Speed and task success: Everyday coding tasksThinking on · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
Ling 3.0 Tiny Q4_08.4 seconds16 of 16 tasks timed122.0 tokens/s15 of 1616 of 16 scored
LFM2.5 1.2B Thinking Q4_K_M11.0 seconds16 of 16 tasks timed293.3 tokens/s2 of 1616 of 16 scored
Ornith 1.5 9B Q4_K_M13.3 seconds16 of 16 tasks timed51.0 tokens/s16 of 1616 of 16 scored
Mellum 2.1 12B-A2.5B Thinking15.2 seconds16 of 16 tasks timed137.3 tokens/s16 of 1616 of 16 scored
Ministral 3 8B Reasoning 2512 Q4_K_M19.0 seconds16 of 16 tasks timed53.6 tokens/s12 of 1616 of 16 scored
Ministral 3 14B Reasoning 2512 Q4_K_M32.7 seconds16 of 16 tasks timed34.3 tokens/s8 of 1616 of 16 scored
Gemma 4 12B QAT Q4_033.6 seconds16 of 16 tasks timed35.3 tokens/s15 of 1616 of 16 scored
Nemotron Nano 9B v2 Q4_034.5 seconds16 of 16 tasks timed53.3 tokens/s10 of 1616 of 16 scored
Granite 4.2 8B Q4_0This model36.7 seconds16 of 16 tasks timed49.9 tokens/s16 of 1616 of 16 scored
SmolLM3 3B Q4_K_M37.7 seconds16 of 16 tasks timed109.3 tokens/s1 of 1616 of 16 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Worth knowing

Redactions, withheld artifacts and absent components limit reader inspection; some papers are linked rather than reproduced. Model-authored project checks were not independently rerun, and detailed repair-check output is unavailable.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Thinking offMean matched task credit87.5%
Thinking onMean matched task credit100.0%

Thinking on took 46.2 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on87.5% → 100.0%
Mean paired credit change+12.5 percentage points
Question outcomes2 improved · 14 unchanged · 0 worse
Mean added task time+46.2s
Mean task time, off → on17.6s → 63.7s
Median task time, off → on9.2s → 36.7s
Median of paired time changes+17.8s
Total added task time+738.6s
Ratio of summed times, on / off3.6×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking on · temperature 0.6

48/48 repeats scored · 47 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 15 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested on · verified
  • Seed 303 · requested on · verified
  • Seed 101 · requested on · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

16 full credit

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-b005f155de21ad1ae2887258
Matching candidate set: paired-b005f155de21ad1ae2887258

Qwen3.5 9B UD-Q3_K_XL

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-291755b68033af9343a74237, fact-4afd936ddf9e3ca0a6f4a16c
Matching candidate set: fact-4afd936ddf9e3ca0a6f4a16c, fact-10a001757ce5e3a4ad0b8662, fact-6650b707149de7ed1c733b20, fact-470bad2baa7a5b9251437b89, fact-d2b59d44b662ae713aeac31a, fact-8911d11609395c5679b85df7, fact-f7b0f0121a8f96c9dfdb8666, fact-d0f093eba79679ef590e1554, fact-328e4a9121e69c97cd7319be, fact-e28b7d3cf40620276ac6c151, fact-d91472a9baf4c9f2b588d3e7, fact-4dd123d36ba8eb2eb1ba71b3, fact-81d2c9b418915e652817cddd, fact-cec7a0963364769cc87c775c, fact-503ad2bbd095726e4510f2a5, fact-c7309f96857ab4d57c685d63, fact-32c8729370f3fa2ea4742bf7, fact-97b75588d5514b8728fe4aa3, fact-300ea84f20ecc412e99cd12e, fact-167aad313fe901c497cd751b

Tested weights

Q4_0 · 5.06 GB

ibm-granite · recorded source

granite-4.2-8b-Q4_0.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.