AgentBench / results
← All models

Qwen3.5 9B

7 tested weight formats · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_0 · 5.38 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_0

Q4_0 · 5.38 GB of weights · rocm · context 32768 · GPU layers 99

Medium · 5–under 20B · dense · 9.0B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: verified · probe 1 · scope Saved profile-level probe; exact historical template/build scope is not fully recorded..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

77.9 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking onDetails
Everyday coding tasksTask pass rate
Pass rate
16/16 · 100%
Answers completed
16/16 · 100%
Correct / completed
16/16 · 100%
16 full credit
16/16 scored12.3s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
90.9%
Full-credit rate
10/11 · 91%
Answers completed
11/11 · 100%
Full credit / completed
10/11 · 91%
10 full credit1 incorrect
11/11 scored50.3s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
50.0%
Full-credit rate
17/34 · 50%
Answers completed
34/34 · 100%
Full credit / completed
17/34 · 50%
17 full credit1 partial credit16 incorrect
34/34 scored61.2s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
37.6%
Full-credit rate
4/11 · 36%
Answers completed
11/11 · 100%
Full credit / completed
4/11 · 36%
4 full credit2 partial credit5 incorrect
11/11 scored74.8s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
18/23 · 78%
Answers completed
23/23 · 100%
Correct / completed
18/23 · 78%
18 full credit5 incorrect
23/23 scored41.9s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
14/16 · 88%
Answers completed
16/16 · 100%
Correct / completed
14/16 · 88%
14 full credit2 incorrect
16/16 scored37.8s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
7/7 · 100%
Answers completed
7/7 · 100%
Correct / completed
7/7 · 100%
7 full credit
7/7 scored16.5s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
56.8%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit7 partial credit
8/8 scored41.6s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
12/12 · 100%
Answers completed
12/12 · 100%
Correct / completed
12/12 · 100%
12 full credit
12/12 scored34.3s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this model for small project edits with thinking off, or longer practical tasks with thinking on. It handled all the assessed small coding tests and most larger tasks in those modes, so it is more than a simple coding assistant. Difficult math and reasoning were weak, however, and strict formatting still needs checking.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Tasks timed16 tasks
Typical time per task (median)7.1 seconds
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking on
Average score90.9/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes10 full credit · 0 partial · 1 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking on
Average score37.6/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 2 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking off
Average score37.6/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 2 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking off
Tasks passed18 of 23 (78.3%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Thinking on
Tasks passed18 of 23 (78.3%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v6
Generation time
2026-10-12T05:53:56.011619Z
Review time
2026-10-22T07:26:27.217016+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-b1aa28e61ea9ce402d5617e636622c22565093f8849fd66537b5f4353d228521
Analysis input hash
9b8a811e7c90fc940fbc4a72a0a96ef9f59f13b42ef338ebe623013b9a1937cf
Original analysis input hash
73b9085d9c2702c6394660f8128cfb70ba5272684747e860c7f1a58668527f31
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-22c9684299ae41b208fb54b0b5c0e524da86038b791b37c29437b066a38b1411
Summary revision
summary-93636e37c8f85399ae95808c22f5175a5f20bd77d4b22a88502ab85c68f9efaf

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (30)
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    5ae751050bef975c356f2bbf362877982f6b0d15884e45c1c05d6db62fb2c064
    Summary revision
    summary-03e194b4f9f54becd2382587d833ce4f8a306e1006ba3b812a129a29b4e3a783
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    e4f895b24b6df45b0995cafba90bdbb9175b4df2494100a37a84c1567d0ce38e
    Summary revision
    summary-0b81ec097668808b3c01f0a3f8acbff6a153c474647dadce29ca2aae739587dd
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    1a29820a4d7d5193c4640ccc5771727ec07705efc27ce82cb8452b9e68a2c16b
    Summary revision
    summary-1350b8b88ea7eec8bd8beff43ac3a2e8547268f570aab2a60d89f00d16d6233b
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    ecae50ec4367ab6bdb3fc104bf93b0befd55797b0ea6054f05e24b75e460a945
    Summary revision
    summary-14f83f624e46fdba5664cf2a2400da5ccafd82e0c1d6bc3b2f605e4526e56080
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    9cda02812fbd33675f91a988dfd1ab1ee60e2be9c7096445cd28df4edaad1d73
    Summary revision
    summary-14f99ea4bf815a135db6cd44ce646177dbc728acc1398df057519399f06c953f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    06c81830a0b917d04b3709f803a019929f153298db05974225cdd7a1f7721f25
    Summary revision
    summary-1b114437948ba5cd86811e69f08c5354a230670a79dfac23c78e8e629f61ceff
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    5f2045dcdac37302bd055b2ed4b66125db15872b7dc2bad615f08b93f76df15c
    Summary revision
    summary-1bfb5c85fd75f50d1258dad69b88f6131ab46b1c09779b7b15e4a5fc6f6bd7d3
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    2626f52f0a37f614f1e6bb2a49bb33364c88d3e0d29a2dc7638c6bdc079777c0
    Summary revision
    summary-2628a5652dc2c093723f31ccd526c8823c3def655dfb91de966e05aeb0204415
    Input summary revisions
    summary-0b81ec097668808b3c01f0a3f8acbff6a153c474647dadce29ca2aae739587dd, summary-14f99ea4bf815a135db6cd44ce646177dbc728acc1398df057519399f06c953f, summary-448d215ee49e71384137232ac04f5f9f86c7003a9013d81300696f4a43706178, summary-837904765ab2d44f4888e70d9b25d34c96f86fe6e5e6773fcd4255daf83eb71a, summary-886aaaf3709cb5af9f8ff4e2553a756b0c0baef162e203f42b085ff841175a70, summary-8b757249c5019917cd50144fff14025f1e5ef735af2e1b291b3e8e368ae34e67, summary-92486cedeb2f090be5c59a96babf9ff088b71aa4620db25747cf1e3f79c132b4, summary-ab5be6eaad3282d89d2398880938bd1d08fb06229ce4110638a14cd79e6653c4, summary-f5a63bf89bfe5bf601170588ca39fc3b03bee7ae64ab96605003f29789d13f56
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    32fef16e940e62b2220580e8ac8dffb805a7af86db1bd9ca0caa40f12feff4e3
    Summary revision
    summary-27999d60ecb33b6144da2ed3ba5bedb2b85f01560548892a621e6e777a1af52a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    370a4a59d7011ff372c4d15f06d569b951c4e4e59c3ce4e545b6069108cfe9b4
    Summary revision
    summary-448d215ee49e71384137232ac04f5f9f86c7003a9013d81300696f4a43706178
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    be4bc23798d825f9b2b7064189df9481c9911e19b257039b10b68192c1460580
    Summary revision
    summary-4785cb7768e770a9dffe38b9b994e80b8725de70954c1799829faaa8e1d3909f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    937a748529172a1674ffbc45f0e7a7bf12a522590bb329151c84efaf1fbaa18a
    Summary revision
    summary-4cd204fde9a43b0ceba63501b2bad0800186635962ce5103283609795b65f7e9
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    ed8e48cda91452edf18c2ce1a68357f420b8f60fd7ae382fa81a19762121c3b1
    Summary revision
    summary-837904765ab2d44f4888e70d9b25d34c96f86fe6e5e6773fcd4255daf83eb71a
    Input summary revisions
    summary-03e194b4f9f54becd2382587d833ce4f8a306e1006ba3b812a129a29b4e3a783, summary-1350b8b88ea7eec8bd8beff43ac3a2e8547268f570aab2a60d89f00d16d6233b, summary-14f83f624e46fdba5664cf2a2400da5ccafd82e0c1d6bc3b2f605e4526e56080, summary-1b114437948ba5cd86811e69f08c5354a230670a79dfac23c78e8e629f61ceff, summary-1bfb5c85fd75f50d1258dad69b88f6131ab46b1c09779b7b15e4a5fc6f6bd7d3, summary-27999d60ecb33b6144da2ed3ba5bedb2b85f01560548892a621e6e777a1af52a, summary-4785cb7768e770a9dffe38b9b994e80b8725de70954c1799829faaa8e1d3909f, summary-4cd204fde9a43b0ceba63501b2bad0800186635962ce5103283609795b65f7e9, summary-9351c8765d68cadb465025388906daea10cb920fed60cbcdda7275d52b69f780, summary-a6466767c78dcbea8fc4c03e3bace55dd277b1777562682c49a27f83a268d221, summary-ab4514835c120c69c71016a30495a40b83eb8f8d1bcccc49c9ef541596dd2c3c, summary-b1ea9a69aead57f3ccb2a1652b7a26eca3a9cb892631ceb6050489d8886e17ec, summary-b84f68c83eb3e83f9dd749b454559d96ee3bbb0b86fcd3006d4ae5008e053644, summary-ca64f2ba15373b8e3b7d9eb10ec1eb644591ee0d24e96cec2343a85d5c32e6b5, summary-d35ee914de661600d1baebe08181efe1802828d24207dbc4cf4b295bc0708d0a, summary-da6a45bb0ce03c0b22ec566ba1beae7a302ec5fbf4bd0c99144ccd74ed012502, summary-dc94c426f653f080fee25fbc5e1f380cf64e731fbc4aa11916710d502a0148eb, summary-edfdf2f8e400f24ce57ad75dbe9d3d525108eac328c4efdacecf6b84c72e0afb, summary-f69174cc6af2f6693752effe15a27baa77377765be180bebb062c6a8a45bba62, summary-f9ffea39f5b4ef2676888252cc4e00cbe4140d4e8d40f8c889a0aeefcb54bb30
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    9e41dc44845d0e682c63e71e283766675b9237b2439a1bca8ffdffa7622454d7
    Summary revision
    summary-886aaaf3709cb5af9f8ff4e2553a756b0c0baef162e203f42b085ff841175a70
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    efea34b869de480cf1b086b92815f3eabedfb7e93cd36c2d29ca86b1bb9f5fce
    Summary revision
    summary-8b757249c5019917cd50144fff14025f1e5ef735af2e1b291b3e8e368ae34e67
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    372ce4b247c26a501aefd875234d01363dbb9999b2cb74c5265a5afd937281f7
    Summary revision
    summary-92486cedeb2f090be5c59a96babf9ff088b71aa4620db25747cf1e3f79c132b4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    ae444936df15f94af6c5159f656ede57e5ef5f334a18ef42d6fe9d226fcfe82a
    Summary revision
    summary-9351c8765d68cadb465025388906daea10cb920fed60cbcdda7275d52b69f780
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    1eaf7e6a7995c987eb92cf8ba6d1a7ae513704f2a773f8f5eb4720948cbc82c7
    Summary revision
    summary-a6466767c78dcbea8fc4c03e3bace55dd277b1777562682c49a27f83a268d221
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    40e1f3112237ee10b9939572b144b94789fd33c13c81331cba9ef6be1bdf35d6
    Summary revision
    summary-ab4514835c120c69c71016a30495a40b83eb8f8d1bcccc49c9ef541596dd2c3c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    1a2d73bdbb9079b34c8d93133acee82660a22399bd4da988f00a67fb4cd67007
    Summary revision
    summary-ab5be6eaad3282d89d2398880938bd1d08fb06229ce4110638a14cd79e6653c4
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    54ae5c96f0da4452d73bc2a531275d01cca434d312e975999f5a10e7725f6e26
    Summary revision
    summary-b1ea9a69aead57f3ccb2a1652b7a26eca3a9cb892631ceb6050489d8886e17ec
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    b315b827da4fef6d820271bc3a823e4482d0fcfba930f25bfab83470a19ce3c4
    Summary revision
    summary-b84f68c83eb3e83f9dd749b454559d96ee3bbb0b86fcd3006d4ae5008e053644
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    55a71fac0f81ff5084d688704e83b35ee16d0c2ced1eaf3b7460151a85a8624a
    Summary revision
    summary-ca64f2ba15373b8e3b7d9eb10ec1eb644591ee0d24e96cec2343a85d5c32e6b5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    example_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    c8d9df3ffac604b7b06aea42ec170bb9fc79192e34f29c499d5acc6793d36488
    Summary revision
    summary-d35ee914de661600d1baebe08181efe1802828d24207dbc4cf4b295bc0708d0a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    ce0504c8a12e1b06110e2addd4d1594586c991a16585c57304de0fa1947681dd
    Summary revision
    summary-da6a45bb0ce03c0b22ec566ba1beae7a302ec5fbf4bd0c99144ccd74ed012502
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    7386d4072f27d2626bea3cc8dfa254a05ffbb23cbaeb8a1102caa4cd4accf927
    Summary revision
    summary-dc94c426f653f080fee25fbc5e1f380cf64e731fbc4aa11916710d502a0148eb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    40c080ac75367ef07058500a28897a91ed8db4fff434194a70a30bd640aa02e9
    Summary revision
    summary-edfdf2f8e400f24ce57ad75dbe9d3d525108eac328c4efdacecf6b84c72e0afb
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    62784d982d89e57cb2ba2f26473ab243657d57b9295a024f738ca27441ec5d27
    Summary revision
    summary-f5a63bf89bfe5bf601170588ca39fc3b03bee7ae64ab96605003f29789d13f56
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    f299f028e93737725a5b4b933c7793147f5ef7a72dc767141f9b6a362bb9c9e3
    Summary revision
    summary-f69174cc6af2f6693752effe15a27baa77377765be180bebb062c6a8a45bba62
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    evidence_segment
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v4
    Analysis input hash
    1913cb3b5dd009b7ea2c668680853a5a44dee4fc4ec0982efbea6360edba97f7
    Summary revision
    summary-f9ffea39f5b4ef2676888252cc4e00cbe4140d4e8d40f8c889a0aeefcb54bb30
    Input summary revisions
    Direct evidence analysis

Good at

Handles small project edits and bug fixes well: all small coding tasks passed with thinking off. That supports everyday assistance, not a promise that arbitrary projects will work.

Evidence for this judgment
Everyday coding tasks · Thinking off
Tasks passed16 of 16 (100.0%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Can carry out longer tasks with tools: with thinking on, most larger practical tasks passed completely. It also passed every source-backed challenge test, retaining correct answers while accepting genuine source updates.

Evidence for this judgment
Larger practical tasks · Thinking on
Average score90.9/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes10 full credit · 0 partial · 1 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking on
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Difficult math and reasoning are a poor fit: average answer scores were low with thinking both on and off. Successful coding results should not be taken as evidence of strong advanced problem-solving.

Evidence for this judgment
Published hard problems · Thinking on
Average score37.6/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 2 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking off
Average score37.6/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes4 full credit · 2 partial · 5 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Can get stuck repeating tool calls rather than changing approach. In the examples checked, duplicate-call warnings did not always stop repetition—even during a larger task that ultimately succeeded.

Evidence for this judgment

Example 1: Larger practical tasks
Thinking off. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 2: Larger practical tasks
Thinking on. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Source limitation
Qualitative examples are a deterministic sample; they do not establish whole-bank behavior.

Should you enable thinking?

Leave thinking off for everyday coding: paired tests showed no accuracy gain but longer task times. It also reduced puzzle scores while taking longer, so extra deliberation is not a general upgrade.

Evidence for this judgment
Everyday coding tasks · Thinking off versus on
Thinking off → on100.0% · Task pass rate → 100.0% · Task pass rate
Paired test coverage16 of 16 tasks compared in both modes
Score change with thinking onNo change
Average time change with thinking on7.3 seconds slower
Tasks timed in both modes16 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking off versus on
Thinking off → on67.6/100 · Mean puzzle credit → 50.0/100 · Mean puzzle credit
Paired test coverage34 of 34 tasks compared in both modes
Score change with thinking on-17.6 score points
Average time change with thinking on13.1 seconds slower
Tasks timed in both modes34 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Enable thinking for longer tool-driven work and for keeping source-backed answers correct under challenge. Both improved substantially in paired tests, making the longer task times worthwhile for those uses.

Evidence for this judgment
Larger practical tasks · Thinking off versus on
Thinking off → on45.5/100 · Mean task credit → 90.9/100 · Mean task credit
Paired test coverage11 of 11 tasks compared in both modes
Score change with thinking on+45.5 score points
Average time change with thinking on27.4 seconds slower
Tasks timed in both modes11 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking off versus on
Thinking off → on58.3% · All-branches pass rate → 100.0% · All-branches pass rate
Paired test coverage12 of 12 tasks compared in both modes
Score change with thinking on+41.7 score points
Average time change with thinking on34.8 seconds slower
Tasks timed in both modes12 pairs
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

With thinking off, small coding produced the best pass result in this three-configuration comparison, though both alternatives finished sooner; Qwen3.5 4B was closer in score, while Gemma 4 E4B scored much lower.

Evidence for this judgment

Speed: Everyday coding tasks
Thinking off. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Speed and task success: Everyday coding tasksThinking off · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
Qwen3.5 4B4.8 seconds16 of 16 tasks timed72.6 tokens/s15 of 1616 of 16 scored
Gemma 4 E4B5.4 seconds16 of 16 tasks timed68.2 tokens/s11 of 1616 of 16 scored
Qwen3.5 9BThis model7.1 seconds16 of 16 tasks timed52.4 tokens/s16 of 1616 of 16 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

With thinking off, its puzzle credit led both matched alternatives. Task time fell between them: slower than the closer-scoring Qwen3.5 4B and faster than the lower-scoring Gemma 4 E4B.

Evidence for this judgment

Speed: Logic and number puzzles
Thinking off. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Speed and task success: Logic and number puzzlesThinking off · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedAverage score
Qwen3.5 4B34.4 seconds34 of 34 tasks timed76.7 tokens/s58.9/10034 of 34 scored
Qwen3.5 9BThis model39.5 seconds34 of 34 tasks timed55.1 tokens/s67.6/10034 of 34 scored
Gemma 4 E4B58.3 seconds34 of 34 tasks timed71.8 tokens/s14.9/10034 of 34 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-12 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Worth knowing

Publication redactions, withheld artifacts and source-link-only paper exports limit readers’ inspection of some examples; they do not establish that the model lacked those inputs.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

Evidence for this judgment

Source limitation
One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

How to interpret these assessments and their limitations →

How do the weight formats compare?

Everyday coding tasks · Task pass rate · requested / displayed reasoning on. Size is the weight download, not RAM. Different runtimes and placement can also affect results.

UD-Q2_K_XL · 4.12 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Recorded · some results unscored or qualifiedThinking on14/16 passed16/16 scored
UD-Q3_K_XL · 5.05 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on16/16 passed16/16 scored
Q4_0 · 5.38 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on16/16 passed16/16 scored
UD-Q4_K_XL · 5.97 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on16/16 passed16/16 scored
UD-Q5_K_XL · 6.74 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on15/16 passed16/16 scored
Q6_K · 7.46 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on16/16 passed16/16 scored
UD-Q6_K_XL · 8.76 GBLocal · test machine with 32 GB RAM · rocm · 99 GPU layers · Fully scoredThinking on15/16 passed16/16 scored

Qualified rows show the reason they are outside the rankings; they cannot establish superiority over a fully tested format. Open the setup to inspect context, runtime and controls.

Choosing a quant

Shared assessment of Q6_K, UD-Q2_K_XL, UD-Q3_K_XL, UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL · 2026-10-22 UTC. The current results table can include newer formats.

The selected weight format was not included in this written comparison. Inspect its recorded results separately.

UD-Q4_K_XL is a sensible starting point for small coding tasks. With reasoning requested off, it earned full credit alongside Q6_K, UD-Q5_K_XL and UD-Q6_K_XL, with a smaller download. UD-Q3_K_XL has a smaller download still and finished close behind requested-off; requested-on, it earned full credit alongside UD-Q4_K_XL and Q6_K. These narrow differences provide little reason to favor the largest download for this task bank.

Evidence for this judgment

Everyday coding tasks · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatTask pass rateTask timeDownloadInspect
Q6_Kunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored7.6s16/16 timed7.46 GBWeights, not required RAMAnswers and code
UD-Q2_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
15/16 passed16/16 scored10.0s16/16 timed4.12 GBWeights, not required RAMAnswers and code
UD-Q3_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
15/16 passed16/16 scored8.7s16/16 timed5.05 GBWeights, not required RAMAnswers and code
UD-Q4_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored7.9s16/16 timed5.97 GBWeights, not required RAMAnswers and code
UD-Q5_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored8.4s16/16 timed6.74 GBWeights, not required RAMAnswers and code
UD-Q6_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored7.8s16/16 timed8.76 GBWeights, not required RAMAnswers and code

Everyday coding tasks · Thinking requested on

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatTask pass rateTask timeDownloadInspect
Q6_Kunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored13.0s16/16 timed7.46 GBWeights, not required RAMAnswers and code
UD-Q2_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
14/16 passed16/16 scored17.0s16/16 timed4.12 GBWeights, not required RAMAnswers and code
UD-Q3_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored13.3s16/16 timed5.05 GBWeights, not required RAMAnswers and code
UD-Q4_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
16/16 passed16/16 scored12.3s16/16 timed5.97 GBWeights, not required RAMAnswers and code
UD-Q5_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
15/16 passed16/16 scored13.0s16/16 timed6.74 GBWeights, not required RAMAnswers and code
UD-Q6_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
15/16 passed16/16 scored16.8s16/16 timed8.76 GBWeights, not required RAMAnswers and code

For requested-off larger tool-driven tasks, UD-Q4_K_XL is a stronger choice than UD-Q3_K_XL: its task-credit lead was substantial here. Its lead over Q6_K was only a single fully credited task, but its download is smaller.

Evidence for this judgment

Larger practical tasks · Thinking requested off

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean task creditTask timeDownloadInspect
Q6_Kunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored69.8s11/11 timed7.46 GBWeights, not required RAMAnswers and code
UD-Q2_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
36.4/10011/11 scored64.1s11/11 timed4.12 GBWeights, not required RAMAnswers and code
UD-Q3_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
36.4/10011/11 scored33.3s11/11 timed5.05 GBWeights, not required RAMAnswers and code
UD-Q4_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
72.7/10011/11 scored53.3s11/11 timed5.97 GBWeights, not required RAMAnswers and code
UD-Q5_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
54.5/10011/11 scored49.7s11/11 timed6.74 GBWeights, not required RAMAnswers and code
UD-Q6_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored36.6s11/11 timed8.76 GBWeights, not required RAMAnswers and code

For requested-on larger tasks, UD-Q6_K_XL led UD-Q3_K_XL and Q6_K by only a single fully credited task. UD-Q3_K_XL is a useful download-size tradeoff: it tied Q6_K on measured task credit with a smaller download and shorter median task time.

Evidence for this judgment

Larger practical tasks · Thinking requested on

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatMean task creditTask timeDownloadInspect
Q6_Kunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
72.7/10011/11 scored57.0s11/11 timed7.46 GBWeights, not required RAMAnswers and code
UD-Q2_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
36.4/10011/11 scored66.3s11/11 timed4.12 GBWeights, not required RAMAnswers and code
UD-Q3_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
72.7/10011/11 scored49.3s11/11 timed5.05 GBWeights, not required RAMAnswers and code
UD-Q4_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored61.2s11/11 timed5.97 GBWeights, not required RAMAnswers and code
UD-Q5_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
63.6/10011/11 scored42.4s11/11 timed6.74 GBWeights, not required RAMAnswers and code
UD-Q6_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
81.8/10011/11 scored53.2s11/11 timed8.76 GBWeights, not required RAMAnswers and code

For requested-on research-paper questions, consider UD-Q5_K_XL: it earned higher answer credit than Q6_K and UD-Q6_K_XL while requiring a smaller download. This supports a task-specific choice, not a universal ranking.

Evidence for this judgment

Reading research papers · Thinking requested on

Matching task controls and 16,384 context. These quality results describe the recorded setups; they do not isolate the effect of bit depth.

Weight formatAnswer token F1Task timeDownloadInspect
Q6_Kunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
46.1/1008/8 scored57.8s8/8 timed7.46 GBWeights, not required RAMAnswers and code
UD-Q2_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
56.3/1008/8 scored53.2s8/8 timed4.12 GBWeights, not required RAMAnswers and code
UD-Q3_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
44.6/1008/8 scored47.9s8/8 timed5.05 GBWeights, not required RAMAnswers and code
UD-Q4_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
41.0/1008/8 scored51.2s8/8 timed5.97 GBWeights, not required RAMAnswers and code
UD-Q5_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
58.8/1008/8 scored54.9s8/8 timed6.74 GBWeights, not required RAMAnswers and code
UD-Q6_K_XLunslothrocm · 32 GB test machineCPU MoE layers: Not recorded
Recorded execution settings
Context
16,384
GPU layer setting
99
Threads
6
Concurrency
1
Runtime build
Not recorded
Tool adapter
saved-campaign-agent-tools
48.1/1008/8 scored63.5s8/8 timed8.76 GBWeights, not required RAMAnswers and code
Scope and limits
  • These are comparisons of tested setups registered to one checkpoint. They do not isolate bit depth from conversion, publisher, runtime-build or template differences.
  • Only complete eligible banks with matching task controls and recorded context enter the quality comparison. Execution profiles are shown per setup; timing is compared only when every recorded execution profile matches and timing coverage is complete. Other results remain coverage, not ranked evidence.
  • Download size is not RAM or VRAM requirement. No memory-fit recommendation is established by this comparison.
  • Small score differences can reflect a single question. Similar results are not proof of equivalence; timings describe this benchmark machine.
About this quant assessment
Requested model
gpt-6-astra
Returned model
Not supplied; request identity only.
Exact provider snapshot
Not supplied.
Generation route
Codex CLI · ChatGPT account
Prompt version
quant-summary-v3
Review time
2026-10-22T06:58:37.216745+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Source edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Revision
quant-summary-320a2b31bea841bea7cef03026053dc9d52b08847aa079d7e75718be421d3aec

What changed in these results

One same-budget retry after context exhaustion. A second context limit is failed to answer (zero task-completion credit), not an incorrect submitted answer. Answer-only averages exclude these no-answer outcomes; completed retry answers enter the normal scoring path. This follow-up is selected after observing failures.

Thinking tradeoff

16/16 matched scored questions · 16/16 matched timed questions

Both modes earned full credit on every matched question. These matched results leave no measured quality headroom; unmatched questions are not covered by this observation.

Thinking offMean matched task credit100.0%
Thinking onMean matched task credit100.0%

Thinking on took 7.3 seconds more per matched timed question, on average.

All matched quality and timing measurements
Matched measurementRecorded result
Average credit, off → on100.0% → 100.0%
Mean paired credit change0.0 percentage points
Question outcomes0 improved · 16 unchanged · 0 worse
Mean added task time+7.3s
Mean task time, off → on7.1s → 14.4s
Median task time, off → on7.1s → 12.3s
Median of paired time changes+5.1s
Total added task time+117.3s
Ratio of summed times, on / off2.0×

Quality and timing can cover different subsets. Median paired change is calculated per question; it is not the difference between the two medians. Task time includes tools and grading.

Inspect the matched modes and their answers →
Inspect individual matched questions

Weight formats compared

All tested weight formats of this checkpoint, using the selected thinking setting. This shared comparison updates with new results; each saved model summary keeps its original date.

Everyday coding tasks

Scores and times describe this task bank. Quick failures are not a speed advantage. Different execution settings are labeled.

VersionTask resultTypical task timeWeight downloadExplore
Qwen3.5 9B Q6_KThinking on · Different or uncertain execution settings100.0%16/16 scored13.0 seconds16/16 timed7.46 GBAnswers and code

Compare

Qwen3.5 9B UD-Q2_K_XLThinking on · Different or uncertain execution settings87.5%16/16 scored17.0 seconds16/16 timed4.12 GBAnswers and code

Compare

Qwen3.5 9B UD-Q3_K_XLThinking on · Different or uncertain execution settings100.0%16/16 scored13.3 seconds16/16 timed5.05 GBAnswers and code

Compare

Qwen3.5 9B UD-Q4_K_XLThinking on · Different or uncertain execution settings100.0%16/16 scored12.3 seconds16/16 timed5.97 GBAnswers and code

Compare

Qwen3.5 9B UD-Q5_K_XLThinking on · Different or uncertain execution settings93.8%16/16 scored13.0 seconds16/16 timed6.74 GBAnswers and code

Compare

Qwen3.5 9B UD-Q6_K_XLThinking on · Different or uncertain execution settings93.8%16/16 scored16.8 seconds16/16 timed8.76 GBAnswers and code

Compare

Qwen3.5 9BThinking on100.0%16/16 scored12.3 seconds16/16 timed5.38 GBAnswers and code

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking on · temperature 0.6

48/48 repeats scored · 48 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 16 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 101 · requested on · verified
  • Seed 202 · requested on · verified
  • Seed 303 · requested on · verified

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

16 full credit

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Thinking on versus off

Verified requested on/off conditions of the same configuration, matched on the recorded test protocol.

Requested thinking off · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template verified-thinking-pair
Facts: paired-7eebd4b946e9473551ead051
Matching candidate set: paired-7eebd4b946e9473551ead051

Qwen3.5 9B UD-Q3_K_XL

Another tested weight format of this checkpoint. Complete results first, then nearest download size.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-checkpoint-matching-controls
Facts: fact-470bad2baa7a5b9251437b89, fact-4afd936ddf9e3ca0a6f4a16c
Matching candidate set: fact-4afd936ddf9e3ca0a6f4a16c, fact-f7b0f0121a8f96c9dfdb8666, fact-d0f093eba79679ef590e1554, fact-e28b7d3cf40620276ac6c151, fact-4dd123d36ba8eb2eb1ba71b3, fact-503ad2bbd095726e4510f2a5

Nemotron Nano 9B v2 Q4_0

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-470bad2baa7a5b9251437b89, fact-6650b707149de7ed1c733b20
Matching candidate set: fact-6650b707149de7ed1c733b20, fact-10a001757ce5e3a4ad0b8662, fact-291755b68033af9343a74237, fact-8911d11609395c5679b85df7, fact-d2b59d44b662ae713aeac31a, fact-328e4a9121e69c97cd7319be, fact-d91472a9baf4c9f2b588d3e7, fact-81d2c9b418915e652817cddd, fact-cec7a0963364769cc87c775c, fact-c7309f96857ab4d57c685d63, fact-32c8729370f3fa2ea4742bf7, fact-97b75588d5514b8728fe4aa3, fact-300ea84f20ecc412e99cd12e, fact-167aad313fe901c497cd751b

Tested weights

Q4_0 · 5.38 GB

unsloth · recorded source

Qwen3.5-9B-Q4_0.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

UD-Q2_K_XL · 4.12 GB

unsloth · recorded source

Qwen3.5-9B-UD-Q2_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

UD-Q3_K_XL · 5.05 GB

unsloth · recorded source

Qwen3.5-9B-UD-Q3_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

UD-Q4_K_XL · 5.97 GB

unsloth · recorded source

Qwen3.5-9B-UD-Q4_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

UD-Q5_K_XL · 6.74 GB

unsloth · recorded source

Qwen3.5-9B-UD-Q5_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Q6_K · 7.46 GB

unsloth · recorded source

Qwen3.5-9B-Q6_K.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

UD-Q6_K_XL · 8.76 GB

unsloth · recorded source

Qwen3.5-9B-UD-Q6_K_XL.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.