AgentBench / results
← All models

Falcon-H1R 7B Q4_K_M

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_K_M · 4.6 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_K_M

Q4_K_M · 4.6 GB of weights · vulkan · context 16384 · GPU layers 99

Medium · 5–under 20B · dense · 7.6B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: ignored switch · probe 2 · scope Observed reasoning despite an off request; toggle is not verified..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

53.0 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteReasoning enabled (observed; requested off)Details
Everyday coding tasksTask pass rate
Pass rate
9/16 · 56%
Answers completed
16/16 · 100%
Correct / completed
9/16 · 56%
9 full credit7 incorrect
16/16 scored92.8s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
27.3%
Full-credit rate
3/11 · 27%
Answers completed
10/11 · 91%
Full credit / completed
3/10 · 30%
3 full credit7 incorrect1 failed to finish
11/11 scored111.8s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
35.4%
Full-credit rate
12/34 · 35%
Answers completed
16/34 · 47%
Full credit / completed
12/16 · 75%
12 full credit4 partial credit18 failed to finish
34/34 scored83.3s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
27.3%
Full-credit rate
3/11 · 27%
Answers completed
5/11 · 45%
Full credit / completed
3/5 · 60%
3 full credit2 incorrect6 failed to finish
11/11 scored79.1s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
6/23 · 26%
Answers completed
7/23 · 30%
Correct / completed
6/7 · 86%
6 full credit1 incorrect16 failed to finish
23/23 scored78.7s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
11/16 · 69%
Answers completed
16/16 · 100%
Correct / completed
11/16 · 69%
11 full credit5 incorrect
16/16 scored7.4s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
7/7 · 100%
Answers completed
7/7 · 100%
Correct / completed
7/7 · 100%
7 full credit
7/7 scored13.3s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
35.7%
Full-credit rate
0/8 · 0%
Answers completed
7/8 · 88%
Full credit / completed
0/7 · 0%
7 partial credit1 failed to finish
8/8 scored26.5s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
12/12 · 100%
Answers completed
12/12 · 100%
Correct / completed
12/12 · 100%
12 full credit
12/12 scored13.1s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this tested configuration for focused passage calculations and source-backed answer updates—not as an unattended project assistant. It handled those focused tests well, but small coding edits were uneven, while larger practical tasks and strict output requirements exposed clear weaknesses. Passage calculations combined strong accuracy with a median task time of 13.3 seconds on this test machine. These are reasoning-enabled results despite an off request; thinking’s added benefit could not be established.

Evidence for this judgment
Resisting incorrect corrections · Reasoning enabled (observed; requested off)
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage · Reasoning enabled (observed; requested off)
Tasks passed7 of 7 (100.0%)
Test coverage7 of 7 tasks scored
Typical time per task (median)13.3 seconds
Tasks timed7 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Reasoning enabled (observed; requested off)
Tasks passed9 of 16 (56.3%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Reasoning enabled (observed; requested off)
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Reasoning enabled (observed; requested off)
Tasks passed6 of 23 (26.1%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Source limitation
Reasoning was observed despite requesting it off. These are reasoning-enabled observations, not verified thinking-off runs or a matched on/off experiment. The unexecuted duplicate toggle is omitted from presentation totals.

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v7
Generation time
2026-10-14T01:15:25.841047Z
Review time
2026-10-22T05:56:51.035394+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-a241e6726b39b4e5b1cf99f9801c22ddffdd62a6885206d83da8ab42772579ef
Analysis input hash
a2e15a2677b16518ffa91a2d05a8d3d62a9cd679e69f98bde5e9dfee8b30e16e
Original analysis input hash
20fc540c8f4bc2dcf0b700679b2a5767bcc164aaed61dc443a9677cd9a4fab3c
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-466210158c03c9535d093d979d1c3b07ad7366aafd64af42a751a04e73710fcd
Summary revision
summary-5e61d90d8ade69921b2bbdf60aacd9fec531bfc04806ad9f6608f31c81652daf

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (10)
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    c6dec759444407e371e480b4b3a2ed6f1ee72ab04ac4214db06d837fbdeb090c
    Summary revision
    summary-0a167c16a353a8648a3a1e7acdbc101909e74eef851b8c5a78546d699d03a25a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    bafd31a33d4ce3d7ac78cb2b3649c0c6c7e548c871292d67544f8a69e4ede867
    Summary revision
    summary-11c55b0fdfb57c6c92ccc14f0943aa4581702c919ef5df40cbf7c7b119869798
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8ac83ac30097f5c2f3caf4a6fbbb49301288af082a985e103226572379cd19bd
    Summary revision
    summary-2d883d1386ee6ccf39702a75f0ce3ad8147eea84c9976ca6fbecf92b25493360
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d3c5f4049d4bcc8b1a18a9c0f4961c3576a8b32b446173044bf2c7436ec4b01d
    Summary revision
    summary-370c2abea1b97b952131a635c1e8136cb6c6a5ab4810405d958c186ec7aab4fe
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f6e238c8d9773e0088c7597ed9b194f938ac255dc7b0d7cd6f1f236406448f7e
    Summary revision
    summary-45b1c9e07aac63e2e2b935e087aa6a93c854d18d9610e33656f57852ff66233d
    Input summary revisions
    summary-0a167c16a353a8648a3a1e7acdbc101909e74eef851b8c5a78546d699d03a25a, summary-11c55b0fdfb57c6c92ccc14f0943aa4581702c919ef5df40cbf7c7b119869798, summary-2d883d1386ee6ccf39702a75f0ce3ad8147eea84c9976ca6fbecf92b25493360, summary-370c2abea1b97b952131a635c1e8136cb6c6a5ab4810405d958c186ec7aab4fe, summary-52f6bf407bfea13244988a22626342a21bd828d0514658befcdbe47477be07e6, summary-8e1984f1fa1519d36cf5470e086769d6bbe1f7c67047ddf5ad5bf2f266dab604, summary-ab6190e17a9852f0bb36b91c7a9c5a583ecf6fd8a528a28bbd274a20cfa2240c, summary-b1e98d130f3b9a9ab95a2b638dcbc9019ffd56fc1b22d9967d6cb2c99f97638f, summary-c8342ae7849ff99f5091782f207bd59f006b7426f1d53908af2ef6c24cae0702
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f3c3b118c995a79d3561c3d252bbf092e65224a30c05b2b91a1c32472a329b55
    Summary revision
    summary-52f6bf407bfea13244988a22626342a21bd828d0514658befcdbe47477be07e6
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    8e3a52a3db92ecaa1c7115a7679d8bbbf2d39032ba4095957507d8e2d7375900
    Summary revision
    summary-8e1984f1fa1519d36cf5470e086769d6bbe1f7c67047ddf5ad5bf2f266dab604
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    dab112a8f8b81ed1782907f889208814e59f01893ce8cd24d6bf75faec45baa1
    Summary revision
    summary-ab6190e17a9852f0bb36b91c7a9c5a583ecf6fd8a528a28bbd274a20cfa2240c
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b34b58eead9714b5ba45ee654fb595c25762af2ab88d306b21476ef5bee9f95a
    Summary revision
    summary-b1e98d130f3b9a9ab95a2b638dcbc9019ffd56fc1b22d9967d6cb2c99f97638f
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    9bf297ee595eec6ab03f52b34b7e44d6b90499b71f49235dd7bc859010149c2c
    Summary revision
    summary-c8342ae7849ff99f5091782f207bd59f006b7426f1d53908af2ef6c24cae0702
    Input summary revisions
    Direct evidence analysis

Good at

Keeps source-backed answers steady under unsupported challenges and updates them when the source changes. It passed every case in this small, controlled test—not proof of general resistance to persuasion.

Evidence for this judgment
Resisting incorrect corrections · Reasoning enabled (observed; requested off)
Tasks passed12 of 12 (100.0%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Calculating from a passage is a useful niche: all included answers were exactly correct, with short total task times in these runs.

Evidence for this judgment
Calculating from a passage · Reasoning enabled (observed; requested off)
Tasks passed7 of 7 (100.0%)
Test coverage7 of 7 tasks scored
Typical time per task (median)13.3 seconds
Tasks timed7 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Do not rely on it for unattended projects or rigid output requirements: larger practical tasks achieved low average scores, and most strict wording, language and formatting tests failed.

Evidence for this judgment
Larger practical tasks · Reasoning enabled (observed; requested off)
Average score27.3/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions · Reasoning enabled (observed; requested off)
Tasks passed6 of 23 (26.1%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Hard reasoning and research reading are weak choices. Both difficult math/reasoning questions and generated puzzles had low average scores; research-paper answers also matched reference answers poorly.

Evidence for this judgment
Published hard problems · Reasoning enabled (observed; requested off)
Average score27.3/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes3 full credit · 0 partial · 8 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Reasoning enabled (observed; requested off)
Average score35.4/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes12 full credit · 4 partial · 18 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Reasoning enabled (observed; requested off)
Average score35.7/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes0 full credit · 7 partial · 1 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Check even small project edits: baseline results were uneven, and success varied across repeated runs. A successful attempt is not enough to establish dependable coding assistance.

Evidence for this judgment
Everyday coding tasks · Reasoning enabled (observed; requested off)
Tasks passed9 of 16 (56.3%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking on · sampled repetitions
Questions successful every repetition7 questions
Distinct questions planned16 questions
Questions with all repetitions scored16 questions
Repetitions scored48 repetitions
Repetitions planned48 repetitions
Repetitions earning full credit31 repetitions
Explore these test results →

View this model’s answers, code and tool activity →

Worth knowing

Reasoning remained active despite requests to turn it off. These results do not establish a working toggle or an on/off tradeoff; the unexecuted duplicate condition is excluded.

Evidence for this judgment

Source limitation
Some requested reasoning conditions lack verified control.

Source limitation
Reasoning was observed despite requesting it off. These are reasoning-enabled observations, not verified thinking-off runs or a matched on/off experiment. The unexecuted duplicate toggle is omitted from presentation totals.

Publication redactions and withheld artifacts limit readers’ inspection of some examples.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

What changed in these results

Reasoning was observed despite requesting it off. These are reasoning-enabled observations, not verified thinking-off runs or a matched on/off experiment. The unexecuted duplicate toggle is omitted from presentation totals.

Thinking tradeoff

No verified on/off comparison is available for this configuration and test suite. Requested modes alone do not establish a controlled thinking comparison.

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Reasoning enabled (observed; requested off) · temperature 0.6

48/48 repeats scored · 31 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 7 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested off · ignored switch
  • Seed 303 · requested off · ignored switch
  • Seed 101 · requested off · ignored switch

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

9 full credit7 incorrect

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

No matching suggestion is available for this selection.

Tested weights

Q4_K_M · 4.6 GB

tiiuae · recorded source

Falcon-H1R-7B-Q4_K_M.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.