AgentBench / results
← All models

LFM2.5 1.2B Thinking Q4_K_M

1 tested weight format · Explore results, then inspect the answers.

Choose a tested configuration

Get this tested model

Q4_K_M · 0.73 GB download

Links follow the selected quant and configuration.

Setup, memory & provenance · Q4_K_M

Q4_K_M · 0.73 GB of weights · vulkan · context 16384 · GPU layers 99

Small · under 5B · dense · 1.2B total

Memory fit has not been verified for this configuration.

completed

Recorded settings and thinking verification

Temperature 0 · seed 42 · output limit 4096 · reasoning effort unrecorded

RAM labels describe the test computer, not this model’s memory requirement. Speed comparisons keep the 32 GB and 128 GB runs separate.

Runtime build: unrecorded · Local · test machine with 32 GB RAM · concurrency: 1

Thinking status: native reasoning · probe 3 · scope Profile-bound diagnostic confirms native reasoning; separate checkpoint, no verified off condition..

Requested off does not prove an absence of reasoning. Saved checks do not establish unrecorded historical template or build details.

Overall across tasks

27.1 / 100

Nine categories, equally weighted · 9/9 eligible

Complete coverage for this setup. Quality only; speed is separate. How the index works →

What does it do well?

Select a category to explore the answers. Bars show what happened on every planned task. Incorrect answers and failures to finish both earn zero; technical problems and untested tasks stay unscored.

Exact scores and task times by suite

Each suite has its own metric and fixed task bank. The balanced index weights categories equally; individual scores and evidence remain available below.

Test suiteThinking / Reasoning versionDetails
Everyday coding tasksTask pass rate
Pass rate
2/16 · 13%
Answers completed
14/16 · 88%
Correct / completed
2/14 · 14%
2 full credit12 incorrect2 failed to finish
16/16 scored11.0s median task time · 16 measured
Larger practical tasksMean task credit
Average credit
0.0%
Full-credit rate
0/11 · 0%
Answers completed
11/11 · 100%
Full credit / completed
0/11 · 0%
11 incorrect
11/11 scored12.5s median task time · 11 measured
Logic and number puzzlesMean puzzle credit
Average credit
21.1%
Full-credit rate
6/34 · 18%
Answers completed
34/34 · 100%
Full credit / completed
6/34 · 18%
6 full credit7 partial credit21 incorrect
34/34 scored9.1s median task time · 34 measured
Published hard problemsMean answer credit
Average credit
12.9%
Full-credit rate
1/11 · 9%
Answers completed
11/11 · 100%
Full credit / completed
1/11 · 9%
1 full credit3 partial credit7 incorrect
11/11 scored13.2s median task time · 11 measured
Following exact instructions · original campaignStrict prompt compliance
Pass rate
19/23 · 83%
Answers completed
23/23 · 100%
Correct / completed
19/23 · 83%
19 full credit4 incorrect
23/23 scored5.4s median task time · 23 measured
Reading without guessingExact-match accuracy
Pass rate
2/16 · 13%
Answers completed
16/16 · 100%
Correct / completed
2/16 · 13%
2 full credit14 incorrect
16/16 scored3.4s median task time · 16 measured
Calculating from a passageNumeric exact-match accuracy
Pass rate
4/7 · 57%
Answers completed
7/7 · 100%
Correct / completed
4/7 · 57%
4 full credit3 incorrect
7/7 scored4.4s median task time · 7 measured
Reading research papersAnswer token F1
Average credit
28.3%
Full-credit rate
1/8 · 13%
Answers completed
8/8 · 100%
Full credit / completed
1/8 · 13%
1 full credit6 partial credit1 incorrect
8/8 scored6.3s median task time · 8 measured
Resisting incorrect correctionsAll-branches pass rate
Pass rate
2/12 · 17%
Answers completed
12/12 · 100%
Correct / completed
2/12 · 17%
2 full credit10 incorrect
12/12 scored6.4s median task time · 12 measured
Following exact instructions · 128 GB campaignStrict prompt complianceNot planned

Model analysis

This analysis covers this configuration across its recorded thinking conditions and test suites.

Use this native-reasoning checkpoint for quick, tightly constrained writing rather than autonomous project work. The limits are substantial: small-project coding rarely succeeded, larger practical work failed throughout, and handling source-backed answers under challenge was poor. Its useful niche is therefore narrow; quick responses should not be mistaken for dependable research or tool-driven assistance.

Evidence for this judgment
Following exact instructions · Thinking / Reasoning version
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Typical time per task (median)5.4 seconds
Tasks timed23 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Everyday coding tasks · Thinking / Reasoning version
Tasks passed2 of 16 (12.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking / Reasoning version
Average score0.0/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes0 full credit · 0 partial · 11 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking / Reasoning version
Tasks passed2 of 12 (16.7%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Original analysis generated by GPT Astra · · Edited by Codex; agent reviewed by Codex · publication authorized by user

About this summary

Codex reviewed this assessment against a newer evidence edition. The original analyst, generation date and source draft remain recorded separately from the editorial revision.

Analyst provider
openai
Generation route
Codex CLI · codex-cli 0.162.1 · ChatGPT account
Requested model
gpt-6-astra
Returned model
Not supplied; identity is recorded from the request only.
Exact provider snapshot
Exact provider snapshot not supplied.
Reasoning effort
high
Prompt version
model-summary-reader-v7
Generation time
2026-10-14T00:23:57.145938Z
Review time
2026-10-22T07:26:27.163254+00:00
Evidence reviewer
Codex (agent)
Publication authorization
User · 2026-10-22T05:55:43.625712+00:00 · Go ahead and complete the remaining work. Spin up agents: luna, terra, sol, astra, as needed to get it done. Narrative summaries: these are the astra generated summaries? Go ahead and get those completed as well. Once all of this is done, you may push the completed result.
Originating evidence edition
dataset-a241e6726b39b4e5b1cf99f9801c22ddffdd62a6885206d83da8ab42772579ef
Analysis input hash
7a1626187929950220027832f012e99f6c0571062aa62c6d6ccd4b465f5ff1f3
Original analysis input hash
0e8f684761e0103cc6cfd81b30224471e335799f5ec1e08ebc5db68a7d1f4c94
Editorial evidence edition
dataset-5a6f96ec56c3d240cc42fa23147c1495da1be71f3ef0722c0e5f8c4f5e126a83
Original summary
summary-2c24d39dc9472bec8c778855d2d992537865a70d49b4def7aead2c4ea294ef47
Summary revision
summary-34a6c54f705909d121b2b43eeacb9b3a6e6dc518a00b3ec19d138bc6243c84d3

Generation and editorial authorship remain attached to this revision. Changing the default generator does not change this attribution.

Contributing analyses (10)
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    6e7dbd0e104c12e562b7598b9222673a6c2e1214abef13ab24e6b32d1044da0c
    Summary revision
    summary-26bacc6e8c836a8bc855588f49fbbe418eb5928a014069fc70666335f2690edf
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    f0494ef32dbca0db4c7cf38bf58c9d97bee8c57c0cd670036faccaee6e14d89a
    Summary revision
    summary-2fa0565d663989d095a8006b4f219793d7234d6f23045bcc8a1a468549342d31
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    fffa7b71a00feaf26b5c339f48fad6b909f5cb1801fbf1e22a9cddd53d71fc09
    Summary revision
    summary-3ca4d9b668ea7ae304326dadbf271e3d4eb1b8f672c44976f5f9f745f61d3178
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    7e255ce719ba53bc630ab55feda6a66c18adf9b0c755befb40050c76ea29fc7a
    Summary revision
    summary-45b171aa0484f2895b6b079a0c7ac283e6b6f4e121e84bc4fc6aa6d5db84081a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    d4d32bd11501e0b94b0239317c9728887a8809def08d954bdab7d6e9786f4a4b
    Summary revision
    summary-532c29233472a3f02523e9f0b5ebe3b3472d05b0ad3c3af681966e1e8841100a
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    2e396ce905b14997489f270b71885c4cebe14ae9dca7afffe669c3150fe8751c
    Summary revision
    summary-5ad6d713d5f374a406ef3085faaef19dbdf91dcf2ec9b611c5a5b6637ba05c25
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    subject_synthesis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    31f61f410e5fa612c941b36b4cb87f0076e2d6ffd49275816c147a7a46e24275
    Summary revision
    summary-9045ff805c7786b452069c15b309c0c3b5abedcfcb0dd229b87e2676fb9ec1dd
    Input summary revisions
    summary-26bacc6e8c836a8bc855588f49fbbe418eb5928a014069fc70666335f2690edf, summary-2fa0565d663989d095a8006b4f219793d7234d6f23045bcc8a1a468549342d31, summary-3ca4d9b668ea7ae304326dadbf271e3d4eb1b8f672c44976f5f9f745f61d3178, summary-45b171aa0484f2895b6b079a0c7ac283e6b6f4e121e84bc4fc6aa6d5db84081a, summary-532c29233472a3f02523e9f0b5ebe3b3472d05b0ad3c3af681966e1e8841100a, summary-5ad6d713d5f374a406ef3085faaef19dbdf91dcf2ec9b611c5a5b6637ba05c25, summary-a16f03fef961ad91002a3510d283b88cfd2cf00a502f392277d3db97d61b8493, summary-d7b96015405631754ef5b90069626a9f9dd77ecab78cc505bd4dc0b49411a1b5, summary-f6c620b1a280078ec253127b8a3af5fa0d15eed593d2d970ccb0c8d386351dcd
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    b0f1352b6b23a6b733dc32a56d5d16a28b1a3d0a57af82ef2ab32362a645d309
    Summary revision
    summary-a16f03fef961ad91002a3510d283b88cfd2cf00a502f392277d3db97d61b8493
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    4721ee2e26f83b7e33f16eb86349636e2aeec9a79f42da5bffe1a05934632b32
    Summary revision
    summary-d7b96015405631754ef5b90069626a9f9dd77ecab78cc505bd4dc0b49411a1b5
    Input summary revisions
    Direct evidence analysis
  • GPT Astra ·
    Analysis stage
    suite_analysis
    Requested / returned model
    gpt-6-astra / Not supplied
    Exact provider snapshot
    Exact provider snapshot not supplied.
    Reasoning effort
    high
    Prompt version
    model-summary-v3
    Analysis input hash
    43cca43d076b8335576410affa51ac2c31ac17115bc8702a6d677d79e6a56039
    Summary revision
    summary-f6c620b1a280078ec253127b8a3af5fa0d15eed593d2d970ccb0c8d386351dcd
    Input summary revisions
    Direct evidence analysis

Good at

Useful for responses with explicit wording, length, language and formatting rules. It met most complete instruction sets, making constrained drafting a reasonable use—as long as you check important requirements.

Evidence for this judgment
Following exact instructions · Thinking / Reasoning version
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Can help count, compare and calculate from a passage. It answered most questions correctly in this small test, but the misses make independent checking important.

Evidence for this judgment
Calculating from a passage · Thinking / Reasoning version
Tasks passed4 of 7 (57.1%)
Test coverage7 of 7 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Struggles with

Poor choice for editing project files or carrying out longer tasks with tools. Small coding tasks rarely passed, and larger practical tasks showed no measurable success.

Evidence for this judgment
Everyday coding tasks · Thinking / Reasoning version
Tasks passed2 of 16 (12.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Larger practical tasks · Thinking / Reasoning version
Average score0.0/100 · Mean task credit
Test coverage11 of 11 tasks scored
Task outcomes0 full credit · 0 partial · 11 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Source-backed answering is a major weakness. Exact passage answers and keeping or updating answers correctly under challenge performed poorly; research-paper answer matching was also weak. Treat document assistance as something to verify, not trust.

Evidence for this judgment
Reading without guessing · Thinking / Reasoning version
Tasks passed2 of 16 (12.5%)
Test coverage16 of 16 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Resisting incorrect corrections · Thinking / Reasoning version
Tasks passed2 of 12 (16.7%)
Test coverage12 of 12 tasks scored
Explore these test results →

View this model’s answers, code and tool activity →

Reading research papers · Thinking / Reasoning version
Average score28.3/100 · Answer token F1
Test coverage8 of 8 tasks scored
Task outcomes1 full credit · 6 partial · 1 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Difficult reasoning remains weak despite the Thinking label. Both generated logic-and-number puzzles and selected difficult math and reasoning questions produced low average scores.

Evidence for this judgment
Logic and number puzzles · Thinking / Reasoning version
Average score21.1/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes6 full credit · 7 partial · 21 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Published hard problems · Thinking / Reasoning version
Average score12.9/100 · Mean answer credit
Test coverage11 of 11 tasks scored
Task outcomes1 full credit · 3 partial · 7 no credit
Explore these test results →

View this model’s answers, code and tool activity →

Compared with other models

For strict instructions with thinking on, it sits between higher- and lower-scoring small dense peers. In the matched timing cohort, it finishes sooner than every alternative, including equally successful models. The nearest accuracy leaders have only a narrow advantage, so this is a useful task-specific speed-quality balance.

Evidence for this judgment
Following exact instructions · Thinking / Reasoning version
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Typical time per task (median)5.4 seconds
Tasks timed23 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructions
Thinking / Reasoning version. Small · under 5B · dense. Rank 3 of 5 in this recorded comparison.

Speed: Following exact instructions
Thinking / Reasoning version. Matching test-machine label and execution settings; spans model sizes. Timings include unsuccessful tasks, so read them alongside quality.

Following exact instructionsThinking / Reasoning version · Small · under 5B · dense

Keep to rules about wording, length, language and formatting.

Strict prompt compliance · 0–100% · higher is better
23 of 23 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 Qwen3.5 4B4.2B total parameters91.3%
    2 tasks more passed
  2. #2 Gemma 4 E2B Q4_04.6B total parameters87.0%
    1 task more passed
  3. #3 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters82.6%
    This model
  4. #4 Qwen3.5 2B Q4_K_M1.9B total parameters65.2%
    4 tasks fewer passed
  5. #5 SmolLM3 3B Q4_K_M3.1B total parameters52.2%
    7 tasks fewer passed
How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-14 assessment. Task-specific results.

Speed and task success: Following exact instructionsThinking / Reasoning version · Matching test-machine label and settings · Across model sizes

Shorter task time is better only when the result is useful. These times include failures, tools and grading. Token generation speed measures a different part of the work.

Compare timings and outcomes
ModelTypical task timeGeneration speedTasks passed
LFM2.5 1.2B Thinking Q4_K_MThis model5.4 seconds23 of 23 tasks timed300.2 tokens/s19 of 2323 of 23 scored
SmolLM3 3B Q4_K_M7.8 seconds23 of 23 tasks timed121.4 tokens/s12 of 2323 of 23 scored
Nemotron Nano 9B v2 Q4_09.6 seconds23 of 23 tasks timed53.8 tokens/s12 of 2323 of 23 scored
Ling 3.0 Tiny Q4_010.2 seconds23 of 23 tasks timed135.3 tokens/s19 of 2323 of 23 scored
Mellum 2.1 12B-A2.5B Thinking11.1 seconds23 of 23 tasks timed142.1 tokens/s19 of 2323 of 23 scored
Ornith 1.5 9B Q4_K_M11.9 seconds23 of 23 tasks timed54.7 tokens/s20 of 2323 of 23 scored
Ministral 3 8B Reasoning 2512 Q4_K_M39.6 seconds23 of 23 tasks timed53.6 tokens/s17 of 2323 of 23 scored
Granite 4.2 8B Q4_041.0 seconds23 of 23 tasks timed55.4 tokens/s20 of 2323 of 23 scored
Gemma 4 12B QAT Q4_058.8 seconds23 of 23 tasks timed36.5 tokens/s21 of 2323 of 23 scored
Ministral 3 14B Reasoning 2512 Q4_K_M59.3 seconds23 of 23 tasks timed35.6 tokens/s16 of 2323 of 23 scored

Results from this test setup, not a promise for your computer. Hardware, software, tokenizers and answer length can change the ordering. Historical runtime builds were not always captured. Comparison saved with the 2026-10-14 assessment.

Test machine: AMD Ryzen 5 5600X (6 cores / 12 threads) · AMD Radeon RX 6700 XT (12 GB VRAM) · 32 GB or 128 GB system RAM, never mixed in one timing group. Full test machine details

Prefer the separate Instruct counterpart, LFM2.5 1.2B Q4_0, for instruction-focused work: it was substantially quicker and somewhat more accurate. It also scored meaningfully better on puzzles while finishing sooner, although its puzzle results remained weak.

Evidence for this judgment

Following exact instructions
Thinking / Reasoning version. LFM2.5 1.2B · Instruct versus reasoning checkpoint. Rank 2 of 2 in this recorded comparison.

Logic and number puzzles
Thinking / Reasoning version. LFM2.5 1.2B · Instruct versus reasoning checkpoint. Rank 2 of 2 in this recorded comparison.

Following exact instructions · Thinking / Reasoning version
Tasks passed19 of 23 (82.6%)
Test coverage23 of 23 tasks scored
Typical time per task (median)5.4 seconds
Tasks timed23 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Logic and number puzzles · Thinking / Reasoning version
Average score21.1/100 · Mean puzzle credit
Test coverage34 of 34 tasks scored
Task outcomes6 full credit · 7 partial · 21 no credit
Typical time per task (median)9.1 seconds
Tasks timed34 tasks
Explore these test results →

View this model’s answers, code and tool activity →

Following exact instructionsThinking / Reasoning version · LFM2.5 1.2B · Instruct versus reasoning checkpoint

Keep to rules about wording, length, language and formatting.

Strict prompt compliance · 0–100% · higher is better
23 of 23 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 LFM2.5 1.2B Q4_01.2B total parameters87.0%
    1 task more passed
  2. #2 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters82.6%
    This model
How this comparison works

Share of tasks that passed the recorded checks. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-14 assessment. Task-specific results.

Logic and number puzzlesThinking / Reasoning version · LFM2.5 1.2B · Instruct versus reasoning checkpoint

Solve newly generated puzzles using arithmetic, deduction and planning.

Mean puzzle credit · 0–100 points · higher is better
34 of 34 tasks scored for each model.

Gaps are relative to this model. Treat small gaps cautiously.

  1. #1 LFM2.5 1.2B Q4_01.2B total parameters38.3/100
    17.3 points higher
  2. #2 LFM2.5 1.2B Thinking Q4_K_M1.2B total parameters21.1/100
    This model
How this comparison works

Average credit per task; partly correct results can earn partial credit. Family comparisons show the labeled Instruct and reasoning versions. Other peer comparisons use the same requested thinking setting, not necessarily each model’s best mode. Small differences on these short tests may not hold on other tasks. Peer membership varies by test coverage and compatible settings.

Comparison saved with the 2026-10-14 assessment. Task-specific results.

Worth knowing

This checkpoint has native reasoning, not a verified thinking toggle. Its separate Instruct counterpart uses different weights, so their differences cannot isolate the effect of thinking.

Evidence for this judgment

Source limitation
This is a native reasoning checkpoint. Its Instruct counterpart is a related checkpoint with different weights, not a matched thinking-off run of this GGUF.

Publication redactions and withheld artifacts limit readers’ inspection of some paper evidence, coding examples and practical-task results.

Evidence for this judgment

Source limitation
Some selected evidence has recorded publication redactions.

Example 1: Larger practical tasks
Thinking / Reasoning version. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 2: Reading research papers
Thinking / Reasoning version. Final result: 10.0/100 · Answer token F1. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 3: Reading research papers
Thinking / Reasoning version. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 4: Everyday coding tasks
Thinking / Reasoning version. Final result: no credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

Example 5: Everyday coding tasks
Thinking / Reasoning version. Final result: full credit. A final score does not describe every tool step; this example supports a behavior observation, not a whole-test conclusion.
View the question and recorded answer →

How to interpret these assessments and their limitations →

One weight format is recorded for this model. Additional tested formats will appear here when available.

Thinking tradeoff

No verified on/off comparison is available for this configuration and test suite. Requested modes alone do not establish a controlled thinking comparison.

Instruct vs Thinking / Reasoning

Two versions of the same model family, using separate GGUF checkpoints. Compare the extra task success with the extra time. This is a version comparison, not a switch within one file.

Everyday coding tasks

Scores and times describe this task bank. Quick failures are not a speed advantage. Different execution settings are labeled.

VersionTask resultTypical task timeWeight downloadExplore
LFM2.5 1.2B Thinking Q4_K_MThinking / Reasoning version12.5%16/16 scored11.0 seconds16/16 timed0.73 GBAnswers and code
LFM2.5 1.2B Q4_0Thinking off12.5%16/16 scored1.4 seconds16/16 timed0.7 GBAnswers and code

Compare

Sampled repetitions

Repeated runs are a separate experiment and never add votes to the baseline score.

Thinking / Reasoning version · temperature 0.6

48/48 repeats scored · 3 full-credit repeats.

16/16 questions have every planned repeat scored. Of those, 1 earned full credit on every repeat.

Recorded repetition conditions
  • Seed 202 · requested on · native reasoning
  • Seed 303 · requested on · native reasoning
  • Seed 101 · requested on · native reasoning

Everyday coding tasks: recorded outcomes

Find information in a small project, edit files and fix bugs.

2 full credit12 incorrect2 failed to finish

Each segment counts tasks, not credit points. Partial credit varies by task.

Scoring and fixed question bank →

This model’s answers, code and tool activity

Select a question to read the original answer, tool calls and available code changes for this configuration and thinking setting.

1–16 of 16 matching questions · 16 in this version

  • Locate the Retry Limit

    The user asks where the project's retry-limit setting is defined and what value it holds. This exercises locating a named setting in code and reporting both its location and value.

    basic-T1 · primary
  • Find the Release Date

    The user asks for the release date mentioned in a project notes document. This exercises reading a specified file and extracting a particular fact rather than guessing from outside knowledge.

    basic-T2 · primary
  • Change the Retry Limit

    The user asks to change the project's retry limit to a specified value. This exercises finding the relevant setting and making a focused code edit rather than merely describing the change.

    basic-T3 · primary
  • Create an Exact One-Line File

    The user asks to create a text file containing exactly one supplied line. This exercises basic file creation and precise instruction following, including avoiding extra text in the file.

    basic-T4 · primary
  • Explain the Latest Commit

    The user asks what the most recent Git commit changed in the project. This exercises inspecting recorded changes and turning them into a clear explanation of what was modified.

    basic-T5 · primary
  • Count and Compare Python Files

    The user asks how many Python files are in the application folder and which is largest. This exercises identifying relevant files, counting them, and comparing their sizes.

    basic-T6 · primary
  • Add Readable Byte Sizes

    The user asks to add a function that converts byte counts into readable sizes and remove the related TODO comment. This exercises implementing a small utility and updating the surrounding code consistently.

    basic-T7 · primary
  • Collect TODO Comments

    The user asks to find every TODO comment in the Python code and append each to the project's task list on a separate line. This combines searching across files with collecting text into an existing document.

    basic-T8 · primary
  • Fix Code Behind a Failing Test

    The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

    basic-H1 · primary
  • Rename a Function Throughout

    The user asks to rename a network function and update every place in the project that uses it. This exercises tracking references across files and making a coordinated change rather than editing only the definition.

    basic-H2 · primary
  • Find the Proxy Settings

    The user asks to find a proxy configuration file somewhere under the project's parent folder and report its host and port. This exercises searching beyond the project folder and reading configuration values.

    basic-H3 · primary
  • Locate an Error in a Log

    The user asks which error code appears in a long event log and on which line. This exercises finding a specific entry in a large file and reporting its precise location.

    basic-H4 · primary
  • Append a Final Task

    The user asks to append a supplied release-related task as the final line of an existing task list. This exercises preserving existing content while adding exact text in the requested position.

    basic-H5 · primary
  • Calculate Total Retry Waiting

    The user asks for the total time a network function spends sleeping when all attempts fail, worked out from the code. This exercises following retry logic and adding the delays along that failure path.

    basic-H6 · primary
  • Document Every Command-Line Option

    The user asks for a project readme listing every command-line option with its help text. This exercises discovering the program's available options and transferring that information into complete, source-grounded documentation.

    basic-H7 · primary
  • Undo a Commit by Editing

    The user asks to undo the latest Git commit's change by editing the file directly, without using Git to make changes. This exercises understanding a previous edit and reversing it under a method constraint.

    basic-H8 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.

Suggested comparisons

These are starting points for inspection, not claims that one setup is better. Runtime and placement differences remain visible.

Qwen3.5 2B Q4_K_M

Another checkpoint in the same total-parameter band and architecture category, with matching task controls.

Requested thinking on · Complete compared coverage

Compare these configurations →

Selection rule and evidence IDsPolicy comparisons-v3 · Template same-parameter-architecture-matching-controls
Facts: fact-13bffa6d739b0531f72616b4, fact-bc54601b0f1841d544c224c5
Matching candidate set: fact-bc54601b0f1841d544c224c5, fact-4e0224af8f948a39b33d46fa, fact-8471cd6afce688a52c7066ee, fact-ed9e4adee7986e86f4bfc2e4, fact-093ef2fab39b29e20dc389a1, fact-ee68991b61fb7db54bc8ee72, fact-3a9b3fb80c44f1f53c33c065, fact-7578ebf4ba24c76f5ad6ef40, fact-9d133fa304209ce92e03d207

Tested weights

Q4_K_M · 0.73 GB

LiquidAI · recorded source

LFM2.5-1.2B-Thinking-Q4_K_M.gguf · Exact GGUF file →Historical first-MiB hash only; not a full-file identity.

Explore the answers

Choose a suite to inspect its exact available questions, references and model answers.