Useful evidence.
Clear boundaries.

Scores and coverage

Each configuration retains its predetermined question set. The observed score averages assessed task outcomes, including confirmed budget failures. A perfect average on a partial run remains a preview, not a complete result.

Incorrect answers and confirmed failures to finish within the test budget both earn zero, with distinct labels. Technical errors, ambiguous answer-channel failures, uncertain context guards, tasks blocked before execution and pending reviews remain unscored. Saved file or code checks can earn credit without a final chat message. A context guard stopping a task before execution is not evidence of poor model performance. The new interpretation is versioned as task-outcomes-v1; older editions retain their original scores. Repeated attempts and corrected grades stay in the history; only the canonical outcome contributes to a baseline score.

Binary suites show passed/scored counts. Other suites report mean credit or F1; those are not pass percentages. Capability groups are navigation labels. The balanced overall index is defined separately below.

Completion, correctness and retries

Pass rate is full-credit tasks divided by assessed tasks, including confirmed failures to finish. Answers completed counts full-credit, partial-credit and incorrect outputs among assessed tasks. Correct / completed counts full-credit outputs among those completed outputs. A saved file can be assessable without a final chat message. Technical failures, untested tasks and unresolved reviews are unknown and are excluded from all three denominators; their counts remain visible. Average-credit categories retain their original credit score alongside the full-credit and completion rates.

Failed to finish means this run stopped at a recorded token, step, time or context boundary, or in a repeated tool loop, without an accepted result. The benchmark measures work under recorded limits. It does not establish that a model could never solve the task, or predict success with a larger allowance. An incorrect delivered answer and an unfinished run both earn zero in the baseline, but are different outcomes with different practical costs.

For interactive work, you may be willing to wait longer, give feedback or try again. For unattended work, dependable completion matters more. Read both rates and the individual stop reason. Giving extra chances only to selected models would make the baseline harder to compare.

There is no new retry allowance in the published score. A future recovery experiment should keep first-attempt results intact, use a fixed additional budget and retry rule for every eligible model, and report recovery success plus total time and tokens separately. Technical recovery attempts remain distinct from retries of assessed model failures. More budget, restarting from scratch and continuing the same run are different experiments.

Reading-page grading limits

The original reading-page checker awards either zero or full credit. It requires an HTML file, all expected titles in the rendered text, no flagged author/page-count substitutions left undisclosed in the final reply, and a data-issue keyword in that reply. The prompt asks for a beautiful, self-contained page, but does not explicitly request a written explanation of data cleanup. The duplicate and browser-error fields are diagnostics; they do not independently set the grade. Browser errors can still cause missing text.

A zero can therefore mean a missing file, a broken page, missing books, or just an explanation the checker did not recognize. The answer page now separates those recorded findings. Keyword and title matching can miss valid presentations; absence of a flag is not proof of factual correctness. The checker does not score visual quality.

Partial credit would better distinguish delivery from data handling, but it needs a common rubric and an evidence review across models. It has not been applied to the current scores. A revised task should state its data-handling requirements explicitly and assess whether the page works, represents the source accurately and communicates unknown values. Any retrospective regrade must be versioned, retain original grades, and leave insufficient evidence unscored rather than assume failure.

Reviewed scoring corrections

The current results retain drop-adjudication-v1: decimal percentages keep their value, two incorrect accepted references are removed, and the passer/receiver ambiguity is excluded. Reading arithmetic uses the same seven questions for every setup.

scoring-adjudication-v2 requires Make-24 expressions to use exactly the supplied numbers. Competition multiple-choice answers must contain an unambiguous five-letter repeated answer token; an explanation may follow because those prompts do not require the token at the end. The three-digit competition questions explicitly require the answer at the end. Presentation markup and ordinary final punctuation are accepted, while matching digits inside unfinished reasoning do not count as a final answer. A format failure does not establish that the mathematical reasoning was wrong.

Three flawed task definitions are excluded uniformly: the cube question whose correct rectangle is absent from the options, and both versions of the itinerary prompt requesting six asterisks while showing seven. The CPU monitor task is also excluded: its check accepts an incorrect calculation, and incomplete saved artifacts prevent a uniform regrade with stronger tests. Missing artifacts are not treated as model failures.

Saved answers were reassessed without model reruns. Original responses, tool evidence and grades remain available in linked history, including the excluded tasks. Only the included primary questions contribute to category scores. The nine category weights are unchanged, and earlier editions keep their original results.

Balanced overall index

The overall index is a 0–100 average of the nine category scores, each with weight 1/9. Each category keeps its existing pass-rate or average-credit metric; sampled repeats do not add extra weight. Scores describe the selected quant, reasoning mode and execution setup. There is no speed component, and this index is not Elo.

Only complete, eligible coverage in every category receives a ranked index. For incomplete results, recorded credit is held fixed and each unscored task ranges from zero to full credit. The resulting bounds show possible overall scores, not statistical confidence intervals. Fully scored setups with comparison qualifications remain unranked even if their bounds coincide. Missing results are not treated as observed failures.

Policy balanced-categories-v1 fixes the nine category weights. Adding a model does not change another setup’s index. A new task family or unverified scoring revision requires a policy review. The two current instruction campaigns have identical question content and scorer revisions, so they count once; original evidence versions remain intact. Hardware-specific timing groups remain separate.

Thinking and repeated trials

Thinking controls are checked with saved profile-level probes. “Requested off” does not establish an absence of reasoning. Historical probe scope may not identify every exact template/runtime build.

Thinking comparisons match question versions, seed, sampling, artifact and execution configuration. Score pairs and valid positive-duration time pairs have separate counts. Incomplete pairs are previews. Basic-task sampled seeds remain separate from the greedy baseline.

Test machine

All local runs used one desktop. Its system RAM was upgraded during the project, so each run keeps the RAM group it recorded.

CPU

AMD Ryzen 5 5600X (6 cores / 12 threads)

GPU

AMD Radeon RX 6700 XT (12 GB VRAM)

System RAM

128 GB now. Earlier campaign runs used 32 GB. Every local run keeps the RAM group it recorded, and speed charts never mix the two groups.

Operating system

Linux Mint 22.3 “Zena” (Ubuntu noble base), observed on 21 October 2026. Earlier runs did not record an OS version, so this observation is not applied to them.

Hosted runs

Cloud runs used provider hardware that the provider did not disclose. They are quality references only and never enter local speed comparisons.

Answer pages show the selected run’s recorded RAM group. A hosted run shows its provider and its exclusion from speed comparison instead of desktop hardware.

Time, file size and memory

The desktop was upgraded from 32 GB to 128 GB system RAM. Results retain their original hardware label. Extra capacity may matter most for large models or setups that were memory-constrained; it does not establish a uniform speed improvement. Speed charts keep the two hardware groups separate. Cloud results contribute quality evidence only.

The overview selects one setup per model using comparison eligibility, then scored coverage, then the declared baseline to break ties. It never picks a setup because it scored higher or ran faster. All other quants and setups remain on the model page.

Benchmark task time includes applicable tools, setup and grading. Model loading is separate. Timings describe this machine and the recorded setup; runtime and offload changes are not isolated quantization experiments.

Parameter categories use total parameters, with separate dense and MoE groups. When derived from GGUF headers, counts describe stored tensor elements; active/effective counts remain separately attributed. Category boundaries are versioned and the roster can grow.

Memory filters use decimal GB budgets for exact quants and runtime profiles. Verified fit requires documented measurements and system headroom; estimates are labeled and unknown requirements remain unclassified. System RAM, GPU VRAM, and unified memory are separate choices. Smaller supported tiers are included in larger budget filters.

Download-size groups use decimal gigabytes of all recorded weight shards. They do not imply a RAM requirement. Process RSS, device-wide VRAM and system available RAM measure different things and are never added together.

Exact evidence and availability

Original final answers, references, captured input envelopes, tool results and reasoning are shown where recorded and permitted. Model-generated code is displayed as inert text. The site does not execute answers.

Components are labeled full, redacted, source-link-only, not recorded or withheld. Named privacy rules replace private host paths; redacted content is visibly marked. Starting workspace snapshots may be unavailable even when tool change records survive.

Full QASPER papers link to their sources until redistribution is established per paper. The question and accepted answer annotations retain dataset attribution. External live filesystem artifacts require per-item publication review.

Analyst summaries

The configured analyst is GPT Astra, model ID gpt-6-astra. Generated summaries require evidence citations, numeric validation and an explicit review of the claims against the saved evidence. Each published summary identifies its reviewer as human or agent. Agent reviews name the agent and separately record the user’s publication authorization; they do not claim human review. Each summary retains its own requested and returned model IDs, prompt/evidence versions, original generation date and editorial history.

Changing the configured analyst cannot relabel older summaries. Factual fallback text does not receive an Astra authorship badge. Current pages explicitly identify analysis that has not yet been generated.