Published hard problems
Answer selected difficult math and reasoning questions from LiveBench.
Compare model results on this test →
How this test is scored · 11 questions · Mean answer credit
What these tests cover
These tasks ask the model to solve selected math problems, complete supplied proofs, and reason through spatial and logic puzzles. They exercise following linked constraints and mathematical steps in these specific problems, while also requiring carefully formatted answers.
About this explanation
Generated by GPT Astra (gpt-6-astra) · 2026-10-12. Shared across models for this version of the test. The original question and checks remain the source of truth.How it is scored
Official pinned LiveBench per-family graders. Generation-based only, no shared-prefix likelihood mode. This tiny selected subset is NOT an official LiveBench score. Reviewed by scoring-adjudication-v2: exact Make-24 input numbers; explicit five-letter choice tokens and final three-digit math answers; invalid cube, contradictory asterisk items, and CPU monitor with invalid check and incomplete replay evidence excluded uniformly. Original evidence and grades retained.
Mean answer credit · 11 fixed primary questions
Version 78f00bdba65082c1. Excluded questions remain available below; retired tasks are kept in the edition archive.LiveBench contributors; Apache-2.0 benchmark distribution; selected historical subset, not an official score. · Original source · Publication terms
Recorded evidence availability
Counts cover question definitions and all saved attempts, including historical attempts. They are evidence-component counts, not scored questions. A zero means no components in that state.
| Component | full | redacted | source link only | not recorded | withheld |
|---|---|---|---|---|---|
| answer | 1762 | 0 | 0 | 462 | 0 |
| artifacts | 0 | 0 | 0 | 2224 | 0 |
| checks | 1774 | 0 | 0 | 450 | 0 |
| context | 0 | 0 | 0 | 2224 | 0 |
| input | 1621 | 0 | 0 | 603 | 0 |
| prompt | 1783 | 0 | 0 | 441 | 0 |
| reasoning | 803 | 0 | 0 | 1421 | 0 |
| reference | 12 | 0 | 0 | 2212 | 0 |
| tools | 0 | 0 | 0 | 2224 | 0 |
Full: recorded material is available. Redacted: explicit publication rules removed identified text. Source link only: follow the cited source. Not recorded: no saved component. Withheld: material is not published.
Question bank
1–12 of 12 matching questions · 12 in this version
- Rectangles Sharing a Circle
Find an unknown segment length in a diagram of two rectangles, using given side lengths and the fact that four vertices lie on a circle. This exercises geometric reasoning; the response must explain the steps and end with a three-digit answer, including leading zeros if needed.
lb-8f3f8e01ff3c82ee · primary - Two-Jump Return Probability
Determine the probability that a cricket ends near its starting point after two equal-length jumps in different directions. This combines geometry with probability and requires selecting a multiple-choice answer, then repeating its letter five times in one string.
lb-01fa9af2dad55a3b · primary - Counting Overlapping Ownership Groups
Use ownership totals and counts of residents owning exactly two or three kinds of objects to find how many own all four. This exercises counting overlapping groups; the model must explain its reasoning and end with a three-digit answer.
lb-75ee28de242ec3fd · primary - Last Digit of a Row Sum
Find the last digit of a distant row’s sum in a triangular number array whose entries depend on the preceding row. This exercises reasoning about repeated numerical rules and digit patterns, with the chosen answer letter repeated five times.
lb-c3dde5118315216f · primary - Completing a Function Proof
Match ten supplied formulas to gaps in a proof about a function on positive real numbers. Rather than inventing a proof from scratch, the model must follow substitutions and logical connections, explain its choices, and finish with an ordered list of formula numbers.
lb-6e12ab2219a4051c · primary - Completing a Divisibility Proof
Fill 29 formula gaps in a supplied proof about when two integer expressions have a greatest common divisor that eventually stays constant. This exercises following divisibility arguments and matching formulas to their roles, ending with an ordered, comma-separated list of formula numbers.
lb-6d9bd0806e27f34f · primary - Four-Person Attribute Puzzle
Use clues about four people’s positions and five kinds of attributes to answer four questions about them. This exercises combining adjacency, ordering, even-or-odd positions, and either-or conditions; answers must appear in question order inside the requested solution tags.
lb-ba951e86124110cf · primary - Two-Person Lineup Puzzle
Deduce jobs, movie preferences, and positions for two people using uniqueness, adjacency, and left-to-right clues. This exercises a short chain of logical deductions, followed by four ordered answers inside solution tags, with positions written as numerals.
lb-20d48b97e524cd64 · primary - Cutting Two Touching Triangles
Imagine two physical equilateral triangles touching at specified vertices, then count the separate pieces after a cut and separation. This exercises tracking spatial arrangements and distinguishing contact from attachment, with an explanation followed by a single bold integer.
lb-c8f81f9118ead46a · primary - Shape From Selected Cube Vertices
Use labeled top and bottom vertices of a cube to identify the shape formed by four selected vertices from a supplied list. This exercises translating a verbal description into a spatial arrangement, with reasoning followed by one bold shape name.
lb-4c6e9d7cd2f5506e · excluded - Truth Claims Across Town
Trace statements among people who always tell the truth or always lie to determine the status of those at the airport, botanical garden, and amusement park. This exercises linked logical deductions amid extra details, ending with three bold yes-or-no answers in order.
lb-8e20b94937c6fd08 · primary - Farm, Vineyard, and Library Claims
Determine whether the people at the farm, vineyard, and library tell the truth, using linked claims and known truth-tellers or liars. This exercises separating relevant statements from distracting details and following their implications, with a bold, ordered list of three yes-or-no answers.
lb-ebae339b57ef0084 · primary