AgentBench / results
← All suites

Logic and number puzzles

Solve newly generated puzzles using arithmetic, deduction and planning.

Compare model results on this test →

How this test is scored · 34 questions · Mean puzzle credit

What these tests cover

These puzzles ask the model to follow explicit rules, combine clues, calculate exact probabilities, and plan routes. The tasks cover truth-teller logic, house arrangements, course prerequisites, arithmetic expressions, and grid navigation, showing how the model handles these specific kinds of structured reasoning.

About this explanationGenerated by GPT Astra (gpt-6-astra) · 2026-10-12. Shared across models for this version of the test. The original question and checks remain the source of truth.

How it is scored

Pinned Reasoning Gym 0.1.25 score_answer, including its partial credit. Fixed seeds/configurations and generated answers saved before any model runs; answer tags extracted uniformly. A score of 1 is a full pass. Reviewed by scoring-adjudication-v2: exact Make-24 input numbers; explicit five-letter choice tokens and final three-digit math answers; invalid cube, contradictory asterisk items, and CPU monitor with invalid check and incomplete replay evidence excluded uniformly. Original evidence and grades retained.

Mean puzzle credit · 34 fixed primary questions

Version 0bafa27052b08d41. Excluded questions remain available below; retired tasks are kept in the edition archive.

Reasoning Gym contributors; Apache-2.0; seeded generated subset. · Original source · Publication terms

Recorded evidence availability

Counts cover question definitions and all saved attempts, including historical attempts. They are evidence-component counts, not scored questions. A zero means no components in that state.

Componentfullredactedsource link onlynot recordedwithheld
answer44950012420
artifacts00057370
checks45310012060
context00057370
input41050016320
prompt45340012030
reasoning20410036960
reference360057010
tools00057370

Full: recorded material is available. Redacted: explicit publication rules removed identified text. Source link only: follow the cited source. Not recorded: no saved component. Withheld: material is not published.

Question bank

1–20 of 36 matching questions · 36 in this version

  • Three Heroes and Villains

    Identify which of three island inhabitants are truth-telling heroes and which are lying villains from their statements about one another. This exercises consistency checking across claims joined by “and” and “or,” with a label required for each person.

    rg-knights_knaves-0-0 · primary
  • Three Saints and Sinners

    Classify three inhabitants as truth-telling saints or lying sinners using their linked statements. This exercises reasoning about “if” and “if and only if,” while keeping each speaker’s assigned role consistent with what they say.

    rg-knights_knaves-0-1 · primary
  • Four Heroes with Linked Claims

    Determine the roles of four inhabitants whose statements combine direct claims, alternatives, and conditional relationships. The model must find a consistent hero-or-villain assignment and report a role for every person.

    rg-knights_knaves-1-0 · primary
  • Four Saints with Nested Claims

    Assign saint or sinner roles to four people using a mix of direct accusations and nested logical statements. This tests whether the model can connect several claims without losing the rule that saints tell the truth and sinners lie.

    rg-knights_knaves-1-1 · primary
  • Five Heroes with Nested Logic

    Classify five inhabitants as heroes or villains from statements containing nested “if and only if,” negation, and combined claims. This exercises tracking several interdependent truth conditions while producing a complete list of roles.

    rg-knights_knaves-2-0 · primary
  • Five Sages and Fools

    Determine which of five inhabitants are truthful sages and which are lying fools. Their statements mix accusations, conditions, and negation, requiring the model to check that all assigned roles and statements fit together.

    rg-knights_knaves-2-1 · primary
  • Three Houses with Drink Clues

    Use clues about three residents, their drinks, and their smoothies to identify who occupies the first house. This exercises combining neighbor relationships with fixed positions and unique attributes, returning only the requested name.

    rg-zebra_puzzles-0-0 · primary
  • Three Houses with Pet Clues

    Identify the first house’s resident from clues linking three people to house positions, pets, and nationalities. This exercises elimination using unique assignments and direct facts, with only a single name requested.

    rg-zebra_puzzles-0-1 · primary
  • Four Houses with Lunch Clues

    Work out who lives in the first of four houses using clues about colors, lunches, and pets. This exercises combining attribute matches with neighbor and immediately-left relationships while keeping every assignment unique.

    rg-zebra_puzzles-1-0 · primary
  • Four Houses with Flower Clues

    Identify the first resident in a four-house row using flower, drink, and smoothie preferences. The model must combine linked preferences, fixed positions, and neighboring-house clues, then return only the resident’s name.

    rg-zebra_puzzles-1-1 · primary
  • Five Houses with Phone Clues

    Find the first house’s resident among five people using clues about phones, pets, colors, and nationalities. This exercises keeping several sets of unique attributes aligned while applying fixed-position and immediately-left constraints.

    rg-zebra_puzzles-2-0 · primary
  • Five Houses with Family Clues

    Identify who occupies the first of five houses from clues linking residents, nationalities, children, cigars, and pets. This exercises multi-step deduction across several attribute categories, including distinguishing residents’ names from children’s names.

    rg-zebra_puzzles-2-1 · primary
  • Check Prerequisites for 11 Courses

    Decide whether all 11 courses can be completed under the listed prerequisites, returning True or False. This exercises following dependency chains and checking whether any circular requirement prevents a valid course order.

    rg-course_schedule-0-0 · primary
  • Check Prerequisites for Eight Courses

    Determine whether eight courses can all be taken while respecting the supplied prerequisite pairs. The task exercises interpreting which course must come first and checking for circular dependencies, rather than producing a full schedule.

    rg-course_schedule-0-1 · primary
  • Check Prerequisites for 17 Courses

    Check whether the prerequisite relationships among 17 courses permit completing them all. This exercises tracing indirect dependencies and detecting circular requirements, with a True-or-False answer rather than a proposed course sequence.

    rg-course_schedule-1-0 · primary
  • Check Prerequisites for 18 Courses

    Decide whether all 18 courses can be completed given a list of prerequisite pairs. This exercises keeping dependency directions straight and checking whether linked requirements form a loop that blocks a valid ordering.

    rg-course_schedule-1-1 · primary
  • Check Prerequisites for 26 Courses

    Evaluate whether the prerequisite list for 26 courses allows every course to be completed. This exercises tracking many interconnected requirements and checking for circular dependencies, then summarizing the conclusion as True or False.

    rg-course_schedule-2-0 · primary
  • Check Prerequisites for 25 Courses

    Determine whether 25 courses can be arranged in an order that respects all supplied prerequisites. This exercises reasoning over direct and indirect dependencies; the requested output is only whether such an order is possible.

    rg-course_schedule-2-1 · primary
  • Make 24 with a Repeated Pair

    Build an expression that makes 24 from four supplied numbers, including a repeated value, using each occurrence once. This exercises arithmetic search and number tracking, with only addition, subtraction, multiplication, and division allowed and an expression-only answer required.

    rg-puzzle24-0-0 · primary
  • Make 24 with Three Repeats

    Make 24 from four supplied numbers, three of which share the same value, using each occurrence once. This exercises choosing arithmetic operations and grouping while respecting the available numbers and returning an expression rather than an equation.

    rg-puzzle24-0-1 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.