AgentBench / results
← All suites

Following exact instructions

Keep to rules about wording, length, language and formatting.

Compare model results on this test →

How this test is scored · 23 questions · Strict prompt compliance

How it is scored

Official strict prompt-level compliance is the primary score. Strict per-instruction fraction and loose prompt/instruction scores are retained separately. Original prompts are unchanged. Reviewed by scoring-adjudication-v2: exact Make-24 input numbers; explicit five-letter choice tokens and final three-digit math answers; invalid cube, contradictory asterisk items, and CPU monitor with invalid check and incomplete replay evidence excluded uniformly. Original evidence and grades retained.

Strict prompt compliance · 23 fixed primary questions

Version a08b1f61cf6ddde8. Excluded questions remain available below; retired tasks are kept in the edition archive.

Google Research IFEval; Apache-2.0. · Original source · Publication terms

Recorded evidence availability

Counts cover question definitions and all saved attempts, including historical attempts. They are evidence-component counts, not scored questions. A zero means no components in that state.

Componentfullredactedsource link onlynot recordedwithheld
answer50400600
artifacts0005640
checks52800360
context0005640
input50400600
prompt52800360
reasoning240003240
reference0005640
tools0005640

Full: recorded material is available. Redacted: explicit publication rules removed identified text. Source link only: follow the cited source. Not recorded: no saved component. Withheld: material is not published.

Question bank

1–20 of 24 matching questions · 24 in this version

  • Lowercase Darth Vader Dialogue

    The model must write a conversation with an angry Darth Vader in three parts separated by asterisks, using only lowercase letters and repeating a required word at least six times. This exercises simultaneous control of wording, capitalization and structure.

    if-2943 · primary
  • Two Haiku Without Commas

    The model is asked for two question-form haiku about foolish behavior and not studying, aimed at young readers. It must separate the responses with six asterisks, include a title in double angle brackets and avoid commas, combining creative writing with precise formatting.

    if-2704 · primary
  • Hamptons Overview With Placeholders

    The model must describe the Hamptons in at least 300 words while including at least three square-bracketed placeholders and exactly two Markdown bullet points. This exercises length control alongside required text elements and list formatting.

    if-2284 · primary
  • Team Comparison With Paragraph Rules

    The model is asked to compare Tottenham Hotspur and Stockton in a specified season using exactly three paragraphs separated by blank lines. Required keywords, a prescribed opening word for the third paragraph and a specially marked postscript exercise detailed placement and structure control.

    if-2549 · primary
  • Weather Template With Text Constraints

    The model must draft a weather chatbot template that accepts a location, stays under six sentences and highlights at least two sections with asterisks. A separate requirement governs use of a particular letter, adding character-level control to the formatting demands.

    if-1174 · primary
  • Long Comparison With Forbidden Words

    The model must discuss whether a beach or a park is more likely to contain a dog in at least 900 words. It must avoid two specified words and end with a fixed phrase, exercising constraint tracking across a long response.

    if-3114 · primary
  • Portuguese Text Inside Quotation Marks

    The model is asked to write about Adam and Eve entirely in Portuguese and enclose the whole response in double quotation marks. This exercises adherence to a requested language and an exact outer format.

    if-2724 · primary
  • All-Caps Critique of Obeying Laws

    The model must critique the claim that bad laws should not be followed, using two labeled paragraphs written entirely in capital letters. This exercises consistent capitalization and section labeling within an argumentative response.

    if-1021 · primary
  • Flood Poem With Capitalized Words

    The model is asked to write a poem about flooding in Donnell, Texas, with a title in double angle brackets and at least three fully capitalized words. This combines creative writing with explicit title and capitalization requirements.

    if-2853 · primary
  • Repeat the Request Before Rewriting

    The model must repeat a supplied rewriting request word for word before giving an unusual version of its biblical sentence, with nothing added beforehand. This exercises exact copying and keeping required material in the correct order.

    if-2482 · primary
  • Character Sheet in JSON

    Acting as a dungeon master, the model must create a character sheet with a randomly chosen class for a solo player. The entire response must be JSON inside a Markdown code block and include two specified keywords, exercising structured output and required wording.

    if-2591 · primary
  • Tennis Opinion With Fixed Responses

    The model is asked whether Roger Federer is a better tennis player than Rafael Nadal, but may respond only with one of three supplied phrases. This tests staying within a fixed response set without adding explanation.

    if-3749 · primary
  • Teen Newspaper Name With Postscript

    The model must suggest a name for a newspaper aimed at Melbourne teenagers and finish with a postscript using the requested marker. The specific instruction-following demand is remembering the required ending while answering a creative naming request.

    if-2622 · primary
  • Riddle Story With Highlighted Sections

    The model is asked to turn a supplied riddle and its answer into a funny story, highlighting at least two sections with asterisks. The task pairs creative expansion with a concrete requirement for marked text.

    if-2905 · primary
  • Formal Rewrites Without Commas

    The model must provide two formal alternatives to a casual greeting and statement about visiting the beach, without using any commas. This exercises following a punctuation restriction while changing the tone of supplied text.

    if-1187 · primary
  • Funny Email With Minimum Sentence Count

    The model is asked to write a long, funny email to a friend about the ingredients of a good life, using at least 20 sentences. The explicit constraint exercises sentence-count control within an informal writing task.

    if-292 · primary
  • Seasons Explanation With Postscript

    The model must explain why seasons occur to elementary school students and add a postscript with the requested marker at the end. This pairs an audience-specific explanation with a required closing element.

    if-1800 · primary
  • Programming Comparison With Bracketed Title

    The model is asked to explain the pros and cons of Kotlin versus Java and include a title enclosed in double angle brackets. The specific formatting demand tests whether a small presentation instruction is retained during a technical comparison.

    if-1322 · primary
  • Long News Post in Few Sentences

    The model must write a blog post about recent US news using at least 250 words but fewer than five sentences, plus a title in double angle brackets. This exercises coordinating word and sentence limits with title formatting.

    if-1392 · primary
  • Levee Comparison Entirely in Korean

    The model is asked to explain the difference between a levee and an embankment while responding only in Korean. This tests following a language restriction when the question itself is written in English.

    if-2225 · primary
Filters change this list, not the fixed scoring denominator. Search currently covers question IDs, titles and available task explanations. Focus a control to load prompt/outcome search where this edition provides it.