AgentBench / results

← Everyday coding tasks

Fix Code Behind a Failing Test

The user asks to investigate a failing test and fix the application code without changing the test. This exercises diagnosing a bug while respecting an explicit limit on what may be edited.

complex

About this descriptionGenerated by GPT Astra (gpt-6-astra) · 2026-10-12. Shared across models for this version of the test. The original question and checks remain the source of truth.

Opening the requested recorded answer… This page needs JavaScript to select and verify evidence. If this message remains, no other model’s answer is shown in its place.

All recorded answers to this question

Each row is one recorded test trial: a model, weight format, thinking setting and seed. Open a row to read that answer above. Filtering this list never changes the open answer.

Showing recorded answers 1–25 of 511 · sorted by model name

  1. Falcon-H1R 7B Q4_K_M Q4_K_M · Reasoning enabled (observed; requested off) · Local · baseline seed 42Full credit
  2. Falcon-H1R 7B Q4_K_M Q4_K_M · Reasoning enabled (observed; requested off) · Local · repeat seed 101Incorrect
  3. Falcon-H1R 7B Q4_K_M Q4_K_M · Reasoning enabled (observed; requested off) · Local · repeat seed 202Incorrect
  4. Falcon-H1R 7B Q4_K_M Q4_K_M · Reasoning enabled (observed; requested off) · Local · repeat seed 303Incorrect
  5. Falcon-H1R 7B Q4_K_M Q4_K_M · Thinking on · Local · baseline seed 42Untested / blocked
  6. Falcon-H1R 7B Q4_K_M Q4_K_M · Thinking on · Local · repeat seed 101Untested / blocked
  7. Falcon-H1R 7B Q4_K_M Q4_K_M · Thinking on · Local · repeat seed 202Untested / blocked
  8. Falcon-H1R 7B Q4_K_M Q4_K_M · Thinking on · Local · repeat seed 303Untested / blocked
  9. Gemma 4 12B Q6_K · Thinking off · Local · baseline seed 42Failed to finish
  10. Gemma 4 12B Q6_K · Thinking off · Local · repeat seed 101Failed to finish
  11. Gemma 4 12B Q6_K · Thinking off · Local · repeat seed 202Full credit
  12. Gemma 4 12B Q6_K · Thinking off · Local · repeat seed 303Failed to finish
  13. Gemma 4 12B Q6_K · Thinking on · Local · baseline seed 42Full credit
  14. Gemma 4 12B Q6_K · Thinking on · Local · repeat seed 101Full credit
  15. Gemma 4 12B Q6_K · Thinking on · Local · repeat seed 202Full credit
  16. Gemma 4 12B Q6_K · Thinking on · Local · repeat seed 303Full credit
  17. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking off · Local · baseline seed 42Failed to finish
  18. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking off · Local · repeat seed 101Failed to finish
  19. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking off · Local · repeat seed 202Failed to finish
  20. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking off · Local · repeat seed 303Failed to finish
  21. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking on · Local · baseline seed 42Incorrect
  22. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking on · Local · repeat seed 101Full credit
  23. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking on · Local · repeat seed 202Full credit
  24. Gemma 4 12B QAT Q4_0 QAT-Q4_0 · Thinking on · Local · repeat seed 303Full credit
  25. Gemma 4 19B-A4B REAP IQ4_XS · Thinking off · Local · baseline seed 42Incorrect

Source item basic-H1. Original agent-harness tasks, published by the benchmark owner.