Reading research papers
Answer a question about a paper and identify the paragraphs supporting the answer.
Compare model results on this test →
How this test is scored · 8 questions · Answer token F1
What these tests cover
These tasks ask the model to answer specific questions about research papers and identify supporting paragraphs. They exercise finding and summarizing definitions, data sources, methods, and experimental details. Full papers are linked rather than included here; the available questions and supporting excerpts establish the scope described below.
About this explanation
Generated by GPT Astra (gpt-6-astra) · 2026-10-12. Shared across models for this version of the test. The original question and checks remain the source of truth.How it is scored
Official QASPER answer token F1 and evidence paragraph F1; primary score is answer F1. Any valid annotator answer can match. Unsupported/malformed evidence is separately reported.
Answer token F1 · 8 fixed primary questions
Version a489b64ee03196e3. Excluded questions remain available below; retired tasks are kept in the edition archive.QASPER, Dasigi et al.; CC BY 4.0 dataset annotations. Paper text retains its original rights. · Original source · Publication terms
Recorded evidence availability
Counts cover question definitions and all saved attempts, including historical attempts. They are evidence-component counts, not scored questions. A zero means no components in that state.
| Component | full | redacted | source link only | not recorded | withheld |
|---|---|---|---|---|---|
| answer | 868 | 0 | 0 | 327 | 0 |
| artifacts | 0 | 0 | 0 | 1195 | 0 |
| checks | 868 | 0 | 0 | 327 | 0 |
| context | 0 | 0 | 8 | 1187 | 0 |
| input | 0 | 0 | 809 | 386 | 0 |
| prompt | 8 | 0 | 868 | 319 | 0 |
| reasoning | 417 | 0 | 0 | 778 | 0 |
| reference | 8 | 0 | 0 | 1187 | 0 |
| tools | 0 | 0 | 0 | 1195 | 0 |
Full: recorded material is available. Redacted: explicit publication rules removed identified text. Source link only: follow the cited source. Not recorded: no saved component. Withheld: material is not published.
Question bank
1–8 of 8 matching questions · 8 in this version
- Define a Medical Coding Term
The reader asks what ICD-10 codes mean in a paper about automatically coding death certificates. This exercises finding a technical term’s definition and grounding the explanation in the paper rather than giving unrelated medical information.
qa-bbfe7e131ed776c85f2359b748db1325386c1af5 · primary - Identify the Experiment’s Dataset
The reader asks which dataset was used in an experiment involving text annotation tools. This exercises locating the experimental data source and distinguishing it from surrounding details about participants and procedures.
qa-f5913e37039b9517a323ec700b712e898316161b · primary - Find the Training Data Amount
The reader asks how much data was used to train numerical representations of Chinese words and characters. This exercises extracting the relevant quantities from a data-preparation passage and keeping their units clear.
qa-5f7f4a1d4380c118a58ed506c057d3b7aa234c1e · primary - Explain the Caption Evaluation Method
The reader asks how a study evaluates generated humorous image captions. This exercises finding and summarizing the evaluation procedures described in the paper, rather than confusing how quality was assessed with the findings themselves.
qa-477da8d997ff87400c6aad19dcc74f8998bc89c3 · primary - Identify the Training Dataset
The reader asks which dataset the authors use to train their model in a study involving gender bias. This exercises combining details from the paper’s data description into a concise, source-backed account.
qa-ab975efc916c34f55e1144b1d28e7dfdc257e371 · primary - Identify Extracted Text Features
The reader asks what information is extracted from text for a classification system. This exercises locating the feature-extraction description and distinguishing the model’s input representation from the categories it predicts.
qa-915e4d0b3cb03789a20380ead961d473cb95bfc3 · primary - Identify the Financial Word Representation
The reader asks which method the authors use to represent words numerically for financial text. This exercises identifying the method in the paper and separating its name from details about its training material.
qa-9c529bd3f7565b2178a79aae01c98c90f9d372ad · primary - Find the Propaganda Categories
The reader asks which propaganda types are described in a news-annotation study. This exercises extracting the categories supported by the text while recognizing that the available excerpt gives examples rather than a complete list.
qa-37b972a3afae04193411dc569f672d802c16ad71 · primary