About EQ Lab

How it works

A tested model holds an eight-turn conversation with a simulated person. Judges compare two models on the same persona in both orders, across six dimensions. The partner is anthropic/claude-haiku-5.5 via OpenRouter. Both judges use subscriptions: Claude judge is Claude Code Opus 5.5 at low effort; Codex judge is Codex GPT-6.1 Sol at low effort.

The plan covers 24 personas. Ratings use the upstream soft Bradley–Terry solver. When at least two anchors are rated, a least-squares fit maps the ratings to the EQ-Bench scale. Intervals are bootstrapped; abilities are relative to the models in this run.

Conversational style

A separate pointwise pass uses the upstream traits rubric on each conversation with both judges. These are style tendencies, not quality scores: higher is not better. They do not contribute to the rating. Model intervals bootstrap conversations, averaging judges within each conversation.

Differences from upstream

• Partner: a cheaper simulated person than upstream's Gemini 3.1 Pro (chosen by a recorded audition). Conversations are therefore not interchangeable with the published EQ-Bench 4 transcripts; agreement with the leaderboard is checked through anchor models instead.
• Judges use subscription clients. Every judge is assigned every comparison in both orders. CLI temperature and token limits cannot be set; effort is set.
• Judge output is constrained with the CLI JSON-schema option; the upstream prompt text is unchanged. A short fixed system text replaces each client's default agent instructions.
• Matchups: each model meets four opponents per persona (two seeded random rings), not upstream's rating-adaptive opponent selection.
• Local models use the conversation-studio launch settings (16K context, studio samplers, 2048-token thinking budget) with upstream temperature 0.7; the reply limit is upstream's 10000 or what fits in 16K.
• Transport: at most one automatic retry for clearly recoverable failures (upstream: 7 HTTP retries). Ambiguous paid timeouts are never retried automatically.
• A tested-model length cutoff stops the scenario as output_truncated (upstream logs a warning and continues).
• Ministral 14B and Mistral Small 4: their chat templates refuse upstream's assistant-first history from turn 2, so upstream's own turn-1 opening message is kept at the start of their history. They see nothing new.
• Respondent codes keep the upstream format but are seeded from the judgment ID, so resume is reproducible.
• Codex tested models use ChatGPT subscription login, not API billing; user/project instructions and tool-capable execution features are disabled.
• Codex still exposes exec/wait utility tool definitions. Any tool attempt invalidates the conversation. Live exec/resume audit found no skill, agent, collaboration, environment, permissions or app instruction blocks.
• Codex tested models retain the opening message and turn-1 system instructions throughout native resume; upstream normally rebuilds current_turn each turn.
• Codex CLI cannot set temperature, seed or a hard output-token limit; model and reasoning effort are pinned.
• Claude Code uses a subscription session rather than the API; customizations, tools, MCP and slash commands are disabled with Claude Code safe mode.
• Native resume retains the opening message and initial system prompt across turns, rather than rebuilding turn-specific instructions.
• Claude Code does not expose temperature, seed or a hard output-token limit.
• Nemotron 3.5 Super VL 120B-A12B (API, Prime Intellect NVFP4) replaces the unstarted local Nemotron 3.5 Super VL 120B-A12B UD-Q4_K_XL slot. This is an API model variant, not a measurement of the original GGUF/runtime. API reasoning and serving settings can differ from local settings.
• GPT-OSS 120B stock (API, Near AI FP4) replaces the unstarted local GPT-OSS 120B stock Q4 slot. This is an API model variant, not a measurement of the original GGUF/runtime. API reasoning and serving settings can differ from local settings.
• Mistral Small 4 119B (API, Mistral) replaces the unstarted local Mistral Small 4 119B UD-Q4_K_XL slot. This is an API model variant, not a measurement of the original GGUF/runtime. API reasoning and serving settings can differ from local settings.

Attribution and licence

Uses EQ-Bench 4, commit 93dcb7f5d430433ad49d501e672a98bcff2c9678, dataset v0.17. Licence: BSD-3-Clause with Commons Clause.