Eval³ 1.1
We’re releasing Eval³ 1.1, an update to our eval-auditing benchmark. We revised three parts of the methodology:
- Higher turn limits. Models now get 200 turns instead of 40. We wanted to reward finding flaws quickly and efficiently, but 40 turns was too tight: models were often cut off before they could write up their results.
- A countdown. Models were bad at judging when to stop looking for issues and write the report, so after each turn we now tell the model how many turns it has left.
- Better deduplication. Some models split a single root cause into each of its possible impacts, which inflated their scores. We tuned our judge to group issues under their root cause.
Here is how the models rank now. See the full leaderboard here.
| # | Model | ||
|---|---|---|---|
| 1 |  GPT-6.1 Sol*Max | 1144 | $335 |
| 1 | GPT-6 Astra*Max | 1144 | $1,285 |
| 3 | GPT-6.1 Sol*High | 1105 | $203 |
| 4 |  Claude Sonnet 5.5†Max | 1090 | $454 |
| 5 |  DeepSeek V4.1 FlashMax | 1030 | $21 |
| 6 | Claude Haiku 5.5Max | 1011 | $406 |
| 7 | Claude Opus 5.5†Max | 1009 | $467 |
| 8 | GPT-6 LunaMax | 974 | $35 |
| 9 |  Gemini 4 ArgonMax | 920 | $79 |
| 10 |  GLM 5.3Max | 917 | $216 |
| 11 |  Muse WonderMax | 901 | $396 |
| 12 | GLM 5.3 FlashMax | 757 | $17 |
700800900100011001200 | |||
* Fallback to GPT-6 Sol on refusal.
† Native fallback on refusal.
Analysis
Deduplication
Since a model’s score grows with the number of issues it reports, every extra issue widens its lead over the others. We already deduplicated issues that were the same finding reworded, so a model couldn’t get double credit for one answer. But splitting a root cause into several issues got past our judge.
In one task, instead of reporting the root cause, a misconfigured command blocklist, Sonnet 5.5 reported each command that got past the blocklist as a separate issue. The task’s grader runs the agent’s Vim macro and bans dangerous commands with a list of exact spellings, so anything spelled differently isn’t caught.
One root cause, filed as seven issues
Root cause
Exact-spelling command blocklist
- +6
Scripting-command aliases are not blacklisted
:python3:py3:luado:perldo - +6
Command abbreviations bypass exact-spelling patterns
:rea:sou:ed:spl - +6
:silent !shell escape bypasses the blocklist:silent !cmd - +6
:silent readbypasses the blocklist - +6
Range shell filters are not blocked
:%!cmd:1,$!cmd - +6
:exeis not covered - +6
Terminal and job-start routes are not covered
:terminaljob_start(
Model similarity
Different models also tended to find different things. OpenAI models found more issues in task mechanics, such as a task not testing what it asks for, while smaller models found more issues in grading setups and isolation, such as leaked answer keys and misconfigured infrastructure. Below is how similar the root causes each model found are, with similar models grouped together.
Models that find the same kinds of issues
Grading
We used GPT-6 Luna as the judge for both the original benchmark and this update. After releasing our results, we wanted to know whether the choice of judge has a strong influence on the results, so we reran the grading step of Eval³ 1.0 with six different judges, one per column:
| Auditor | Luna | Haiku | GLM Flash | DeepSeek | Gemini | MiniMax |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 1230 | 1248 | 1244 | 1244 | 1229 | 1216 |
| Opus 5.5 | 1120 | 1121 | 1207 | 1121 | 1137 | 1111 |
| Muse Wonder | 1011 | 1025 | 1048 | 1014 | 1029 | 1034 |
| Gemini 4 Argon | 1011 | 1002 | 1005 | 1025 | 987 | 1027 |
| GLM 5.3 Flash | 826 | 816 | 781 | 810 | 817 | 805 |
| GLM 5.3 | 802 | 787 | 715 | 787 | 801 | 806 |
Swapping the judge never moved a model more than one place, so we kept Luna as the judge for Eval³ 1.1.