90% threshold
25 Original 198 configurations
45 Clean 189 configurations
+20 cross upward
What cleaning changes
Nine answer-key and eligibility defects were removed from GPQA Diamond.
The effect is measurable, configuration-specific, and large enough to change interpretation.
Results 0.1.0Data through Aug 17, 2026
235of 243 improved
+2.43 ppMedian change
+1.05–+2.98 ppMiddle 50%
8decreased
Clean score versus original score
Points above the diagonal improved after the nine exclusions.
Chart loads from the release bundle when visible.
View chart data table
| Configuration | Original 198 | Clean 189 | Change |
|---|---|---|---|
| GPT 5.5 Pro pre-releaseOpenAI · mean@7 · xhigh | 93.94% | 97.35% | +3.42 pp |
| Gemini 3.6 FlashGoogle · mean@1 · high | 93.94% | 97.35% | +3.42 pp |
| GPT 5.5 pre-releaseOpenAI · mean@8 · xhigh | 94.00% | 97.22% | +3.22 pp |
| Gemini 3.1 Pro PreviewGoogle · mean@8 · default | 94.51% | 97.02% | +2.52 pp |
| GPT 5.6 SolOpenAI · mean@1 · max | 93.43% | 96.83% | +3.39 pp |
| GPT 5.4OpenAI · mean@8 · xhigh | 93.24% | 96.69% | +3.45 pp |
| Gemini 3.1 Pro PreviewGoogle · mean@1 · high | 94.44% | 96.30% | +1.85 pp |
| Gemini 3.5 FlashGoogle · mean@8 · high | 92.80% | 96.10% | +3.29 pp |
| Grok 4.6xAI · mean@1 · high | 92.42% | 95.77% | +3.34 pp |
| DeepSeek V4 FlashDeepSeek · mean@1 · max | 92.93% | 95.77% | +2.84 pp |
| Gemini 3 Pro PreviewGoogle · mean@8 · default | 92.61% | 95.50% | +2.89 pp |
| DeepSeek V4 ProDeepSeek · mean@1 · max | 92.42% | 95.24% | +2.81 pp |
| Claude Opus 5Anthropic · mean@1 · none | 92.93% | 95.24% | +2.31 pp |
| GPT 5.2OpenAI · mean@8 · xhigh | 91.41% | 95.04% | +3.63 pp |
| GPT 5.6 LunaOpenAI · mean@1 · max | 92.42% | 94.71% | +2.28 pp |
| Qwen3.7 MaxQwen · mean@1 · max | 90.91% | 94.71% | +3.80 pp |
| Kimi K3Moonshot AI · mean@1 · high | 91.92% | 94.71% | +2.79 pp |
| GPT 5.5OpenAI · mean@8 · low | 90.66% | 93.85% | +3.19 pp |
| Kimi K2.6Moonshot AI · mean@6 · default | 90.91% | 93.83% | +2.92 pp |
| GPT 5.6 TerraOpenAI · mean@1 · max | 90.91% | 93.65% | +2.74 pp |
| MiniMax M3MiniMax · mean@1 · default | 90.91% | 93.65% | +2.74 pp |
| GPT 5.4OpenAI · mean@1 · high | 89.90% | 93.65% | +3.75 pp |
| DeepSeek V4 FlashDeepSeek · mean@1 · high | 90.40% | 93.65% | +3.25 pp |
| Claude Opus 4.7Anthropic · mean@8 · xhigh | 90.15% | 93.65% | +3.50 pp |
| GPT 5.6 SolOpenAI · mean@1 · low | 89.90% | 93.65% | +3.75 pp |
| Claude Opus 4.6Anthropic · mean@8 · default · 32K context | 90.53% | 93.65% | +3.12 pp |
| GLM 5.1Z.ai · mean@1 · default | 89.90% | 93.12% | +3.22 pp |
| DeepSeek V4 ProDeepSeek · mean@1 · high | 90.91% | 93.12% | +2.21 pp |
| Gemini 3.5 FlashGoogle · mean@1 · low | 88.89% | 92.59% | +3.70 pp |
| GPT 5.4OpenAI · mean@1 · medium | 88.89% | 92.06% | +3.17 pp |
| Gemini 3 Flash PreviewGoogle · mean@1 · high | 89.39% | 92.06% | +2.67 pp |
| Claude Opus 4.6Anthropic · mean@7 · default · 64K context | 88.53% | 91.69% | +3.16 pp |
| Claude Opus 4.6Anthropic · mean@1 · max | 88.38% | 91.53% | +3.15 pp |
| GPT 5.2OpenAI · mean@8 · high | 88.19% | 91.40% | +3.21 pp |
| Claude Opus 5Anthropic · mean@1 · low | 87.88% | 91.01% | +3.13 pp |
| GLM 5Z.ai · mean@1 · default | 87.88% | 91.01% | +3.13 pp |
| Qwen3.6 PlusQwen · mean@1 · default | 88.38% | 91.01% | +2.62 pp |
| Kimi K2.7 CodeMoonshot AI · mean@1 · default | 87.88% | 91.01% | +3.13 pp |
| Claude Opus 4.8Anthropic · mean@1 · low | 88.38% | 91.01% | +2.62 pp |
| GPT 5.2OpenAI · mean@8 · medium | 87.88% | 90.94% | +3.06 pp |
| Kimi K2.5Moonshot AI · mean@8 · default | 87.56% | 90.48% | +2.91 pp |
| Qwen3.7 PlusQwen · mean@1 · default | 87.88% | 90.48% | +2.60 pp |
| GPT 5.6 TerraOpenAI · mean@1 · low | 87.37% | 90.48% | +3.10 pp |
| Qwen3.6 Max PreviewQwen · mean@1 · max | 87.37% | 90.48% | +3.10 pp |
| GPT 5.1OpenAI · mean@8 · high | 87.63% | 90.41% | +2.78 pp |
| Gemini 3.6 FlashGoogle · mean@1 · low | 86.36% | 89.95% | +3.58 pp |
| GPT 5.4 miniOpenAI · mean@1 · xhigh | 86.87% | 89.95% | +3.08 pp |
| Gemini 3.5 FlashGoogle · mean@1 · minimal | 86.36% | 89.95% | +3.58 pp |
| Grok 4.3xAI · mean@1 · high | 86.87% | 89.95% | +3.08 pp |
| Claude Sonnet 4.6Anthropic · mean@8 · default · 32K context | 87.37% | 89.68% | +2.31 pp |
| GPT 5OpenAI · mean@8 · high | 86.17% | 89.55% | +3.38 pp |
| Claude Opus 4.7Anthropic · mean@1 · max | 86.36% | 89.42% | +3.05 pp |
| Claude Fable 5Anthropic · mean@1 · max | 85.86% | 89.42% | +3.56 pp |
| Qwen3.5 397B A17BQwen · mean@1 · default | 85.86% | 89.42% | +3.56 pp |
| Qwen3.5 397B A17BQwen · mean@1 · none | 86.36% | 89.42% | +3.05 pp |
| Qwen3.6 27BQwen · mean@1 · default | 85.86% | 88.89% | +3.03 pp |
| Claude Opus 4.5Anthropic · mean@8 · default · 32K context | 86.05% | 88.76% | +2.71 pp |
| GPT 5OpenAI · mean@7 · medium | 85.43% | 88.74% | +3.31 pp |
| Claude Opus 4.5Anthropic · mean@8 · default · 16K context | 85.48% | 88.69% | +3.21 pp |
| Gemini 3.6 FlashGoogle · mean@1 · minimal | 85.86% | 88.36% | +2.50 pp |
| Qwen3.5 PlusQwen · mean@1 · default | 84.85% | 88.36% | +3.51 pp |
| Kimi K3Moonshot AI · mean@1 · low | 84.85% | 88.36% | +3.51 pp |
| Gemini 2.5 ProGoogle · mean@8 · default | 85.29% | 88.29% | +3.00 pp |
| Gemini 2.5 Pro PreviewGoogle · mean@1 · default | 84.85% | 87.83% | +2.98 pp |
| Claude Opus 4.8Anthropic · mean@1 · none | 85.35% | 87.83% | +2.48 pp |
| Qwen3.6 27BQwen · mean@1 · none | 84.85% | 87.83% | +2.98 pp |
| GPT 5.4OpenAI · mean@1 · low | 84.85% | 87.83% | +2.98 pp |
| Qwen3.6 35B A3BQwen · mean@1 · none | 84.85% | 87.83% | +2.98 pp |
| GPT 5.1OpenAI · mean@8 · medium | 85.04% | 87.76% | +2.73 pp |
| Gemini 2.5 Pro ExpGoogle · mean@1 · default | 83.84% | 87.30% | +3.46 pp |
| Qwen3.6 35B A3BQwen · mean@1 · default | 83.84% | 86.77% | +2.93 pp |
| Claude Fable 5Anthropic · mean@1 · high | 83.33% | 86.77% | +3.44 pp |
| Claude Sonnet 4.6Anthropic · mean@1 · high | 83.33% | 86.77% | +3.44 pp |
| GPT 5.4 miniOpenAI · mean@8 · high | 83.59% | 86.77% | +3.19 pp |
| Kimi K2 Thinking TurboMoonshot AI · mean@8 · default | 84.22% | 86.51% | +2.29 pp |
| Claude Sonnet 4.6Anthropic · mean@1 · medium | 83.33% | 86.24% | +2.91 pp |
| GPT 5.6 SolOpenAI · mean@1 · none | 82.83% | 86.24% | +3.42 pp |
| DeepSeek ReasonerDeepSeek · mean@7 · default | 83.33% | 86.17% | +2.83 pp |
| Gemini 3 Flash PreviewGoogle · mean@8 · default | 83.21% | 85.91% | +2.71 pp |
| Claude Sonnet 4.5Anthropic · mean@1 · default · 59K context | 82.32% | 85.71% | +3.39 pp |
| Qwen3.5 FlashQwen · mean@1 · default | 82.32% | 85.71% | +3.39 pp |
| Qwen3.6 FlashQwen · mean@1 · default | 83.33% | 85.71% | +2.38 pp |
| Gemini 3.5 Flash LiteGoogle · mean@1 · high | 83.33% | 85.71% | +2.38 pp |
| GPT 5.2OpenAI · mean@8 · low | 82.70% | 85.65% | +2.95 pp |
| GPT 5.6 LunaOpenAI · mean@1 · low | 82.32% | 85.19% | +2.86 pp |
| Qwen3.7 FlashQwen · mean@1 · default | 82.32% | 85.19% | +2.86 pp |
| o3OpenAI · mean@8 · high | 81.82% | 85.05% | +3.23 pp |
| Claude Sonnet 4.5Anthropic · mean@1 · default · 32K context | 81.82% | 84.66% | +2.84 pp |
| Gemini 3.1 Flash LiteGoogle · mean@1 · high | 81.82% | 84.66% | +2.84 pp |
| Qwen3.7 FlashQwen · mean@1 · none | 80.81% | 84.13% | +3.32 pp |
| Qwen3.7 PlusQwen · mean@1 · none | 81.82% | 84.13% | +2.31 pp |
| Claude Sonnet 5Anthropic · mean@1 · max | 80.30% | 83.60% | +3.29 pp |
| o3OpenAI · mean@1 · medium | 80.81% | 83.60% | +2.79 pp |
| GLM 4.7Z.ai · mean@1 · default | 80.30% | 83.07% | +2.77 pp |
| o4 miniOpenAI · mean@8 · high | 79.61% | 82.94% | +3.33 pp |
| Claude Opus 4.5Anthropic · mean@8 · default | 80.68% | 82.74% | +2.06 pp |
| o3OpenAI · mean@1 · low | 79.80% | 82.54% | +2.74 pp |
| Claude Sonnet 4.5Anthropic · mean@1 · default · 16K context | 78.79% | 82.54% | +3.75 pp |
| Qwen3 235B A22B ThinkingQwen · mean@2 · default | 80.05% | 82.28% | +2.22 pp |
| Claude Fable 5Anthropic · mean@1 · low | 78.79% | 82.01% | +3.22 pp |
| Claude Sonnet 4.6Anthropic · mean@1 · max | 78.79% | 81.48% | +2.69 pp |
| GPT 5.4 nanoOpenAI · mean@8 · high | 78.47% | 81.02% | +2.55 pp |
| Claude Sonnet 4Anthropic · mean@1 · default · 32K context | 78.28% | 80.42% | +2.14 pp |
| Claude Opus 4.1Anthropic · mean@1 · default · 16K context | 77.27% | 80.42% | +3.15 pp |
| o4 miniOpenAI · mean@1 · medium | 77.78% | 80.42% | +2.65 pp |
| Claude 3.7 SonnetAnthropic · mean@1 · default · 16K context | 76.77% | 79.89% | +3.13 pp |
| Claude 3.7 SonnetAnthropic · mean@1 · default · 32K context | 76.77% | 79.89% | +3.13 pp |
| Claude Opus 4.1Anthropic · mean@1 · default · 27K context | 76.77% | 79.89% | +3.13 pp |
| GPT 5.5OpenAI · mean@1 · none | 77.27% | 79.89% | +2.62 pp |
| o3 miniOpenAI · mean@8 · high | 77.02% | 79.56% | +2.54 pp |
| o1OpenAI · mean@1 · high | 76.77% | 79.37% | +2.60 pp |
| Grok 3 mini BetaxAI · mean@1 · low | 76.26% | 79.37% | +3.10 pp |
| GPT 5.6 TerraOpenAI · mean@1 · none | 77.27% | 79.37% | +2.09 pp |
| DeepSeek R1DeepSeek · mean@8 · default | 76.33% | 78.84% | +2.51 pp |
| Claude Opus 4Anthropic · mean@1 · default · 16K context | 76.26% | 78.84% | +2.57 pp |
| Claude Sonnet 4Anthropic · mean@1 · default · 16K context | 75.76% | 78.31% | +2.55 pp |
| GPT OSS 120BOpenAI · mean@2 · high | 75.76% | 78.31% | +2.55 pp |
| Gemma 4 31B ITGoogle · mean@1 · minimal | 75.76% | 78.31% | +2.55 pp |
| Gemini 3.5 Flash LiteGoogle · mean@1 · low | 75.76% | 78.31% | +2.55 pp |
| GPT 5 miniOpenAI · mean@8 · high | 75.00% | 77.84% | +2.84 pp |
| o4 miniOpenAI · mean@1 · low | 75.25% | 77.78% | +2.53 pp |
| o1OpenAI · mean@1 · medium | 75.76% | 77.25% | +1.49 pp |
| GPT 5.4OpenAI · mean@1 · none | 74.75% | 77.25% | +2.50 pp |
| o1OpenAI · mean@1 · low | 74.24% | 76.72% | +2.48 pp |
| Gemini 3.1 Flash LiteGoogle · mean@1 · low | 74.24% | 76.72% | +2.48 pp |
| Gemini 3.5 Flash LiteGoogle · mean@1 · minimal | 74.24% | 76.72% | +2.48 pp |
| o3 miniOpenAI · mean@16 · medium | 74.27% | 76.52% | +2.25 pp |
| GPT 5.2OpenAI · mean@1 · none | 73.23% | 76.19% | +2.96 pp |
| Claude Opus 4.1Anthropic · mean@1 · default | 73.23% | 75.66% | +2.43 pp |
| Gemini 3.1 Flash LiteGoogle · mean@1 · minimal | 73.74% | 75.66% | +1.92 pp |
| DeepSeek V4 ProDeepSeek · mean@1 · none | 73.23% | 75.66% | +2.43 pp |
| Claude Sonnet 4.5Anthropic · mean@1 · default | 73.74% | 75.13% | +1.39 pp |
| Qwen3 MaxQwen · mean@8 · max | 72.60% | 75.13% | +2.53 pp |
| GPT 5OpenAI · mean@1 · minimal | 71.72% | 74.60% | +2.89 pp |
| GPT 5 miniOpenAI · mean@8 · medium | 71.65% | 74.54% | +2.88 pp |
| GPT 5.4 nanoOpenAI · mean@1 · low | 72.22% | 74.07% | +1.85 pp |
| GPT 5 miniOpenAI · mean@1 · minimal | 71.72% | 74.07% | +2.36 pp |
| DeepSeek ChatDeepSeek · mean@1 · default | 71.21% | 73.54% | +2.33 pp |
| Claude Haiku 4.5Anthropic · mean@1 · default · 32K context | 71.21% | 73.54% | +2.33 pp |
| Qwen3 235B A22BQwen · mean@8 · default | 70.71% | 73.35% | +2.64 pp |
| GLM 5.2Z.ai · mean@1 · none | 71.21% | 72.49% | +1.27 pp |
| GPT 5 nanoOpenAI · mean@8 · high | 69.44% | 72.16% | +2.71 pp |
| Claude Opus 4Anthropic · mean@1 · default | 69.19% | 71.43% | +2.24 pp |
| o3 miniOpenAI · mean@1 · low | 68.18% | 70.90% | +2.72 pp |
| GPT 4.5 PreviewOpenAI · mean@1 · default | 68.69% | 70.90% | +2.21 pp |
| GPT 5 nanoOpenAI · mean@8 · medium | 67.42% | 70.17% | +2.75 pp |
| DeepSeek V3DeepSeek · mean@8 · default | 67.61% | 69.64% | +2.03 pp |
| GPT 4.1OpenAI · mean@8 · default | 66.92% | 69.31% | +2.39 pp |
| GPT 5.1OpenAI · mean@1 · none | 66.67% | 69.31% | +2.65 pp |
| Claude Sonnet 4Anthropic · mean@1 · default | 66.67% | 68.78% | +2.12 pp |
| Gemini 2.5 Pro PreviewGoogle · mean@1 · default | 66.67% | 68.78% | +2.12 pp |
| Llama 4 Maverick 17B 128E Instruct FP8Meta · mean@8 · default | 66.98% | 68.25% | +1.27 pp |
| GPT 4.1 miniOpenAI · mean@8 · default | 65.85% | 68.19% | +2.34 pp |
| Claude 3.7 SonnetAnthropic · mean@8 · default | 66.04% | 68.19% | +2.15 pp |
| QwQ PlusQwen · mean@8 · default | 65.40% | 67.26% | +1.86 pp |
| Gemini 2.0 Pro ExpGoogle · mean@1 · default | 65.66% | 67.20% | +1.54 pp |
| GPT 5.4 miniOpenAI · mean@1 · none | 64.14% | 66.14% | +2.00 pp |
| GPT 5.6 LunaOpenAI · mean@1 · none | 63.64% | 65.61% | +1.97 pp |
| Gemini 2.0 FlashGoogle · mean@16 · default | 64.14% | 65.54% | +1.40 pp |
| o1 miniOpenAI · mean@8 · high | 62.37% | 64.29% | +1.91 pp |
| Claude Haiku 4.5Anthropic · mean@8 · default | 60.48% | 61.77% | +1.29 pp |
| o1 miniOpenAI · mean@16 · medium | 59.50% | 61.64% | +2.14 pp |
| Mistral MediumMistral · mean@8 · medium | 59.53% | 60.98% | +1.45 pp |
| GPT 5 nanoOpenAI · mean@1 · low | 57.58% | 59.26% | +1.68 pp |
| Gemini 2.0 Flash Thinking ExpGoogle · mean@1 · default | 57.07% | 59.26% | +2.19 pp |
| Gemini 1.5 ProGoogle · mean@16 · default | 57.23% | 58.63% | +1.40 pp |
| DeepSeek V3DeepSeek · mean@16 · default | 56.53% | 57.84% | +1.30 pp |
| DeepSeek R1 Distill Llama 70BDeepSeek · mean@8 · default | 55.74% | 57.80% | +2.06 pp |
| Magistral SmallMistral · mean@1 · default | 56.06% | 57.14% | +1.08 pp |
| GPT 5.4 nanoOpenAI · mean@1 · none | 55.56% | 57.14% | +1.59 pp |
| Qwen MaxQwen · mean@8 · max | 56.12% | 57.08% | +0.95 pp |
| Phi 4Microsoft · mean@16 · default | 56.06% | 56.94% | +0.88 pp |
| Claude 3.5 SonnetAnthropic · mean@16 · default | 55.30% | 56.45% | +1.15 pp |
| Claude 3.5 SonnetAnthropic · mean@16 · default | 54.04% | 55.69% | +1.65 pp |
| Grok 2xAI · mean@16 · default | 53.79% | 54.07% | +0.28 pp |
| Llama 4 Scout 17B 16E InstructMeta · mean@8 · default | 51.83% | 52.65% | +0.81 pp |
| o1 PreviewOpenAI · mean@16 · default | 50.32% | 52.55% | +2.23 pp |
| Llama 3.1 405B InstructMeta · mean@16 · default | 50.92% | 52.51% | +1.60 pp |
| Mistral LargeMistral · mean@8 · default | 51.33% | 51.98% | +0.66 pp |
| GPT 4oOpenAI · mean@16 · default | 49.21% | 50.46% | +1.25 pp |
| Qwen2.5 72B InstructQwen · mean@16 · default | 49.15% | 50.03% | +0.89 pp |
| GPT 4oOpenAI · mean@16 · default | 48.90% | 49.97% | +1.07 pp |
| GPT 4.1 nanoOpenAI · mean@8 · default | 48.93% | 49.93% | +1.01 pp |
| Mistral LargeMistral · mean@16 · default | 49.02% | 49.80% | +0.78 pp |
| GPT 4oOpenAI · mean@16 · default | 47.89% | 49.14% | +1.26 pp |
| GPT 5 nanoOpenAI · mean@1 · minimal | 48.48% | 48.68% | +0.19 pp |
| Qwen PlusQwen · mean@8 · default | 48.11% | 48.35% | +0.24 pp |
| Llama 3.3 70B InstructMeta · mean@16 · default | 47.44% | 48.15% | +0.70 pp |
| GPT OSS 20BOpenAI · mean@1 · high | 45.96% | 48.15% | +2.19 pp |
| Gemini 1.5 FlashGoogle · mean@16 · default | 47.32% | 48.05% | +0.73 pp |
| Mistral SmallMistral · mean@8 · default | 47.47% | 48.02% | +0.54 pp |
| Claude 3 OpusAnthropic · mean@16 · default | 47.16% | 47.88% | +0.72 pp |
| GPT 4 TurboOpenAI · mean@8 · default | 46.59% | 46.96% | +0.37 pp |
| Llama 3.1 Tulu 3 70B DPOAllenAI · mean@16 · default | 46.28% | 46.96% | +0.68 pp |
| Qwen2.5 32B InstructQwen · mean@16 · default | 46.09% | 46.63% | +0.54 pp |
| DeepSeek R1 Distill Qwen 14BDeepSeek · mean@8 · default | 44.70% | 46.49% | +1.80 pp |
| Gemini 1.5 ProGoogle · mean@16 · default | 45.86% | 46.43% | +0.56 pp |
| Mistral SmallMistral · mean@16 · default | 45.30% | 46.06% | +0.77 pp |
| Gemma 3 27B ITGoogle · mean@1 · default | 43.94% | 44.97% | +1.03 pp |
| Llama 3.1 70B InstructMeta · mean@16 · default | 44.19% | 44.87% | +0.68 pp |
| WizardLM 2 8x22BMicrosoft · mean@16 · default | 43.43% | 44.28% | +0.84 pp |
| GPT 4 PreviewOpenAI · mean@16 · default | 42.36% | 43.42% | +1.06 pp |
| GPT 4 PreviewOpenAI · mean@16 · default | 42.27% | 43.39% | +1.12 pp |
| Qwen TurboQwen · mean@8 · default | 41.79% | 41.80% | +0.01 pp |
| Llama 3.2 90B Vision InstructMeta · mean@16 · default | 41.04% | 41.63% | +0.60 pp |
| Llama 3 70B InstructMeta · mean@16 · default | 40.56% | 41.57% | +1.01 pp |
| Claude 3 SonnetAnthropic · mean@16 · default | 40.59% | 41.30% | +0.71 pp |
| Qwen2 72B InstructQwen · mean@16 · default | 40.78% | 40.91% | +0.12 pp |
| Gemini 1.5 FlashGoogle · mean@16 · default | 40.37% | 40.38% | 0.00 pp |
| Mistral LargeMistral · mean@16 · default | 38.76% | 39.42% | +0.66 pp |
| Claude 3.5 HaikuAnthropic · mean@8 · default | 38.13% | 38.96% | +0.82 pp |
| GPT 4o miniOpenAI · mean@16 · default | 37.72% | 38.82% | +1.10 pp |
| Hermes 2 Theta Llama 3 70BNous Research · mean@16 · default | 37.47% | 38.26% | +0.79 pp |
| Claude 3 HaikuAnthropic · mean@16 · default | 36.30% | 36.38% | +0.08 pp |
| Gemma 2 27B ITGoogle · mean@16 · default | 36.49% | 36.24% | -0.25 pp |
| GPT 4OpenAI · mean@8 · default | 35.73% | 36.04% | +0.31 pp |
| Gemini 1.0 ProGoogle · mean@16 · default | 33.96% | 34.59% | +0.63 pp |
| Open Mixtral 8x22BMistral · mean@16 · default | 34.06% | 34.26% | +0.20 pp |
| Claude 2.0Anthropic · mean@16 · default | 34.66% | 34.19% | -0.47 pp |
| Eurus 2 7B PRIMEPRIME-RL · mean@16 · default | 33.90% | 34.16% | +0.26 pp |
| Claude 2.1Anthropic · mean@16 · default | 32.95% | 33.47% | +0.51 pp |
| DBRX InstructDatabricks · mean@16 · default | 32.89% | 33.40% | +0.51 pp |
| Gemini 1.5 Flash 8BGoogle · mean@8 · default | 32.95% | 33.00% | +0.05 pp |
| Yi 1.5 34B Chat01.AI · mean@16 · default | 31.98% | 32.24% | +0.27 pp |
| GPT 4OpenAI · mean@16 · default | 30.65% | 31.32% | +0.67 pp |
| Mixtral 8x7B Instruct v0.1Mistral · mean@16 · default | 30.59% | 31.25% | +0.66 pp |
| Qwen1.5 32B ChatQwen · mean@16 · default | 30.74% | 31.02% | +0.27 pp |
| Open Mixtral 8x7BMistral · mean@16 · default | 29.83% | 30.29% | +0.46 pp |
| Open Mistral NemoMistral · mean@16 · default | 29.89% | 29.99% | +0.10 pp |
| Qwen1.5 72B ChatQwen · mean@16 · default | 28.82% | 29.50% | +0.68 pp |
| GPT 3.5 TurboOpenAI · mean@16 · default | 28.03% | 27.88% | -0.15 pp |
| Phi 3 Medium 128K InstructMicrosoft · mean@16 · medium · 128K context | 27.59% | 27.58% | -0.01 pp |
| Gemma 2 9B ITGoogle · mean@16 · default | 27.46% | 27.58% | +0.12 pp |
| GPT 3.5 TurboOpenAI · mean@16 · default | 27.18% | 27.31% | +0.14 pp |
| Llama 2 70B Chat HFMeta · mean@16 · default | 26.33% | 27.08% | +0.76 pp |
| Ministral 8BMistral · mean@16 · default | 27.15% | 26.46% | -0.69 pp |
| Llama 3 8B InstructMeta · mean@16 · default | 26.07% | 26.36% | +0.28 pp |
| Llama 3.1 8B InstructMeta · mean@16 · default | 25.95% | 26.22% | +0.28 pp |
| Ministral 3BMistral · mean@16 · default | 25.25% | 24.60% | -0.65 pp |
| DeepSeek LLM 67B ChatDeepSeek · mean@16 · default | 24.62% | 24.54% | -0.08 pp |
| Mistral 7B Instruct v0.3Mistral · mean@16 · default | 15.18% | 15.44% | +0.26 pp |
| Yi 34B Chat01.AI · mean@16 · default | 14.74% | 15.21% | +0.47 pp |
| Open Mistral 7BMistral · mean@16 · default | 13.23% | 13.10% | -0.13 pp |
Distribution of score changes
The median and middle 50% come from all 243 accepted configurations.
Chart loads from the release bundle when visible.
8 decreased; 235 improved. Range -0.69 to +3.80 percentage points.
Threshold interpretation changes
These are benchmark-definition changes, not new model progress.
95% threshold
0 Original 198 configurations
14 Clean 189 configurations
+14 cross upward
Where ranks remain fragile
| Observed rank | Configuration | 95% rank interval | Change |
|---|---|---|---|
| 1 | GPT 5.5 Pro pre-releaseOpenAI · mean@7 · xhigh | 1–10 | +3.42 pp |
| 1 | Gemini 3.6 FlashGoogle · mean@1 · high | 1–14 | +3.42 pp |
| 3 | GPT 5.5 pre-releaseOpenAI · mean@8 · xhigh | 1–10 | +3.22 pp |
| 4 | Gemini 3.1 Pro PreviewGoogle · mean@8 · default | 1–12 | +2.52 pp |
| 5 | GPT 5.6 SolOpenAI · mean@1 · max | 1–17 | +3.39 pp |
| 6 | GPT 5.4OpenAI · mean@8 · xhigh | 1–14 | +3.45 pp |
| 7 | Gemini 3.1 Pro PreviewGoogle · mean@1 · high | 1–23 | +1.85 pp |
| 8 | Gemini 3.5 FlashGoogle · mean@8 · high | 2–20 | +3.29 pp |
Why people should care
- Leaderboard claims can hinge on defective items.Small benchmark flaws can invert or exaggerate differences.
- A Clean score is paired evidence.It is recomputed from the same evaluated attempts after the exclusions.
- Provenance makes disagreement inspectable.Every accepted path remains explicitly labelled.
What this does not claim
- Cleaning does not make GPQA contamination-free.
- Combined rank does not erase protocol differences.
- Targeted recoveries remain labelled composites.
- No private prompts, answers, or reasoning traces are published.