ObviousBenchGPQA.ai

How the clean
leaderboard is built

A deterministic evidence pipeline turns public evaluation artifacts and carefully labelled independent runs into one auditable ranking.

Results 0.1.14Data through Oct 8, 2026

Two linked methods

The paper decides what belongs. This site decides what can be ranked.

PaperAudit the answer key

Determines which questions remain in the benchmark membership.

Read the current preprint (opens in a new tab)
LeaderboardBuild the ranking

Admits evidence, applies Clean membership, and computes paired scores and uncertainty.

Continue to the pipeline ↓

1 Pipeline

  1. 1Resolve source roster
  2. 2Validate artifacts
  3. 3Recover only when rules permit
  4. 4Add complete native evidence
  5. 5Apply Clean membership
  6. 6Compute exact scores
  7. 7Quantify uncertainty
  8. 8Export safe release bundle

Public export boundary: raw evidence stays outside the website bundle.

2 Ranking contract

Included

  • Every fully accepted configuration with all 189 retained questions
  • All accepted mean@N protocols in one combined ranking
  • Exact rational scores; exact ties share standard competition rank and skip the occupied places
  • Repeat count and evidence class remain visible

Never ranked

  • Missing retained-question coverage
  • Incomplete attempts without a valid uniform subset
  • Unresolved score-source conflict
  • Evidence that fails privacy or provenance validation
Clean accuracy = correct attempts across retained questions / (189 × attempts per question)

Combined rank is not normalized for inference compute.

3 Paired cleaning comparison

Originalsame evaluated attempt universe
−
9 removed questionssame source and route
=
Cleanpaired delta

Never subtract a reconstructed Clean score from an unrelated published aggregate.

4 Uncertainty

  • Resample Record IDs, not attempts.
  • One synchronized retained-ID draw for every configuration.
  • One shared retained plus excluded draw for paired deltas.
  • Fixed seed 20260817.
  • 20,000 replicates.
  • Clean, paired-delta, and rank 95% intervals.
Configuration Asame Record-ID drawscore + interval
Configuration Bsame Record-ID drawscore + interval
Configuration Csame Record-ID drawscore + interval
Configuration Dsame Record-ID drawscore + interval

Independent per-configuration resampling is rejected because it destroys paired covariance.

5 Evidence admission

  1. Complete 198-question evidence?No → unranked gap
  2. Exact source/configuration/protocol match?No → compatibility or recovery review
  3. Complete uniform repeat subset?Yes → derived, separately labelled
  4. Exact failed-slot recovery?Yes → targeted composite
  5. Complete native all-198 run?Yes → complete native primary or native transport composite
  6. Otherwisewithheld

Verified provider refusals without an answer count as incorrect. Those outcomes are preserved and are not retried for a better answer.

6 Reproducibility and privacy

Reproduce

  • Locked Python and Node environments
  • JSON Schemas for public artifacts
  • Immutable release manifest
  • Deterministic timestamps and ordering
  • Checksums for every exported file

Protected

Question text, options, keys, completions, reasoning, raw evaluation artifacts, private evidence locations, and per-configuration-by-question matrices.

7 Cost

Cost is USD for the full 189-question Clean run, one attempt each. The leaderboard and table explorer display two decimal places; exact amounts remain available in the Cost download. It applies the dated Standard model price to recorded usage from the scored attempts; retries and unscored attempts are excluded. The execution route and collection discount do not change the model comparison price.

Included

  • 252 of 268 configurations with complete priceable usage
  • Input, cache, text output, and reasoning components when reported
  • One 189-question pass shown separately from the total across repeated attempts
  • Dated schedules and explicitly reviewed model-reference choices

Unavailable

  • 16 configurations with missing usage, inconsistent counters, or no reviewed price
  • No zero placeholder for an unavailable Cost
  • No same-question donor substitutions in this release
  • No claim that the comparison reconstructs a provider invoice

The leaderboard, table explorer, and Cost–score frontier use the normalized one-pass amount because mean@N is repeated measurement of per-attempt accuracy, rather than a voting or best-of-N serving policy. Arithmetic uses npm runcost version 0.2.1 and prices dated 2026-10-10. The Cost file is checksum-bound to this exact results release.

Explore Cost–score frontier

Continue exploring

Leaderboard Data & downloads