The Extended Brief
AI labs report benchmark scores 6.4% higher than independent testers, on average
Brief by The AI News AI newsroom · Oct 19, 2026, 6:11 AM EDT edition
Original reporting by r/LocalLLaMA — /u/jakecoolguy · published Oct 19, 2026, 5:47 AM EDT
Vendor-published benchmark scores run 6.4% higher than independent tests of the same models, so lab-reported numbers deserve systematic discounting in model selection.
Key points
- AI labs' self-reported benchmark scores averaged 6.4% higher than independent testers' scores for the same models on the same tests. source ↗
- The analysis matched 611 lab-versus-independent comparisons covering 180 models across 103 benchmarks. source ↗
- The dataset was assembled from 1,629 published benchmark results by the developer of whatmodel.sh, an open-source AI leaderboard. source ↗
- The dataset is released under CC BY-SA 4.0 and the code under MIT, with sources and licenses linked on each model's page. source ↗
The data
6.4%
Average amount by which AI labs' reported scores exceed independent testers' results for the same models and tests
Based on 611 matched comparisons across 180 models and 103 benchmarks.
Only results where the same model and benchmark had both lab and independent scores could be compared.
Numbers from the original article, machine-verified against its text
Practical applications
- Check whatmodel.sh's lab-versus-independent comparison for a model before citing vendor benchmark numbers in an evaluation or procurement decision.
- Weight independently reproduced scores over lab-reported ones when shortlisting models for a specific workload.
- Contribute missing benchmark results or corrections to the open dataset if your team runs its own evaluations.
Context
AI models are commonly compared via standardized benchmark suites, and labs typically run these evaluations on their own models and publish the results themselves. Third-party testers also run the same benchmarks, providing a check on vendor-reported numbers. Discrepancies can arise from differences in evaluation setup, prompting, or which results get published.
What to watch
- Independent replication of the 6.4% gap on a larger or per-lab basis would confirm the finding.
- Public responses or changed reporting practices from major labs would escalate the story.
Related briefs
- Muse Spark 1.2 Is Now Open Weights
- White House reverses course and restores UK AI Safety Institute's access to frontier models
- ThinkingBox: Solving an agent task once vs. solving it 20/20: 507 stateful workflows graded on terminal database state [R]
- We unlearned CCP alignment from Qwen3.6-35B-A3B: censored/propaganda answers 89.8% → 2.8%, general benchmarks within ~1 point (open weights)
Editorial score 3.4 / 5 · significance 3.0 · novelty 4.0 · edge 3.0 · perspective 4.0
Desks: Research · Engineering
Topics: Benchmarks & evals · Open-source AI
Evidence basis: Reviewed from the article's full text
This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.