The Extended Brief
I spoke to seven physics experts grappling with the AI revolution

Brief by The AI News AI newsroom · Nov 17, 2026, 10:11 AM EST edition
Original reporting by Understanding AI — Dan Kagan-Kans · published Nov 17, 2026, 9:51 AM EST
Frontier models now solve most unpublished theoretical physics problems that occupy top experts for a day, so labs need contamination-free ways to measure machine reasoning.
Key points
- GPT-6 Astra solved nearly 90% of problems in a new benchmark of unpublished theoretical physics questions. source ↗
- Leading models solved about 60% of the problems in preliminary results Raju presented at a September colloquium. source ↗
- The benchmark uses never-published problems because models trained on the internet may have memorized answers to public ones. source ↗
- Contributing physicists supplied private findings designed to take top human researchers roughly a day to solve. source ↗
- Asked to find where it went wrong using human-written solutions, Astra argued it had not erred. source ↗
The data
Preliminary colloquium results showed about 60%; the completed benchmark reached nearly 90% with GPT-6 Astra.
Numbers from the original article, machine-verified against its text
Practical applications
- Evaluate models on private, never-published problems from your own domain before trusting public benchmark scores, since training data can leak answers.
- Adopt the article's error-check probe: hand the model the reference solution, ask it to locate its mistake, and study whether it corrects or defends its answer.
- If you maintain a benchmark, recruit domain experts to contribute unpublished findings as test items, as Raju's team did with physicists.
Context
Large language models are trained on much of the public internet, so widely discussed problems may already sit in their training data, making public benchmarks an unreliable gauge of reasoning. Testing on problems that have never been published removes that memorization shortcut. Theoretical physics problems that take expert humans about a day rank among the harder reasoning tests available.
What to watch
- Formal publication of the benchmark and full results, which would let other labs attempt to replicate the roughly 90% figure.
- Whether expert review settles the problems where Astra disputed that it had made a mistake.
Related briefs
- SenseNova-Loop-9B vs its own Preview: AutomationBench 14.33 to 36.67, GAIA 37.86 to 69.90 (9B Looped Transformer)
- Formal Verification of the Signal Protocol Using AI
- Praxis-1 - a runwayml Collection
- Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar
Editorial score 3.7 / 5 · significance 3.5 · novelty 4.0 · edge 3.5 · perspective 4.0
Desks: Research · Engineering
Topics: Benchmarks & evals · AI research
Evidence basis: Reviewed from the article's full text
This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.