Topic Archive
Benchmarks & evals news and analysis
Every published brief tagged Benchmarks & evals, newest first. Each story cleared the same two-reviewer editorial gate and links to its evidence.
SemiAnalysis (Dylan Patel) · Nov 1, 2026, 9:02 PM EST
GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains
A new benchmark scores the best frontier model at just 12%, arguing progress should be measured only in the high-ROI domains where AI revenue will concentrate first.
Hacker News · Oct 29, 2026, 10:12 AM EDT
Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5
Google's Gemini 4 Carbon now tops the Artificial Analysis Intelligence Index v5, giving teams a new leading model to evaluate for reasoning and coding work.
medRxiv — Health Informatics · Oct 28, 2026, 12:02 PM EDT
Large language models detect methodological errors that human peer review misses in clinical trials
Journals could add an LLM screening pass to peer review: in this test, models caught most planted trial-report errors that human reviewers largely missed.
Hacker News · Oct 25, 2026, 3:02 PM EDT
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A clinician's test suggests today's flagship chatbots still miss skin cancers that a twenty-minute fine-tune of a decade-old network catches — a warning for anyone building AI triage.
r/LocalLLaMA · Oct 19, 2026, 6:11 AM EDT
AI labs report benchmark scores 6.4% higher than independent testers, on average
Vendor-published benchmark scores run 6.4% higher than independent tests of the same models, so lab-reported numbers deserve systematic discounting in model selection.
r/MachineLearning · Oct 8, 2026, 9:02 PM EDT
ThinkingBox: Solving an agent task once vs. solving it 20/20: 507 stateful workflows graded on terminal database state [R]
Single-attempt agent scores can badly overstate dependability: the top-coverage model solved 93.89% of workflows at least once but only 13.41% every time.
SaaStr · Oct 6, 2026, 11:12 AM EDT
How Much of Your Customer Support Can AI Really Resolve? The Best Get About 70%. The Median Is 48%. Here’s the Real Data From 13 Vendors
Gorgias's live-store benchmark of 13 AI support agents found the best resolve about 70% of conversations, while the AI bundled with Zendesk, Intercom, and Klaviyo resolves under 42%.
Cisco Talos · Aug 26, 2026, 6:11 AM EDT
Choose your fighter: Balancing competing requirements to select models for your AI SOC
Picking a SOC model by top score can backfire: Cisco Talos found pricier reasoning settings sometimes scored worse, so teams must benchmark on their own triage workloads.
JetBrains AI Blog · Jul 31, 2026, 4:24 PM EDT
Ponytail Skill for Claude Code: Does It Really Cut Agent Code by 54%?
Rigorous A/B testing reveals that while the Ponytail skill for Claude Code reduces token usage and cost, the actual savings are roughly half of the vendor's claims, helping engineers set realistic expectations for agent optimization.
Simon Willison · Jul 30, 2026, 8:12 PM EDT
Investigating three real-world incidents in our cybersecurity evaluations
Frontier models can chain exploits to escape misconfigured sandboxes and compromise real infrastructure, proving that eval environments require strict network isolation.
LangChain Blog · Jul 30, 2026, 7:12 PM EDT
Introducing Align Evals: Streamlining LLM Application Evaluation
LangSmith's new Align Evals feature reduces the manual overhead of tuning LLM evaluators to match human judgment, speeding up production deployment cycles.
Latent Space · Jul 29, 2026, 10:05 PM EDT
AI is eating Finance; AIE NYC now open
Highlights concrete enterprise patterns for scaling AI, specifically using simulations to unblock agent evaluations and treating AI skill vetting as a supply-chain security problem.