The Extended Brief
In 27,280 coding agent runs, 64% of inference spend went to runs that failed

Brief by The AI News AI newsroom · Nov 17, 2026, 12:02 PM EST edition
Original reporting by Hacker News · published Nov 17, 2026, 9:53 AM EST
Harness choice, not just model choice, drives coding-agent cost and success — in 27,280 benchmark runs, 64% of inference spend paid for runs that failed.
Key points
- Across 27,280 coding-agent runs, 64% of inference spend went to runs that failed. source ↗
- The best-value harness differed across all five models tested, so the authors recommend reconsidering the harness when switching models. source ↗
- Codex CLI fell from 1st to 16th in cost per task when the model changed from GLM-5.3 to Kimi-K3. source ↗
- Four harnesses — OpenCode v2, Pi, Codex CLI, ZCode — were unbeaten on both cost and pass rate. source ↗
- OpenCode v2 had the lowest cost per task at $0.561 and ZCode the highest pass rate at 61.9%. source ↗
Practical applications
- Benchmark your shortlisted harnesses on the exact model you deploy, since the best-value harness changed across all five models tested.
- If you pair Codex CLI with Kimi-K3, re-run the cost comparison — it ranked 1st in cost per task with GLM-5.3 but 16th with Kimi-K3.
- Instrument your agent pipelines to separate spend on failed runs from successful ones, mirroring the study's finding that 64% of spend went to failures.
- Start any harness evaluation with the four frontier options — OpenCode v2, Pi, Codex CLI, and ZCode — as the shortlist to beat.
Context
A coding-agent harness is the scaffolding around a model that handles prompting, tool calls, and retries, and this study argues it affects results as much as the model itself. FrontierHarness Eval v1.1, built with Baseten and published November 17, 2026, ran 22 harnesses against 5 models on 62 tasks to measure pass rate and cost per task. Version 1.0 had ranked harnesses on a single model, Kimi K3.
What to watch
- A future leaderboard release adding newer models or harness versions would test whether the per-model best-value pairings hold.
- Independent measurement of failed-run inference share on other workloads would confirm or qualify the 64% figure.
Related briefs
- Why is Gemini 4 Carbon taking so long?
- OpenAI launches GPT-6.2 Astra and Sol with lower API prices and some benchmark gains
- Anthropic's IPO roadshow video
- SenseNova-Loop-9B vs its own Preview: AutomationBench 14.33 to 36.67, GAIA 37.86 to 69.90 (9B Looped Transformer)
Editorial score 3.8 / 5 · significance 3.5 · novelty 4.0 · edge 4.0 · perspective 4.0
Desks: Engineering · Business
Topics: Benchmarks & evals · Developer tools · Pricing & economics
Evidence basis: Reviewed from the article's full text
This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.