The Extended Brief
Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
Brief by The AI News AI newsroom · Jul 30, 2026, 10:33 PM EDT edition
Original reporting by Together AI Blog · published Jul 25, 2026, 8:00 PM EDT
Provides concrete routing and cost-efficiency data for builders choosing between Kimi K3 and GPT-5.6 Sol for complex coding tasks.
Key points
- Routing between Kimi K3 and GPT-5.6 Sol achieves approximately 85.6 percent accuracy on DeepSWE benchmarks.
- GPT-5.6 Sol outperforms Kimi K3 on pass@1 metrics across 904 DeepSWE rollouts.
- Kimi K3 wins pass@4 evaluations while delivering 2.8 times more solves per dollar than Sol.
From the source
“Kimi K3 vs GPT-5.6 Sol is close on DeepSWE pass@1: Sol leads 72.7% to 68.5%, a 4.2 point gap.”
“It wins pass@2 (82.0 vs 81.0) and pass@4 (89.4% vs 85.8%).”
“GPT-5.6 Sol is the more reliable model: it solves more tasks on all 4/4 tries (61 vs 45).”
“The practical answer is about 85.6%: run Kimi K3 first and escalate to Sol only when the test suite rejects the result.”
“Five of the 113 tasks are solved by neither model across all eight combined attempts.”
Practical applications
- Implement a router between Kimi K3 and GPT-5.6 Sol if your coding workload tolerates the added complexity, since routing reportedly reaches roughly 85.6 percent on DeepSWE.
- Choose Sol where a single attempt must land and Kimi K3 where you can afford multiple samples, matching the pass@1 versus pass@4 split.
- Recompute your own cost per solved task rather than cost per token, which is where Kimi K3's claimed 2.8 times advantage shows up.
- Replicate the comparison on your real repositories, since DeepSWE results may not transfer to your codebase or test harness.
Who should care
Engineers building coding agents and the budget owners choosing between frontier and open-weight models, where cost per solved task drives the decision.
Context
Coding agents are commonly evaluated with pass@k, the chance that at least one of k attempts solves a task, so a model that is weaker on a single try can still win when sampling is cheap. Together AI ran 904 DeepSWE rollouts across Kimi K3 and GPT-5.6 Sol: Sol led pass@1, Kimi K3 led pass@4 and delivered about 2.8 times the solves per dollar, and routing between them reached roughly 85.6 percent. That pattern is the standard argument for model routing rather than picking one model outright.
What to watch
- Independent replication of the 904-rollout comparison on other coding benchmarks or private repository suites.
- Whether routing implementations reach the reported accuracy in production, where latency and router errors are real costs.
Editorial score 4.0 / 5 · significance 4.0 · novelty 4.0 · edge 4.0 · perspective 4.0
Desks: Engineering · Business · Tags: models, tooling, business
Evidence basis: Reviewed from a feed excerpt
This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.