The Extended Brief

4090 + 5060 Ti + 64GB RAM: 206 t/s on a 35B-A3B, and a 122B at 37 t/s

Brief by The AI News AI newsroom · Jul 30, 2026, 5:10 PM EDT edition

Original reporting by r/LocalLLaMA — /u/Dry_Long3157 · published Jul 30, 2026, 2:46 AM EDT

Prosumer hardware can now run 122B parameter MoE models at interactive speeds by strategically spilling layers to system RAM, redefining local inference economics.

Key points

• A dual-GPU setup achieved 206 tokens per second on a 35B parameter model using llama.cpp.
• The hardware generated 37 tokens per second for a 122B parameter model with 17 layers in system RAM.
• The benchmarking system combined an RTX 4090, an RTX 5060 Ti, and 64GB of DDR5 memory.
• Using CUDA 13.1 caused silent fallbacks to cuBLAS, reducing prompt processing speed by approximately six times.

Practical applications

• Benchmark your own workloads on mixed-GPU setups before assuming a 122B-class MoE model requires datacenter hardware.
• Pin your CUDA version and verify llama.cpp is using its fast paths, since CUDA 13.1 silently fell back to cuBLAS with roughly a 6x prompt-processing slowdown.
• Test spilling a limited number of MoE layers to system RAM as a cheaper alternative to buying more VRAM.
• Recheck local-versus-API cost assumptions for inference now that interactive speeds are reachable on prosumer parts.

Who should care

Engineers running local inference and teams costing self-hosted deployments, because these numbers show prosumer dual-GPU boxes reaching interactive speeds on models previously assumed to need server hardware.

Context

Mixture-of-experts (MoE) models activate only a fraction of their parameters per token, which is why a 35B-total model with about 3B active parameters can run far faster than a dense model of the same size. This benchmark paired an RTX 4090 with an RTX 5060 Ti and 64GB of DDR5, hitting 206 tokens per second on a 35B-A3B model and 37 tokens per second on a 122B model with 17 layers offloaded to system RAM via llama.cpp. It also surfaced a practical pitfall: on CUDA 13.1 the stack silently fell back to cuBLAS, cutting prompt processing speed by roughly six times.

What to watch

• Independent reproductions of these numbers on similar mixed-GPU configurations, or a llama.cpp/CUDA fix for the silent cuBLAS fallback.
• Whether upcoming open-weight MoE releases keep active-parameter counts low enough to sustain this local-inference trend.

This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.