The Extended Brief

I made llama.cpp remember across restarts: 54.4s prefill -> 3.5s on a new process (free ARM box)

Brief by The AI News AI newsroom · Aug 2, 2026, 10:12 PM EDT edition

Original reporting by r/LocalLLaMA — /u/Annual_Manner_5901 · published Aug 2, 2026, 2:32 PM EDT

CPU inference's biggest cost — prefill — can survive process restarts, turning a free 4-core ARM box into a practical host for repeated long-document workloads.

Read the original at r/LocalLLaMA →

Key points

  • Persisting llama.cpp's KV cache to disk cut a 3356-token prefill from 54.4 seconds to 3.5 across process restarts.
  • Pre-digesting predictable prefixes via a 03:00 systemd timer cut one document's TTFT from 89.7s to 16.7s.
  • Speculative decoding, on by default, silently bypassed the warm-ahead cache: 90.5s versus 16.7s with it off.
  • Halving active MoE experts during prefill was 44% faster but corrupted the cache, scoring 11/20 versus a 14/20 control.
  • Q4_0 quantization sped prefill 37% but dropped 5 of 20 facts on the author's extraction test.

The data

llama.cpp latency without vs with persisted KV cache (4-core ARM)
Default (recompute)Persisted/warmed cache
3356-token prefill, fresh process−50.9seconds
54.4seconds
3.5seconds
2815-token TTFT at 09:00−73seconds
89.7seconds
16.7seconds

Author's measurements on an Oracle free-tier ARM box; reads from page cache drop prefill to 0.10s.

Numbers from the original article, machine-verified against its text

From the source

“A 3356-token document costs 54.4 seconds before the model writes a single token.”

“A new process inherits that prefill for 3.5 seconds from disk, 0.10 seconds if the blob is still in page cache.”

“With a systemd timer that pre-digests predictable prefixes at 03:00, a 2815-token document goes from 89.7s to 16.7s TTFT (5.4x), and the request that arrives at 09:00 pays nothing for the prefill.”

“Halving active experts during prefill on an MoE: 44% faster, and it silently corrupts the cache — a KV built with 4 experts and read back with 8 scores 11/20 against a 14/20 control.”

Quoted verbatim from the original article at r/LocalLLaMA by /u/Annual_Manner_5901

Practical applications

  • If you serve repeated long documents on llama.cpp, persist the KV cache to disk and reload it at process start instead of re-prefilling.
  • Before combining speculative decoding with prefix caching, verify the speculative path actually consults the cache — here it silently skipped it.
  • Schedule a systemd timer or cron job to pre-warm known prompt prefixes ahead of predictable traffic windows.
  • Validate any prefill speedup (quantization, expert reduction) against reloaded caches with an extraction test, since errors can hide in the cached representation.

Who should care

Engineers self-hosting LLMs on CPUs or cheap ARM VMs with repeated long-context prompts, and llama.cpp contributors working on caching and speculative decoding.

Context

llama.cpp is an open-source engine for running LLMs on commodity CPUs and consumer hardware. On CPU, most latency is prefill — building the key-value cache from the prompt — and that cache normally lives only in RAM, so every process restart forces full recomputation.

What to watch

  • Whether llama.cpp upstream adds native KV-cache persistence or fixes the speculative-decoding path that bypassed the shared-prefix cache.
  • Independent reproductions of the reported 4.8x end-to-end speedup on other hardware and models.

Editorial score 4.1 / 5 · significance 3.5 · novelty 4.0 · edge 4.0 · perspective 5.0

Desks: Engineering · Tags: tooling, models

Evidence basis: Reviewed from the article's full text

This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.