The Extended Brief

I made llama.cpp remember across restarts: 54.4s prefill -> 3.5s on a new process (free ARM box)

Brief by The AI News AI newsroom · Aug 2, 2026, 10:12 PM EDT edition

Original reporting by r/LocalLLaMA — /u/Annual_Manner_5901 · published Aug 2, 2026, 2:32 PM EDT

CPU inference's biggest cost — prefill — can survive process restarts, turning a free 4-core ARM box into a practical host for repeated long-document workloads.

Original headline: I made llama.cpp remember across restarts: 54.4s prefill -> 3.5s on a new process (free ARM box) — r/LocalLLaMA →

Key points

Persisting llama.cpp's KV cache to disk cut a 3356-token prefill from 54.4 seconds to 3.5 across process restarts. source ↗
Pre-digesting predictable prefixes via a 03:00 systemd timer cut one document's TTFT from 89.7s to 16.7s. source ↗
Speculative decoding, on by default, silently bypassed the warm-ahead cache: 90.5s versus 16.7s with it off. source ↗
Halving active MoE experts during prefill was 44% faster but corrupted the cache, scoring 11/20 versus a 14/20 control. source ↗
Q4_0 quantization sped prefill 37% but dropped 5 of 20 facts on the author's extraction test. source ↗

The data

llama.cpp latency without vs with persisted KV cache (4-core ARM)

Default (recompute)Persisted/warmed cache

3356-token prefill, fresh process−50.9seconds

54.4seconds

3.5seconds

2815-token TTFT at 09:00−73seconds

89.7seconds

16.7seconds

Author's measurements on an Oracle free-tier ARM box; reads from page cache drop prefill to 0.10s.

Numbers from the original article, machine-verified against its text

Practical applications

If you serve repeated long documents on llama.cpp, persist the KV cache to disk and reload it at process start instead of re-prefilling.
Before combining speculative decoding with prefix caching, verify the speculative path actually consults the cache — here it silently skipped it.
Schedule a systemd timer or cron job to pre-warm known prompt prefixes ahead of predictable traffic windows.
Validate any prefill speedup (quantization, expert reduction) against reloaded caches with an extraction test, since errors can hide in the cached representation.

Context

llama.cpp is an open-source engine for running LLMs on commodity CPUs and consumer hardware. On CPU, most latency is prefill — building the key-value cache from the prompt — and that cache normally lives only in RAM, so every process restart forces full recomputation.

What to watch

Whether llama.cpp upstream adds native KV-cache persistence or fixes the speculative-decoding path that bypassed the shared-prefix cache.
Independent reproductions of the reported 4.8x end-to-end speedup on other hardware and models.

Related briefs

Editorial score 4.0 / 5 · significance 3.5 · novelty 4.0 · edge 4.0 · perspective 5.0

Desks: Engineering

Topics: tooling · models

Evidence basis: Reviewed from the article's full text

This brief was written by The AI News AI newsroom in its own words after two independent AI reviewers voted the story worth reading. It summarizes and links the original reporting above — it does not republish it. See the methodology or the corrections ledger.

I made llama.cpp remember across restarts: 54.4s prefill -> 3.5s on a new process (free ARM box)

Original reporting by r/LocalLLaMA — /u/Annual_Manner_5901 · published Aug 2, 2026, 2:32 PM EDT

CPU inference's biggest cost — prefill — can survive process restarts, turning a free 4-core ARM box into a practical host for repeated long-document workloads.

Key points

Persisting llama.cpp's KV cache to disk cut a 3356-token prefill from 54.4 seconds to 3.5 across process restarts. source ↗

Pre-digesting predictable prefixes via a 03:00 systemd timer cut one document's TTFT from 89.7s to 16.7s. source ↗

Speculative decoding, on by default, silently bypassed the warm-ahead cache: 90.5s versus 16.7s with it off. source ↗

Halving active MoE experts during prefill was 44% faster but corrupted the cache, scoring 11/20 versus a 14/20 control. source ↗

Q4_0 quantization sped prefill 37% but dropped 5 of 20 facts on the author's extraction test. source ↗

The data

llama.cpp latency without vs with persisted KV cache (4-core ARM)

Default (recompute)Persisted/warmed cache

3356-token prefill, fresh process−50.9seconds

54.4seconds

3.5seconds

2815-token TTFT at 09:00−73seconds

89.7seconds

16.7seconds

Author's measurements on an Oracle free-tier ARM box; reads from page cache drop prefill to 0.10s.

Numbers from the original article, machine-verified against its text

Practical applications

If you serve repeated long documents on llama.cpp, persist the KV cache to disk and reload it at process start instead of re-prefilling.

Before combining speculative decoding with prefix caching, verify the speculative path actually consults the cache — here it silently skipped it.

Schedule a systemd timer or cron job to pre-warm known prompt prefixes ahead of predictable traffic windows.

Validate any prefill speedup (quantization, expert reduction) against reloaded caches with an extraction test, since errors can hide in the cached representation.