The π daemon runs qwen3-1.7b-instruct CPU-only through llama.cpp. We wanted to know, on a 16-logical-core host with dual-channel memory, where the knobs actually matter. So we restarted the daemon clean for every data point and swept n_threads, context_tokens, and max_output_tokens. This post is the resulting map.
More threads is not always better. At 16 logical cores the same model becomes an order of magnitude slower than at 8, because CPU inference is memory-bandwidth limited and oversubscription causes cache thrashing.
— Finding from the thread sweep
1. Methodology
All measurements used the production π daemon (pi-daemon) reached through the Crypt-authorized native Unix socket. The client was pi_agent --once driven by ./scripts/pi.sh --get. Before every run we stopped and restarted the daemon so the model.n_threads, model.context_tokens, or model.max_output_tokens change in the transient TOML took effect.
The host has 16 logical cores, 64 GB RAM, and the model is a Q4_K_M GGUF of Qwen3-1.7B. Debug telemetry was enabled so each response printed wall time, prompt/generation tokens per second, CPU%, and peak RSS.
During this work we also removed the per-turn Agent::shutdown()/init() cycle from the persistent pi_agent --worker path. Crypt capabilities are multi-use, so the worker now reuses one capability across all Arena turns and only calls Agent::reset() between prompts. A live 1-round Arena run confirmed the worker stays stable.
2. Thread Sweep
Prompt: "What is 2+2?" — context 8192, max output 1024, batch 2048 / ubatch 512, CPU-only. We tested 4, 8, 12, and 16 generation/batch threads.
Bars: wall time in seconds. Rust line: generation tokens per second. 16 threads collapses throughput.
| n_threads | wall (s) | CPU% | prompt tok/s | gen tok/s | peak mem (MB) |
|---|---|---|---|---|---|
| 4 | 3.450 | 393.0 | 126.52 | 13.46 | 2912 |
| 8 | 2.194 | 778.6 | 208.67 | 19.44 | 2912 |
| 12 | 2.210 | 1145.2 | 216.48 | 18.01 | 2912 |
| 16 | 27.284 | 1344.9 | 94.36 | 0.66 | 2912 |
Analysis: 8 threads is the sweet spot on this host. 4 threads is under-provisioned; 12 threads is competitive but burns 47% more CPU for slightly lower generation throughput. 16 threads is pathological: two logical cores per physical core fight over the same memory bandwidth, context switches dominate, and generation speed falls to 0.66 tok/s. This matches the well-known behavior of CPU-bound llama.cpp inference: Johannes Gaessler's llama.cpp CPU benchmarks show throughput is usually limited by memory bandwidth, and oversubscribing cores hurts rather than helps.
3. Context Sweep
Fixed at 8 threads / 1024 max output, we swept the KV-cache budget: 2048, 4096, 8192, and 16384 tokens.
Bars: wall time in seconds. Rust line: peak RSS in GB. Memory grows roughly linearly with context budget.
| context_tokens | wall (s) | prompt tok/s | gen tok/s | peak mem (MB) |
|---|---|---|---|---|
| 2048 | 2.423 | 201.11 | 15.99 | 2222 |
| 4096 | 2.232 | 197.82 | 20.35 | 2453 |
| 8192 | 2.568 | 193.23 | 14.71 | 2912 |
| 16384 | 2.557 | 176.88 | 17.03 | 3830 |
Analysis: 4096 context is the fastest and most efficient sweet spot for short prompts. The 8192 and 16384 runs are a little slower despite the larger cache, because the model still has the same small prompt; the extra KV memory only adds allocation cost. 16384 uses 3.83 GB, a 72% increase over 2048. For the Arena's long, multi-turn debates, 8192 remains the safe default; for quick one-shot queries, 4096 would save 459 MB and run 9% faster.
4. Max-Output Sweep
Fixed at 8 threads / 8192 context, we set max_output_tokens to 128, 256, 512, and 1024 and asked for a three-paragraph summary of internet history.
Bars: wall time in seconds. Rust line: actual output tokens produced. The limit is honored; the model naturally stopped at ~408 tokens.
| max_output | actual output | wall (s) | gen tok/s | peak mem (MB) |
|---|---|---|---|---|
| 128 | 128 | 8.823 | 17.42 | 2912 |
| 256 | 256 | 15.504 | 18.37 | 2912 |
| 512 | 408 | 23.281 | 18.69 | 2912 |
| 1024 | 408 | 20.567 | 21.30 | 2912 |
Analysis: The output ceiling works exactly as intended. At 128 and 256 the model hits the cap and stops; at 512 and 1024 the same three-paragraph answer runs to ~408 tokens and then terminates. The 1024 run finished about 2.7 s faster than the 512 run despite producing the same 408 tokens, which is ordinary run-to-run variance at this short scale. Memory is unchanged because the KV cache is sized by context, not output length.
5. Conclusion
The π daemon is now configured with a safe, measured default: 8 generation threads, 8 batch threads, 8192 context tokens, 1024 max output tokens, batch 2048 / ubatch 512. This is the knee of the curve for a 16-logical-core CPU running Qwen3-1.7B Q4_K_M.
The most important finding is that CPU inference is not a "use all cores" problem. Past the physical-core count, hyperthreads fight for memory bandwidth and throughput collapses. The second finding is that context size is a memory trade-off: 4096 is the sweet spot for short prompts, 8192 is necessary for long multi-turn sessions like the Arena, and 16384 is cheap only if you truly need the window.
For anyone tuning their own local daemon, the recipe is:
- Set
n_threadsto the number of physical cores, not logical cores. - Set
n_threads_batchequal ton_threadsfor this small model. - Use 4096 context for one-shot tools, 8192 for dialogue, and 16384 only when the prompt justifies the 72% memory increase.
- Leave
max_output_tokensas the budget, not the target; the model will stop when it is done.
6. Possible Next Experiments
- Flash Attention on CPU: Test
flash_attn = enabledvsautoon the CPU backend to see if it changes prompt throughput. - NUMA strategies: On multi-socket hosts, sweep
disabled,distribute,isolate, andnumato quantify cross-socket penalty. - Batch / ubatch sweep: Fix context and threads, then vary
n_batchandn_ubatchto find the prompt-processing knee. - KV-cache quantization: Reduce
cache_type_kandcache_type_vto Q4_0 and measure context-memory savings at 8192+. - Hyperthreading off: Compare 8 physical cores with HT disabled to 8 logical cores with HT enabled.
- Instruction-set variants: Build with AVX-512, AVX2, and SSE and compare prompt tok/s on the same model.
- Larger models: Repeat the thread sweep with Qwen3-4B or Qwen3-8B to see if the memory-bandwidth knee shifts.
7. Links
On this site: π Arena: endless debatePolicy-gated local inferenceBenchLabAPIVisitors
External references: llama.cpp CPU perfllama-bench docsQwen3-1.7BQwen3 repoQwen3 analysis