The Starting Point
We set out to build a pure C++ inference engine for sentence-transformers/all-MiniLM-L6-v2 — a 22M parameter model used for semantic embeddings. No PyTorch dependency, no BLAS library, no framework overhead. Just hand-tuned AVX2 SIMD kernels and a custom model format (.vnn).
The initial unoptimized implementation ran at 35.87 samples/s on an Intel i7-11800H. PyTorch on the same CPU with batch=1 managed 221 s/s. We were 6x slower than the baseline we wanted to beat.
Optimization Journey
Hashmap to Sorted Array Tokenizer
Replaced std::unordered_map with sorted array + binary search for vocab lookup. Modest gain but eliminated hash collision overhead.
AVX2 Vectorized Attention
Tiled Q×K^T and softmax with 8-wide SIMD. Attention dropped from 27ms to 20ms per 100 sentences.
Fused FFN1+GELU Kernel
Eliminated separate GELU pass by fusing activation into the GEMM output loop. Fast AVX2 exp/sigmoid for vectorized GELU — 2.7x FFN speedup.
FP16 Weight-Only Quantization
Stored FFN weights as FP16 (uint16_t), converted on-the-fly with F16C intrinsics (_mm256_cvtph_ps). Halved weight memory bandwidth. Accuracy preserved: 0.999999 cosine.
2x Batch-Interleaved GEMM
The key innovation: process 2 input vectors simultaneously, sharing FP16→FP32 weight conversions. 2:1 FMA-to-conversion ratio amortizes the F16C bottleneck. FFN dropped from 136ms to 112ms.
The Batch Interleaving Insight
The F16C instruction _mm256_cvtph_ps converts 8 FP16 values to FP32 in one instruction — but it can only execute at 1 per clock. Fused multiply-add (_mm256_fmadd_ps) runs at 2 per clock. This means FP16 GEMM is conversion-bound, not compute-bound.
Batch interleaving solves this by processing 2 input vectors per weight row. Each weight conversion feeds 2 FMAs — one for each input. This doubles the FMA-to-conversion ratio from 1:1 to 2:1, effectively hiding the conversion latency behind useful computation.
The 2:1 FMA-to-conversion ratio is the key innovation. Don't sacrifice it for anything.
The Preconversion Failure
We hypothesized that separating conversion from FMA would improve instruction-level parallelism. The idea: pre-convert FP16 weights to FP32 in L1-resident stack buffers, then run pure FP32 FMA without conversion bottleneck.
L1 Preconversion Kernel
255 s/s — 16.8% SLOWER than baseline. Pre-converting loses the batch interleaving benefit (1:1 ratio). Stack buffer overhead adds latency. The 2:1 sharing was the entire point.
Vectorized GELU
Fast exp_ps was 17% slower than scalar expf for GELU. GELU is <1% of FFN time — sometimes scalar is faster than vectorized for non-bottleneck operations.
Fused FFN Pipeline
Analysis showed the intermediate buffer between FFN1 and FFN2 is only 0.63ms (0.5% of FFN). Fusion would add complexity for negligible gain. Not worth it.
The CPU Governor Trap
powersave, causing the i7-11800H to run at 1.9-4.0 GHz inconsistently instead of sustained 3.6+ GHz. Setting performance governor recovered throughput from 245 to 312 s/s. Always check /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor before benchmarking. Head-to-Head: VNN vs PyTorch
100 sentences, 5 iterations, performance governor, i7-11800H + RTX 3060:
| Runtime | Device | Batch | Throughput (s/s) | Memory (MB) | vs VNN |
|---|---|---|---|---|---|
| VNN C++ (release) | CPU | 100 | 311.75 | 151.35 | — |
| PyTorch 5.6.0 | CPU | 1 | 221.30 | 18.63 | 0.71x |
| PyTorch 5.6.0 | CPU | 8 | 537.27 | 22.41 | 1.72x |
| PyTorch 5.6.0 | CUDA | 1 | 376.66 | 95.27 | 1.21x |
| PyTorch 5.6.0 | CUDA | 8 | 1,200.83 | 187.40 | 3.85x |
VNN beats PyTorch CPU batch-1 by 1.41x — our hand-tuned FP16+SIMD kernel wins on single-sentence inference where PyTorch's framework overhead dominates. But PyTorch CPU batch-8 beats VNN by 1.72x, because BLAS libraries have highly optimized batch GEMM that amortizes weight loads across multiple inputs.
Per-Stage Breakdown
| Stage | Time (ms) | % of Inference | Optimization Status |
|---|---|---|---|
| FFN (GEMM + GELU) | 112.49 | 52.3% | FP16 + batch-interleaved |
| QKV GEMM | 48.11 | 22.3% | Still FP32 — next target |
| Attention | 20.29 | 9.4% | AVX2 vectorized |
| Tokenize | 1.04 | 0.5% | Sorted array + binary search |
| Embed lookup | 0.74 | 0.3% | Direct index |
| Pool + L2 norm | 0.49 | 0.2% | AVX2 horizontal sum |
What's Next
QKV FP16 Conversion
22.3% of inference, still using FP32 weights. Converting to FP16 with batch-interleaved kernels should yield 3-5% throughput gain. 5MB memory savings across 6 layers.
Batch GEMM
The biggest opportunity: process 8 sentences simultaneously through GEMM layers. Could close the 1.72x gap with PyTorch CPU b8. Requires input layout refactor.
Benchmark Environment Guard
Auto-detect CPU governor and warn if powersave. Record environment metadata in telemetry. Prevents false regression alarms.
Lessons Learned
Benchmarking environment matters as much as kernel optimization. The best kernel in the world won't help if your CPU is throttling to 1.9 GHz.
- 5-run averaging minimum for NN benchmarks. 3-run averages produced 274-340 s/s range; 5-run converged to 307 stable.
- 8 AVX2 accumulators (2KB register pressure) is the safe limit. 12 spills to stack. Always count: rows × batch_items = total accumulators.
- Don't sacrifice batch sharing for ILP separation. Preconversion failed because it broke the 2:1 ratio. The sharing IS the optimization.
- Scalar can beat vectorized for non-bottleneck operations. GELU scalar was faster than vectorized because GELU is <1% of FFN time.
- Check CPU governor before benchmarking. Powersave caused 30-40% regression. This was the hardest bug to find because it wasn't in the code.