State-of-the-Art CPU Optimization for Math Kernels and Thread Pools

The Two Bottlenecks

Modern CPU inference for transformer-style models is rarely limited by raw arithmetic. The cores can issue fused multiply-add instructions every cycle, but only if the data is already in registers. The real bottlenecks are data movement and dispatch overhead: how many bytes travel between DRAM, cache, and registers, and how many times the runtime wakes threads up to do a small piece of work.

This post is a high-level survey of the techniques production CPU engines use to attack both problems: SIMD microkernels, cache tiling and packing, and lock-free thread pool scheduling. No source code, no proprietary internals — just the engineering shape of the solution.

1. The Memory Pyramid

Every optimized kernel starts by admitting a hardware fact: registers are fast and small, caches are bigger and slower, and DRAM is enormous and glacial. A vector floating-point operation on data already in registers takes about one cycle. A load from L1 takes four, L2 twelve, L3 forty, and DRAM hundreds.

The goal of kernel design is to move each byte as few levels down as possible. Once a cache line is pulled into L1, the kernel should do every operation it can on that line before it is evicted. That principle drives everything that follows.

2. SIMD and the Microkernel

A microkernel is the smallest unit of compute that still exposes the architecture's vector width. On x86 with AVX2, one vector register holds eight 32-bit floats; with AVX-512 it holds sixteen. A microkernel multiplies and accumulates a small tile of the output matrix while streaming through the reduction dimension.

The classic 8×8 microkernel keeps all sixty-four output values in vector registers for the entire K-loop. The two source operands are streamed from packed buffers, so each load pulls a full cache line and every float is reused many times before it is evicted. The accumulator tile is written back to memory only once, at the very end. This is the same register-blocking strategy used by oneDNN and BLIS.

3. Tiling and Packing

The microkernel alone only wins when its inputs are already contiguous. Real weight and activation tensors are rarely laid out that way. Tiling partitions the M, N, and K dimensions into blocks sized for each cache level. Packing copies those blocks into contiguous panels so the microkernel can stream them with unit stride.

The trade-off is extra memory traffic for the pack step. Packing only pays off when the packed buffer is reused enough times to amortize the copy. In matrix-matrix multiplication the weights are reused across the whole M-dimension, so the cost is tiny. In matrix-vector multiplication the reuse is much lower, which is why GEMV kernels use a different strategy than GEMM kernels.

4. Thread Pool Execution

Parallelizing a kernel is the second bottleneck. The simplest approach is a static partition: split the output rows evenly across threads. That works for uniform, large tiles. It fails when the workload is small, irregular, or has dependencies. For those cases, work-stealing thread pools give better utilization by letting idle threads take tasks from busy peers.

Modern work-stealing queues are lock-free, often based on the Chase-Lev deque. Owners push and pop from one end; thieves steal from the other. A subtle but important choice is continuation stealing versus child stealing. Continuation stealing keeps the current execution path on the same worker and pushes the continuation onto the queue, which tends to preserve depth-first traversal and better memory behavior. Child stealing does the opposite and can create breadth-first expansion. Libraries like libfork use continuation stealing with C++20 coroutines to keep memory usage low, reporting order-of-magnitude reductions in memory compared to OpenMP and TBB on fork-join benchmarks.

5. Putting It Together: Dispatch Policy

The final piece is deciding which kernel and which thread schedule to run for each operator. A production inference engine treats dispatch as a first-class concern:

Recent research reinforces this layered approach. The EXO compiler generates micro-kernels from user schedules, showing that auto-generated C can match hand-written intrinsic code. IntrinsicML demonstrates that a transparent, well-documented AVX2 GEMM can reach 40–50% of OpenBLAS single-thread performance while remaining understandable. KleidiAI ships Arm microkernels as standalone .c/.h files with no dependencies, no memory management, and no scheduler, letting the framework decide how to call them. And X-OpenMP reports dramatic reductions in tasking overhead on fine-grained workloads through lock-less work stealing on many-core systems.

6. Reported Landscape

Conclusion

State-of-the-art CPU kernel performance is not a single trick. It is a stack: vector arithmetic at the bottom, register blocking above it, cache tiling and packing above that, and a carefully chosen thread schedule at the top. Each layer has to fit the others. A great microkernel is wasted on strided data; a perfect packing strategy is wasted if threads spend more time waking up than computing.

The modern landscape offers two complementary paths. One is libraries of hand-optimized or auto-generated microkernels — BLIS, oneDNN, KleidiAI, Exo — that expose small, fast building blocks. The other is runtime research — libfork, X-OpenMP, work-stealing deques, continuation stealing — that reduces the cost of orchestrating those blocks. The engines that exceed general-purpose framework throughput are the ones that design every layer for the exact shapes, formats, and batch sizes they actually see.