llama.cpp on the GPU: One Runtime, Many Backends

“GPU support” is not one feature

When a local model suddenly becomes usable after compiling llama.cpp with a GPU backend, it is tempting to credit the hardware and move on. The more useful story is underneath: llama.cpp builds on ggml, a tensor graph and backend system that can dispatch work to different devices. The backend changes the execution vocabulary, but model fit, quantization, memory placement, synchronization, and driver quality decide whether that vocabulary produces a faster program.

This is a map of the current support surface, based on the project’s own README and build documentation. It is not a leaderboard. llama.cpp changes quickly, and “supported” can mean anything from a mature first choice to an experimental path worth testing on a specific device.

The backend map

CUDA is the natural first experiment for NVIDIA hardware, with custom kernels and CUDA libraries. Metal is enabled by default on macOS and targets Apple GPUs. HIP brings the CUDA-shaped path to AMD through ROCm. Vulkan is the broad, cross-platform option when a portable GPU API matters more than a vendor-specific stack. SYCL is primarily aimed at Intel GPUs and depends on the oneAPI ecosystem.

The wider project also lists paths for MUSA, CANN, OpenCL, OpenVINO, WebGPU, VirtGPU, and RPC, alongside CPU-oriented BLAS, BLIS, ZenDNN, and zDNN backends. That list is useful as a discovery map, not a promise that every operation or quantization format has equal coverage everywhere.

Build the path that matches the machine

The build flag selects compiled capability. It does not guarantee that a device is visible, that a model fits, or that every graph operation has an optimized kernel.

Example: compile Vulkan support

cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

Example: compile two backend families

cmake -B build -DGGML_CUDA=ON -DGGML_VULKAN=ON
cmake --build build --config Release

Offload is a memory decision

llama.cpp can run in a hybrid CPU-GPU mode, which is why a model larger than available VRAM can still be useful. Some layers or buffers remain in system memory while other work lives on the accelerator. That is a capacity escape hatch, not a free speed multiplier: every boundary crossing adds transfer and scheduling cost.

The practical controls include --n-gpu-layers (also written -ngl) for layer offload, --device for choosing visible devices, and --list-devices for discovering what the binary can actually see. The exact defaults and accepted values evolve, so use the installed binary’s help output as the final contract.

When one GPU is not enough

The official multi-GPU guide describes layer split mode as the default pipeline-parallel choice: each GPU owns a contiguous slice of layers. tensor split mode is experimental and divides tensors across devices, trading more cross-GPU communication for a path that can improve token generation on suitable hardware and interconnects. The older row mode is deprecated.

Multi-GPU allocation is not merely “divide VRAM by GPU count.” Interconnect speed, available memory, KV-cache placement, split mode, and model shape all matter. Start with --list-devices, make one controlled change at a time, and record prompt-processing and token-generation rates separately.

Quantization changes the question

Quantization stores weights with fewer bits, reducing memory traffic and model size. llama.cpp’s backend matrix is not uniform: support and performance vary by quantization family, operation, backend, and hardware generation. A Q4 model that fits comfortably may outperform a larger, nominally more precise model simply because it avoids memory pressure.

Do not turn a feature table into a benchmark claim. Validate output quality against a reference, then benchmark the workload you care about: prompt length, context size, batch size, KV-cache type, and generation length can all change the winner.

What this means for CatGame

llama.cpp and CatGame’s custom Vulkan compute work solve different problems. llama.cpp is an inference runtime with a broad backend ecosystem and GGUF model tooling. CatGame is exploring explicit Vulkan resource ownership, shaders, training experiments, and game-engine integration. The useful connection is methodological: keep a CPU reference, make data residency visible, and treat synchronization as part of the algorithm.

Our Vulkan ML compute report documents the local version of that lesson. The GEMM and tiling post explains why matrix layout and reuse dominate so many accelerator kernels, while the GRU training report shows why numerical validation survives every backend choice.

Research checked against the official ggml-org/llama.cpp README, build documentation, multi-GPU guide, and SYCL backend documentation on 2026-08-11. Backend maturity and exact flags are version-sensitive; consult the checked-out revision.