The Problem With Fixed LoRA
LoRA (Low-Rank Adaptation) has become the default way to fine-tune large language models without touching the base weights. You freeze the pretrained model, inject low-rank A * B update matrices into selected layers, and train only those. The rank r controls the expressivity of the update: too low and you underfit, too high and you waste compute and risk overfitting.
The problem is that most practitioners pick a single rank, apply it uniformly across every target layer, and hope for the best. The attention Q, K, V, and O projections in a transformer do different things. The MLP gate and down projections have different sensitivity profiles. Treating them all the same leaves performance on the table.
The varying importance of different weight matrices means a uniform rank allocation is provably suboptimal. The question is: what do you do about it?
The Adaptive LoRA Family
Between 2023 and 2025, a family of methods emerged that dynamically allocate the parameter budget across layers. They share a common pattern: score the importance of each adapter, then prune or grow ranks accordingly. The differences are in how they score and how they adjust.
AdaLoRA (ICLR 2023)
SVD-based parameterization. Prunes singular values by sensitivity (Fisher information). The foundational method that most others build on or compare against.
DoRA (ACL 2024)
Decomposes LoRA into rank-1 components, then dynamically prunes based on task-specific importance. Competitive with full fine-tuning under the same parameter budget.
AutoLoRA (NAACL 2024)
Meta-learning approach. Associates each rank-1 component with a selection variable learned via meta-learning. Determines optimal rank per layer automatically.
ALoRA (NAACL 2024)
AB-LoRA importance scoring, then gradual pruning and reallocation. Prunes negatively-impacting ranks and redirects budget to layers that need it.
GeLoRA (EMNLP 2025)
Estimates intrinsic dimensionality of hidden representations to set ranks. Theoretically grounded: intrinsic dimension is a lower bound for optimal LoRA rank. Up to 64% faster than baselines.
AROMA (EMNLP 2025)
Iteratively builds rank-one components with a dual-loop architecture. Inner loop extracts information, outer loop determines optimal rank. Parameters diminish to zero as training converges.
SubLoRA (2025)
Uses second-order (Hessian) information instead of first-order approximations. Reformulates rank determination as submodular function maximization with greedy approximation guarantees.
PLoRA (2025)
System-level: orchestrates concurrent LoRA fine-tuning jobs across a hyperparameter search space. Up to 12.8x throughput improvement via packed adapters and parallel job scheduling.
Quantized + Dynamic: QDyLoRA and Q-BLoRA
QLoRA (4-bit quantization + LoRA) made it possible to fine-tune multi-billion-parameter models on consumer GPUs. But QLoRA trains at a fixed rank, so you still need a separate run for each candidate rank. QDyLoRA combines DyLoRA's dynamic rank training with QLoRA's quantization, enabling fine-tuning across a spectrum of ranks (1 to 64) in a single run on a 32 GB GPU.
Q-BLoRA (TACL 2025) identifies a different problem: an imbalance between adapter complexity and trainability in quantized models. The adapter's inputs and outputs are overly complex while the effective rank is too low, causing underfitting. Q-BLoRA simplifies the adapter interface while increasing rank to restore balance.
GPU-Side Adaptivity: Batch Size and Pipeline Scheduling
Adaptive rank is only half the story. The other half is GPU utilization. Even with optimal LoRA ranks, training throughput depends on batch size, context length, and thread count being matched to the hardware.
Research on adaptive batch size scheduling (Lau et al., PMLR 2025) shows that constant large batches improve throughput but hurt generalization due to reduced gradient noise. Their adaptive schedule, compatible with data and model parallelism, outperforms both constant and heuristic warmup schedules for Llama-2 family models up to 3B parameters, with theoretical convergence guarantees for Adam.
On the system side, WLB-LLM (OSDI 2025) addresses workload imbalance in 4D parallelism training. The key insight: tokens at tail positions of long documents require more attention computation, so equal token distribution across GPUs creates stragglers. Their workload-balanced packing eliminates this heterogeneity.
For long-context fine-tuning, Skrull (NeurIPS 2025) uses dynamic data scheduling to balance long and short sequences, and SPPO (2025) uses adaptive sequence pipeline parallel offloading to train 7B models with 4M-token sequences on 128 A100s.
Vulkan Backward Passes: The GPU Training Frontier
Most LLM training happens on CUDA. But Vulkan compute shaders are catching up. In early 2025, the llama.cpp project merged Vulkan backward passes for key training ops: GEGLU backward, SiLU backward, cross-entropy loss backward, and optimizer step (AdamW) via OPT_STEP_ADAMW. PR #11769 added SUM, ARGMAX, SUB, COUNT_EQUAL, and REPEAT_BACK to the Vulkan backend, making it possible to run test-opt on Vulkan.
NVIDIA's Vulkanised 2025 talks on Cooperative Matrix 2 (CoopMat2) and Cooperative Vectors (CoopVec) are directly relevant. CoopMat2 exposes tensor cores through compute shaders with group-wide matrix multiply, flexible dimensions, and workgroup-scope operations. CoopVec provides a SIMT-friendly interface for neural network layers in any shader stage. Both support outer-product accumulation, which is exactly what LoRA backward passes need.
dW = dY * X^T outer product and CoopVec for the element-wise activation backward. This eliminates the CPU-GPU round-trip that currently bottlenecks mixed CPU-GPU training pipelines. CatGame's Implementation: Pre-Flight Benchmarking + Persistent TOML
CatGame's pi_train tool implements a practical auto-tune system that combines the GPU-side adaptivity research with persistent configuration. The system has three phases:
On startup, pi_train --auto checks for a <model_name>.autotune.toml file next to the model. If found and the hardware identity matches (GPU name, VRAM, CPU cores, system memory), it loads the saved settings and skips the benchmark. If the file is missing, the hardware has changed, or --auto-refresh is passed, it runs a pre-flight benchmark.
The pre-flight benchmark sweeps across candidate batch sizes (1, 2, 4, 8), context lengths (128, 256, 512), and thread counts (8, 16, 32). For each combination, it creates a temporary llama.cpp context, runs 3 forward passes with a test prompt, and measures throughput (tokens/sec), GPU utilization (via NVML), and batch time. The best-performing configuration is saved to the TOML file.
During training, an AdaptiveTuner monitors GPU/CPU utilization and throughput every 10 batches. If GPU utilization is below 70% and CPU is below 50%, it increases the thread count. If GPU utilization is above 85% and VRAM is above 80%, it decreases the batch size. Every adjustment is logged with the telemetry context and saved back to the TOML file, so the next run starts from the improved configuration.
Measured Results on Qwen3.5-2B (F16)
| Config | Batch | Ctx | Threads | Tok/s | GPU Util | VRAM |
|---|---|---|---|---|---|---|
| Default (no auto-tune) | 2 | 128 | 16 | 131 | 66% | 6.0/8.0 GiB |
| After pre-flight | 8 | 128 | 32 | 172 | 71% | 6.0/8.0 GiB |
| After 8 auto-tune events | 4 | 256 | 32 | 189 | 63% | 6.0/8.0 GiB |
The pre-flight benchmark found batch=8, ctx=128, threads=32 as optimal for the RTX 3080 Laptop GPU with the Qwen3.5-2B F16 model. The 31% throughput improvement (131 to 172 tok/s) comes from better GPU saturation: the larger batch keeps the Vulkan compute shaders busier per kernel launch. The auto-tuner then adjusted threads upward during training as CPU headroom was detected.
What's Next: DoRA, QLoRA, and Per-Layer Adaptive Rank
The current implementation tunes batch size, context length, and thread count at the system level. The next step is per-layer adaptive rank, following the AdaLoRA/DoRA/GeLoRA research. This requires:
Importance Scoring
Sensitivity-based (AdaLoRA), intrinsic dimension (GeLoRA), or Hessian-based (SubLoRA). The sensitivity metric is cheapest to compute and well-validated.
SVD Parameterization
Store LoRA updates as U * S * V instead of A * B. Pruning singular values in S is equivalent to reducing rank without retraining.
Rank Pruning Schedule
Gradual pruning over training steps, not one-shot. AdaLoRA uses a cubic schedule; AROMA uses a dual-loop with optimizer state resets.
QLoRA Integration
4-bit base model + LoRA adapters. QDyLoRA enables dynamic rank within quantized training. Requires NF4 quantization support in the GGML backend.
Roadmap (not yet implemented)
Per-layer adaptive rank (AdaLoRA/DoRA style), QLoRA with NF4 quantization, and Vulkan CoopMat2 backward shaders are roadmap items for CatGame's PI training stack. The current system implements system-level auto-tuning (batch/ctx/threads) and persistent TOML configs. The SVD-based rank allocation and quantized training are next-phase work.
The Convergence Point
The research converges on a clear direction: adaptive everything. Adaptive rank per layer, adaptive batch size per training phase, adaptive thread count per GPU utilization window. The hardware and the loss landscape should decide the hyperparameters, not the practitioner's intuition.
CatGame's implementation is a first step: system-level adaptivity with persistent state. The pre-flight benchmark eliminates the cold-start problem. The TOML config eliminates repeated benchmarking. The adaptive tuner handles runtime drift. The next phase brings the same philosophy to the LoRA adapters themselves: let the singular values decide their own rank.
Related: Distillation Farming