L40S Cloud Pretrain: 75M Parameters, 17.7k Tokens/Sec, $7.76

The 8GB Wall

Our GRU-L3 V3a model has 75.9M parameters. At D=2048, B=128, T=1024, K=1, the training footprint is 26.4 GB of VRAM. The RTX 3080 Laptop GPU in our dev machine has 8 GB. We could shrink the model, cut the batch size, or segment the sequence — but all of those compromise training quality. The real answer was simple: rent a bigger GPU.

We picked the NVIDIA L40S. It has 46 GB of VRAM, 91.6 TFLOPS of FP32 throughput, and costs $1.06/hour on Brev's cloud platform. That's enough to run our full production config — B=128, T=1024, K=1 — using 57% of available VRAM, with 43% headroom for future scaling. This post covers the first production pretraining run: epoch 0, from random initialization to a 64.5% top-1 eval accuracy in 6.9 hours.

The Setup

The model is a GRU-L3-V3a: three stacked GRU layers with hidden dimension 2048, untied classifier, byte-level tokenization (vocabulary size 100), and an embedding dimension of 2048. The training config uses Nadam optimization with a WSD (warmup-stable-decay) learning rate schedule, gradient accumulation of 2 steps (effective batch 256), and no gradient checkpointing (K=1) — the L40S has enough VRAM to hold all activation buffers in full.

The training corpus is 395 million characters of mixed dialogue text (7.99 million lines, 378 MB), encoded into 384,426 sequences at T=1024 with byte-level tokenization. The vocabulary is 100 tokens — the 95 printable ASCII characters plus 5 special tokens. No BPE merging, no subword tokenization. Every byte is a token.

The Cloud Workflow

We built a unified cloud interface called cloud.sh that wraps the Brev CLI. It handles the full instance lifecycle: create, deploy, build, train, monitor, download, snapshot, and destroy — with guardrails to prevent accidental data loss.

The key design decision was stateful instance handling. If an instance already exists, cloud.sh reconnects to it instead of creating a new one. Training state is tracked in a local .state file — status, epoch, PID, timestamps, download flags. If the SSH connection drops, training continues on the remote (protected by nohup), and cloud.sh resume reconnects and polls until completion.

Training Results

Epoch 0 ran for 3,004 optimizer steps across 384,426 training sequences. The loss descended from 4.6033 (log(100), the random initialization baseline) to 1.4240 — a 69% reduction. Bits-per-byte (BPB) dropped from 6.64 to 1.88.

Eval accuracy was measured on 187 held-out sequences at 8 checkpoints. The model learned fast — top-1 jumped from 58.4% at step 390 to 63.3% at step 1950, then continued climbing steadily to 64.5% by the final checkpoint. Top-5 accuracy reached 89.3%, meaning the correct next byte was in the top 5 predictions nearly 9 out of 10 times. There was one non-monotonic dip at step 1560 (top-1 dropped from 62.9% to 62.5%), which recovered by the next checkpoint.

The free-running eval (teacher forcing = 0.0, the model feeds its own predictions back) tells a different story: loss 5.53, top-1 7.42%, top-5 23.15%. This is expected — the model has only seen one epoch and hasn't been trained with scheduled sampling or teacher-forcing decay yet. The 20-epoch TF decay schedule will address this in future runs.

Throughput

The L40S sustained an average of 17,666 tokens per second throughout training. Each batch of 128 sequences (1024 tokens each) took ~7.4 seconds: 2.25 seconds for the forward pass, 4.17 seconds for the backward pass, and the remainder for gradient clipping, optimizer update, and telemetry. GPU utilization was 100%, drawing ~317W at 48°C. The throughput variance (14k–18k) comes from periodic eval checkpoints and the final-batch tail effect.

Learning Rate Schedule

The learning rate followed a WSD (warmup-stable-decay) schedule: linear warmup from 0 to 0.002 over the first ~60 steps, a stable plateau at 0.002 for ~2,700 steps, then a smooth decay to 0.0002 over the final ~250 steps. The gradient norm stayed well-controlled throughout, dropping from 4.64 at initialization to 0.52 at the end — the sqrt-scaled clip norm of 2.5 was conservative enough to avoid instability but not so aggressive as to slow learning.

What's Next: Mixed Precision

This run used pure FP32 compute. We've since implemented a mixed-precision path using Vulkan cooperative matrix operations — FP16 tensor-core multiplication with FP32 accumulation and storage. On the local RTX 3080 at D=512, B=64, the mixed path delivers +9.2% throughput overall and +14.6% on the backward pass, with negligible numerical drift (L2 relative weight difference of 1.45e-04 vs FP32).

The key optimization was skipping the scalar per-element post-processing pass for weight gradient GEMMs. When a GEMM only needs accumulation (no bias, no activation, no output remapping), the cooperative matrix store writes the final result directly — no barrier, no scalar read-modify-write cycle. That single optimization accounts for the entire backward speedup.

The L40S has the same cooperative matrix support (compute capability 8.9, Ada Lovelace architecture) and significantly more SMs (142 vs 48 on the RTX 3080 Laptop). At the production config D=2048, the GEMMs are 4x larger, which means better tensor core utilization and more work per cooperative matrix dispatch. We expect the mixed-precision speedup to scale to 15–20% on the L40S at production scale.

Cost Analysis

At $0.02 per million tokens processed, the L40S via Brev is competitive with much larger infrastructure. The model checkpoint (869 MB) and vocab embeddings (800 KB) were downloaded, verified locally (md5 checksums, no NaN/Inf, weight statistics within expected range), and the instance was destroyed after a final 2.5 GB snapshot was taken for future restoration.

Caveats and Limitations

Cross-References