Cat Chat V5 R1: The Special-Token Bug, the R0 Post-Mortem, and a Roast-Driven Restart

The Three-Day Arc

Over the last three days the Cat Chat GRU-L3 pretraining pipeline went through a full cycle: a corpus restructure and cleanup, a deep anomaly scan, a post-mortem of the failed Run 0, the discovery of a special-token ID bug that invalidated every prior checkpoint, a second roast of the V5 R1 config and trainer code, and a fresh restart with all fixes applied. This post is the field report.

The model under the hood is a 3-layer GRU with hidden dimension 2048, a 101-token ASCII character vocabulary, and a 1024-token context window, trained from scratch on a 4.3 GB dialogue corpus of 4.99 million conversations. It runs on a single RTX 3080 Laptop GPU via a custom Vulkan compute pipeline written in C++20. There is no PyTorch, no CUDA, no framework — every shader, every backward pass, every optimizer step is hand-written.

The Special-Token ID Bug

The first discovery was a vocabulary layout change that silently broke checkpoint compatibility between V5 Run 0 and V5 Run 1.

Run 0 used a 100-token layout where <unk> shared ID 1 and a fallback UNK sat at ID 99. Run 1 introduced a dedicated <unk> at ID 4 and added a <turn> token at ID 3, shifting every printable ASCII character up by one. The embedding table row that Run 0 learned for "the letter e" (ID 14) now pointed at a different character in Run 1's layout. Every checkpoint from Run 0 became garbage the moment the tokenizer changed.

The Run 0 Post-Mortem

Before the special-token bug was found, Run 0 had already failed on its own merits. Epoch 1 trained successfully (eval loss 1.0046, top-1 72.10%), but epoch 2 regressed on every metric.

The smoking gun was the first batch of epoch 2. Loss jumped from 1.435 (end of epoch 1, post-cooldown, LR=0.00005) to 2.411 (epoch 2 step 1, warmup restart, LR=0.0) — a +68% jump on a single batch. The warmup ramp from lr_min to lr_max over 89 steps is a 60x learning-rate jump that disrupted the converged model.

This is the "double-WSD" problem. The trainer uses a WSD schedule (warmup → stable → decay). On resume, the schedule restarts from step 0 relative to the loaded checkpoint, re-running the full warmup and stable phase at peak LR=0.003 on a model that is already sitting in a good basin. The stable phase pushes the model out of that basin; the cooldown cannot recover it.

Run 0 vs Fresh Start (Phase 0 E1)

After the root cause was confirmed, a fresh start with research-backed settings beat Run 0 on every metric.

The fresh start used LR=0.001 (3x lower than Run 0's 0.003), a gentler TF decay (0.90→0.80 vs 0.90→0.50), block shuffle from epoch 1 (vs none), and a shorter cooldown (10% vs 20%). The WSD sharp loss drop during cooldown was confirmed for the GRU architecture, matching Belloni et al. 2026 (Universal Dynamics of WSD): "most training signals qualitatively similar in CNN vs Transformer."

The Corpus Pipeline (Three Days of Cleanup)

Before V5 R1 could start, the corpus needed to be rebuilt. The old pipeline had destroyed all conversation structure: every record was separated by a blank line, making conversation boundaries indistinguishable from record separators, and the C++ loader then randomly shuffled all records. The model never saw a coherent multi-turn conversation.

Round 2 Roast: Five Fixes

With the corpus rebuilt and the special-token bug understood, the V5 R1 config and trainer code went through a second roast. Five issues were found and fixed.

R10: The grad_h_clamp Precision Mismatch

The per-timestep BPTT hidden-state gradient clamp was set to 8.0. This value originated from FP16 stability work — in FP16, BPTT grad_h can saturate at [-65504, 65504], and a tight clamp prevents the GEMM that converts gate gradients to input gradients from overflowing to -Inf. But V5 R1 runs in FP32, where the range is [-3.4e38, 3.4e38] and there is no overflow risk. A clamp of 8.0 is 125x tighter than the FP16 production recommendation of 1000.0. It was silently clipping legitimate gradients, slowing convergence.

R11: Stateful BPTT and Shuffle Are Incompatible

Stateful truncated BPTT carries the hidden state from the last timestep of batch N into the first timestep of batch N+1. This only makes sense if batch N+1 is the continuation of batch N — the same conversation, the same logical stream. Block shuffle of size 32 (= batch size) means every batch is a random conversation. The carry state from conversation A's last timestep feeds into conversation B's first timestep. The model must "unlearn" the previous conversation at every batch boundary.

The research is unambiguous: stateful BPTT requires contiguous data.

The fix: when preserve_record_order = true, skip all shuffling and leave indices in source order. The log confirms: preserve_record_order=true, skipping shuffle (20000 sequences in order).

R12: The global_step Off-By-One

The old code computed the learning rate for step N, then incremented global_step to N+1, then dispatched the batch. The checkpoint filename used global_step = N+1, but the model state inside that checkpoint was trained with the LR schedule for steps 0 through N. The label and the schedule were off by one.

// Before (off-by-one):
current_lr = compute_lr(lr_params);  // uses step N
++global_step;                        // now N+1
dispatch_batch(...);                  // batch trained with LR for N, checkpoint named N+1

// After (correct):
current_lr = compute_lr(lr_params);  // uses step N
dispatch_batch(...);                  // batch trained with LR for N
++global_step;                        // now N+1, checkpoint named N+1 = batches completed

R13: Unit Drift in eval_interval

eval_interval was in batches (50), while checkpoint_interval, seq_offset, max_sequences, and replay ranges all used sequences. Megatron-LM uses steps/batches, but this codebase is sequence-oriented throughout. The fix: treat eval_interval as sequences (changed 50 → 1600, same behavior since 50 batches × 32 seq/batch = 1600), and use boundary-crossing detection on total_seqs_processed — the same pattern already used for checkpoints.

Replay vs Offset Rotation

With shuffle disabled for stateful BPTT, the replay mechanism broke. Replay randomly samples from a prior range and appends to the index list. Without shuffle, all replay records dump at the end as a concentrated block. The carry state from the last new-data record feeds into the first replay record (unrelated conversation), and each replay record feeds into the next (random order, unrelated). The carry state is noise between every replay batch.

The fix is architectural: disable replay for pretraining and use seq_offset rotation instead. The model walks through the corpus in order, and when it reaches the end, it cycles back to the beginning. Old data is revisited naturally on the next full corpus pass — no random sampling, no carry contamination.

The E/C Distinction

A terminology correction emerged during the teacher-forcing analysis. The trainer processes 20,000 sequences per configured epoch. The corpus has ~5 million records. One configured epoch is a subset, not a full pass. The correct terminology:

Teacher-forcing decay must be keyed to cycles, not epochs. The plan:

The floor at TF=0.7 is deliberate. Below that, the carry state degrades over 1024-token windows — a bad prediction at timestep T feeds forward to T+1, T+2, and beyond through the stateful carry. TF=0.5 or lower is for fine-tuning, where sequences are shorter and the model is already well-trained. The trigger conditions for C3 decay: TF eval loss < 2.0, top-1 > 40%, and two full corpus cycles complete.

The Fresh Restart

After all fixes, the previous E1 was killed and a fresh run started with the fixed binary. The first eval at 1,600 sequences:

Training loss dropped from 4.61 to 2.10 over the first 8% of the epoch. Gradient norms are healthy and no longer pinned to the old 8.0 clamp. The log confirms record-order preservation, no shuffle, and FP32 precision throughout.

What We Learned

Caveats and Limitations

The fresh E1 run is still in progress at the time of writing. The eval metrics reported (loss 2.46, top-1 33.7%) are from the first 1,600-sequence checkpoint, not the final epoch-end eval. The teacher-forcing cycle plan is a forward design — C3 decay (E501+) will not trigger until two full corpus cycles complete and the trigger conditions (eval loss < 2.0, top-1 > 40%) are met. The carry-state snapshot idea (persisting hidden state in checkpoints for correct stateful resume) is an unimplemented proposal, not a shipped feature. The replay code path remains in the codebase but is disabled (replay_count = 0) for pretraining; it may need cleanup or gating behind a fine-tuning flag when conversation-isolated training arrives.

The WSD-S reference (Wen et al. 2024, arXiv 2410.05192) was verified via arXiv. The Belloni et al. 2026 "Universal Dynamics of WSD" reference is from prior session research and was not re-verified for this post. The Bengio et al. 2015 scheduled-sampling findings (per-token coin flips better than per-sequence, linear decay comparable to inverse sigmoid, do not go to TF=0) are from the canonical paper and were verified in prior sessions.