The Short Story
Three epochs into pretraining a character-level GRU language model on a 100-token ASCII vocabulary, the model has crossed 60% top-1 teacher-forced accuracy and bits-per-byte has broken below 2.0. The work spans three threads: disjoint sequence slicing across epochs so each epoch sees fresh data, teacher-forcing decay from 1.0 to 0.95 to begin scheduled sampling, and a bard-talks-to-bing mode that loads two checkpoints on a single GPU and streams an endless dialogue as JSONL. Along the way we visualized per-layer loss landscapes using the filter-normalized random-direction method and watched the basins widen as training progressed.
The Model
Cat Chat GRU-L3-V3 is a stacked gated recurrent network for next-token prediction over a fixed 100-token ASCII vocabulary. The tokenizer maps printable ASCII characters, space, and newline to integer IDs, with reserved slots for pad, turn boundary, begin-of-sequence, and end-of-sequence. The model uses three GRU layers with a 2048-dimensional hidden state, learnable token embeddings of dimension 2048, residual connections between layers, RMSNorm after each residual add, and a linear classifier over the 100-token vocabulary. Training runs on a single NVIDIA RTX 3080 Laptop GPU through a custom Vulkan compute path, with a CPU reference implementation used for validation and loss landscape evaluation.
Stateful BPTT
Hidden state carries across 1024-token windows within a conversation, with gradients truncated at window boundaries. Reset to zeros on EOS.
Residual + RMSNorm
Each layer adds its input to the GRU output, then applies RMSNorm. Layer 0 also receives a residual from the scaled token embedding.
WSD schedule
Warmup, stable, and cosine cooldown phases. Learning rate holds at 0.002 during the stable phase, then decays toward 0.0002 over the final 10% of steps.
Label smoothing
0.10 label smoothing on the cross-entropy loss, with EOS-aware loss masking so padding tokens after end-of-sequence do not contribute to gradients.
Dropout and clipping
Variational hidden dropout at 0.15 between layers, embedding dropout at 0.10, and gradient clipping at 2.5. Weight decay 0.01 with decoupled AdamW.
Disjoint Sequence Slicing
The pretraining corpus contains roughly 3.8 million records and encodes into approximately 1.56 million training sequences of length 1024. Rather than retraining on the same slice each epoch, we introduced a seq_offset parameter that skips the first N generated sequences and trains on the next window. This keeps each epoch on fresh data without reshuffling the corpus.
| Epoch | Sequence range | Train seqs | Source records | Eval loss | Top-1 | Top-5 | BPB |
|---|---|---|---|---|---|---|---|
| 1 | 0 to 9,999 | 10,000 | 1.56M | 1.5503 | 58.64% | 85.48% | 2.237 |
| 2 | 10,000 to 15,658 | 5,659 | 30,000 | 1.3688 | 60.80% | 88.23% | 1.975 |
| 3 | 15,659 to 25,658 | 10,000 | 200,000 | 1.4262 | 59.98% | 87.14% | 2.058 |
max_samples parameter caps source records, not generated sequences. With 30,000 records and a roughly 5.25 records-per-sequence yield, only 5,659 sequences fell in the offset window. Epoch 3 raised the cap to 200,000 records to reach the full 10,000-sequence target.Scheduled Sampling: Teacher Forcing 1.0 to 0.95
Epoch 3 introduced teacher-forcing decay. At the start of the run, every timestep uses the ground-truth previous token as input (teacher forcing = 1.0). Over the course of the run, the rate linearly decays to 0.95, meaning 5% of timesteps feed the model's own previous prediction back as input. This is scheduled sampling, a technique introduced by Bengio et al. (2015) to reduce the train-test mismatch in autoregressive models.
The decay is linear over the run's total steps. At step 0 the rate is 1.00, at the midpoint it is 0.975, and at the final step it reaches 0.95. The training log confirms the schedule: tf=1.00 at step 32, tf=0.99 at step 1056, tf=0.96 at step 7200, and tf=0.95 by step 9120. The effect on eval loss is visible: epoch 3's final eval loss (1.4262) narrowed the gap with epoch 2 because the scheduled sampling introduces a train-time distribution shift that the fully teacher-forced eval does not match. This is expected and is the cost of preparing the model for free-running generation.
Loss Landscape Visualization
To understand how the loss surface changes between epochs, we implemented the filter-normalized random-direction method from Li et al. (2018). For each GRU layer, we pick two random directions in the update-gate weight matrix W_z, normalize each to the same Frobenius norm as the original weights, and evaluate the loss on a fixed text prompt at every point on a 2D grid. The result is a contour plot of the loss surface around the trained weights.
The forward pass was reimplemented in Python from the C++ reference: token embedding lookup scaled by sqrt(E), three GRU layers with update, reset, and candidate gates, residual connections before RMSNorm, and a linear classifier with softmax cross-entropy. The reimplementation was validated against the C++ model by matching the baseline loss on a 25-character prompt (24 input/target token pairs after shifting) (1.0153 for epoch 1, 1.1227 for epoch 2).
The weight norms grew 15 to 25 percent from epoch 1 to epoch 2 across all three gates and all three layers. This is consistent with the model moving away from its small random initialization and settling into a deeper, wider basin. The loss landscape contours (generated as 5x5 grids with a 0.3 alpha range) show the epoch 2 basin is wider and flatter around the trained weights, which is the expected signature of a model that has converged further.
Bard Talks To Bing: Two-Model Dialogue
To probe whether the model has learned conversational structure, we built a bard-talks-to-bing mode that loads two different checkpoints and has them talk to each other. The mode keeps both models resident on the GPU for the duration of the dialogue, alternating between speaker A and speaker B, and streams each turn as a JSONL record with provenance fields.
Each response is generated with a rolling context window of approximately 800 characters to fit within the 1024-token limit. Speaker prefixes and <turn> separators match the training corpus format. N-gram blocking (default n=5) prevents repetitive loops. Temperature sampling with optional top-k and top-p controls the stochasticity. The output is a JSONL file where each line contains an id, turn index, speaker name, text body, and model checkpoint path.
A 20-turn run between epoch 1 (Bard) and epoch 2 (Bing) showed distinct personalities: Bard leaned toward news and political vocabulary (president, campaign, House), while Bing produced more sci-fi dialogue (station, ship, Starfleet, shields, Chief O'Brien). Both models generated questions, conversational markers (Well, you know), and character names from the Star Trek and Stargate training corpus. The dialogue structure is present even though the text is far from coherent, which is consistent with the model's free-running loss being much higher than its teacher-forced loss.
Training Trajectory
The three epochs show a clear progression. Epoch 1 established the baseline at 58.64% top-1 and 2.237 BPB. Epoch 2, despite training on fewer sequences (5,659 vs 10,000), improved all metrics substantially: eval loss dropped 11.7%, top-1 crossed 60%, and BPB broke below 2.0. Epoch 3, with scheduled sampling and a fresh 10,000-sequence slice, finished at 59.98% top-1 and 2.058 BPB, narrowing the gap with epoch 2. The scheduled sampling cost is visible: the eval metrics do not improve over epoch 2 because the training distribution has shifted, but the model is better prepared for free-running generation.
What Comes Next
More epochs
Continue training on fresh 10k-sequence slices. The corpus has 1.56M sequences, so there is room for dozens of disjoint epochs before reuse.
Free-running eval
Epoch 2's free-running loss is 5.63 with 6.78% top-1; epoch 3 improved to 5.51 with 7.55% top-1. More epochs and deeper scheduled sampling should improve coherent generation.
Optimizer state fix
The NVIDIA Vulkan path segfaults on optimizer-state upload. Currently skipped, causing a transient loss spike at the start of each resumed epoch. A proper fix needs buffer-layout compatibility between CPU Adam state and GPU optimizer buffers.
Synthetic dialogue
The JSONL output from bard-talks-to-bing is valid training data. A future corpus revision could incorporate generated dialogue as synthetic augmentation, with provenance preserved.
Caveats and Limitations
The loss landscape visualizations use a 5x5 grid with a 0.3 alpha range, which is coarse. The filter-normalized random-direction method is sensitive to the choice of directions; different random seeds produce different contours. The baseline losses (1.0153 for epoch 1, 1.1227 for epoch 2) are computed on a single 24-token prompt ("Hello, how are you today?"), not on the full eval set, so they are not directly comparable to the held-out eval loss reported in the training table.
The eval loss for epoch 3 (1.4262) is the final held-out eval after the WSD cooldown completed. The scheduled sampling schedule (1.0 to 0.95) is linear and mild; more aggressive decay schedules may produce different tradeoffs between teacher-forced and free-running metrics.
The two-model dialogue observations (Bard leaning political, Bing leaning sci-fi) are qualitative and based on a single 20-turn run. The character names and vocabulary that appear in the generated text come from the training corpus, which includes Star Trek and Stargate dialogue; the models have not learned to generate coherent original dialogue, only to reproduce statistical patterns from the corpus.
The weight norm growth between epoch 1 and epoch 2 is a measured value from the checkpoint files. The claim that this indicates a "wider basin" is an interpretation based on the loss landscape literature (Li et al., 2018), not a direct measurement of basin width. The contour plots themselves were generated but are not embedded in this post; the bar chart above shows the weight norm growth that motivates the basin-width interpretation.