GRU-L3 V3: Character-Level Pretraining on a 4.8M Parameter RNN

The Premise

Modern language model research has overwhelmingly converged on transformer architectures with subword tokenization. But there is a quieter thread of research — character-level recurrent neural networks — that asks a different question: how small can a model be and still speak coherent English?

At CatGame Research, we built a character-level GRU language model from scratch in C++20 with a custom Vulkan compute backend. No PyTorch, no CUDA, no HuggingFace. Every gradient is computed by hand-written compute shaders running on an RTX 3080 Ti. This post covers the architecture, the research that guided our hyperparameters, a regression and recovery cycle that taught us how not to resume training, the fresh-start results that beat our initial run on every metric, and the road ahead.

The question isn't whether RNNs can match transformers. It's whether a 4.8M parameter model with 100-token vocabulary can learn to produce coherent dialogue — and what that tells us about the minimum viable language model.

Architecture: GRU-L3 V3

The model is a 3-layer Gated Recurrent Unit (GRU) network with 512 hidden dimensions, operating on a fixed vocabulary of 100 ASCII tokens. Each character is one token. There is no BPE, no sentencepiece, no subword merging — just raw bytes.

4.8M

Parameters

512

Hidden Dim

GRU Layers

100

Vocab Size

1024

Seq Length

Layer Structure

Each of the 3 GRU layers follows the same pattern: GRU cell → RMSNorm → residual connection → variational hidden dropout. The residual connections allow gradients to flow through the stacked layers without vanishing, while RMSNorm stabilizes the hidden state distribution. The output layer uses tied softmax cross-entropy loss — the embedding matrix and output classifier share the same weight matrix, reducing parameters and improving regularization.

Figure 1: GRU-L3 V3 architecture — 3 stacked GRU layers with RMSNorm, residual connections, variational dropout, and tied softmax output. Spectral norm constraints on all GRU weight matrices.

Regularization Stack

The model uses five complementary regularization mechanisms, each guided by published research on RNN training stability:

Variational Hidden Dropout

Same dropout mask applied across all timesteps within a sequence (variational dropout), rate 0.15. This prevents the model from memorizing specific timestep patterns and encourages robust hidden state representations.

Embedding Dropout

Rate 0.10 applied to the input embeddings. Drops entire token embeddings during training to prevent over-reliance on specific characters.

Spectral Norm Constraint

GRU weight matrices constrained to spectral norm ≤ 1.5. This bounds the Lipschitz constant of each layer, preventing gradient explosion in deep recurrent stacks.

Gradient Clipping

Global gradient norm clipped to max_norm = 1.0. Standard practice for RNN training — prevents rare large gradients from destabilizing the optimizer.

Weight Decay

0.1 L2 weight decay applied via the NAdam optimizer. Decouples regularization from the learning rate schedule.

Label Smoothing

0.10 label smoothing on the cross-entropy loss. Prevents the model from becoming overconfident on individual characters, which improves calibration and generalization.

Training Configuration

The training pipeline uses a custom C++20/Vulkan implementation — no PyTorch, no CUDA kernels. All forward passes, backward passes, and optimizer steps run as Vulkan compute shaders on the GPU. The optimizer state (NAdam first and second moments) lives in GPU memory and is checkpointed alongside model weights for resume training.

Our initial run (Run 0) used aggressive defaults that produced a working epoch 1 but caused a severe regression when we attempted epoch 2. After root-cause analysis and research, we launched a fresh start (Run 1) with research-backed settings. After 3 epochs of Run 1 revealed that 0.10/epoch TF decay is too aggressive, we planned Run 2 with a gentler 0.05/epoch schedule starting at TF=1.0. The table below shows all three configurations side by side — the differences are the subject of this post.

Parameter	Run 0 (initial)	Run 1 (3 epochs)	Run 2 (planned)	Research Basis
Optimizer	NAdam	NAdam	NAdam	Adam-family optimizers are robust for RNN training; NAdam adds Nesterov momentum
Learning Rate	0.003	0.001	0.001	GRU consensus: 6e-4 to 1e-3 (NanoGPT). Run 0 was 3× above consensus
LR Schedule	WSD	WSD-S	WSD-S	WSD-S (Wen et al., 2024): skip warmup on resume, start at peak LR immediately
Warmup Fraction	5% (89 steps)	2% (36 steps)	2% (36 steps)	WSD paper: 2k/100k = 2% for 124M GPT-2 (Wen et al., 2024)
Cooldown Fraction	20% (358 steps)	10% (179 steps)	10% (179 steps)	WSD paper: "insensitive to precise fraction near 10%" (Wen et al., 2024)
LR Minimum	0.00005	0.0001	0.0001	10% of peak — NanoGPT: "low floor near 10% of peak rate"
Batch Size	64	64	64	~3.1GB VRAM; leaves headroom for future model scaling
Teacher Forcing Start	0.90	0.90	1.00	Bengio et al., 2015: canonical schedule starts at ε=1 (TF=1.0)
TF Decay Rate	0.40/epoch	0.10/epoch	0.05/epoch	Run 1 showed 0.10/epoch too aggressive; 0.05 gives gentler adaptation
TF End (final epoch)	0.30	0.60 (E3)	0.00 (E20)	Full decay to free-running over training horizon
Planned Epochs	2	6 (stopped at 3)	20	Run 1 stopped at 3 due to plateau; Run 2 extends for gentler schedule
Block Shuffle	None (epoch 1)	64 (from start)	64 (from start)	UniChunk (Zhao et al., ACL 2024); Best-fit Packing (Ding et al., ICML 2024)
Eval Mode	TF=1.0 only	TF=1.0 + free-running	TF=1.0 + free-running	Liu et al., EMNLP 2021: perplexity "fails to capture error accumulation"
Cooldown Shape	Linear	Cosine	Cosine	Dremov et al., 2025: "shapes that balance exploration and exploitation outperform"
Sequence Length	1024	1024	1024	Long enough for multi-turn dialogue; fits comfortably in GRU memory
Precision	FP32	FP32	FP32	FP32 for numerical stability during validation; FP16 path exists for future
Prefix Dropout	0.20	0.20	0.20	Drops up to 20% of sequence prefix for robustness to varied context lengths

WSD-S Learning Rate Schedule

The WSD (Warmup-Stable-Decay) schedule has three phases: linear warmup from 0 to peak LR, a stable phase at peak LR, and a cooldown phase that decays LR to a minimum. Compared to cosine decay, WSD allows easy resume training (the stable phase is restartable) and often produces better final loss when the cooldown is sufficiently long.

WSD-S (Wen et al., 2024) extends WSD with a critical resume behavior: when resuming from a decayed checkpoint, the scheduler skips warmup entirely and jumps directly to peak LR. This avoids the "double WSD" anti-pattern — re-running a full warmup→stable→cooldown cycle on a converged model, which causes catastrophic forgetting. The LR schedule uses absolute/global step counts, so the cooldown phase of epoch N and the stable phase of epoch N+1 are correctly positioned in the global training trajectory.

The "double WSD" anti-pattern. Our Run 0 epoch 2 re-ran a full WSD cycle (warmup 89 steps → stable 1341 steps at LR 0.003 → cooldown 358 steps) on top of the converged epoch 1 model. The first batch loss jumped 68% (1.435 → 2.411) — a forward-pass measurement before any gradient update, proving the jump was caused by data ordering changes, not LR. But the subsequent 1341-step stable phase at LR 0.003 pushed the model out of its good basin, and the 358-step cooldown could not recover. Eval regressed on all metrics: loss +28.8%, top-1 -5.08pp, top-5 -3.57pp. WSD-S fixes this by skipping warmup on resume and starting at peak LR without re-exploration.

Teacher Forcing Schedule

Teacher forcing — feeding ground-truth tokens as input during training rather than the model's own predictions — is the standard RNN training method. But it creates a train/inference mismatch: at inference time, the model must consume its own predictions. Scheduled sampling (Bengio et al., 2015) addresses this by gradually reducing the teacher forcing rate over training, forcing the model to handle its own predictions.

Our Run 1 schedule linearly decays teacher forcing by 0.10 per epoch across the full training horizon — a gentler approach than Run 0's 0.40 drop in a single epoch. Bengio et al. proposed three decay schedules:

Linear decay: ε = max(ε_min, k − ci) — simple, predictable
Exponential decay: ε = k^i — faster transition to free-running
Inverse sigmoid decay: ε = k/(k + exp(i/k)) — slow start, fast middle, slow end

The key lesson from our regression: TF decay should span the full training horizon, not be reset per-epoch. Run 0 dropped TF from 0.90 to 0.50 in epoch 1, then 0.50 to 0.30 in epoch 2 — a 0.40 jump in a single epoch that confounded with block shuffle introduction and WSD repeat. Run 1 initially planned a 0.10 per-epoch decay across 6 epochs:

Epoch	TF Start	TF End	Decay	Status
1	0.90	0.80	0.10	Completed
2	0.80	0.70	0.10	Completed
3	0.70	0.60	0.10	Completed — plateaued
4–6	0.60→0.30	—	0.10	Abandoned — too aggressive

This matches the Bengio et al. curriculum learning philosophy: the model should face inference-like conditions more often as it becomes capable of handling them. The Flipped Classroom approach (Teutsch et al., 2022) confirms that "increasing training scale curricula consistently outperforms previous training strategies."

0.10/epoch is too aggressive for this model. After 3 epochs of 0.10/epoch TF decay, the training loss plateaued at 1.578 for 60% of epoch 3 — the model was spending the entire epoch adapting to the new TF level rather than improving. Free-running top-1 gains collapsed from +18.8% (E2) to +2.7% (E3). Teacher-forced eval loss degradation accelerated from +2.0% (E2) to +5.6% (E3). The model was running to stand still. Run 2 will use 0.05/epoch decay over 20 epochs instead.

Bengio's canonical schedule starts at TF=1.0, not 0.90. Bengio et al. (2015) explicitly state the canonical schedule starts at ε=1 (full teacher forcing): "at the beginning of training, sampling from the model would yield a random token since the model is not well trained, which could lead to very slow convergence." Our Run 1 epoch 1 at TF=0.90 introduced 10% free-running from the very first batch on an untrained model — violating this guidance. Run 2 will start at TF=1.0 with a full epoch of pure teacher forcing to establish a solid foundation before any free-running exposure.

Dual evaluation. Run 1 adds a free-running evaluation (TF=0.0) after the primary teacher-forced eval (TF=1.0). Liu et al. (EMNLP 2021) showed that perplexity "fails to capture error accumulation" — a model can score well on TF=1.0 eval while producing incoherent free-running text. Feng et al. (2022) found that "lower exposure bias correlates with better generation quality." The free-running eval measures what scheduled sampling actually improves: autoregressive generation quality.

Figure 2: WSD-S learning rate schedule (Run 1) — 2% warmup (36 steps), 88% stable at peak LR (1573 steps), 10% cosine cooldown (179 steps). Total 1,788 optimizer steps for one epoch. On resume, warmup is skipped and training starts at peak LR immediately.

Research Foundations

The hyperparameter choices were guided by published research on character-level RNN training, small model scaling, and continual pretraining. Key references:

TinyStories (Eldan & Li, 2023)

Showed that models below 10M parameters can produce coherent English when trained on sufficiently large, clean data. Their 1M-parameter transformer produced fluent stories on 2.19B characters of simple text. Key insight: data quality and quantity matter more than model size for coherence.

Scaling Laws (Kaplan et al., 2020)

Established that loss scales as a power law in parameters, data, and compute. For our 4.8M parameter model, the Chinchilla-optimal data ratio is ~20 tokens per parameter. Character-level tokens carry less information than subword tokens (~2.3 chars per subword token), so our 119M character corpus provides ~11 effective tokens per parameter — below Chinchilla-optimal, but "Beyond Chinchilla" (Gadre et al., 2024) shows overtraining small models is acceptable.

char-rnn (Karpathy, 2015)

The seminal character-level RNN work. Trained 2-layer LSTMs with 256–512 hidden units on Shakespeare, Linux source code, and Wikipedia. Produced recognizable text at ~1.0 loss. Our architecture extends this with residual connections, RMSNorm, and spectral norm constraints.

WSD-S Scheduling (Wen et al., 2024; Hägele et al., 2024)

"Warmup-Stable-Decay" schedule matches or beats cosine decay for large-scale pretraining. WSD-S extends it with resume-safe behavior: skip warmup on resume, start at peak LR immediately. The stable phase allows indefinite resume training, and the cooldown phase produces a sharp final loss drop. We use 2% warmup, 88% stable, 10% cosine cooldown. arXiv: 2410.05192

WSD Cooldown Dynamics (Dremov et al., 2025)

"Training Dynamics of the Cooldown Stage in WSD LR Scheduler" analyzes how cooldown shape affects final loss. Cosine cooldown balances exploration and exploitation — linear is too aggressive, square-root too conservative. We use cosine cooldown based on this finding.

Universal WSD Dynamics (Belloni et al., 2026)

"Universal Dynamics of Warmup Stable Decay" shows WSD dynamics are "qualitatively similar" across CNN, Transformer, and other architectures. This validates applying WSD to GRU — a non-transformer architecture — without architecture-specific tuning.

Scheduled Sampling (Bengio et al., 2015)

Curriculum learning from full teacher forcing (ε=1) toward free-running (ε=0). Three decay schedules: linear, exponential, inverse sigmoid. Lower ε improves sample quality for long sequences. Per-token sampling (which our code already does) outperforms per-sequence.

Dual Evaluation (Liu et al., EMNLP 2021; Feng et al., 2022)

Perplexity "fails to capture error accumulation" — a model can score well on TF=1.0 eval while producing incoherent free-running text. We run dual eval: TF=1.0 (token prediction) + TF=0.0 (generation quality) to measure both axes independently.

Block Shuffle (Zhao et al., ACL 2024; Ding et al., ICML 2024)

UniChunk and Best-fit Packing show that block-level shuffle prevents position memorization and improves generalization without the overhead of full random shuffle. We use block size 64 — same as batch size — for locality-preserving shuffle. Critical: introduce from epoch 1 to avoid mid-training distribution shift.

Critical Batch Size (McCandlish et al., 2018)

"An Empirical Model of Large-Batch Training" relates optimal batch size to gradient noise scale. For small models, the critical batch size is small — meaning smaller batches provide more optimizer updates per epoch. Our batch 64 vs 128 provides ~2× the optimizer updates (1,788 vs ~906 steps) at the cost of ~50% more epoch time, a worthwhile tradeoff for a model that needs many updates to learn character-level patterns.

Continual Pretraining with Replay

Research on continual pretraining (Gururangan et al., 2020; Xie et al., 2023) shows that adding new domains with replay mixing and learning-rate re-warming prevents catastrophic forgetting. Our curriculum plan uses simple in-corpus replay mixing rather than a separate replay buffer.

Phase 0 Pretraining: Dialogue-First

The training curriculum is deliberately staged. Phase 0 trains on a dialogue corpus only — no code, no math, no general knowledge. The goal is to establish whether the model can learn conversational structure before introducing broader domains. If the model cannot produce coherent dialogue on a single-domain corpus, adding more domains will only make the problem harder.

Run 0: Dialogue-Only Pretraining Epoch 1 CompleteEpoch 2 RegressedArchived

119M character dialogue corpus, 1 epoch at batch 64. Completed in 134.4 minutes. Final: train loss 1.435, eval loss 1.005, top-1 72.1%, top-5 90.8%. Epoch 2 regressed on all metrics due to double-WSD anti-pattern. Archived to catchat_work/archive/phase0_run0/.

Run 1: Fresh Start with Research-Backed Settings 3 Epochs Complete0.10/epoch Too Aggressive

Fresh start with LR 0.001, 2% warmup, 10% cosine cooldown, TF 0.90→0.60 over 3 epochs, block shuffle 64 from start. WSD-S resume validated 3 times. Epoch 1 beat Run 0 on all metrics. Epoch 2 showed healthy scheduled sampling tradeoff (FR top-1 +18.8%). Epoch 3 plateaued — FR top-1 gains collapsed to +2.7%, TF eval loss accelerating degradation. Conclusion: 0.10/epoch TF decay is too aggressive for a 4.8M param GRU. Also: TF=0.90 start violated Bengio canonical schedule (should be TF=1.0).

Run 2: 20-Epoch Schedule at 0.05/epoch from TF=1.0 6 Epochs CompleteFR Top-1 Broke 5% (E4)

Fresh start with TF 1.00→0.00 at 0.05/epoch over 20 epochs (~44 hours). All Run 1 lessons applied: TF=1.0 start (Bengio canonical), 0.05/epoch decay (gentler adaptation), WSD-S resume, block shuffle 64 from E1, dual eval, LR 0.001, FP32. E1-E6 complete: TF eval top-1 peaked at 74.09% (E3), FR top-1 broke 5% at E4 (surpassing Run 1 E3's 4.94%), FR top-5 jumped to 18.31%. E5 showed adaptation dip (FR top-1 to 4.37%), E6 held TF at 0.75 and partially recovered (FR top-1 to 4.75%). The dip→recovery pattern is now confirmed with two data points. Adaptive stopping: if FR eval plateaus for 3 consecutive epochs, hold current TF level.

Knowledge Domain Planned

Add dictionary, thesaurus, and WordNet-style data with in-corpus replay mixing. Re-warm learning rate for distribution shift.

Code & Technical Planned

Add source code, shaders, config files, shell scripts, and man pages. These introduce structured reasoning and precise syntax.

Math & Fiction Planned

Add mathematical reasoning data and fiction corpus. These test the model's ability to handle logical structure and narrative coherence.

Broad Knowledge (Optional) Conditional

Simple English Wikipedia (~87M chars) only if the model demonstrates a need for broader world knowledge. Adding this too early risks diluting dialogue quality.

Training Telemetry: Run 0 vs Run 1

Both runs trained for 1,788 optimizer steps per epoch over 114,418 training sequences (117.2M tokens). All telemetry was collected from the custom Vulkan GPU backend with per-step timing breakdowns. Run 1's block shuffle added ~10 minutes of data-access overhead (127ms/batch slower), bringing total epoch time from 134 to 144 minutes — an acceptable tradeoff for improved generalization.

Run Comparison (Epoch 1)

Metric	Run 0 E1	Run 1 E1	Delta
Training loss	1.435	1.427	-0.008 (better)
Eval loss (TF=1.0)	1.0046	0.9781	-0.0265 (-2.6%)
Eval top-1	72.10%	72.74%	+0.64pp
Eval top-5	90.75%	91.02%	+0.27pp
Eval BPB	1.6459	1.411	-0.235 (-14.3%)
Free-running loss	not measured	6.6728	new baseline
Free-running top-1	not measured	4.05%	new baseline
Free-running top-5	not measured	14.44%	new baseline
Epoch time	134 min	144 min	+10 min (block shuffle)
Cooldown drop	0.020 (358 steps)	0.028 (179 steps)	+40% per step

Run 0 E2 vs Run 1 E2-E3: Catastrophic vs Gradual

Metric	Run 0 E2	Run 1 E2	Run 1 E3
TF eval loss	1.2940 (+28.8%)	0.9976 (+2.0%)	1.0539 (+5.6%)
TF eval top-1	67.02% (-5.08pp)	72.87% (+0.13pp)	72.18% (-0.69pp)
TF eval top-5	87.18% (-3.57pp)	90.88% (-0.14pp)	90.36% (-0.52pp)
FR eval loss	not measured	6.1327 (-8.1%)	5.5666 (-9.2%)
FR eval top-1	not measured	4.81% (+18.8%)	4.94% (+2.7%)
FR eval top-5	not measured	15.40% (+6.6%)	18.69% (+21.4%)

Run 0's epoch 2 was catastrophic — all metrics regressed severely. Run 1's epoch 2 was a healthy scheduled sampling tradeoff, and epoch 3 showed the tradeoff becoming unproductive at 0.10/epoch. Different failure modes require different fixes: Run 0 needed WSD-S and consistent block shuffle, Run 1 needs gentler TF decay.

WSD cooldown is more efficient at 10% than 20%. Run 1's 10% cosine cooldown produced a 0.028 loss drop in 179 steps, while Run 0's 20% linear cooldown produced only 0.020 in 358 steps. The shorter cosine cooldown is 40% more efficient per step. This matches the WSD paper's finding that cooldown is "insensitive to precise fraction near 10%" — and suggests 10% is not just sufficient but actively better for our GRU architecture.

Loss Trajectory (Run 1)

The loss descended through four distinct phases, each corresponding to a different level of linguistic structure being learned:

Phase	Steps	Batch Loss	What was learned
Random init	1	4.60	Uniform prediction over 100 chars
Char frequency	1–76	4.60→1.70	Character frequencies and common bigrams
Word formation	76–376	1.70→1.48	Common words and short phrases
Stable refinement	376–1609	1.48→1.455	Longer-range dependencies, rarer patterns
Cooldown (10%)	1609–1788	1.455→1.427	LR cosine decay sharpens final model (0.028 drop in 179 steps)

Final Eval Results (Run 1, Epoch 1)

After epoch 1, the model was evaluated on 99 held-out sequences (100,650 character predictions) under two regimes: teacher-forced (TF=1.0, ground-truth context) and free-running (TF=0.0, autoregressive generation):

0.978

Eval Loss (TF=1.0)

72.74%

Top-1 (TF=1.0)

91.02%

Top-5 (TF=1.0)

1.411

BPB (TF=1.0)

6.67

Free-Run Loss

4.05%

Free-Run Top-1

After 3 epochs of scheduled sampling, the free-running metrics improved substantially while teacher-forced metrics slightly degraded — the expected scheduled sampling tradeoff:

5.57

FR Loss (E3)

4.94%

FR Top-1 (E3)

18.69%

FR Top-5 (E3)

-16.5%

FR Loss Improvement

+22%

FR Top-1 Gain

+29%

FR Top-5 Gain

The teacher-forced eval loss (0.978) is significantly lower than the train running average (1.427) — a gap of 0.45. This is expected and healthy: training loss includes dropout (variational hidden 0.15 + embedding 0.10) and teacher forcing noise, while eval runs without dropout and with full ground-truth context. A negative gap would indicate overfitting; a gap >1.0 would indicate underfitting. 0.45 is in the healthy range.

The exposure bias gap. The free-running eval (TF=0.0) reveals what teacher-forced eval cannot: the model's autoregressive generation quality. A 72.74% → 4.05% drop in top-1 accuracy (18× degradation) is the exposure bias gap — the model can predict the next character given correct context but cannot sustain coherent generation from its own predictions. After 3 epochs of scheduled sampling, the gap narrowed: FR top-1 improved from 4.05% to 4.94% (+22%), FR top-5 from 14.44% to 18.69% (+29%), FR loss from 6.67 to 5.57 (-16.5%). The gap is closing, but slowly — 0.10/epoch proved too aggressive to sustain the improvement rate. Run 2's 0.05/epoch schedule should allow steadier progress.

Generated Samples (Run 1)

After epoch 1, we ran inference with three sampling configurations to assess generation quality. Samples are saved to epoch1_samples.txt for epoch-to-epoch comparison as training progresses.

Greedy decoding (temperature 1.0, argmax):

DOC MCSTUFFINS: I don't know. DOC MCSTUFFINS: I don't know. DOC MCSTUFFINS: I don't know. DOC MCSTUFFINS: I don't know.

Sampled (temperature 0.8, seed 42):

WORDGIRL: I don't know what I can do WISHBONE: and the same story is a big ball. PAW PATROL: I don't know what I can do GO DIEGO GO: So we can do it.

Sampled (temperature 0.5, seed 123) — best coherence:

DOC MCSTUFFINS: Oh, well, I mean, DANIEL TIGERS NEIGHBORHOOD: We did it! GO DIEGO GO: So we can do it. SISKO: I don't know what they were a lot of the same time.

What the model learned in 1 epoch:

Character-level prediction at 72.74% top-1 accuracy (teacher-forced)
Word formation — all generated words are real English
Dialogue format — speaker labels with colons (NAME:)
Common phrases — "I don't know", "We did it!", "So we can do it"
Recognizable speaker names from the corpus (DOC MCSTUFFINS, SISKO, WORDGIRL, WISHBONE, PAW PATROL, GO DIEGO GO, DANIEL TIGERS NEIGHBORHOOD)
Sentence structure — periods, question marks, capitalization

What the model lacks (needs more epochs):

Diversity — greedy decoding collapses into repetition loops
Long-range coherence — no sustained topic or narrative
Character-specific voice — all speakers sound the same
Context awareness — doesn't respond to prompt content meaningfully
Free-running stability — 4.05% top-1 vs 72.74% teacher-forced

The repetitive loop behavior is classic for a 1-epoch character-level RNN. The model has learned the statistical structure of dialogue but not the semantic content. Greedy decoding enters a deterministic attractor — the model is confident in its repetition. Sampling at temperature 0.5 produces the most coherent fragments, suggesting the model has learned real structure that is masked by the greedy attractor.

CLI temperature bug discovered after epoch 3. The --temperature flag in our cat_chat CLI was silently ignored — the parser had no else clause for unknown arguments, so the flag was dropped without warning. All "sampled" outputs above were actually greedy. The correct flag was --softmax-temperature with --sample. Fixed in commit 18bc429a: added --temperature as alias, auto-enable sampling when temperature ≠ 1.0, and warn on unknown arguments. Epoch 3 samples were regenerated with the fix and show real stochastic output.

Run 1 Complete: 3-Epoch Scheduled Sampling Experiment

After epoch 1, we resumed training for 2 more epochs with WSD-S scheduling (skip warmup, start at peak LR). Each epoch decayed teacher forcing by 0.10: E2 at TF 0.80→0.70, E3 at TF 0.70→0.60. The results reveal a clear pattern — scheduled sampling works, but 0.10/epoch is too aggressive for this model.

3-Epoch Results Comparison

Metric	E1 (0.90→0.80)	E2 (0.80→0.70)	E3 (0.70→0.60)	Trend
Train loss	1.4274	1.4080	1.5735	E3 higher (lower TF)
TF eval loss	0.9781	0.9976 (+2.0%)	1.0539 (+5.6%)	Accelerating degradation
TF eval top-1	72.74%	72.87%	72.18%	Below E1 by E3
TF eval top-5	91.02%	90.88%	90.36%	Declining
FR eval loss	6.6728	6.1327 (-8.1%)	5.5666 (-9.2%)	Steady improvement
FR eval top-1	4.05%	4.81% (+18.8%)	4.94% (+2.7%)	Gains collapsing
FR eval top-5	14.44%	15.40% (+6.6%)	18.69% (+21.4%)	Still improving
Runtime	144 min	132 min	132 min	Stable

The Scheduled Sampling Tradeoff

The 3-epoch experiment reveals a fundamental tradeoff in scheduled sampling: teacher-forced metrics degrade while free-running metrics improve. This is expected — the model is being trained to handle its own predictions, which makes the easier teacher-forced task slightly worse but the harder free-running task substantially better. The question is whether the tradeoff is productive.

Epoch 2: Healthy Tradeoff

TF eval loss worsened +2.0%, but FR top-1 improved +18.8% relative. The model gained more free-running ability than it lost teacher-forced quality. This is the productive scheduled sampling regime.

Epoch 3: Collapsing Gains

TF eval loss worsened +5.6% (accelerating), but FR top-1 only improved +2.7% (collapsing). The model spent the epoch adapting to the new TF level rather than improving. The tradeoff became unproductive.

FR Top-5 vs Top-1 Gap

FR top-5 improved +21.4% at E3 while FR top-1 only improved +2.7%. The model is learning to narrow plausible next tokens but not pick the exact right one — broader context learning without precise token prediction.

Training Plateau

Running avg flatlined at 1.578 for 60% of epoch 3. Descent rate dropped from 0.0008/64-batch (10-29%) to 0.0002/64-batch (48-66%). The model was running to stand still.

WSD-S Resume Validation

WSD-S resume was validated 3 times across Run 1. The signature is consistent: the first batch after resume shows a small loss increase (~11% from the previous epoch's final running average), the second batch spikes to ~2.0-2.2 (from the 10× LR jump after cooldown), and the model recovers within 7 batches. This is the expected behavior — not a failure.

Resume	First batch	Second batch	Gradient norm	Recovery
E1→E2	1.4395 (+2.2%)	2.0611	9.5 (clipped to 1.0)	~7 batches
E2→E3	1.5676 (+11.3%)	2.1755	8.9 (clipped to 1.0)	~7 batches

The E3 first-batch loss is higher than E2's because TF dropped from 0.80 to 0.70 — the model faces 30% free-running exposure instead of 20%. The 0.13 loss difference is proportional to the 0.10 TF drop, confirming the higher loss is from the harder task, not a bug.

Block Shuffle Verification

Block shuffle (size 64) was verified working correctly. The implementation uses std::mt19937 rng(seed=42) created once at startup, with std::shuffle advancing the RNG state each epoch — producing a different block order per epoch while preserving within-block sequence order. 1788 blocks (1787 full + 1 partial), with the partial block moved to the end after shuffling to avoid misaligning batch boundaries. The higher epoch 3 initial loss was not caused by block shuffle — it was caused by the lower teacher forcing rate.

Cooldown Drop Shrinks Across Epochs

The WSD cooldown loss drop shrinks with training: E1=0.028, E2=0.0036, E3=0.0024. The model enters a flatter loss landscape with more training, and the cooldown provides diminishing returns. Most learning happens in the stable phase; the cooldown sharpens the final model but cannot compensate for a plateaued stable phase.

Epoch 3 Generated Samples (CLI Bug Fixed)

After fixing the CLI temperature bug, epoch 3 samples show real stochastic output for the first time:

Greedy decoding (temperature 1.0, argmax):

. I was a little bit of the same o the seat of the same time. DONKEY HODIE: Oh, I don't know what to do. DONKEY HODIE: Oh, I don't know what to do. DONKEY HODIE: Oh, I don't know what to do. DONKEY

Sampled (temperature 0.5) — most coherent:

. we're doing the side. The was no name in support. DAX: (to Carding) I come along with the conncious. WORDGIRL: who wants to get the word WORDGIRL: the party called WORDGIRL: around the best DAN

Sampled (temperature 0.8) — more diverse:

. we're becaZIO: : TW: This is not a demhaZAok aITTLE HEI9XANA: (tution Admiral) Wait!POCOYO: Have to step WONDER PETS: To. Yheo! WONDER PETS IN THE CITY: Connie % helpful toar's srop LAZYTOWN:

The sampled output at temperature 0.5 shows real progress — recognizable speaker names (DAX, WORDGIRL), dialogue structure, and partial coherence. The greedy output still collapses to repetition (DONKEY HODIE loops), but the stochastic output reveals the model has learned more than greedy decoding shows.

Figure 3: Training loss (running average) over the first epoch. The rapid descent phase (steps 1–76) corresponds to learning character frequencies. The word-formation phase (76–376) learns common words. The refinement phase (376+) learns longer-range patterns. Cooldown begins at step 1430.

Anomaly Scan

A full anomaly scan was performed on Run 0's telemetry CSV at 42% completion (751 steps). Run 1's telemetry showed the same clean profile:

Check	Result
Non-finite gradients (NaN/Inf)	0 / 1788 steps — clean
Running loss increases >0.01	0 — monotonically decreasing
Gradient norm spikes (>5× mean)	7, all in steps 8–12 (warmup) — expected
Throughput anomalies	None — stable ~15,000 tok/s (Run 1 slightly lower due to block shuffle)
Memory growth	Stable (no leak)
LR schedule correctness	Peak 0.001, stable phase, cosine cooldown to 0.0001 — correct
Teacher forcing schedule	0.90→0.80 linear decay across epoch — correct
WSD-S resume readiness	Absolute step counter persists; resume will skip warmup — verified by unit tests

Early gradient spikes are normal. Steps 8–12 showed gradient norms of 2.2–2.5 (vs mean 0.41). These are from the random model encountering real data for the first time. The gradient clip (max_norm=1.0) handled them correctly — all post-clip norms were ≤1.0. No spikes occurred after step 12, confirming the 2% warmup phase served its purpose.

GPU Performance Breakdown

The Vulkan compute backend provides per-step timing for forward, backward, and optimizer phases. All times are for batch 64, sequence length 1024, FP32 precision on RTX 3080 Ti:

~1600ms

Forward (mean)

~2400ms

Backward (mean)

~0.7ms

Optimizer

~4300ms

Total Batch (Run 1)

~15K

Tok/s (Run 1)

3.1GB

VRAM Used

The backward pass takes 1.5× longer than the forward pass — expected for GRUs since the BPTT (backpropagation through time) computation requires storing and differentiating through all timesteps. The optimizer step is negligible (NAdam is a simple elementwise update). Run 1's total batch time is ~127ms slower than Run 0 due to block shuffle's reduced data locality — an acceptable overhead for improved generalization. VRAM usage of 3.1GB at batch 64 leaves significant headroom on the 12GB RTX 3080 Ti for future model scaling.

Run 2: 0.05/epoch Schedule — First 6 Epochs

Run 2 launched with all Run 1 lessons applied: TF=1.0 start (Bengio canonical), 0.05/epoch decay (half of Run 1's aggressive 0.10), WSD-S resume, block shuffle 64 from E1, dual eval, LR 0.001, FP32. The critical question: would the gentler schedule avoid the E3 plateau that killed Run 1?

The answer is yes. Run 2 E3 did not plateau. The model continued learning throughout the epoch, with a training running average that descended steadily from 1.225 (10%) to 1.2032 (final). Run 1 E3 was flat at 1.578 for 60% of the epoch. The 0.05/epoch schedule is validated through the critical epoch. Then E4 broke through: FR top-1 exceeded 5% for the first time and surpassed Run 1 E3's best, confirming the long-game bet. E5 showed the expected adaptation dip — the same pattern as E2, but larger — as the model adjusts to TF=0.75. E6 held TF at 0.75 to test recovery: FR top-1 recovered from 4.37% to 4.75%, confirming the dip was adaptation shock, not a capacity ceiling.

Run 2 Schedule

Epoch	TF Start	TF End	Decay	Status
1	1.00	0.95	0.05	Complete
2	0.95	0.90	0.05	Complete
3	0.90	0.85	0.05	Complete — Critical
4	0.85	0.80	0.05	Complete — Breakthrough
5	0.80	0.75	0.05	Complete — Adaptation Dip
6	0.75	0.75	HOLD	Complete — Partial Recovery
7–20	0.75→0.00	—	0.05	Planned

Run 2 E1-E6 Results

Metric	E1 (1.00→0.95)	E2 (0.95→0.90)	E3 (0.90→0.85)	E4 (0.85→0.80)	E5 (0.80→0.75)	E6 (0.75 HOLD)	Trend
Train loss	1.2222	1.1363	1.2032	1.2848	1.3722	1.4636	Rising with TF (expected)
TF eval loss	0.9362	0.9231	0.9372	0.9584	0.9828	1.0118	E2 best, gradual regression
TF eval top-1	73.30%	74.04%	74.09%	73.82%	73.46%	73.00%	E3 peak, slow regression
TF eval top-5	91.44%	91.71%	91.61%	91.42%	91.09%	90.87%	Stable
FR eval loss	6.8606	6.6090	6.3111	6.0526	6.1871	6.1613	E4 best, E6 partial recovery
FR eval top-1	4.47%	4.20%	4.43%	5.00%	4.37%	4.75%	E4 peak, E6 recovering
FR eval top-5	13.71%	13.91%	14.55%	18.31%	16.11%	17.28%	E4 peak, E6 recovering
Runtime	127 min	132 min	132 min	132 min	132 min	132 min	Stable

Run 2 vs Run 1: The Critical E3 Comparison

Epoch 3 was where Run 1 failed. The 0.10/epoch schedule dropped TF from 0.80 to 0.70, and the model spent the entire epoch adapting rather than improving. Run 2 E3 dropped TF from 0.90 to 0.85 — a smaller decrement on a stronger foundation:

Metric	Run 1 E3 (TF 0.70)	Run 2 E3 (TF 0.85)	Delta
Train loss	1.5735	1.2032	-0.370 (-23.5%)
TF eval loss	1.0539	0.9372	-0.117 (-11.1%)
TF eval top-1	72.18%	74.09%	+1.91pp
FR eval loss	5.5666	6.3111	+0.744 (Run 1 better)
FR eval top-1	4.94%	4.43%	-0.51pp (Run 1 better)
FR eval top-5	18.69%	14.55%	-4.14pp (Run 1 better)

The tradeoff is clear. Run 1 had better FR metrics at E3 because it had much more free-running exposure (TF=0.70 vs 0.85). But Run 1 paid for this with catastrophic TF degradation (+5.6% eval loss). Run 2 maintains TF quality (only +1.5% regression, top-1 still improving) while slowly building FR capability. The bet: by E10-E20, Run 2 will surpass Run 1's FR metrics without the TF collapse.

The Plateau That Didn't Happen

The most important result from Run 2 E3 is what did not happen. Run 1 E3's training running average was flat at 1.578 for 60% of the epoch — the model was running to stand still. Run 2 E3 descended steadily throughout:

E3 Progress	Run 1 avg	Run 2 avg	Gap
10%	1.6009	1.2255	-0.375
25%	1.5889	1.2161	-0.373
50%	1.5817	1.2103	-0.371
75%	1.5783	1.2075	-0.371
Final	1.5735	1.2032	-0.370

Run 1 E3 descended only 0.027 over the full epoch (1.6009 → 1.5735). Run 2 E3 descended 0.022 (1.2255 → 1.2032) — similar absolute descent, but Run 2 started from a much lower base and never plateaued. The gap was rock-solid at -0.371 throughout, confirming both models descended at the same rate but Run 2 maintained its head start from E1-E2.

FR Recovery Pattern

Run 2's free-running metrics show a dip-then-recovery pattern across E1-E3:

E1: TF=1.00→0.95 (baseline)

FR top-1: 4.47%. Nearly pure teacher forcing. The model has minimal free-running exposure. FR metrics are baseline — the model can barely generate coherent text on its own.

E2: TF=0.95→0.90 (dip)

FR top-1: 4.20% (-0.27pp). The first epoch with meaningful free-running exposure. The model's generation quality dipped as it encountered its own predictions during training. This is the expected "adaptation shock" — the model is learning to handle free-running for the first time.

E3: TF=0.90→0.85 (recovery)

FR top-1: 4.43% (+0.23pp). The model recovered from E2's dip and is back near E1's level. FR loss improved significantly (-4.5%), and FR top-5 continued climbing (+0.64pp). The adaptation shock is over — the model is now improving at free-running.

WSD-S Resume: E2→E3

The WSD-S resume signature for E3 was consistent with E1→E2. The first batch showed a small loss increase from E2's final (1.1363 → 1.1955), and the second batch spiked from the 10× LR jump after cooldown (1.5307, gradient norm 5.9). The spike was smaller than Run 1's E3 (2.1755, gn 8.9) — Run 2's stronger foundation absorbs the LR jump better.

Resume	First batch	Second batch	Gradient norm	Recovery
E1→E2 (Run 2)	1.4395	1.5870	6.3	~7 batches
E2→E3 (Run 2)	1.1955	1.5307	5.9	~7 batches
E2→E3 (Run 1)	1.5676	2.1755	8.9	~7 batches

Run 2 E3 Generated Samples

Greedy decoding:

. I was a second station. I was a second station. DOC MCSTUFFINS: I don't know DOC MCSTUFFINS: the best friend DOC MCSTUFFINS: and the best friends DOC MCSTUFFINS: I don't know DOC MCSTUFFINS: the best friend

Sampled (temperature 0.5) — most coherent:

. we're all about to bring 100 seconds of last minutes... THE ADVENTURES OF PADDINGTON BEAR: Oh, being the power THE ADVENTURES OF PADDINGTON BEAR: to share anything DOC MCSTUFFINS: and the most game

. we need to go to the stra crew of the place. FETCH WITH RUFF RUFFMAN: All right, Benjamin, but I wnow the sun is aomfort. DRAGON TALES: and I was still doing the DRAGON TALES: and the sea place

Sampled (temperature 0.8) — most diverse:

. we're became. NIRA: Your EREN: Happy Uddness so itend when I would dS: defense DONKEY HODIE: Fi--Say, something, like being more gonna look at the Blch

The E3 samples show richer character diversity than E2 — new speaker names appear (PADDINGTON BEAR, FETCH WITH RUFF RUFFMAN, DRAGON TALES, NIRA, EREN, DONKEY HODIE). The greedy output still loops but with more varied phrases ("the best friend" / "and the best friends" instead of just "I don't know"). Sampled output at temperature 0.5 produces near-coherent fragments: "we need to go to the stra crew of the place" is close to real English.

What Run 2 E3 Tells Us

0.05/epoch Avoids the Plateau

Run 1 E3 plateaued because the 0.10 TF drop was too large — the model spent the epoch adapting. Run 2 E3's 0.05 drop was small enough that the model continued learning throughout. This validates the 0.05/epoch schedule through the critical epoch.

TF Quality is Preserved

Run 1 E3 TF eval degraded +5.6%. Run 2 E3 TF eval degraded only +1.5%, and top-1 continued improving (+0.05pp). The gentler schedule preserves the teacher-forced foundation while building free-running capability.

FR Recovery Confirms Adaptation

The E2 dip (4.47% → 4.20%) and E3 recovery (4.43%) confirm the "adaptation shock" hypothesis. The model needs one epoch to adapt to new free-running exposure, then starts improving. This pattern should repeat at each TF decrement, with smaller dips as the model gains FR experience.

The Long Game

Run 1 had better FR metrics at E3 (4.94% vs 4.43%) because it reached TF=0.70 faster. But Run 1's TF quality collapsed. Run 2 is playing the long game: maintain TF quality, build FR slowly. By E10-E20, Run 2 should reach TF levels Run 1 never saw while maintaining its foundation.

E4: The Breakthrough — Long Game Confirmed

Epoch 4 was where the long-game bet paid off. With TF dropping to 0.80 (20% free-running during training), the model's free-running metrics surged:

Metric	Run 1 E3 (TF 0.70)	Run 2 E4 (TF 0.80)	Delta
FR eval top-1	4.94%	5.00%	+0.06pp (Run 2 surpasses!)
FR eval top-5	18.69%	18.31%	-0.38pp (nearly matched)
FR eval loss	5.5666	6.0526	+0.49 (Run 1 better)
TF eval top-1	72.18%	73.82%	+1.64pp (Run 2 better)
TF eval loss	1.0539	0.9584	-0.10 (Run 2 much better)

Run 2 E4 surpassed Run 1 E3's FR top-1 (5.00% vs 4.94%) while maintaining dramatically better TF quality (73.82% vs 72.18% top-1, 0.9584 vs 1.0539 loss). Run 2 reached this at TF=0.80 — Run 1 was at TF=0.70. The long-game prediction is confirmed: the gentler schedule builds FR capability without sacrificing the TF foundation.

The FR top-5 jump from 14.55% (E3) to 18.31% (E4) — a +3.76pp single-epoch gain — is the largest FR improvement in any epoch of either run. The model is rapidly improving at narrowing plausible next tokens during free-running generation. The FR top-1 gain (+0.57pp) is also the largest single-epoch gain.

E4 Training Stability

E4 showed the same stable training pattern as E3, with a rock-solid gap vs E3's trajectory:

Progress	E3 avg	E4 avg	Gap
10%	1.2265	1.3014	+0.075
25%	1.2161	1.2941	+0.078
50%	1.2099	1.2898	+0.080
75%	1.2069	1.2878	+0.081
Final	1.2032	1.2848	+0.082

The gap widened only 0.007 from 10% to final — both epochs descend at nearly the same rate. The +0.082 gap is fully explained by the TF difference (E3 at TF~0.88, E4 at TF~0.82). No plateau, no instability.

E4 Generated Samples

Sampled (temperature 0.5) — most improved:

PICARD: Daniel, I said than you shall have an ancient way to do anything about it. (beat) The best wish is probably whi

WORDGIRL: Ha ha! WORDGIRL: Huh? MAGGIE AND THE FEROCIOUS BEAST: Whoa! MAGGIE AND THE FEROCIOUS BEAST: Well, we e this is

DOC MCSTUFFINS: Iay! PUPPY DOG PALS: So did you see the animal PUPPY DOG PALS: Okay. Maybe we can't go to the parer to

E4 samples show major improvements over E3: near-grammatical sentences ("I said than you shall have an ancient way to do anything about it"), stage directions ("(beat)"), emotional variety ("Ha ha! Huh?"), and diverse speaker transitions (PICARD, ODD SQUAD, PUPPY DOG PALS, MAGGIE AND THE FEROCIOUS BEAST, WORDGIRL). The model is learning dialogue structure and character interactions, not just character-level statistics.

E5: Adaptation Dip at TF=0.75

Epoch 5 showed the expected adaptation dip — the same pattern as E2, but larger. When TF drops, the model needs one epoch to adapt to the new free-running exposure before FR metrics resume improving:

Dip	Epoch	TF drop	FR top-1 before	FR top-1 after	Dip size	Recovery?
E2	2	0.95→0.90	4.47%	4.20%	-0.27pp	Yes (E3: 4.43%)
E5	5	0.80→0.75	5.00%	4.37%	-0.63pp	TBD (E6)

The larger dip at TF=0.75 makes sense — 25% free-running is a bigger jump from the model's TF foundation than 10% was at E2. The model is being asked to generate more autonomously, and it struggles initially. E6 will determine whether this is a normal adaptation dip (like E2→E3) or a sign that TF=0.75 is too aggressive.

Despite the FR regression, E5 training was rock-solid stable — zero anomalies, steady descent, and the gap vs E4 was perfectly stable at +0.087 throughout the entire epoch. The model is still learning; it just hasn't yet adapted to the new TF level for free-running generation.

E5 inference samples show more character names correctly spelled (O'BRIEN, TEAL'C, WORF, DIEGO GO, MILO, READY JET GO, JAKERS THE ADVENTURES OF PIGGLEY WINKS) but less sentence-level coherence than E4 — consistent with the adaptation dip.

E6: TF Hold at 0.75 — Partial Recovery Confirmed

Epoch 6 held TF flat at 0.75 (no decay) as a diagnostic epoch to test whether E5's dip was adaptation shock or a capacity ceiling. The result: partial recovery confirmed.

Metric	E5 (dip)	E6 (hold)	Recovery	E4 (pre-dip)
FR eval top-1	4.37%	4.75%	+0.38pp	5.00%
FR eval top-5	16.11%	17.28%	+1.17pp	18.31%
FR eval loss	6.1871	6.1613	-0.026	6.0526

E6 recovered most of E5's dip but did not surpass E4. The model is adapting to TF=0.75, not stuck at a ceiling. The recovery pattern mirrors E2→E3 but is larger in absolute terms (+0.38pp vs +0.23pp). The hold strategy worked: giving the model time to consolidate without additional TF pressure allowed FR quality to recover.

The dip→recovery pattern now has two data points:

Dip→Recovery	TF drop	FR top-1 dip	FR top-1 recovery	Net	Surpassed pre-dip?
E2→E3	0.95→0.90→0.85	-0.27pp	+0.23pp	-0.04	No (4.43 < 4.47)
E5→E6	0.80→0.75→0.75	-0.63pp	+0.38pp	-0.25	No (4.75 < 5.00)

Both recovery patterns show partial recovery — the model recovers most but not all of the dip. This suggests that each TF decrement leaves a small permanent FR quality cost, but the model continues adapting. The question for E7 is whether resuming the 0.05/epoch decay (TF 0.75→0.70) produces another dip, or whether the model has now adapted enough to handle the next level.

E6 training showed convergence behavior — the running average barely moved in the second half (1.4677 at 50% → 1.4655 at 90%, -0.0022). This is expected for a hold epoch: the model is consolidating, not learning new free-running skills. One isolated gradient spike (gn=13.18 at step 9835) was fully clipped and recovered in 1 step.

Scaling Considerations

The batch-64 configuration leaves ~4GB of VRAM headroom. This raises the question: when should the model be scaled up, and by how much?

Config	Params	VRAM (B=64)	Time/Epoch	Tokens/Param	Coherence Threshold?
H=512, L=3 (current)	4.8M	3.1 GB	~130 min	11	Below 7M threshold
H=640, L=3	7.5M	~4.3 GB	~170 min	7	Crosses 7M threshold
H=768, L=3	10.8M	~5.7 GB	~200 min	5	Well above threshold
H=896, L=3	14.7M	~7.2 GB	~230 min	3.5	Near VRAM limit

Scaling law caveat: The "7M coherence threshold" is derived from transformer scaling studies on WikiText, not from GRU character-level models. RNNs are more parameter-efficient for character-level modeling because gated recurrence naturally handles temporal dependencies. Published examples of coherent char-level RNNs at 5.8M (LSTM) and 10M (GRU) parameters suggest our 4.8M GRU is in the viable range. The threshold should be treated as a rough guide, not a hard cutoff.

The recommended approach is to validate H=512 first, then scale to H=640 only if the model fails to produce coherent text after 5–10 epochs. Scaling before validating would confound the experiment — if the model produces word salad, we need to know whether the cause is insufficient training, insufficient data, or insufficient capacity. Changing multiple variables at once makes this diagnosis impossible.

Future Work

Run 2: Continue E7-E20

Run 2 E1-E6 complete. E4 broke through (FR top-1 5.00%), E5 dipped (4.37%), E6 partially recovered (4.75%) with TF held at 0.75. E7 decision: resume 0.05/epoch decay (TF 0.75→0.70) or hold again at 0.75. The dip→recovery pattern suggests resuming decay — the model adapts. If E7 dips again, E8 hold would respond. Hold TF if FR plateaus for 3 consecutive epochs.

Top-K Constrained Free-Running

Idea: during scheduled sampling, when using the model's prediction, sample from top-k logits instead of argmax. The FR top-5/top-1 gap (18.69% vs 4.94%) suggests the model narrows plausible tokens but can't pick the exact one. Top-k sampling would feed "near-miss" tokens instead of wildly wrong argmax, with k decayed alongside TF for a two-dimensional curriculum.

Curriculum Pretraining

After dialogue coherence is achieved, add knowledge domains (dictionary, thesaurus, WordNet), then code (C++, Python, GLSL, shell), then math and fiction. Each phase uses in-corpus replay mixing to prevent catastrophic forgetting, with learning-rate re-warming for distribution shifts.

Mixture of Experts (MoE)

Architectural roadmap: convert the feedforward path to a sparse MoE with 2–4 experts. This increases parameter count without proportional compute increase, allowing the model to specialize different experts for different domains (dialogue vs code vs math).

Long-Context Extension

Current sequence length is 1024 characters (~14 dialogue turns). Extending to 2048–4096 would allow multi-scene narrative generation. This requires careful memory management in the Vulkan backend — GRU hidden states are O(seq_len × hidden_dim) per layer.

FP16 Mixed Precision

The FP16 training path exists but was disabled for Phase 0 to ensure numerical stability during validation. Once the FP32 baseline is established, enabling FP16 would roughly double throughput and halve VRAM, allowing larger batch sizes or model dimensions.

On-Policy Fine-Tuning

After pretraining, supervised fine-tuning on curated dialogue examples could improve response quality. Reinforcement learning from human feedback (RLHF) is a longer-term goal, but requires a reward model and preference data — infrastructure that does not yet exist in the Cat Chat pipeline.

Why Not Just Use a Transformer?

This is the question every ML researcher asks when they see a new RNN project. The honest answer has three parts:

1. We wanted to understand the fundamentals. Building a GRU from scratch in C++20 with Vulkan compute shaders forces you to understand every gradient, every gate, every normalization. You cannot hide behind PyTorch's autograd. This understanding is valuable for building any future architecture — including transformers.

2. RNNs are parameter-efficient for character-level modeling. A transformer needs attention heads to model temporal dependencies, which requires O(seq_len²) memory and compute. A GRU models the same dependencies in O(seq_len) with gated recurrence. For a 100-token vocabulary and 1024-character sequences, this is a significant efficiency win.

3. The research question is interesting. TinyStories showed that 1M-parameter transformers can produce coherent English. But what about RNNs? How small can a GRU be and still speak coherent English? Our 4.8M parameter model is in the same range as the TinyStories 1M–10M models, but with a fundamentally different architecture. The answer informs our understanding of what architectural features are necessary for language coherence.

Caveats and Limitations

Single-domain corpus. Phase 0 trains on dialogue only. The model will learn conversational patterns but will not have broad world knowledge. This is by design — the goal is to establish dialogue coherence before broadening the corpus.

3 epochs reported for Run 1. The telemetry covers Run 1's 3 epochs (E1-E3). The 0.10/epoch TF decay schedule was abandoned after E3 plateaued. Run 2 will use 0.05/epoch over 20 epochs from TF=1.0. The 3-epoch data is sufficient to conclude that 0.10/epoch is too aggressive but not sufficient to determine the model's coherence ceiling.

Free-running quality is poor after 3 epochs. The 4.94% free-running top-1 accuracy is not a failure — it is the expected state after only 3 epochs of scheduled sampling. The FR top-5 of 18.69% shows the model is learning to narrow plausible tokens. Teacher-forced metrics (72.18% top-1) measure token prediction, not generation quality. Do not conflate the two.

0.10/epoch conclusion is model-specific. The finding that 0.10/epoch TF decay is too aggressive applies to this 4.8M parameter GRU on 117M tokens/epoch. Larger models or different architectures may tolerate faster decay. The 0.05/epoch Run 2 schedule is a hypothesis, not a validated result.

Chinchilla ratios are approximate. The "11 effective tokens per parameter" figure assumes ~2.3 characters per subword token, which is a rough heuristic. Actual information density varies by corpus and tokenizer. The "Beyond Chinchilla" finding that overtraining small models is acceptable is based on transformer experiments — it may not transfer exactly to GRUs.

Coherence threshold is from transformer data. The 7M-parameter "gibberish to readable" threshold comes from transformer scaling studies on WikiText. GRU character-level models may have different scaling properties. Published examples of coherent RNNs at 5.8M (LSTM on Shakespeare) and 10M (GRU on Wikipedia) suggest the threshold is in the right ballpark but should not be treated as a hard cutoff.

No comparison baseline. We do not have a transformer baseline trained on the same corpus with the same parameter count. Without this, we cannot make direct architecture comparison claims. The research question is whether this specific GRU can achieve coherence — not whether it is better or worse than a transformer.

WSD dynamics validated for GRU qualitatively, not quantitatively. Belloni et al. 2026 showed WSD dynamics are "qualitatively similar" across architectures. Our cooldown drops (0.028, 0.0036, 0.0024 across 3 epochs) confirm the cooldown behavior exists for GRU and shrinks with training, but we have not measured whether the optimal cooldown fraction differs from the transformer-derived 10%.

CLI temperature bug affected all Run 1 samples. All "sampled" inference samples from epochs 1-2 were actually greedy outputs due to the silent --temperature flag ignore. Only epoch 3 samples (regenerated after the fix) show real stochastic output. This means the sampled samples shown for epoch 1 above are not representative of the model's stochastic generation capability at that point.

Cross-References

Previous: CAT VLM V3.5 Stability — the FP16 BPTT and RMSNorm fixes that made this run possible.
Live Training Telemetry — interactive charts from the current pretraining run, updated on each rebuild.