Scaling the Logic Model: 96K Records, a Four-Epoch Wall, and Gradient-Spike Forensics

Last dev-log we introduced the logic model: a small, stateful, locally-trained GRU that answers Jev-style state + question → options requests, plus LRL, a compiled language that chains those answers into branching decision programs. Since then the work moved from "does it run" to "why does it move." This entry is the scaling story: a corpus jump to 96,506 records across 51 task families, a model that kept its accuracy after losing half its parameters, and a telemetry dig that ended with our gradient spikes matching a finding from the PaLM paper almost line for line.

96,506L9 records

51task families

18.7MBPE tokens

19.4Mparameters

49.9%best test acc

~ep4repeat-value knee

The contract, briefly

Every record is the same shape: a state (the world the decision is about), a question ("which tool?", "is this safe?", "what next?"), and a set of options. The model encodes state and question through a three-layer GRU, then forks the resulting hidden state once per option and scores each fork independently — no positional signal, so permuting the options cannot change the answer. A second head emits confidence, and a NOUL answer type ("none of the above") handles abstention. The whole point is locality: users roll their own domain-specific decision model from a TOML config and JSONL data, no hosted dependency. The previous post covers the format and the LRL compiler pipeline (lexer → AST → optimization passes → bytecode → VM) in detail.

The dataset ladder

The corpus grew in steps, each one a distinct dataset build with its own split files — L6 through L8 lived at the ~20K–27K-record scale, dominated by procedural families. Each taught us something different: L6 exposed cross-dataset generalization gaps, L7 was the first large Jev-style expansion, L8 rebalanced eval coverage. But 20K records turned out to be a ceiling, not a plateau we could tune our way off.

L9 is the answer to that ceiling: 96,506 unique records — 69,123 train / 11,264 val / 16,119 test — across 51 families spanning tool calls, model routing, retrieval, safety verdicts, agent decisions, classification, ranking, extraction, and the Jev/System-One-style decision patterns that seeded the project. Measured with the run's real BPE tokenizer it comes to 18.7M tokens (49.6M characters, ~2.65 chars/token), 1.19M scored option texts, mean arity 3.9 options per record with a 64-option maximum. Labels are 59% procedural generators and 41% real Hugging Face-derived records, and the paraphrase pressure is genuine rewording — new question framings, option orderings, state details — not word-swap templating.

The splits passed the full quality-gate suite before launch: no exact record collisions, no state/question or state-seal collisions, no cross-split leakage, no ambiguous options, no positional-skew cells, integer-only NOUL answers, and a permuted test variant (test_perm.jsonl) that remaps cleanly. Known limits we are honest about in the card: tool_risk has only 307 records, model_route 369, and agent_audit reuses states heavily — three of the families we most want to grow next.

The runs: v5 → v8

Four training lineages bracket the dataset ladder. The naming is honest bookkeeping: v<n>_L<m> means run n on dataset m.

run	dataset	params	test acc	ECE	note
v5	L5-era (~20K)	39.9M	63.5% / 55.8%*	0.052	feature-stack experiments; *on L6 eval
v6_L7	L7 (~25K)	39.9M	48.6%	0.079	first Jev-family expansion; eval GPU path
v7_L8	L8 (27,178)	19.4M	49.9%	0.074	half-size model, peak at ep4
v8_L9	L9 (96,506)	19.4M	in progress	—	ep1 val 35.7% on a much harder val set

Two of those rows carry the story. v7 halved the model (39.9M → 19.4M parameters — embedding 512, three GRU layers of width 1024, a 4,123-token BPE vocab) and gained accuracy: 49.9% test versus v6's 48.6%, with calibration actually improving (ECE 0.074, near-diagonal reliability). On a problem this size, parameters were already ahead of data — shrinking the net cost nothing because the bottleneck was records, not weights.

Validation accuracy and loss across runs — v8_L9's epoch-1 point sits lower because L9's validation set is 4x larger and includes eval-only families — Validation across runs. The early v-series (L6-era corpora) reach into the 60s on small eval sets; v6–v8 evaluate on progressively harder splits. v8's epoch-1 point (35.7%) is not directly comparable — L9's val is 11,264 records including eval-only blocks the smaller sets never tested.

Checkpoint scorecards — accuracy, ECE, and Brier score per checkpoint; v7_best at 49.9 percent with 0.074 ECE — Checkpoint scorecards. v7_best (epoch 4) holds the best calibrated accuracy: 49.9% test, ECE 0.074, Brier 0.229. v5's higher raw acc reflects its smaller, easier eval set — the cross-eval column (v5 scored on L6 data) shows the generalization drop.

Per-domain accuracy trajectories across v5, v6, and v7 checkpoints — code and config domains near the top, story and game near the bottom — Domain trajectories across the last three scorecards. Structured/code domains lead (python 1.00, build_config 0.95, shell 0.89 on v7) while narrative domains trail (story ~0.31, game ~0.29) — the same split visible in the family-diff chart.

The four-epoch wall

Both completed runs peaked around epoch 4 and then stopped gaining. v6's validation loss bottomed at epoch 3 (1.140) and crept up at epoch 4 while accuracy flatlined at ~47%; v7's accuracy topped at epoch 4 (47.0% val / 49.9% test) and epochs 5–6 slid backward while confidence kept sharpening — the classic memorization signature, visible family-by-family: dialogue-walk −17pp, story-continue −10pp, pathfinding and puzzle −6–7pp, while lookup-style families held or even gained.

That knee is not a bug — it is a published law arriving on schedule. Muennighoff et al.'s Scaling Data-Constrained Language Models (NeurIPS 2023, 400+ runs) finds that repeated data is worth nearly as much as fresh data for about four epochs, after which its value decays toward zero. Our 19.4M-parameter model on 13.1M unique training tokens sits at 0.67 unique tokens per parameter — miles under the Chinchilla ~20:1 compute-optimal ratio — so the binding constraint is data value, and the knee showed up exactly where the paper predicts.

"The model did not stop learning because it was broken. It stopped because it had already learned the data."

The arithmetic that follows: to reach the ~20 tokens-per-parameter point while staying inside the useful-repeat window (~4 epochs), the corpus needs roughly 97M unique tokens — about 500K records at L9's shape. That is the L10 target, and it reframes the roadmap: the next lever is corpus scale, not hyperparameters.

Gradient-spike forensics

v8 opened with a wall of gradient spikes — window-max norms hitting ~9,000 in the first 500 batches, 13 of them bad enough to trigger the grad_skip guard (which clears the accumulated gradients and refuses the optimizer step — records stay in the dataset, the update just does not land). Then, on schedule, it calmed: post-transient window-maxes settled into the 8–14 band with isolated outliers about once per epoch.

v8 gradient-norm timeline — dense spike cluster in the first 500 batches with 13 skip markers, then scattered outliers decaying to window-max around 10 — v8's gradient-norm timeline. The dense cold-init cluster decays by ~batch 500–700; the 13 orange triangles are `grad_skip` firings. The lone mid-run outliers (~1/epoch) are batch-by-state events, not a trend.

The investigation produced three findings worth writing down:

1. The spikes are head-dominated, not recurrent. Per-layer norms (gL) consistently showed the score head carrying 5–11 while GRU layers sat at 0.2–2 — this is the loss geometry of sparse decision supervision, not recurrent instability.

2. Halving the learning rate stretched the transient ~2.3×. v7's transient resolved by ~batch 224 at lr=1e-3; v8's at lr=5e-4 resolved by ~batch 500–700. Adam's per-step escape magnitude is ~lr regardless of gradient size, so halving lr roughly doubles the basin-exit distance in batches. The transient was never an instability — it was a cold-init basin geometry the optimizer walks out of at lr-controlled speed.

3. Record length is not the driver — and accumulation was not the fix. We considered grad_accum=2 to dilute the spikes and rejected it: dilution would have let a gn≈8,800 batch evade the 5,000 skip threshold, and the recipe deliberately sits in Masters & Luschi's small-batch regime (m=2–32), where the gradient noise is a mild regularizer we want to keep. So we tested the record-length hypothesis directly — sliding-window BPTT should scale with record length, and slow windows (rec/s dips) did visually coincide with spikes. Post-transient, at constant LR: corr(gn, tokens/record) = +0.01, corr(gn, rec/s) = +0.04 across 81 windows. Flat. The visual correlation was a cold-init confound — spikes and low throughput shared the same early wall-clock region for unrelated reasons. What actually correlates: batch-by-model-state interactions — the same conclusion PaLM reached at 540B parameters (spikes recur only for specific batch×parameter-state combinations, not bad data), now observed at 19.4M.

Per-family accuracy difference between best and final checkpoints — state-tracking families dropped 6 to 17 points while lookup families held — Per-family accuracy, v7 best vs final. The ep5–6 regression concentrated in state-tracking families (dialogue_walk, story_continue, pathfinding, puzzle, lrl_branch) — skills that cannot be memorized — while lookup families held or gained. The overfit signature, made visible.

The walk-vs-record gap

The most honest number in the scorecard is the smallest one. v7 scores 49.9% on independent test records, but in the LRL suite — where the model's own answers feed the next ask in a program — node accuracy is 13.8%. Sequential decisions compound: a model that is right half the time per record does not get to half-right over a five-node walk.

That gap is a distribution problem, not a capacity problem — training records contain gold prefixes, but at inference the model conditions on its own prior decisions. The fix direction is DAgger-style self-trajectory records (Ross et al., 2011): train on the state distribution the model actually visits, annealing between gold and model-generated prefixes. That is the next dataset family on the list — L9 added the coverage; the walk data has to teach compounding.

Tooling shipped this cycle

None of the analysis above was visible until the telemetry got honest. This cycle's instrumentation work:

Windowed train telemetry — loss= now reports the reporting-window mean separately from the epoch average; a cumulative-mean display had been manufacturing a phantom "loss rises at epoch start" pattern.
Eval progress heartbeats — the trainer's post-epoch validation sweep, cat_chat logic eval, and suite walks all emit ~10s progress lines with rate and ETA. L9's 11K-record val made silent multi-minute sweeps unacceptable.
Live shard logs — the scorecard wrapper now streams each worker's stderr to per-shard log files and prints a pending-shard status line every ~20s, instead of hiding worker output in pipes until exit.
plot-run.py and compare-runs.py — per-run diagnostics (curves, gradient timeline, reliability, arity, per-family diffs) and cross-run aggregation (val curves, scorecards, domain trajectories, spike/skips, efficiency), including segmentation of polluted telemetry from restarted runs.

v7 reliability diagram — predicted confidence vs measured accuracy tracks near the diagonal — v7_best calibration. Near-diagonal reliability — the confidence head means what it says, which is what makes the abstention and routing families trustworthy.

Where this goes

The current run — v8 on L9 at lr=5e-4 — is the data-scaling test: does 3.5× the records with the same 19.4M weights move the knee? Epoch 1 landed at 35.7% val on a validation set four times larger and deliberately harder (eval-only blocks L8 never tested), so raw comparison waits for the per-family readout.

v8 mid-run training curves — windowed and epoch-mean train loss descending through epoch 2, epoch-1 validation point at 35.7 percent — v8 mid-run, epoch 2 of 10. Train loss tracking below epoch 1's mean while the epoch-1 validation point (35.7%) anchors the run — the per-epoch scorecard sweep at run end will show whether the knee still lands at ~ep4.

Whichever way it lands, the roadmap is set by arithmetic, not vibes:

L10 corpus (~500K records) — the Chinchilla-scaled target; proportional family growth, with tool_risk/model_route/agent_audit prioritized.
Self-trajectory records — DAGGER-style walk data to close the 3.6× record-vs-node gap.
Per-epoch scorecard sweeps — every checkpoint scored post-run on the now-free GPU lane, turning "which epoch peaked" from a telemetry guess into a curve.
Family balancing — gradient mass is currently ~25× skewed toward the largest families; per-family weighting is the next trainer feature.

Caveats

Comparisons across runs are cross-dataset. Each run's eval set differs in size, family count, and difficulty; raw accuracy deltas are directional evidence, not apples-to-apples. The per-family and per-domain views are the honest comparison.
v8 is in progress. Epoch-1 metrics (35.7% val, 1.2176 loss) are one data point on a harder validation set — the verdict needs the full run plus the post-run scorecard sweep.
The repeat-wall framing is an inference, not a measurement. Muennighoff et al. studied transformer LMs up to 9B params; our 19.4M GRU matching the ~4-epoch knee twice is suggestive, not proof the same law binds here.
Suite metrics sample ~325 programs. Walk/node accuracies carry meaningful variance at that n.
Scaling arithmetic is order-of-magnitude. The ~500K-record target comes from a token-per-parameter ratio designed for compute-optimal LM pretraining; it is a planning number, not a fitted law.

References

Muennighoff et al., Scaling Data-Constrained Language Models, NeurIPS 2023 — arXiv:2305.16264
Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla), NeurIPS 2022 — arXiv:2203.15556
Chowdhery et al., PaLM: Scaling Language Modeling with Pathways, 2022 — arXiv:2204.02311
Masters & Luschi, Revisiting Small Batch Training for Deep Neural Networks, 2018 — arXiv:1804.07612
Ross, Gordon & Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger), AISTATS 2011 — arXiv:1011.0686

Previous: the logic model + LRL introduction