Every mined dialogue corpus has a dirty secret, and ours is no exception. The 306 MB corpus of organic conversations we assembled for Cat Chat's phase-0 pretraining — CHILDES child-speech transcripts, IMSDb film scripts, fandom and open-domain dialogue dumps, TinyDialogues, Gutenberg dialogue — is rich in exactly the things a small character-level model needs: real slang, real dialect, real interruptions. It is also full of mechanical garbage. When we sampled three thousand records from the untouched portion, 98% carried at least one mechanical defect — missing terminal punctuation, spacing artifacts, stutter loops, stray markup, truncated turns.
The obvious fix is to run a teacher model over every record and let it clean. We built that: a local 2B-parameter teacher that rewrites each record to repair grammar, punctuation, and formatting while preserving the speakers' slang, tone, and turn structure. It works. But a full pass over ~800k records is roughly thirty hours of GPU time, and we intend to keep mining and keep pretraining — phase 0 is not the last corpus. The interesting question became: how do you make each future cleanup cheaper than the last? The answer we landed on is not a smarter model. It is a procedural judge — a deterministic validator that decides what needs the teacher, what gets fixed for free, and what goes back for another try. This post is about that judge, and the flywheel it creates.
Measured vs. estimated. Corpus size, record counts, slice composition, acceptance rates, and gate fire rates in this post are measured from live pipeline output. The 98% figure comes from a seeded 3,000-record sample of the unrefined portion — real but sampled, not a full census. GPU-hour estimates are extrapolations from observed throughput (~485 tok/s aggregate on a consumer laptop GPU).
The three-layer judge
Our pipeline routes every candidate record through three judges of increasing cost. The key property: each cheaper layer removes work from the more expensive layer beneath it.
Layer 1: fix it for free
The largest single defect class in our mined corpus is boring: a turn that ends without terminal punctuation. Transcript-mined dialogue drops periods constantly — <spk0>how come we can't have a cat here instead of ...cat here? A rules engine repairs this in microseconds: append terminal punctuation where a turn ends declaratively, collapse spacing artifacts, capitalize after sentence boundaries, trim runaway stutter loops while keeping natural two- or three-word disfluencies that are authentic child speech.
This layer matters because of the 76% problem. Our teacher run covers a 191,391-record stratified slice — about 17% of the corpus by bytes. The remaining ~604k records were never sampled; they weren't excluded for quality, they were excluded for GPU budget. Sending them all to the teacher is the brute-force answer (~25–30 hours). Normalizing them deterministically is the free answer — and the right one for most of them, because most of what's wrong with them is mechanical, not semantic.
Layer 2: the gate validator as procedural judge
Normalization fixes what rules can fix. The gate stack flags what they can't — and this is where the judge earns its name. Every record gets a typed verdict: pass, or a specific reject reason. The current taxonomy, built up over weeks of staring at real teacher output:
- Structural — not_canonical, sequence_mismatch, turn_count, truncated: the output broke the record's shape. The sampler locks each input's speaker-tag sequence as a required prefix; the validator enforces it.
- Cross-turn contamination — turn_shift (a line moved into a wrong-speaker turn), turn_dup (a line copied into a turn that kept its own text), content_drop (a turn silently lost most of its content), drift (per-turn token overlap below threshold).
- Degeneracy — word_loop, phrase_loop, stutter_run, echo_turn, turn_repeat, degenerate: the repetitive-garbage family.
- Leakage — meta: teacher commentary inside the record. Deliberation prefixes were easy; infix edit-notes like (no changes needed) and Cleaned Text: scaffolds took a second pass — the gate now catches all 930 observed carriers (a re-scan over live output found zero slipping through).
- Artifacts — dangling_quote (a turn holding an unbalanced quote — the teacher "fixes" a stray source quote by inventing a closing mark, sometimes a whole invented turn), colon_marker, digit_run, non_ascii, spk_leak.
- Append discipline — extra_len, extra_echo, extra_q, ragged_end: the teacher may append up to two short closing turns to a conversation cut mid-stream, but closers can't be long, can't echo an existing line, and can't answer a question with a question.
The verdict does real work. On the unrefined 76%, the gate report is the triage: pass → ship after normalization; fail → teacher queue. On the refine side, the same verdict drives the recursive loop.
Layer 3: the teacher, recursively
The teacher is a local 2B-parameter model (no API, no daemon — in-process GGUF inference) with a strict contract: fix only mechanical problems — grammar, punctuation, spacing, capitalization, malformed tags, stutters — while preserving slang, style, tone, and word choice. It may trim a dangling fragment, append up to two short closing turns, or emit DROP when a record can't be cleaned without invention. Its speaker-tag sequence is locked to the input's as a required prefix, so it can rewrite turn text but never reattribute a speaker.
Refinement runs in passes, and the judge routes every output:
The other half: synthetic curriculum
The organic side is only half the phase-0 blend. In parallel runs a syllabus-driven synthetic generator — a CSV curriculum manifest with 24 topic rows across six developmental stages (single-word toddler utterances at vocab-ceiling 150 up to full conversational turns), each row producing count records at a fixed seed. The synthetic side exists for a specific reason: TinyStories showed that restricted-vocabulary synthetic text can train tiny models to fluency — our p0a is the same idea applied to dialogue, a clean easy head that warms the GRU before it sees noisy organic conversation.
Both halves flow through the same judge — the synthetic builder shares the gate stack — then blend at a fixed ratio. And the organic anchor is deliberate, not incidental: the model-collapse literature (Shumailov et al.) shows recursively generated data erodes distributional tails, while Gerstgrasser et al. show that accumulating real and synthetic data avoids collapse entirely. Fifty percent organic is the insurance policy.
Why this is a flywheel, not a pipeline
Pipelines end. Flywheels spin up. The three-layer judge makes each future corpus phase cheaper than the last in four compounding ways:
The student as a fourth judge — roadmap
There's one more judge waiting in the wings: the GRU itself. A character-level model trained on clean dialogue develops a distributional sense of what clean dialogue looks like — garbage reads as high perplexity. Once a trustworthy checkpoint exists, student perplexity becomes a cheap secondary signal for scoring future candidate records: the student filters its own next-phase training data. We deliberately keep it secondary — a model grading its own future diet needs the deterministic gates as a hard floor, or blind spots compound. That integration is roadmap, not shipped infrastructure.
Where this design sits in the literature
None of this is exotic — which is the point. The field has been converging on the same shape:
- Small teachers clean corpora well.ProX (ICML'25) showed even 0.3B models refine pretraining corpora effectively; our 2B teacher fits the same paradigm.
- Per-example triage beats uniform processing.DataOrchestra routes each chunk to drop / untouch / clean — our normalize → gate → teacher stack is the same three-way decision, implemented procedurally instead of learned.
- Iterative refinement converges fast.Self-Refine (NeurIPS'23) gets ~20% absolute improvement from feedback→refine loops that converge in 2–3 iterations — our fixpoint detection and reason-injected retries are the same mechanism with a drift anchor.
- Organic anchors prevent collapse. The curse of recursion and its accumulation-based mitigation are why synthetic stays at 50%, never 100%.
Caveats — what the judge can't see
- The judge shares the designers' blind spots. Every gate was written after eyeballing a failure. Defect classes we haven't seen yet pass silently — each new class discovered in monitoring gets a gate, but the taxonomy is only as complete as our surveillance. Periodic manual audit of confident accepts is non-optional.
- Deterministic normalization can over-edit. Capitalization and terminal-punctuation rules occasionally "fix" intentional transcript style. We accept mild over-normalization as the price of a free pass; the teacher pass stays the semantic-quality layer.
- Distribution shift is real. New mining sources bring novel garbage types; a gate set tuned on CHILDES + subtitles may under-fire on forum text or ASR transcripts. The validator needs re-auditing per source.
- Perplexity-as-filter is unproven here. The student-signal idea is roadmap. A GRU trained on imperfect data may learn to like the imperfections — the student must never be its own unchecked judge.
Status
Shipped and running: the shared gate stack, deterministic body normalization, teacher refine with speaker-prefix locking and DROP, recursive polish/retry passes with reason-injected feedback, fixpoint detection, the syllabus generator (22.6k of 140k synthetic records banked), and the blend tooling. In progress: refine pass-1 (~58% at time of writing), then pass-2, then the synthetic resume. Roadmap: corpus-wide normalize → gate-report → teacher-on-flagged for the unrefined 76%, difficulty-ordered merge, and the student perplexity signal.