π Eval and the Road to Self-Training: Benchmarks, DoRA, and C++ Native Fine-Tuning

The PI agent — our local LLM harness for the V Engine — needed a honest mirror. We built a 68-task CatGame-specific evaluation suite, ran Qwen3.5-2B on GPU across every category, and mapped a C++-native training pipeline that needs no Python. The results were humbling: the model aces tool use but fails every C++ coding task. Here is what the data says and what we are going to do about it.

The Benchmark Suite

We expanded the PI eval harness from 17 original tasks to 68 tasks across 9 dataset families. Each dataset targets a specific capability the agent needs to develop CatGame — a C++20 Vulkan game and physics engine using EnTT ECS and GLM math. The datasets are generated by POSIX shell scripts using jq — no Python runtime dependency anywhere in the eval path.

The frontier code dataset is our answer to LiveCodeBench and HumanEval for C++. Twelve competitive programming tasks — Two Sum, FizzBuzz, Fibonacci, Binary Search, GCD, Matrix Transpose, Linked List Cycle — that the agent must solve by writing a C++20 program with write_file, which is then compiled and run against test cases. The AIME 2025 dataset is 30 competition math problems from the American Invitational Mathematics Examination, converted from HuggingFace's math-ai/aime25 dataset with a shell converter.

The Baseline: Qwen3.5-2B on GPU

We ran the full suite on Qwen3.5-2B quantized to Q4_K_M, with all 26 layers offloaded to an RTX 3080 Laptop GPU via llama.cpp's Vulkan backend. The context was 16,384 tokens with 8 threads for batch processing. Here is what happened.

The pattern is clear. When the task is "read this file and tell me what is in it" or "search for skills matching a pattern," the model succeeds 76.5% of the time. When the task is "use write_file to create a C++ program that computes the cross product of two GLM vec3 vectors," the model fails 100% of the time. The reason is not that the model cannot write a cross product function. The reason is that it outputs the code in its final text answer instead of using the write_file tool.

A secondary issue surfaced on the Vulkan dataset: one task triggered runaway generation of 16,383 tokens — the maximum output limit — which took over 100 seconds and corrupted the daemon state. Subsequent tasks received zero turns and zero tokens until the daemon was restarted. This is a known failure mode of small models when they enter repetitive generation loops on complex prompts.

The Training Strategy: DoRA, Not LoRA

Once we knew what was broken, the question was how to fix it. The standard approach is supervised fine-tuning (SFT) with LoRA — low-rank adaptation that trains a small number of parameters on top of a frozen base model. But 2024 and 2025 research has produced consistently better PEFT methods.

A 2025 study by Yin et al. evaluated 12 PEFT methods under RLVR and found that structural variants like DoRA and AdaLoRA consistently outperform standard LoRA, while SVD-informed methods (PiSSA, MiLoRA) suffer from spectral collapse. The evidence is clear: DoRA is the right choice for our use case.

C++ Native Training: No Python Required

The biggest discovery was llama.cpp PR #20453 by srossitto79, which adds native QLoRA training with reward-weighted SFT and GRPO directly in C++. This is not a wrapper around PyTorch — it is a from-scratch training loop using GGML's optimizer infrastructure with custom CUDA kernels for the backward pass.

The PR was tested on Qwen3 1.7B Q4_K_M with an RTX 4060 Ti 16GB — hardware comparable to our RTX 3080 Laptop 8GB. The VRAM budget works: the model takes 1.9 GiB, LoRA adapters at rank 16 take 0.5 GiB, and training activations with gradient checkpointing fit in 2-3 GiB. Total: approximately 4.5-5.5 GiB, comfortably within 8GB.

Saved adapters are directly compatible with llama_adapter_lora_init and can be merged into the base model with llama-export-lora. This means our existing pi-daemon infrastructure can load trained adapters without any changes — the same daemon that serves inference can serve the fine-tuned model.

The Self-Training Loop: Teacher, Judge, Actor

The user asked about a teacher model and a judge model. This maps directly onto recent research on self-improving LLMs. The architecture has three roles:

The Teacher generates correct trajectories — either by human annotation, by prompting a larger model, or by using the PI eval graders themselves as deterministic oracles (the compile grader already knows the correct answer). The Judge evaluates the Actor's outputs and assigns rewards. The Actor is the PI agent itself — Qwen3.5-2B with DoRA adapters.

Recent research makes this concrete. The Self-Taught Evaluator approach (Weston et al., 2024) shows that an LLM-as-a-Judge can improve itself without human annotations by generating contrasting synthetic preference pairs and training on its own judgments. Meta-Rewarding (EMNLP 2025) adds a meta-judge step where the model judges its own judgments, improving Llama-3-8B's AlpacaEval win rate from 22.9% to 39.4%. RLSR (2025) demonstrates that LLMs can provide reliable reward signals without ground truth answers, achieving performance comparable to formal verification.

The Teacher can be bootstrapped from the eval datasets themselves: each task has an expected field with the correct answer or expected file content. We can generate correct trajectories by constructing the tool-call sequence that produces the expected output, then train the Actor on those trajectories via SFT. Once the Actor has basic competence, we switch to GRPO with the eval graders as reward functions — the Actor generates solutions, the Judge (eval grader) scores them, and the Actor's weights are updated.

Preventing Catastrophic Forgetting

The user raised a critical concern: if we fine-tune only on CatGame coding tasks, will the model forget its general knowledge? The research says yes — and the fix is surprisingly cheap.

This result comes from Bethune et al. (PMLR 2025), who derived scaling laws for forgetting during fine-tuning with pretraining data injection. The recipe is simple: mix a small fraction of general-domain data — chat dialogues, wiki pages, tiny stories, code from unrelated languages — into the SFT training set alongside the task-specific data.

A complementary approach is regularized approximate replay (arXiv 2512.22337), which penalizes KL divergence with respect to the initial model and interleaves next-token prediction data from a corpus similar to what was used in pretraining. Applied to Qwen instruction-tuned models, this recipe preserves general knowledge without hindering plasticity on new tasks.

For our pipeline, the plan is to intermix 5% general-domain data from the OpenAssistant conversations dataset, Wikipedia summaries, and unrelated C++ code snippets (competitive programming solutions from non-CatGame sources). This keeps the model's general conversational ability and coding knowledge intact while it learns CatGame-specific tool-use discipline.

The Four-Phase Training Sequence

Phase 1 (Week 1): Train on 100-200 trajectories of correct write_file usage for code tasks. The model must learn that code goes into files, not into the final text answer. Target: 90%+ write_file usage rate on coding tasks. Eval: GLM, Physics, EnTT datasets — target 50%+ pass rate (up from 0%).

Phase 2 (Week 2): Train on 200-500 C++ code solutions spanning Vulkan, EnTT, GLM, Physics, and Frontier Code datasets. The model must produce code that compiles and passes tests. Target: 70%+ pass rate on frontier code. Eval: frontier-code, Vulkan datasets — target 40%+ pass rate.

Phase 3 (Week 3): Train on 100-200 multi-turn trajectories (read → write, plan → implement). The model must complete all steps in a multi-turn task. Target: 90%+ pass rate on multi-turn tasks. Eval: Archon, core eval multi-001 — target 80%+ pass rate.

Phase 4 (Week 4+): Switch to GRPO with PI eval graders as verifiable reward functions. The Actor generates solutions, the Judge (eval grader) scores them, and GRPO updates the DoRA adapters. This is where the model starts improving beyond what SFT alone can achieve. Target: improve pass rate on all benchmarks, measured by running the full suite with --model-id to compare base vs fine-tuned.

Agent Infrastructure: Skills, Rules, Memories

The user asked about getting the PI agent's skills, rules, AGENTS.md, abilities, workflows, memories, ideas, and plans working like our Devin agent setup. This is the other half of the vertical slice — the model is not just a code generator, it is an agent that needs to navigate a project ecosystem.

Our Devin agent has a rich instruction ecosystem: AGENTS.md for repository policy, .devin/rules/ for style and self-model constraints, .devin/skills/ for imperative usage patterns, .devin/workflows/ for orchestration, .devin/abilities/ for higher-order capabilities, .devin/memories/ for persistent context, .devin/ideas/ for autonomous ideation, and .devin/plans/ for structured execution plans.

The PI agent needs the same stack. The eval harness already tests skills (pi-eval-skills.jsonl) and Archon orchestration (pi-eval-archon.jsonl). The next step is to build a PI-specific instruction compiler that can compress our .devin/ tree into a context window that fits Qwen3.5-2B's 16K token budget — likely using the Compact Instruction Compiler (CIC) we already built for Devin. The agent would load compiled instructions at startup, query memories via a tool, and follow skills as imperative recipes — exactly like Devin does, but running locally on the engine's own inference runtime.

What We Built

Caveats and Limitations

Next Steps

The eval infrastructure is done. The training strategy is written. The next session focuses on implementation: vendoring the QLoRA training code, building the pi-finetune binary, collecting SFT training data from successful eval trajectories, and running the first SFT pass on tool-use discipline. If the model can learn to use write_file instead of dumping code into its final answer, the C++ coding pass rate should jump from 0% to something measurable — and the self-training loop begins.