Don't Ship Skills Without Evals: A Companion Guide

Source: Don't Ship Skills Without Evals by Philipp Schmid (Google DeepMind) at the AI Engineer World's Fair · used as the primary text for this companion

Philipp Schmid's talk Don't Ship Skills Without Evals begins with an uncomfortable number. SkillsBench counted roughly 47,000 unique skills across more than 6,300 repositories — and almost none of them had evaluations. Most were AI-generated. Many were never tested. Schmid's argument is blunt: you would not ship a function without a test, so why are you shipping the instructions that steer your agent's brain without one?

* Order-of-magnitude figure attributed to SkillsBench in the talk; methodology not detailed in the sources used here. See caveats.

What this talk is about

The video is framed around a single failure mode that Schmid sees everywhere: teams write a skill, run it twice by hand, say "looks fine," and ship it. Then production breaks in ways nobody can attribute. Was the skill bad? Was the task too hard for the model? Did the skill fire when it shouldn't have, or stay silent when it should have spoken? Without an eval, you are guessing. With one, you have a number.

What makes the talk useful is that it is not a call to build a heavy platform. Schmid's own eval harness for the Gemini Interactions API skill was two files: a JSON list of test cases and a Python runner that shells out to the agent CLI and checks the output with regex. The whole thing fits in a few hundred lines. The discipline is in the prompt set and the checks, not the infrastructure.

What a skill actually is

Before the eval discussion, Schmid pins down the object under test. An agent skill is a folder of instructions, scripts, and reference files that augments an agent's capabilities without retraining or fine-tuning the model. The minimum viable skill is a single SKILL.md with three layers, loaded on demand.

This is progressive disclosure, and it is the single most important architectural idea in the talk. The frontmatter description is paid for on every model call — it lives in the context window whether the skill fires or not. The body and resources are paid for only when the model decides to pull them. So a bloated description taxes every turn, while a bloated reference file taxes only the turns that need it. That asymmetry is why Schmid keeps repeating: keep the description tight, push the depth into resources.

The minimum frontmatter is just a name and a description, and the description does almost all of the triggering work:

---
name: gemini-interactions-api
description: Use this skill when writing code that calls the Gemini API
  for text generation, multi-turn chat, image generation, streaming responses,
  function calling, structured output, or migrating from generateContent SDK.
---

# Gemini Interactions API Skill
The Interactions API is a unified interface for interacting
with Gemini models and agents...

Two kinds of skills, two different fates

Not all skills age the same way. Schmid splits them into two categories, and the split decides how you test them and when you retire them.

The distinction matters because it changes the goal of the eval. For a capability skill, you are climbing a pass-rate hill from low to ~100%, and once you arrive the eval becomes a retirement detector. For a preference skill, you start near 100% and the eval's job is to scream when something drops. Same harness, opposite trajectory.

Triggering: who decides the skill fires

Schmid draws a line between two triggering modes, and it changes what you even need to test.

• Model-invoked — the model reads the description and decides to load the skill itself. This is the hard case: the eval must check both did it fire? and did it fire correctly? Most of the talk's guidance targets this mode.
• User-invoked — the user or system explicitly requests the skill (a slash command, a staged workflow). Triggering is not in question; the eval only grades the outcome. Schmid's advice: if a workflow is a fixed step-by-step script, do not burn tokens on model-triggered discovery — make it user-invoked, or just write a script.

Mentions beyond the main pipeline: agents we use vs agents we build

The talk draws a distinction that shapes how you even think about triggering. Agents we use — external coding and productivity tools like Cursor, Claude Code, and similar coding agents — often trigger skills implicitly, and their users notice fast when a skill misfires (they reprompt, they reach for a slash command). Agents we build for customers are different: the end user generally does not know what a "skill" is. A customer asking "help me with support" will not say "use the refund skill," so the model-invoked trigger has to do all the work of recognizing intent and loading the right capability. This is why Schmid's eval guidance focuses on model-invoked skills — that is where customer-facing agents live or die, and where a vague description silently loses users.

How to write a skill that triggers correctly

Before any eval code, Schmid lists the writing rules that prevent most failures upstream. The description is the trigger; everything else is downstream of it.

The eval harness, end to end

This is the core of the talk. Schmid walks through the harness he used for the Gemini Interactions API skill — a brand-new API released after the last Gemini training run, so older models had no context for it. The skill's job was to teach the model to generate valid Interactions API code. The harness took it from a 66.7% pass rate to 100%.

Step 1 — Write the prompt set first

Before any code, write down what "success" means in measurable terms. Schmid grades three dimensions, in this order of priority:

• Outcome — did it produce a usable result? Code compiles, the API returned a valid response, the image rendered. If the output does not work, nothing else matters.
• Style & instructions — does the output follow your conventions? Right SDK, current model IDs, team naming, the formatting the skill specified.
• Efficiency — how much time, tokens, and effort did it take? No unnecessary retries, reasonable token count. The most undervalued dimension: two runs can produce identical correct output, but one burned 3× the tokens.

For a single skill, 10–20 prompts are enough to start. Each prompt declares its own success criteria. Crucially, include negative prompts where the skill should not fire:

[
  {
    "id": "py_basic_generation",
    "prompt": "Write a Python script that sends a text prompt to Gemini and prints the response.",
    "language": "python",
    "should_trigger": true,
    "expected_checks": ["correct_sdk", "no_old_sdk", "current_model", "interactions_api"]
  },
  {
    "id": "negative_unrelated",
    "prompt": "Write a Python script that reads a CSV and plots a bar chart using matplotlib.",
    "language": "python",
    "should_trigger": false,
    "expected_checks": []
  }
]

Schmid's case study started with 17 tests across five categories: core capabilities (7), guardrails for deprecated models (4), extended features not in the inline examples (3), and negative controls (2), spanning Python (12) and TypeScript (5). The talk summary separately cites 117 test cases for the same skill — likely the full regression suite grown from that initial prompt set as real user traces were folded in. Either way, the negative controls are not optional — a skill with a too-broad description will fire on every coding prompt and pollute the context.

Step 2 — Run the agent the way it really runs

Test the skill through the same surface the agent uses. Schmid's harness shells out to the Gemini CLI inside a Python script, captures stdout as JSON, and extracts the response. The point is to test the skill the way the model experiences it — not a sanitized unit-test mock that hides the real failure modes.

def run_gemini_cli(prompt):
    cmd = [
        "gemini",
        "-m", "gemini-3-flash-preview",
        "--output-format", "json",
        "--yolo",
        "-p", prompt,
    ]
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=3600)
    data = json.loads(result.stdout.strip())
    return CLIOutput(
        response_text=data.get("response", ""),
        stats=data.get("stats", {}),
        exit_code=result.returncode,
    )

Step 3 — Write deterministic checks first

Each check is a small function that runs regex against the extracted code and returns a boolean. Most skill evals do not need an LLM-as-judge at all — regex is millisecond-fast and free. Schmid's Interactions API skill used 11 checks, all deterministic:

# Does the code use the correct SDK?
def check_correct_sdk(code, language):
    if language == "python":
        return bool(re.search(r"from\s+google\s+import\s+genai", code))
    return bool(re.search(r"""['"]@google/genai['"]""", code))

# Does the code use a current (non-deprecated) model?
DEPRECATED_MODELS = ["gemini-2.0-flash", "gemini-1.5-pro", "gemini-1.5-flash"]

def check_current_model(code, language):
    return not any(model in code for model in DEPRECATED_MODELS)

Register every check by ID, then loop each test case through its declared checks:

CHECK_REGISTRY = {
    "correct_sdk":      check_correct_sdk,
    "current_model":    check_current_model,
    "interactions_api": check_interactions_api,
    "no_old_sdk":       check_no_old_sdk,
    # ... 11 checks total
}

def run_eval(test_case):
    output = run_gemini_cli(test_case["prompt"])
    code = extract_code_blocks(output.response_text)
    results = {}
    for check_id in test_case["expected_checks"]:
        results[check_id] = CHECK_REGISTRY[check_id](code, test_case["language"])
    return results

Step 4 — Add LLM-as-judge only when regex cannot

Some checks are qualitative — "distinctive, production-grade interfaces," cohesive color palette, intentional layout. Regex cannot grade those. For them, Schmid uses a second model pass with structured output constrained to a typed schema, so the result is parseable and trackable rather than a free-text blob. Use it selectively: deterministic checks are fast and free, LLM grading adds cost and latency. The Interactions API skill needed no judge at all.

The two fixes that mattered most

Across ~20 test cases the Interactions API skill went from 66.7% to 100%. Schmid is precise about which edits moved the needle:

This is the recurring lesson of the talk. The description is not documentation — it is the trigger. Write it in the user's vocabulary, not the API's. And "the old SDK is deprecated" is a passive statement the model can ignore; "do not use gemini-2.0-flash, use gemini-3-flash" is a directive it follows.

The 10 best practices, condensed

Schmid closes with ten rules. They are worth pinning to the wall above your skill folder.

1. Start with the description. Name and description are the trigger. Vague means not used when it should be, or used when it shouldn't be.
2. Directives, not information. "Always use interactions.create()" works. "The Interactions API is the recommended approach" doesn't.
3. Include negative tests. Add prompts where the skill should not trigger. Broad keywords will fire on every request.
4. Start small, extend from failures. Begin with 10–20 prompts drawn from real usage. Every user-reported wrong output becomes a new test case.
5. Grade outcomes, not paths. Agents find creative solutions. Don't penalize an unexpected route to the right answer.
6. Isolate each run. Clean environment per case. Accumulated context bleeds between tests and masks real failures.
7. Run multiple trials. Agents are nondeterministic. A single pass/fail is not signal — run 3–5 trials and read the distribution. Prefer pass^k (all k must pass) over pass@k (any one of k passes) for production gating.
8. Test across harnesses. The same skill behaves differently in Cursor, Claude Code, Gemini CLI. If it ships to multiple agents, eval it in each.
9. Graduate your evals. Capability evals start at a low pass rate and give you a hill to climb. Once they hit ~100%, they graduate into regression evals that protect against backsliding.
10. Detect skill retirement. Run your evals with the skill unloaded. If they still pass, the model has absorbed the skill's value — retire it to save cost and maintenance.

Regression gating and the lifecycle

At DeepMind, Schmid says, evals live alongside every skill and run on every change. The rule is diff gating: a merge happens only if it improves the test cases; otherwise failures block the merge. This turns the eval from a one-time launch checklist into a permanent guardrail.

The retirement step is the one most teams skip. Schmid's method is an ablation test: run the full eval suite with the skill unloaded. If the pass rate holds without it, the base model has caught up — the skill is now dead weight that costs tokens and maintenance for no gain. Keep the evals even after retirement, so you can detect if a future model regression makes the skill relevant again.

What this means for the C.A.T. platform

This is not an abstract talk for us. The C.A.T. game platform runs a pi-agent backed by the pi-daemon inference runtime, and that agent is steered by a skill library living in .devin/skills/ — build, test, run, frontier-post, grill, dream, ideas, and dozens more. Every one of them is, in Schmid's terms, a skill: a SKILL.md with frontmatter, a body, and reference files. As of this writing, none of our 53 skills ship with a formal eval harness — which is exactly the gap Schmid is describing.

Schmid's framework maps directly onto decisions we already face:

• Capability vs preference. Most C.A.T. skills are preference skills — they encode our repo conventions (run from root, use wrappers, never pipe build.sh). Those are durable and need regression gates. A few are capability skills that may not survive the next model upgrade; those need ablation tests so we know when to delete them.
• The description is the trigger. Our skills are model-invoked via the skill tool. A vague description means the agent loads /dream when it should load /sleep, or skips /grill when a plan needs roasting. Tightening descriptions is the cheapest win available.
• Negative cases for overlapping skills./update-self, /update-memories, /level-up, and /sleep all touch the self-model. Without negative tests, the agent fires the wrong one and clobbers state. Schmid's negative-prompt discipline is exactly the guard we need.
• Retire skills the model has absorbed. As pi-daemon's backing model improves, some skills become no-ops. An ablation run per release would tell us which to prune — saving context budget on every turn.

The practical takeaway is small and concrete: pick one skill, write 10 prompts (5 happy, 5 negative), write regex checks for its canonical outputs, run it 3 times, and gate changes on the score. That is the whole harness. The discipline is in the prompt set, not the plumbing.

Companion field guide

Quick reference for the terms that recur across the talk and the accompanying blog post.

1. Agent skill — a folder of instructions, scripts, and resources that augments an agent without retraining the model; minimum viable form is a SKILL.md.
2. Progressive disclosure — the three-layer load pattern: frontmatter always in context, body loaded on trigger, resources consulted on demand. Minimizes per-turn token cost.
3. Capability skill — teaches the model something it cannot reliably do yet; temporary, retired when the base model catches up.
4. Preference skill — encodes team-specific workflows and conventions; durable, protected by regression evals.
5. Model-invoked trigger — the model decides to load the skill from its description; the eval must check both firing and outcome.
6. User-invoked trigger — the user or system explicitly requests the skill; the eval grades outcome only.
7. No-op — an instruction in a skill that changes no behavior; common in AI-generated skills, costs tokens for nothing.
8. Ablation test — running the eval suite with the skill unloaded; if it still passes, the skill is ready to retire.
9. pass^k — all k trials must pass; stricter and more production-realistic than pass@k (any one of k passes).
10. Diff gating — a merge is allowed only if it improves the eval test cases; otherwise it is blocked.
11. LLM-as-judge — a second model pass, constrained to a typed schema via structured output, used for qualitative checks regex cannot grade.
12. SkillsBench v1.1 — an analysis of ~47,000–50,000 skills across 6,300+ repositories, finding almost none had evals and many were AI-generated.
13. Regression eval — an eval whose job is to scream when a previously-passing case starts failing, not to climb a pass-rate hill.

Six study questions

1. Why does the frontmatter description cost tokens on every model call, while the skill body does not? How does this shape where you put detail?
2. Give one concrete example where a fixed step-by-step workflow should be a script, not a skill. What is the cost of using a skill anyway?
3. The Interactions API skill went from 66.7% to 100% after two edits. Why did rewriting the description in user vocabulary fix more failures than rewriting the body?
4. What is the difference between pass^k and pass@k, and why is pass^k the right metric for production gating?
5. How does an ablation test tell you a capability skill is ready to retire? Why should you keep the evals after the skill is deleted?
6. For the C.A.T. pi-agent, which of our current skills are capability skills and which are preference skills? What would an ablation run across a model upgrade reveal?

Caveats and limitations

This companion is built from the video transcript (via YouTubeSummary), the accompanying blog post on philschmid.de/testing-skills, and a LinkedIn summary by Brendan Graetz. The transcript is machine-generated; the blog post is the author's own written version and is treated as the more authoritative source where they differ. The following caveats apply.

Related transmissions

Skill evals sit at the intersection of several threads we have been working on the Frontier. If you are building the agent's brain, the cognition blueprint is the architectural companion. If you care about how instructions themselves get compiled and compacted, that post is the direct predecessor. And the self-improving instructions post is the same lifecycle problem — write, test, ship, monitor, retire — applied to the agent's own rule files.

Primary sources