The benchmark that actually matters
When was the last time you asked an LLM to answer a multiple-choice question? If you're like most developers, the answer is never. You ask it to build things. A browser-based operating system. A 3D flight simulator. A C++ skateboarding game with a California boardwalk aesthetic. You give it one prompt, you hit enter, and you see if what comes back actually runs.
That's the gap that Bijan Bowen saw. A Boston-based AI consultant and YouTube content creator, Bowen has produced over 50 model evaluation videos using a consistent, standardized suite of generative coding tasks. He's tested every frontier model from GPT-5.6 to Claude Fable 5 to Kimi K3, and he's done it the way real users do: one prompt, zero-shot, does the thing run in the browser or compile without errors?
We've formalized his methodology into what we call the Bijan Bowen Benchmark ($B^3$). This post summarizes the full research paper, which defines the task structure, proves mathematical properties, and introduces a novel metric for measuring the gap between academic benchmark performance and real-world coding capability.
The nine tasks
Each $B^3$ task requires a model to produce a complete, runnable artifact from a fixed natural-language prompt. No follow-up. No iteration. No "let me fix that for you." One shot, one artifact, one score.
The scoring rubric: a difficulty ladder
Every task is scored on five binary criteria. The key insight is that these criteria form a total order---a difficulty ladder. You can't have aesthetic polish if your code doesn't run. You can't have theme fidelity if you don't have aesthetic quality. Each criterion is a prerequisite for the next.
This means a score of 3 on any task is a sufficient statistic: the artifact runs, is functional, and is aesthetically polished, but lacks theme fidelity and final polish. No need to specify which criteria passed---the score tells you everything.
Six capability dimensions
The nine tasks span six orthogonal capability dimensions. This is what makes $B^3$ fundamentally different from SWE-bench or MMLU---those measure one dimension each. $B^3$ produces a capability profile, not just a score.
We prove three mathematical properties in the paper:
- Separability: Models with different capability profiles receive distinct score vectors.
- Coverage: Every capability dimension is exercised by at least one task.
- Monotonicity: If model A dominates model B in every dimension, A scores weakly higher.
The Generative Separability Gap
Here's the novel concept. We define the Generative Separability Gap (GSG) as the difference between a model's academic benchmark score and its normalized $B^3$ score:
Σ(M) is the $B^3$ score (0–45), and
α is a normalization constant mapping both to the same scale.
A positive GSG means the model scores better on academic benchmarks than on generative coding. Its real-world capability is overestimated by traditional benchmarks. A negative GSG means the model excels at generative coding beyond what academic benchmarks predict---it's underestimated.
This matters because organizations select models based on SWE-bench scores, and a model with SWE-bench at 85% but a GSG of +15 is going to disappoint when you ask it to build something from scratch. The GSG quantifies the disconnect between benchmark performance and practical capability.
Case study: Kimi K3
Moonshot AI's Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with a 1-million-token context window. Bowen's evaluation video tested it on four $B^3$ tasks:
- Browser OS (S=5): Fully functional, high aesthetic quality. Strong D1 + D6.
- Subway FPS (S=5): 3D scene with FPS mechanics, sound effects, enemy spawning in a single pass. Strong D2 + D3.
- C++ Skate (S=2): Compiled and ran but lacked the California boardwalk aesthetic. Weak D4 + D6 in compiled context.
- Frontend (S=4): Strong output with functional and aesthetic quality, missing final polish layer.
The profile reveals a model that excels at browser-based generative coding but has a relative weakness in compiled-language generation. Its GSG is approximately zero---academic benchmarks are consistent with generative performance, unlike some competitors with large positive gaps.
$B^3$ vs. traditional benchmarks
| Dimension | $B^3$ | SWE-bench | MMLU | ARC-AGI |
|---|---|---|---|---|
| Task type | Generative | Patch | MCQ | Pattern |
| Artifact | Complete app | Code patch | Answer | Grid |
| Zero-shot | Yes | No | Yes | Yes |
| Ecological validity | High | Medium | Low | Medium |
| Eval method | Human judge | Test suite | Exact match | Exact match |
| Reproducibility | Medium | High | High | High |
| Capability dims | 6 | 1 | 1 | 1 |
The honest limitations
Subjective scoring. Criteria 3–5 (Aesthetic, Fidelity, Polish) require human judgment. There's no automated test suite for "does this look good."
Single evaluator. All evaluations to date are by one person. His consistency across 50+ videos provides longitudinal reliability, but systematic bias is possible.
Non-reproducibility. Model outputs are stochastic. The same model at the same temperature can produce different artifacts. $B^3$ doesn't control for sampling parameters.
Web-chat dependency. Many evaluations use the model's web interface rather than API, introducing UI-dependent variability.
The paper proposes three extensions to address these: a multi-evaluator protocol using median consensus scoring, temperature-controlled evaluation with best-of-k runs, and API-standardized evaluation with fixed system prompts.
Why this matters
The LLM evaluation field has a construct validity problem. We're measuring what's easy to measure (multiple-choice accuracy, function-level code completion, GitHub issue resolution) instead of what matters (can this model build the thing I need from a single prompt?).
Bijan Bowen's methodology, applied across 50+ videos and 20+ frontier models, has created a de facto industry standard for practical coding evaluation. By formalizing it, proving its mathematical properties, and introducing the Generative Separability Gap, we provide a principled foundation for the next generation of LLM benchmarks.
Three principles for next-generation benchmarks emerge from this work:
- Generative over selective: Tasks should require complete artifact generation, not answer selection.
- Multi-dimensional over single-dimensional: Benchmarks should span multiple capability dimensions to produce rich profiles.
- Ecological validity over reproducibility: When trade-offs are necessary, tasks that mirror real-world usage should win.
The full paper, with formal proofs of separability, coverage, and monotonicity, is available at the CatGame Research papers portal.