The one-minute map
Kimi K3 is not simply “a bigger Transformer.” The technical report organizes its design around three information-flow axes: sequence (how tokens communicate), depth (how layers reuse earlier representations), and width (how much specialized computation is available). Its headline system is a 2.78T-parameter native multimodal Mixture-of-Experts model with about 104.2B activated parameters and a context window up to 1,048,576 tokens.
Source of model figures: the Kimi K3 technical report and released model repository. “Activated” is a compute-path count, not a claim that the inactive weights disappear from memory.
Pause point 1: MoE is conditional computation
A dense feed-forward layer applies the same large function to every token. An MoE layer stores many feed-forward “experts,” then a router chooses a small subset for each token. In simplified form:
y = Wo(x) + Σi∈TopK(r(x)) pi(x) · Ei(x)
Here r(x) produces router scores, TopK selects experts, pi are normalized gate weights, and Ei is one expert MLP. This buys parameter capacity without paying the full MLP cost for every token. It also creates new failure modes: router collapse, overloaded experts, token dropping, all-to-all communication, and a training objective that can improve while a minority of experts starves.
The arithmetic matters. 16 / 896 ≈ 1.79% of the routed expert pool is selected per token. The activated-parameter fraction, 104.2B / 2.78T ≈ 3.75%, is a different quantity because two shared experts, attention, embedding, and routing parameters also execute. Never turn either ratio into a blanket “K3 uses 3.75% of the model” statement.
Pause point 2: what “latent” changes
Standard MoE routes a high-dimensional hidden vector into expert MLPs. LatentMoE first projects into a smaller latent width, runs the expert computation there, and projects back. With d as model width and ℓ < d as latent width, the expert-side matrix traffic scales roughly with ℓ, not d. That is not a free theorem—projection cost, quality loss, and communication still have to be measured—but it gives architecture a direct compute/communication knob.
For a rough two-layer expert with expansion m, dense cost is proportional to 2dm multiply-adds per token. A latent version adds down/up projections and performs the expert core around ℓ; the useful comparison is therefore:
Clatent ≈ 2dℓ + 2ℓm versus Cdense ≈ 2dm.
For example, with d=7168, m=3072, and ℓ=3584, the simple estimate is about 74.7M versus 44.0M multiply-adds per expert token before gating, activation, and implementation overhead. Treat this as an explanatory model, not a reproduced K3 benchmark.
Pause point 3: the three-axis architecture
The useful lesson is not “copy all three blocks.” It is to assign each bottleneck to the axis where it occurs. KDA attacks sequence cost; AttnRes attacks depth-wise information dilution; Stable LatentMoE attacks width and specialization. The K3 report attributes an approximate 2.5× overall scaling-efficiency improvement over K2 to the combined architecture and training recipe—not to one isolated trick.
Translation to CAT VLM: borrow the principle, not the parameter count
Our CAT VLM is a different machine. The current ContinuousOutputConfig in nn/systems/lm/lm_config.hpp exposes a compact recurrent model: hidden state, stacked GRU layers, optional residuals and RMSNorm, a ring-buffer attention context, and output modulation. The project’s GRU-L3-V3 direction should stay honest about that scale and execution target. A 2.8T MoE design is a research reference, not a drop-in feature request.
Sequence lesson → gated memory lanes
Prototype a KDA-inspired delta-memory lane beside the GRU, not inside the first recurrence. A bounded state update can extend useful context while preserving a GRU fallback. Compare perplexity, hidden-state drift, and bytes moved per token before calling it attention.
Depth lesson → attentive layer skip
For L3, expose the three layer outputs to a tiny learned softmax mixer. Start with block-level weights, not full per-token AttnRes. This directly tests whether the upper layer benefits from revisiting layer 0/1 without adding another large recurrent matrix.
Width lesson → micro-experts
Use 4–8 small experts over the existing hidden state and select top-1 or top-2. Keep expert weights resident and route locally on CPU/GPU. At CAT scale, avoiding all-to-all communication is more important than imitating 896 experts.
Stability lesson → telemetry first
Measure router entropy, per-expert token counts, dropped capacity, gradient norms, RMSNorm statistics, and sequence-length failure thresholds. The K3 lesson is co-design: routing policy, kernels, memory layout, and diagnostics must be designed together.
A concrete GRU-L3-V3 experiment ladder
- Baseline lock: freeze the current CPU/GPU parity fixture and record loss, gradient norms, activation RMS, throughput, and peak memory for L3.
- Depth mixer: add a parameter-light softmax over cached layer outputs. Verify identity initialization reproduces the old residual path within tolerance.
- Local experts: add four experts with top-1 routing, capacity statistics, and a load-balancing diagnostic. Do not add distributed dispatch.
- Delta lane: add one bounded recurrent memory lane with an explicit decay/retention parameter. Compare it against the existing attention ring buffer rather than assuming it is better.
- Precision gate: run FP32 reference, FP16 weights, and mixed accumulation separately. Preserve the loss-scale-aware gradient clamp already used for long-sequence stability.
- Ablation: test GRU, GRU + depth mixer, GRU + experts, GRU + delta lane, and the combined model at equal training tokens and equal wall-clock budget.
The chart is a planning heuristic, not measured CAT VLM data. Its point is sequencing: each addition must earn its complexity through a controlled ablation.
What the video gets right—and what it cannot prove
The video is an excellent architecture tour, but a tour is not an independent reproduction. Kimi's headline numbers come from Moonshot AI's report and repository; the approximate scaling-efficiency claim is the authors' aggregate comparison. Benchmark scores and kernel case studies are useful evidence, but they are not universal laws about MoE. Hardware, compiler, batch shape, routing balance, sequence length, and evaluation protocol all matter.
Follow the evidence
Start with the video, then read the primary sources: Kimi K3: Open Frontier Intelligence, the official Kimi K3 announcement, and the Attention Residuals paper. For the sequence-side lineage, see Kimi Linear. These references are deliberately separated from secondary explainers so claims can be checked at the source.
Related CatGame work
Our own trail covers the opposite direction: making a small, inspectable model trustworthy before making it wider. Read the Dual-Stream GRU-L3 GPU report for the recurrent/context path, CAT VLM V3.5 stability work for FP16 BPTT guards, RMSNorm and spectral stability for recurrent bounds, and Vulkan ML compute for the dispatch-versus-kernel lesson.