The companion post for How AI Learned to Teach Itself [JEPA] traced three decades of self-supervised video and image learning, from random-dot stereograms to V-JEPA 2-AC. This post takes the next step: a concrete, end-to-end blueprint for an app that turns unlabeled video into an interactive moving image of everything — a world model you can query, steer, and render in real time. The stack is the V Engine, the PI Agent VLM, and the model we are now calling CAT VLM.
The moving-image-of-everything idea
The phrase is deliberately old-fashioned. A camera obscura projects moving images of everything happening outside onto a dark wall. The metaphor here is the same: a single, live, continuous picture of a scene that lets you look around, ask questions, and imagine what happens next. The difference is that the image is not light on a wall — it is a learned latent world model inside an app, rendered by the V Engine, and steerable by language.
The goal is an app with three modes:
- Watch mode — point the camera (or load a video) and let the model learn a latent representation of the scene from raw pixels, no labels, no captions, no bounding boxes.
- Ask mode — query the scene with the PI Agent VLM: “What happens if the red ball rolls left?” The agent returns a natural-language answer grounded in the learned world model, not a search result.
- Steer mode — ask for an action sequence. The CAT VLM predictor rolls forward in latent space, the V Engine renders the result, and the user can accept, reject, or edit the plan.
The app is not a generative video model. It is a planning and explanation model: JEPA-style latent prediction plus a renderer plus an agentic language interface. The pixels are not the target; the structured, controllable world state is.
What JEPA gives the blueprint
The JEPA family is the right starting point because it separates three problems cleanly: what to predict, how to avoid collapse, and how to act.
Latent prediction
JEPA does not reconstruct pixels. It predicts masked or future embeddings in a learned representation space. This is what makes the model compact and what makes it possible to plan without a hand-specified state space.
Collapse prevention
Stop-gradient EMA target encoders, asymmetry between context and predictor, and distribution-level regularization (VICReg, SIGReg) replace the negative-sampling crutches of contrastive learning.
Action conditioning
V-JEPA 2-AC adds action variables to the predictor. The model can answer “what happens if I do this?” by rolling forward in latent space, which is exactly the planner the app needs.
Planning in latent space
The Cross-Entropy Method (CEM) searches action sequences, scores them against a goal embedding, and refits a Gaussian over the elite set. Model Predictive Control without hand-coded states or dynamics.
For the full walk through these ideas, see the JEPA companion post and the video embedded at the top of this page. The companion post also contains the fact-checked arXiv references for I-JEPA, V-JEPA, V-JEPA 2, LeJEPA, and LeWorldModel.
End-to-end pipeline and architecture
The app has four interacting subsystems. The diagram below shows the data flow, the runtime boundaries, and where each piece lives today. Solid lines are implemented or in progress; dashed lines are the roadmap steps that turn the current tools into the full app.
Component status: implemented / in progress / roadmap
| Subsystem | Role in the app | Status |
|---|---|---|
| V Engine | Custom C++20 Vulkan renderer, ECS, asset pipeline, PBR, deterministic job scheduling, cross-platform Linux/Windows presets. | Implemented |
| PI Agent VLM | Local pi-daemon agent with Crypt-authorized Unix sockets, tool calling, streaming, and the new live console. Default model is Qwen via llama.cpp. | Implemented |
| CAT VLM text/embedding base | Dual-stream GRU L3 decoder, BPE tokenizer, MiniLM sentence embedding, libv_nn native inference, Vulkan training pipeline. | Implemented |
| CAT VLM video encoder | JEPA-style spatiotemporal tubelet encoder, EMA target encoder, and latent predictor for unlabeled video. The text base becomes the decoder and interface. | Roadmap |
| Action-conditioned predictor | Predictor g_ψ(s_t, a, z) that rolls forward given a goal position and an action sequence, the V-JEPA 2-AC style planning primitive. | Roadmap |
| CEM / MPC planner | Cross-Entropy Method over action sequences in latent space, scoring against a user-supplied goal embedding. | Roadmap |
| Latent → scene decoder | Mapping from predicted world-state embeddings to V Engine entities, transforms, and materials. This is the moving-image renderer. | Roadmap |
| End-to-end app shell | Camera/file input, live render, natural-language query panel, and a plan-accept/reject loop in one packaged binary. | Roadmap |
What each subsystem does
1. CAT VLM video encoder — learn the world without labels
The encoder ingests unlabeled video as 3D spatiotemporal tubelets (e.g., 2 frames × 16×16 pixels). A context encoder f_θ processes the visible tubelets, a slow EMA target encoder f_ϕ produces target embeddings for masked or future blocks, and a predictor g_ψ(s_t, z) estimates those target embeddings in the learned latent space. The loss is an L2 distance in representation space, not a pixel reconstruction loss. This is exactly the I-JEPA/V-JEPA recipe.
The difference for the app is that the embeddings are not just a pretraining signal. They are the world state that the V Engine renders and the PI Agent queries. The model must learn that objects persist, that motion has continuity, and that the same scene can be viewed from new angles — all without captions or bounding boxes.
2. V Engine — render from the latent state
The V Engine turns a structured scene description into a PBR-rendered frame. In the app, the decoder (the latent → scene block) produces entity/component data that the engine already understands: transforms, materials, VFX, audio, and physics. The engine’s deterministic job scheduler and ECS make it possible to run the render loop at a fixed timestep, which is important for training and for replaying plans deterministically.
3. PI Agent VLM — the language interface
The PI Agent is the user-facing language layer. In the blueprint it has two jobs. In Ask mode, it interprets the user query and the current world-state summary and returns a grounded explanation. In Steer mode, it translates the user’s goal into an embedding and a constraint set, then hands the planning problem to the CEM loop. The agent’s tool-calling surface can already read files, list directories, and write files; in the app it would gain a predict_rollout and render_scene tool that call the CAT VLM and V Engine.
4. CEM planner — search in latent space
The planning loop is the same zeroth-order optimization V-JEPA 2-AC uses for robot actions:
- Sample N candidate action sequences from a Gaussian prior.
- For each sequence, roll the predictor forward T steps from the current latent state s_t.
- Score each final embedding by its distance to the goal embedding (e.g., the red ball near the left wall).
- Keep the top-k “elite” action sequences.
- Refit the Gaussian to the elite set and repeat for a few rounds.
- Execute the best sequence, render the result, and feed the new frames back into the encoder.
This is Model Predictive Control without a hand-coded state or dynamics model. The state and the dynamics are both learned from video.
Local inference and the Crypt boundary
The current PI stack already enforces a strict runtime boundary. The pi-daemon owns the model context and the Unix-domain inference socket; every request is validated against the Crypt Keeper before it runs. There is no HTTP, TCP, FIFO, or llama-server fallback. The app would extend the same boundary to the video encoder: the camera feed never leaves the local machine, the model weights live in .pi/ or a VPak, and the planner is a local tool call inside the same Crypt-authorized session. This is not a privacy marketing claim; it is the current architecture of the PI runtime.
Caveats and what is not proven
This post is a blueprint, not a shipping app. Most of the end-to-end pieces are still on the roadmap. Be specific about the gaps:
- The CAT VLM video encoder and action-conditioned predictor are not yet implemented. The current CAT VLM is a text/embedding model, and the jump to spatiotemporal video latents is a research project, not a configuration change.
- The latent → scene decoder does not exist. Mapping from learned embeddings to V Engine scene entities is an open graphics-and-representation problem; the diagram assumes it will be solved.
- JEPA’s strongest results (V-JEPA 2-AC on DROID) come from internet-scale pretraining and specific downstream robot tasks. Generalizing to arbitrary household or outdoor video is not demonstrated.
- The numbers in the JEPA post (79.3% ImageNet linear probe, 81.9% Kinetics, etc.) are reported paper numbers, not independent CatGame reproductions.
- The app’s CEM planner would require careful engineering around action parameterization, collision, and temporal consistency; the video does not prove these are easy.
References and related work
- How AI Learned to Teach Itself [JEPA] — companion post — the fact-checked walkthrough of the video and the primary arXiv references.
- How AI Learned to Teach Itself [JEPA] — the source YouTube video.
- Bardes et al., “Revisiting Feature Prediction for Learning Visual Representations from Video” (V-JEPA), 2024.
- Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning”, 2025 — V-JEPA 2-AC is the action-conditioned variant in Section 4.
- Assran et al., “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” (I-JEPA), 2023.
- LeJEPA — SIGReg regularization.
- Maes et al., “LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels”, 2026.