How AI Learned to Teach Itself: A JEPA Companion Guide

The video at the top is How AI Learned to Teach Itself [JEPA], a 37-minute walk through roughly 30 years of self-supervised representation learning. It starts with a 1992 random-dot stereogram experiment and ends with an action-conditioned world model that plans robot actions in latent space. This page is a standalone companion: read it alongside the video, or read it on its own. Every method below is pinned to its timestamp, fact-checked against the primary paper, and framed by the single question that drives the whole field — how do you learn useful representations when no one gives you labels?

The three families: where the distance is measured

The video's spine is a taxonomy. Every self-supervised method scores an (x, y) pair with a single scalar "energy" — a distance D — trained low when x and y belong together and high otherwise. What differs is where in the pipeline that distance is measured. That single choice determines what the model spends its capacity on.

Joint-embedding methods match two views in latent space — compact and abstract, but dependent on hand-crafted augmentations that define what counts as "the same." Generative methods reconstruct pixels — detailed, but they waste capacity on texture, lighting, and noise you would happily throw away. JEPA sits between them: it keeps the predictive challenge of generative models but moves the prediction target into the learned representation space. No augmentations, no pixel reconstruction.

The origin: Becker & Hinton and the collapse problem 00:00–04:19

The video opens with a random-dot stereogram: two images that individually look like noise but, when fused, reveal a 3D shape. The question Becker and Hinton asked in 1992 was whether a network could discover that hidden depth without a depth label — just from the relationship between the two views.

Their idea: take two networks, show each a different view of the same scene, and train their outputs to agree. If both views come from the same surface, the networks should learn what they have in common. But there is an immediate trap — the easiest way to make two outputs agree is to make both constant, regardless of input. This is representation collapse: the loss is minimized, but nothing useful is learned.

Their fix was IMAX (Information Maximization): maximize the variance of the shared signal (the sum A+B) and minimize the variance of the disagreement (the difference A−B). Under a Gaussian assumption, this is equivalent to maximizing mutual information between the two representations. It worked — a network with no depth labels discovered depth from stereograms. The limitation: IMAX relies on strong assumptions (Gaussian variables, symmetric agreement) and only scaled to toy problems.

Contrastive learning: positives, negatives, and InfoNCE 04:19–08:13

To scale beyond handcrafted agreement rules, the field moved to a broader objective: correctly match positive pairs and distinguish them from negative pairs. Call one view the anchor (later renamed the query), another view of the same image the positive (later k+), and unrelated images the negatives. Negatives give the model a reason to preserve differences, not just similarities.

The loss evolved through three stages:

• Pairwise contrastive loss — push positives together, push negatives apart, with a fixed distance threshold. Absolute, not relative.
• Triplet loss — make the comparison relative: the anchor only needs to be closer to its positive than to its negative by a margin. Still depends on choosing useful negatives, and a single negative is a weak signal.
• InfoNCE — compare against many negatives at once, framed as a classification problem: given a query q, pick its one true match k+ from a crowd of distractors.

InfoNCE scores the similarity of the query to each key (usually a dot product), applies a softmax to normalize into a probability distribution, and minimizes the negative log-probability of the positive. This does two things at once: it pulls the query toward its positive and pushes it away from every negative. A temperature parameter τ controls how sharp the distribution becomes.

InfoNCE originates from Contrastive Predictive Coding (CPC), where it was used to predict future latent representations from past context. The bottleneck then shifted: InfoNCE needs many high-quality negatives, and the dictionary of negatives is capped by the mini-batch size when training end-to-end.

Scaling contrastive: memory banks, MoCo, and SimCLR 08:13–14:38

The simplest end-to-end approach caps the negative dictionary at the mini-batch size. An intermediate fix is a memory bank: precompute embeddings for every image in the dataset and reuse them as negatives without backpropagation, allowing a much larger dictionary. But the encoder updates every step while stored embeddings were computed by old encoder versions — they go stale and produce unreliable updates. Two architectural answers emerged to solve this consistency problem:

Distillation: do we even need negatives? 14:38–20:12

Contrastive learning works, but it pays for success with negatives. The next question: can we learn without negatives at all? The answer turned out to be yes — through asymmetry, not opposition.

The lesson: negatives were a crutch. The real collapse-prevention mechanism is asymmetry — a slow teacher, stop-gradient, and a predictor the teacher lacks. But these methods still depend on hand-designed augmentations, and there is still no explicit anti-collapse loss. That pushes the field toward a different source of supervision: hide part of the input, and make the model predict what is missing.

Masked autoencoders: predicting missing pixels 20:12–23:16

Take an image, split it into patches, and hide most of them — about 75%. Train an encoder that sees only the visible patches (no mask tokens), then use a lightweight decoder, conditioned on mask tokens and positional information, to reconstruct the missing pixels. This is MAE.

The high masking ratio is critical: it forces the encoder to learn high-level structure rather than copying from neighboring pixels. The asymmetry is the design insight — a heavy encoder does the representational work; a throwaway decoder exists only to shape the encoder's gradients. After training, you discard the decoder and keep the encoder for downstream tasks.

Because MAE reconstructs pixels, collapse is not the main danger: a constant representation cannot reconstruct different missing patches. But there is a tradeoff. When the encoder is frozen, contrastive methods (e.g. MoCo v3) outperform MAE — the discriminative classification-style loss transfers better to linear probes. MAE pulls ahead only when you allow fine-tuning with labeled data, because its reconstructive pretraining gives a strong initialization that adapts fast. The deeper criticism the video builds toward: forcing a model to predict every pixel spends capacity on texture, noise, color variations, and compression artifacts that may be irrelevant for downstream tasks.

I-JEPA: predicting in embedding space 23:16–26:00

This is the pivot. I-JEPA keeps the predictive challenge of MAE but moves the prediction target from pixel space into the learned representation space. No hand-crafted augmentations. No pixel reconstruction. No negatives.

Three components:

• Context encoder f_θ — encodes the visible patches into s_x.
• Target encoder f_ϕ — an EMA copy of the context encoder with stop-gradient, producing target embeddings s_y for the masked blocks.
• Predictor g_ψ — conditioned on the context embedding s_x and a positional variable z (which specifies where to predict), produces a predicted embedding ŝ_y.

The variable z is the model's instruction: "predict this part of the image." Different z-values require different target locations, so the predictor cannot ignore where it is being asked to predict — it must learn spatially-conditioned features that faithfully match the encoding of the actual content at that position. This is what distinguishes JEPA from a generic embedding-matching loss.

Reported result: a ViT-Huge trained on ImageNet in under 72 hours on 16 A100s reaches 79.3% top-1 on the standard linear probe, and wins on Clevr/Count and Clevr/Dist where invariance-based methods stall — because it preserves spatial and relational structure that crop-augmentation methods destroy. See the I-JEPA paper for the primary numbers.

V-JEPA: latent prediction in video 26:00–28:33

The same architecture extends naturally to video. Images test whether the model understands what is missing in space; video raises a deeper challenge — can the model infer what is missing over time? The change is minimal in concept: swap 2D image patches for 3D spatiotemporal tubelets (e.g. 2 frames × 16×16 pixels), and swap 2D rectangle masks for spatial tubes (a spatial rectangle extended through every temporal slice). The EMA target encoder, the predictor with mask-token queries, and the latent-space loss all survive unchanged.

To do this well, the model has to learn that objects persist, motion continues, and actions have physical consequences — all from raw video, with no labels and no captions. On frozen-backbone benchmarks, V-JEPA outperforms video pixel-prediction methods, and the same encoder handles both appearance-heavy tasks (Kinetics, where what an action looks like is enough) and motion-heavy tasks (Something-Something v2, where labels are deliberately disambiguated by what is moving).

Reported: a ViT-H/16 trained on 2 million unlabeled videos reaches 81.9% on Kinetics-400, 72.2% on Something-Something v2, and 77.9% on ImageNet-1K, all with a frozen backbone and a linear probe. See the V-JEPA paper (titled Revisiting Feature Prediction for Learning Visual Representations from Video — V-JEPA is the model name, not the paper title).

V-JEPA 2-AC: from prediction to planning 28:33–31:51

This is the talk's payoff. V-JEPA 2 scales the recipe to a ViT-g encoder (over 1 billion parameters), trained on VideoMix22M (more than 1 million hours of internet video), with two technical upgrades: mask denoising with an L1 loss replaces mask prediction, and 3D Rotary Position Embeddings replace fixed sinusoidal encodings. The frozen encoder then adapts to many downstream tasks — action recognition via an attentive probe, visual question answering by feeding video features as tokens to a language model.

V-JEPA 2-AC goes further: it adds action conditioning. The predictor is conditioned not just on position but on the agent's chosen actions, so the model can answer: "if the agent is here and does this, what happens next?" After post-training on fewer than 62 hours of unlabeled robot videos from the DROID dataset, the model supports robotic planning in the physical world.

The planning loop uses the Cross-Entropy Method (CEM), a zeroth-order optimization procedure: randomly sample candidate action sequences, roll the predictor forward for each, measure how close the predicted final embedding is to the goal, keep the best (the "elite set"), fit a Gaussian distribution over those actions, and sample again. After a few rounds, the candidates zero in on actions that achieve the goal. This is Model Predictive Control (MPC), but planning inside a learned latent state space with a learned action-conditioned dynamics model, rather than hand-specified states and dynamics.

Reported: 77.3% top-1 on Something-Something v2 (a benchmark that requires genuine temporal reasoning), and 39.7 recall@5 on Epic-Kitchens-100 action anticipation — a 44% relative improvement over all previous methods. See the V-JEPA 2 paper.

Preventing collapse: the recurring sub-theme 31:51–36:27

The video returns to collapse explicitly here, and it is worth isolating because it is the field's central engineering problem. The mechanisms, in order of appearance:

The endgame: LeWorldModel 36:27–end

The closing thesis puts the pieces together. JEPA tells the model what to predict: future or missing embeddings, not raw pixels. SIGReg tells the representation how not to collapse. Combined as LeJEPA, the model trains end-to-end with only two loss terms — one predicts the next embedding, the other regularizes the latent space so embeddings stay well-distributed. Applying LeJEPA to world modeling gives LeWorldModel.

The point is not that LeWorldModel is the final answer — the video is honest that these results are early and small-scale. The point is the direction: a principled way to train a world model from raw pixels with a minimal, well-understood objective, on a single GPU, that plans faster than systems orders of magnitude larger. World models need not be giant foundation models to be useful.

What the video gets right — and what it cannot prove

The video is an excellent architecture tour, but a tour is not an independent reproduction. The headline numbers (79.3% linear probe, 81.9% K400, 77.3% SSv2, 39.7 recall@5, 48× faster planning) come from the papers' own reported benchmarks, not from third-party reproductions. Benchmark scores depend on hardware, evaluation protocol, dataset curation, and training schedule; they are evidence, not universal laws about JEPA.

Verified references

Primary sources, deliberately separated from secondary explainers so claims can be checked at the source. Each link was opened to verify the title and authors.

1. How AI Learned to Teach Itself [JEPA] — the source video and chapter timestamps.
2. Becker & Hinton, "Self-organizing neural network that discovers surfaces in random-dot stereograms," Nature 355, 161–163 (1992).
3. van den Oord et al., "Representation Learning with Contrastive Predictive Coding" (CPC, InfoNCE), 2018.
4. He et al., "Momentum Contrast for Unsupervised Visual Representation Learning" (MoCo), 2020.
5. Chen et al., "Improved Baselines with Momentum Contrastive Learning" (MoCo v2), 2020.
6. Chen et al., "A Simple Framework for Contrastive Learning of Visual Representations" (SimCLR), 2020.
7. Grill et al., "Bootstrap your own latent: A new approach to self-supervised Learning" (BYOL), 2020.
8. Caron et al., "Emerging Properties in Self-Supervised Vision Transformers" (DINO), 2021.
9. He et al., "Masked Autoencoders Are Scalable Vision Learners" (MAE), 2022.
10. Assran et al., "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture" (I-JEPA), 2023.
11. Bardes et al., "Revisiting Feature Prediction for Learning Visual Representations from Video" (V-JEPA), 2024.
12. Assran et al., "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning," 2025 — V-JEPA 2-AC is the action-conditioned variant in Section 4 of the same paper.
13. LeCun, "A Path Towards Autonomous Machine Intelligence" (JEPA position paper), 2022.
14. Zbontar et al., "Barlow Twins: Self-Supervised Learning via Redundancy Reduction," 2021.
15. Bardes et al., "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning," 2022.
16. Khazatsky et al., "DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset," 2024 — see also the project page.
17. LeJEPA — the SIGReg regularization paper that LeWorldModel builds on.
18. Maes et al., "LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels," 2026 — ~15M params, single-GPU training, 48× faster planning.
19. Rubinstein, "The Cross-Entropy Method for Combinatorial and Continuous Optimization," 1999 — the CEM planning procedure used by V-JEPA 2-AC.
20. Zhuang et al., "How Well Do Unsupervised Learning Algorithms Model Human Real-time and Life-long Learning?" NeurIPS 2022 — the benchmark cited in the caveats section showing BYOL/SwAV/MAE underperforming SimCLR/MoCo-v2 on sparse, low-diversity datastreams.

Related CatGame work

Our own trail runs the opposite direction: make a small, inspectable model trustworthy before making it wider. The JEPA lesson that transfers is not "build a 1B-parameter video encoder" — it is the principle of predicting in representation space and the discipline of collapse prevention as a first-class engineering concern. Read the Kimi K3 MoE lessons for the same translation exercise at the language-model scale, Dual-Stream GRU-L3 GPU for our recurrent context path, RMSNorm and spectral stability for recurrent bounds, and Vulkan ML compute for the dispatch-versus-kernel lesson that underpins any on-device representation learner.