JEPA SOLUTIONS

LeCun proposed JEPA as the world model at the center of a modular cognitive architecture for autonomous machine intelligence. The five modules: perception (extract representations), world model (JEPA) (predict future states in latent space), cost (evaluate desirability), actor (propose actions), and short-term memory.

Why Latent Prediction?

Predicting all details of the future (pixel-level) is both intractable and unnecessary. A self-driving car doesn't need to predict every leaf — it needs to predict "the car ahead will brake." JEPA achieves this by encoding observations into compact representations and predicting future representations conditioned on actions.

The first concrete implementation of JEPA. Demonstrates that predicting in latent space produces representations superior to MAE for downstream tasks, especially in low-shot and transfer settings.

Architecture

• Context Encoder: ViT-B/16, ViT-L/16, or ViT-H/14 (632M params)
• Target Encoder: EMA copy, momentum cosine 0.996→1.0
• Predictor: Narrow transformer (~6 layers, smaller hidden dim)

Multi-Block Masking (Key Innovation)

• 4 target blocks: scale 0.15-0.2, aspect ratio 0.75-1.5
• 1 context block: ~0.85 of image, with target regions removed
• Predictor must predict from spatially distant context patches

Why I-JEPA Outperforms MAE on Transfer

MAE reconstructs pixels, forcing the encoder to retain low-level texture/color. I-JEPA predicts abstract representations, so the encoder focuses on high-level structure. This is why I-JEPA excels at object counting and depth estimation (structure, not texture).

# I-JEPA training pseudocode
x = sample_image()
ctx_patches, tgt_patches = multi_block_mask(x)

z_ctx = context_encoder(ctx_patches)          # ViT on visible patches
with no_grad():
    z_tgt = target_encoder(x)                 # Full image through EMA

z_pred = predictor(z_ctx, mask_tokens)         # Predict at target positions
loss = mse_loss(z_pred, z_tgt[tgt_positions])

loss.backward(); optimizer.step()
target_encoder.params = tau * target_encoder.params + (1-tau) * context_encoder.params

Extends I-JEPA with multi-scale hierarchical representation learning via a Feature Pyramid Network (FPN). Learns representations at 3 hierarchy levels simultaneously.

Architecture

• Encoder: ViT-Tiny (5.5M) + RoPE + Flash Attention
• Predictor: 4-layer transformer (2.8M params)
• FPN: 3 levels, 128-channel fusion
• Total: ~13.8M params (8.3M trainable)
• Loss: Combined VICReg + prediction + SigReg (sigmoid regularization)

# H-JEPA structure
src/
  models/       # encoder, predictor, H-JEPA module
  losses/       # VICReg, SigReg, combined
  masks/        # masking strategies
  data/         # datasets and transforms

# Config
model: { encoder: vit_tiny, embed_dim: 192, num_hierarchies: 3 }
loss: { type: combined }  # vicreg + prediction

Extends JEPA to video with disentangled representations for motion and content using factored masking and two separate predictors.

Factored Masking (Key Innovation)

• Motion masking (temporal): Entire frames masked → predictor must learn temporal dynamics
• Content masking (spatial): Same spatial region masked across ALL frames → predictor must learn appearance

Each predictor is a ~6-block transformer (384-dim). Both losses backpropagate through the shared context encoder.

Results

• Something-Something v2: Motion features alone approach full-model performance (validates disentanglement)
• Optical flow estimation competitive with unsupervised methods
• Unifies self-supervised visual learning and optical flow in a single encoder

JEPA for self-supervised video understanding. Predicts latent representations of spatiotemporal masked regions without pixel-level reconstruction.

Architecture

• Encoder: ViT-L/16, ViT-H/16; spatiotemporal tube tokens (2×16×16)
• Masking: Contiguous spatiotemporal blocks, ~75-90% masking ratio
• Data: VideoMix2M (~2M video clips)

Key Results (Frozen Encoder)

• Kinetics-400: ~82% top-1 (frozen + attentive probe)
• Something-Something v2: ~71.4% top-1

# V-JEPA spatiotemporal masking
video = load_video(T=16, H=224, W=224)
tokens = patchify_3d(video, patch_size=(2,16,16))

target_tubes = sample_tube_masks(
    num_targets=4, spatial_scale=(0.15, 0.2),
    temporal_span=(0.5, 1.0), aspect_ratio=(0.75, 1.5)
)
z_ctx = context_encoder(tokens[~target_tubes])
z_pred = predictor(z_ctx, mask_tokens_3d)
z_tgt = target_encoder(all_tokens)
loss = mse(z_pred, stop_grad(z_tgt[target_tubes]))

Adapts JEPA to audio by treating mel-spectrograms as 2D images. Uses ViT backbone with random patch masking on spectrograms.

• Input: 10-second AudioSet clips @32kHz → mel-spectrograms
• Masking: Random patch masking on the 2D spectrogram
• Result: Comparable to wav2vec 2.0 and data2vec with <1/5 the training data
• Evaluated on: X-ARES suite (speech, music, environmental sound)

JEPA for 3D point clouds. Introduces a sequencer module that orders patch embeddings by proximity for efficient context/target selection in 3D space.

Architecture

• Tokenization: FPS (center selection) + kNN grouping + mini-PointNet embedding
• Masking: Proximity-based 3D block masking via the sequencer
• Advantage over Point-MAE: No reconstruction ambiguity (many point sets = same surface)
• No handcrafted 3D augmentations needed

A distinct approach to 3D JEPA with emphasis on context-aware decoding and geometry-aware masking.

• Context-aware decoder: Continuously feeds context during target prediction (vs standard one-shot)
• Multi-block 3D sampling: One informative context block + multiple representative target blocks
• Result: 88.65% on ScanObjectNN PB_T50_RS with only 150 pretraining epochs

Bridges imitation learning and self-supervised learning by jointly predicting action sequences and latent observation sequences end-to-end.

• Insight: "Predicting latent observation sequences effectively generalizes to predicting action sequences"
• +40% improvement in world model understanding
• +10% higher task success rate across all environments
• Enables learning from unlabeled data while maintaining policy relevance

The major scale-up of JEPA: trained on >1 million hours of internet video. First JEPA model to demonstrate zero-shot robotic manipulation.

Key Results

• SSv2: 77.3%  |  Epic-Kitchens: 39.7 R@5
• Video QA (8B): 84.0 PerceptionTest, 76.9 TempCompass
• Zero-shot robot: Object picking/placing on Franka arms across two labs, no environment-specific training

Systematically diagnoses and fixes training fragilities in I-JEPA. Key finding: I-JEPA is more fragile than it appears.

Key Fixes

• Adaptive masking curriculum: Start low, increase over training (prevents early collapse)
• Wider, shallower predictor: More stable than I-JEPA's narrow-deep design
• VICReg regularization: Explicit variance + covariance terms
• Modified EMA warmup: Prevents premature target convergence

Introduces object-level masking that acts as latent interventions with counterfactual-like properties. Moving from patches to semantically meaningful objects.

• Masking entire objects forces inference of object state from other objects → causal inductive bias
• +20% absolute improvement in counterfactual reasoning (VQA)
• Comparable control performance with only 1% of latent features

Unlocks dense features in JEPA: spatially structured, semantically coherent, temporally consistent.

Four Innovations

• Dense predictive loss: Both visible AND masked tokens in training loss
• Deep self-supervision: Hierarchical loss across intermediate encoder layers
• Multi-modal tokenizers: Unified image + video training
• Scaling: Improved capacity + data volume

Combines JEPA with Vision-Language Model reasoning through a dual-temporal pathway.

• Dense JEPA Branch: Consecutive frames, fine-grained motion
• VLM Thinker Branch: Sparse frames, semantic reasoning
• Hierarchical Pyramid: Aggregates multi-layer VLM outputs into JEPA-compatible features
• Result: Outperforms both VLM-only and JEPA-only baselines on long-horizon manipulation

A minimalist JEPA world model: two loss terms, one hyperparameter, no EMA, trainable on a single GPU in hours.

Radical Simplification

• Loss: next-embedding prediction + Gaussian regularizer (replaces EMA for collapse prevention)
• One hyperparameter (down from six in prior approaches)
• ~15M parameters; single GPU, hours to train
• 48x faster planning than foundation-model world models
• Competitive on 2D and 3D control tasks
• Latent space encodes physical structure; detects physically implausible events