Motivation
When JEPAs are trained on time series, their embeddings often cluster by the underlying dynamical regime even without any labels — series governed by the same dynamics land near each other. This is a striking and useful emergent property, but it had only been observed empirically. This analytical work asks why it happens, seeking a principled, dynamical-systems explanation rather than treating the clustering as a fortunate accident, so that one can predict when it should occur.
How it works
The study takes a trained time-series JEPA — context encoder $f_\theta$, EMA target encoder $\bar f_\theta$, and predictor $g_\phi$ matching masked future/segment latents, with temporal windows as the masking unit — and analyzes its learned predictor through the lens of Koopman operator theory.
- A Koopman operator linearizes nonlinear dynamics by acting on observables (functions of the state).
- The authors show the JEPA predictor approximates such a linear evolution in latent space.
- Its invariants — Koopman eigenfunctions and spectral quantities preserved under the dynamics — coincide with the directions along which embeddings separate.
Thus the analysis connects the geometry of the embedding to the spectral structure of the generating dynamical system.
The objective
The underlying model is trained with the standard temporal JEPA regression,
$$\mathcal{L} = \big\lVert\, g_\phi(z_{\text{ctx}}, m) - \operatorname{sg}\big[\bar f_\theta(x)_{\text{future}}\big]\,\big\rVert_2^2,$$
with $\operatorname{sg}$ the stop-gradient and EMA targets. The contribution is analytical, not a new loss: the authors argue that minimizing this latent next-step prediction objective implicitly forces $g_\phi$ to act as an approximate linear (Koopman) operator on the encoder's observables, so the encoder is driven to discover Koopman-invariant coordinates of the generating system.
Key results & what's novel
The work gives a dynamical-systems account of representation geometry in time-series JEPAs. Latent next-step prediction implicitly forces the encoder to discover Koopman-invariant coordinates of the generating dynamical system; series sharing a dynamical regime collapse onto the same invariants, producing the emergent clustering. By connecting self-supervised latent prediction to Koopman theory, it explains why and when such clustering should be expected and offers a principled diagnostic for what these models learn — turning an empirical observation into a theoretical prediction.
Strengths & limitations
- + Principled, dynamical-systems explanation for emergent regime clustering.
- + Connects JEPA latent prediction to the well-developed Koopman framework.
- + Offers a diagnostic for when clustering should appear.
- − Analytical: explains rather than improves a model.
- − The linear-Koopman approximation holds best for systems well-described by a finite set of observables; strongly chaotic or non-autonomous dynamics may break it.
- − Conclusions are tied to the specific predictor and masking regime analyzed.