Now published in Nature Machine Intelligence!
nature.com/articles/s4225…

Huge thanks to the reviewers for a constructive review process. The paper is better for it.

really enjoyed working with @mosh_levy on this paper! anyone interested in learning abstractions for decision making should check it out 👇

A short video on how to make sense of which features JEPA keeps in its latent.
Paper: arxiv.org/abs/2610.27364

8/ Being predictable isn’t enough: the set must also fit in the latent. We grow C/D from a pair into a cycle of 4, 8, then 16 images. On average, it stays as predictable as before, yet its representation drops at every step until A/B wins.

11/ These findings matter for anyone building world models with JEPA: what the latent leaves out can't inform planning, however accurate the predictions. Predictive closure helps us reason about what stays in, and how to steer it.

12/
Code: github.com/Mosh0110/predi…
Paper: arxiv.org/abs/2610.27364

Joint work with Aly Lidayan (@a_lidayan) and Yoav Goldberg (@yoavgo).

7/ A feature can lose its place in the latent because of features it has nothing to do with. Image pairs A/B and C/D compete for the latent, each kept or dropped as a whole. Making C/D more predictable pushes A/B out, though nothing about A/B changed.

6/ An unpredictable feature is kept if it helps predict others enough. In our sequences, a traffic light's color decides which image changes next, while its own next color is unpredictable. The more often it decides, the better it's decoded: from chance (50%) to 86%.

4/ Our claim, intuitively: JEPA keeps the set of features that is most "closed under prediction". More precisely: among the sets that fit in the latent, JEPA's objective favors the set with the most information about its own future. We call this set the predictive closure.

5/ This agrees with earlier findings when features evolve independently, and makes new predictions when they interact. To test them we train JEPAs on sequences of a few small images side by side, with rules we set for how the images change, and a latent too small for everything.

3/ Reasoning about one feature at a time feels like chasing your own tail: what's worth representing depends on what must be predicted, and what must be predicted depends on what's represented. So our characterization judges features as members of sets instead of individually.

2/ Two factors decide the traffic light's fate. Can it be predicted? No: there's no telling when it will change, so keeping it costs a prediction error. Earlier work stops here. We also consider: does the light help predict the cyclist? Yes, so dropping it costs an error too.

1/ JEPA selects which features its world model keeps.
But how?
Will it keep a feature it can't predict, like a traffic light?
Judging each feature alone, earlier work says no.
We show evidence that this is wrong and introduce a new characterization that explains why. 👇🧵

On the one hand, I think a strong case can be made that we have made much less progress on AI capabilities than many assume. Frontier AI systems still struggle to learn from limited examples, learn continually from experience, and interact effectively with the physical world. More broadly, their competence is highly uneven. Superhuman performance on tasks they excel at (e.g., maths and coding, where training data can be generated at scale) doesn’t mean they will excel at the much messier tasks involved in most white-collar work.

On the other hand, nobody should find this assessment that comforting. It suggests there is vast room to improve the capabilities of existing systems by changing their design and algorithms. In other words, we could have vastly more powerful systems with the same amount of computing power.

Given that many of the world’s smartest people, assisted by already very capable AI systems and backed by vast resources, are working to accelerate AI research and development, even those sceptical of the current paradigm need to be prepared for the possibility of explosive change once certain capabilities are cracked.

Free to read here (no paywall): rdcu.be/fNGVrPiPBvza

Lucky to have worked on this with @ElyosephZohar, @ravfogel and @yoavgo

"AlphaFold and Claude are very similar, but we don't worry that AlphaFold will become conscious . . . the fact that we worry about it for Claude says more about us than about it."

what if this is the jagged frontier?

"GPT-6 Astra can solve"

-> "The puzzle geometry and object-space reference path come from the supplementary material for C-Space Tunnel Discovery for Puzzle Path Planning. This project did not discover that reference solution from scratch."