Yann LeCun’s Latest JEPA Paper: Where Does End-to-End Planning Go Next?

When it comes to embodied AI, JEPA is a technical approach within world models that cannot be avoided and deserves repeated discussion. It possesses many promising traits: better robustness to interference, more efficient computation and training efficiency, and extremely low dependence on data annotation. More importantly, JEPA captures the laws of the physical world through representation learning to perform real-time planning and predict the future. This mode of thinking aligns closely with the fundamental path of human learning and evolution, and it already has a fairly complete theoretical framework and research foundation. At one point, JEPA satisfied many researchers' technical imaginations for achieving general-purpose embodied intelligence.

「Maybe This Is It.」

Unfortunately, JEPA's current shortcomings in reasoning are as prominent as its strengths. Unlike pixel-based visual learning, JEPA does not directly generate pixels or images, so it cannot output intuitive visual content like generative models. In scenarios requiring visual generation, JEPA typically needs to be combined with rendering or generative models, which currently limits its general capabilities. Compared to training paradigms for 3D display output or video diffusion, JEPA inevitably falls short.

Because JEPA's performance relies heavily on the quality of the representation space, if the encoder learns poor representations or suffers from "representation collapse," prediction effectiveness will be significantly reduced, making it difficult to fully explain the model's reasoning process for complex physical scenes. Therefore, it still faces challenges in engineering implementation regarding planning for complex long-horizon tasks, generalization in real-world open environments, and the stability of real-time control for robotic components.

However, this should not be considered an unsolvable problem. In the long run, this "defect" of JEPA is not fatal. The relationship between representations and latent spaces may improve with long-term research and eventually be resolved.

Recent research has shown that JEPA can support embodied interaction for robots. V-JEPA 2 builds on video learning by introducing robot action data to plan toward target images. V-JEPA 2.1, released in 2026, further improved spatial details and temporal consistency in images and videos, increasing the real-robot grasping success rate in its tests.

What should be done? Currently, the approach is to layer the model.

Yann LeCun’s team has released a new paper proposing H-JEPA, a hierarchical world model for planning in latent space. This research not only introduces an architectural innovation that transcends single latent spaces but also addresses certain limitations of previous JEPA models. The team reports that the three-layer model increased success rates to reach targets from approximately 18% to 73%, while improving computational efficiency.

The paper is led by Wancong Zhang and Basile Terver, with guidance from Yann LeCun and Randall Balestriero. H-JEPA enables different layers to predict futures at varying horizons and learn the information each needs to retain. The lower layer tracks finer details, while higher layers form coarser judgments, breaking down distant goals into smaller sub-goals manageable by lower layers.

Placing the Model in a "Maze"

World models help robots anticipate the consequences of actions. Before executing a move, the system can compare candidate plans: If the arm moves this way, where will the object end up? If it changes direction, does it get closer to the goal? The machine then selects one option to execute and adjusts its plan based on new observations.

However, these models often struggle when predicting and planning within a single latent space and time scale. At a single time scale, long-horizon predictions require unfolding numerous fine-grained steps, leading to accumulating prediction errors and expanding the action search space. Because JEPA relies on data from a single shared latent space, it must align representation relationships with task objectives. While latent variables containing all rapid-changing details may work well, they fail when goals are more abstract than states. For instance, if the goal is simply to reach a location rather than match a specific pose, the machine finds it harder to interpret.

So, what exactly is H-JEPA? In simple terms, it is a world model with an internal "pyramid-style hierarchical" structure.

The Yann LeCun team attempts to execute longer tasks by decomposing the model into hierarchical layers, learning a hierarchy of latent predictive models from observations and action trajectories. In the first layer, the model processes the finest-grained observation streams, encoding each state and action before passing partial information to the second layer, which observes overall changes at intervals.

During this process, decaying states and actions are retained in the first layer. Level 2 samples the state every two lower-level time steps for further processing, while synthesizing the two actions within that period into a single action record. Consequently, each higher layer receives potential states generated by the layer below at coarser time steps, resulting in a stack of JEPA world models, where each layer possesses its own observation encoder, action encoder, and latent predictor.

All layers predict future latent states based on past latent states and actions, comparing these predictions against actual outcomes. Simultaneously, they employ SI-GReg to maintain state stability, preventing the model from collapsing all visual inputs into a single indistinguishable state.

After end-to-end training across all layers, H-JEPA sets objectives and sub-objectives for models at different levels, enabling them to learn planning across varying time horizons: lower layers focus on short-term dynamics, while higher layers plan for more distant states. Each higher level of the hierarchical world model can predict over longer time ranges, retaining predictable content within its latent space. Higher layers process slow features with large time steps to serve long-term prediction and high-level planning; lower layers process fast features with small time steps to support short-term prediction and low-level control. Thus, H-JEPA reveals a hierarchical structure akin to the cerebral cortex: cortical regions form a hierarchy of intrinsic time scales, with representations at higher levels changing more slowly.

How is this executed in specific scenarios?

The team placed the model in a digital maze to observe how it plans movement strategies to navigate out. The top layer first plans a broad progression toward the goal, the middle layer refines this into nearer waypoints, and the bottom layer decides on execution. During this process, the system re-observes the environment and updates its plan.

Through this approach, H-JEPA enables the model to complete planning from top to bottom. The model first deduces a general route out of the maze at a high level—for example, "go to the corridor entrance, then turn, and finally reach the destination." The states predicted along the way by the high layer become small sub-goals for the lower layers. Subsequently, Level 1 and Level 2 calculate specific actions to take, outputting executable movements.

Trade-offs in Details

The above design further changes how each layer of the model uses latent space to observe the world. The team thus constructed four levels of Latent Space for H-JEPA to conduct further experiments. The model's Level 1 JEPA uses LeWM as its base, employs a ViT-Tiny encoder to summarize each image into a global representation, and pairs it with a causal Transformer predictor. In this iteration, the team used FourRoom Distractors for navigation, placing a controlled agent alongside an independently moving distractor that moves smoothly with occasional random teleports, while Visual AntMaze required a quadruped robot to traverse a maze.

Researchers made interesting findings. When reconstructing internal information from each layer back into images, representations of position relationships such as the ant's location and the maze layout remained clearly discernible across all levels. However, body errors grew larger, causing the rapidly changing body postures of robots at Level 2 and Level 3 to become increasingly blurred, to the point where they could not be recovered.

Meanwhile, the distractor (the blue dot in the figure) moving around in the maze and serving as a task interference object for H-JEPA became increasingly blurred as the hierarchy rose, until it almost disappeared. Meanwhile, the model's high-level perception of its own body remained very stable.

Two findings indicate that H-JEPA, designed in this manner, achieves highly consistent perception of the robot's body and position. The more abstract goal-planning layer and the lower-level planning layer dedicated to specific details each perform their respective roles while maintaining strong robustness against interference. During this process, some data is absorbed so effectively by the encoders at the first and second layers that there is no need to pass it further up. This 'focus on the big picture while letting go of the small' approach may encourage each model layer to maintain its own judgment regarding representations that are unimportant or should be appropriately ignored, which then informs actual actions. Consequently, the model learns to make trade-offs in detail.

However, JEPA's most significant challenge remains long-horizon tasks, as representation relationships tend to degrade or collapse when the time scale is extended. So, how does the H-JEPA model perform when the time scale is lengthened?

Compared to prior models such as LeWM and HWM, H-JEPA not only achieves higher task success rates in testing but also shows marked improvement despite inherent limitations. The team evaluated models at different hierarchical levels, comparing the granularity of planning across various latent spaces. In a maze environment, the calculated cost (distance) to the asterisk position decreased as the model's internal layers increased. Blue areas indicate where the model perceives a small gap to the target position, while red indicates a growing gap.

The results revealed that with each additional layer, the model achieved higher success rates with less planner computation. At the second and third layers, the color changes along the corridor became more continuous, and the regions with smaller gaps near the asterisk widened. This provided the model with clearer cues to determine which candidate actions were approaching the target. High-level representations make it easier for the model to distinguish progress toward distant goals, providing clearer guidance for route planning in the maze.

Thanks to hierarchical decomposition, the computational cost for H-JEPA to plan movements has also been reduced. The Yann LeCun team split and compared internal records from the first and second layers of the model. It was evident that whether dealing with interference resistance in FourRoom Distractors or visual recognition in Visual AntMaze, the cost for tracking closer sub-goals decreased more stably along the trajectory at the lower level. This indicates that the planning cost at the model's base is kept within a relatively limited range, and the downward trends of both levels remain largely consistent. Thus, setting objectives layer by layer and decomposing abstract goals helps reduce the computational load of JEPA for long-horizon tasks. The lower-level model makes more stable judgments about progress, facilitating step-by-step task advancement and systematic planning. Of course, the cost difference between Level 1 and Level 2 occasionally rises, stagnates, or fluctuates in certain intervals; the planning cost for Level 1 sub-goals still exhibits varying degrees of stepwise decreases as each sub-goal is generated. Nevertheless, these flaws do not obscure the overall merits.

Offline Deployment

When researchers evaluate models on offline datasets from real-robot deployment, a more troublesome problem emerges.

The DROID dataset contains manipulation videos recorded by humans teleoperating real robots. Different episodes may take place in different rooms, with varying tables, objects, and camera angles. Within a single video, the room background is typically more stable than the robotic arm. If training only encourages the model to predict future representations while requiring sufficient difference between them, the model might find an easy shortcut: it can distinguish many videos simply by recording the room type (e.g., kitchen, office, or a specific workbench). Since the background of a given video changes little, predicting the next moment’s description becomes easy. Details we consider critical—such as how the robotic arm moves or how the gripper approaches an object—can be discarded by the machine.

This phenomenon is called "slow feature collapse." Here, "collapse" does not mean all images are compressed into a single number; the model can still distinguish many frames but loses important information required for action. It is like the model remembering which room the robot is in, but failing to adequately remember what the robot should do.

Therefore, the research team adopted a different approach: inferring intermediate execution actions based on two consecutive states. For example, the model must determine the direction of the robotic arm’s movement using information before and after the motion, thereby preserving changes related to action. This does not require manual data annotation for each object, but it does require that the training data provide action information.

In this evaluation, adding action-related constraints transformed a nearly useless single-layer model into a useful baseline, after which H-JEPA’s hierarchical structure came into play. Only after ensuring the model retained action information did planning trajectories improve. This also demonstrates that not all progress can be attributed solely to "hierarchy."

The finding suggests that ignoring hard-to-predict details can sometimes make long-term planning more effective, but "easy to predict" does not automatically equal "important for the task." A static background is easy to predict, whereas a fleeting grasping contact may determine task success or failure. H-JEPA has not fully resolved how to draw this boundary. Under specific conditions, abstract goal planning is indeed effective, but it can also mislead the model.

The study also tested four simulated navigation and manipulation environments. In the task of pushing T-shaped objects, performance improved when the model was increased to two layers but degraded when increased to three layers. The research team attributes this decline to insufficient data length. Because higher layers require training segments spanning longer time horizons, and the episodes in this task are short, the three-layer model could utilize only 58% of the training transitions per episode compared to the single-layer model to assemble sufficiently long segments.

H-JEPA provides a crucial directional insight: focusing attention on the internal latent space architecture of the model. Theoretically, a world model requires two capabilities simultaneously: preserving details that cannot be missed during operation, and compressing long tasks into stages that are easy to schedule. How these two capabilities connect clearly raises new questions. For instance, how does planning efficiency change after splitting the latent space? Can hierarchical prediction of sub-goals help improve JEPA’s interaction latency? If physical-world changes occur extremely rapidly, how should hierarchical planning JEPA integrate with embodied intelligence? Perhaps dividing the latent space into more levels is not always better. Additionally, updating the model’s higher layers based on the completion of events, rather than mechanically at fixed frame intervals, is another future direction proposed by the authors. These possibilities expand the imaginative space for connecting world models with generative models that produce robot actions.

Looking Further Ahead

Notably, this paper can be viewed as another significant contribution from Advanced Machine Intelligence (AMI), founded by Yann LeCun, further deepening its focus on the JEPA framework. Since Yann LeCun left Meta in November 2025, his startup AMI has completed $1.03 billion in funding, reaching a pre-money valuation of $3.5 billion. Investors include NVIDIA, Samsung, and Toyota Ventures.

Currently, the company is dedicated to building intelligent systems capable of understanding their environment, retaining memory, and continuously planning. Earlier, medical AI company Nabla announced an exclusive strategic partnership with AMI, gaining priority access to relevant world model technologies to explore medical agents that can understand workflows and anticipate consequences. Given that the team’s next research steps focus on closed-loop validation and reliable operations on physical robots, it is not difficult to glimpse AMI’s future commercial trajectory.

One point worth discussing is that the trajectory of world models closely integrated with embodied AI is becoming increasingly diverse. As an internal simulator of the physical world for agents, a world model endows embodied systems with the capacity to think and anticipate. Models like WAM facilitate a mutual integration between world models and embodied models, while JEPA opts to predict abstract states, concentrating computational efficiency on a set of effective representational planning methods; there is also overlap between these approaches. NVIDIA’s DreamZero eliminates the separation between the world perceived by the model and its actions, whereas DeepMind first constructs environments for agent training. Video observations, latent states, and structured representations all contribute to "predicting the future," which constitutes the most critical challenge for the embodied brain.

The divergent yet converging problem-solving approaches in these frontier studies reflect distinct trade-offs made by different teams across objectives, environments, structures, reasoning, and efficiency. However, their core purpose remains the same: enabling machines to adapt to a rapidly changing physical world. Regardless of the starting route, practitioners in this field believe that data is inherent in embodiment, and constructing machine neurons is far from a fantasy. As leading figures in the industry continue their forward-looking research, we may discover that the connection between embodied architectures and world models grows tighter, until they truly merge into one.

Why not maintain patience and observe?