In the fourth installment of the World Models series, we discuss Motus, a unified world model designed to make full use of multimodal, heterogeneous data: internet videos, image–text data, robot trajectories, task-agnostic trajectories, and more. The aim is to use different modalities rather than confine learning to one data format.
We invited Dr. Hengkai Tan, one of the researchers behind the project, to explain it. The views in this interview are his personal views. I highly recommend this episode: some technical details may initially seem demanding, but the explanations are very intuitive.
Original article outline
Contents
👇
1. Why Was Motus Called a Latent Action World Model?
2. How Motus Differs from VLAs and World Models
3. Unified versus Traditional World Models
4. Different Routes to a Robot Policy
5. The Motus Training Framework
6. Motus Training Data
7. Data Requirements at Different Levels
8. Post-Training
9. From Next-Token to Next-State Prediction
10. Why Does China Have So Many World Model Startups?
11. Comparing the Approach with Interactive World Simulators
1. Why Was Motus Called a Latent Action World Model?
We released this work last year, in 2025, when the concept of a “world action model” did not yet exist. NVIDIA introduced that framing around late January or early February this year, roughly two months after our work. [Editorial note: Motus's first arXiv submission was December 15, 2025; NVIDIA's DreamZero paper, World Action Models are Zero-shot Policies, was submitted February 17, 2026. These dates establish the sequence, not who first used the term.]
We therefore did not begin from the world-action-model perspective. We had two other motivations: a unified world model and latent actions. We combined them into one model framework.
More fundamentally, the goal was to make full use of multimodal, heterogeneous data: internet video, image–text pairs, robot trajectories, task-agnostic trajectories, and so on. We wanted to use different modalities rather than be restricted to one data format.
That is the most important point for us. Approaches focused on a particular modality, such as action-only prediction, struggle to incorporate video prediction. That limits their ability to use data and can affect generalization.
More broadly, it reflects a pattern in AI development: disruptive breakthroughs often come from finding a new paradigm that can absorb enormous amounts of data. From AlexNet to GPT and then video models, each represented a leap in the range of data that could be used.
On the theoretical side, Google DeepMind published General agents contain world models in June 2025, proving that a general agent necessarily contains a world model. Any two of the goal, dynamics model (world model), and policy can be used to derive the third. With the goal given, world modeling and policy modeling are therefore interconnected and can be considered from a more unified perspective: unified world models. [Editorial note: the paper proves a result for agents generalizing across a sufficiently rich class of multistep goals under its assumptions; it does not state that any arbitrary pair of a goal, dynamics model, and policy uniquely determines the third.]

2. How Motus Differs from VLAs and World Models
You can understand it by drawing a three-by-three grid.

Prepared by Dr. Tan. Figure legend: the red dot marks the existing foundation we build on—the “shoulders of giants”; the yellow star marks our target, a robot policy.
Here, o₀ is the current observation, or current frame; o₁ through oₖ are future frames to be predicted; and a₁ through aₖ are future actions. Of course, observation can mean a broader observation of the world, rather than vision alone.

I redrew this from Dr. Tan's notes. Both figures are included here to make the surrounding explanation easier to follow.
VGM: Video Generation Model.
WM: World Model.
IDM: Inverse Dynamics Model.
VLA: Vision-Language-Action Model.
O: Observation, or state. In the explanation below, observations are two-dimensional video frames.
a: Action.
Q (query): the information we want to retrieve or predict.
K (key) / V (value): the information index / its content—in other words, the source of information.
Attention: a query reads information from values according to how well it matches their keys.
From the attention perspective, one side is the query and the other is the key/value. We use current information as conditioning to predict the future. Information must flow through the model, but different modeling choices correspond to joint, conditional, or marginal distributions.
The four quadrants represent different regions: predicting video, predicting action, inferring action from video, and modeling dynamics by predicting video from action.
VGM: Video → Future Video
A video generation model represents how the world evolves. It uses current or past frames to predict future frames, learning visual dynamics such as object movement, contact, and occlusion. In probabilistic terms, it models:

IDM: Future Video → Future Action
An IDM works in the opposite direction, using future observations to infer the intervening actions. It reasons backward from an outcome: given states before and after a transition, usually video frames or observations, what action occurred between them? Its central role is to turn visually observed changes into action information usable for control. Data without action labels is especially important for this direction. It models:

IDM means inverse dynamics model. The route is video → IDM → action → policy: first predict video, then infer the action with the IDM.
WM: Action → Future Video
A world model represents how actions change the world. Given a current state and action, it predicts the next state or observation: the classic dynamics model. Traditional world modeling mainly follows a single direction, for example from action to video, predicting future states conditioned on actions. It models:

VLA (Policy): Video + Action → Future Action
A VLA mainly operates on the action side. Actions attend to one another, for example within an action-expert module, while also being conditioned on the initial observation or current frame. It does not predict future observations or video in this setup. It fits the action distribution without incorporating video prediction, so it cannot absorb video data through that prediction objective. It directly predicts control actions from images or states. It models:

Unified WM (Bidirectional): Jointly Modeling Video ↔ Action
What we call a unified world model or world action model brings these blocks together. Rather than learn only one direction, such as dynamics or a policy, it models the relationships within a single framework. A direct consequence is that different data types become usable: video-only data, data without task conditioning, and robot trajectories can all contribute to training.
Video and action have four interaction combinations: video → video, video → action, action → video, and action → action. We cover these paths and learn their distributions during training. Essentially, we model a joint distribution that enables bidirectional multimodal interaction. One trained model supports different inference modes, including marginal and conditional distributions.
How a Unified World Model / World Action Model Differs from a VLA
By comparison, a VLA is essentially one-way, usually predicting actions from observations without this bidirectional joint modeling. Our unified world model aims to encompass all four information flows. In that sense, it is a more general framework covering several predictive relationships at once.
Why do we think this direction is the future? It can absorb more types of data, use existing pretrained models, and support different inference modes. Ultimately, broader and more comprehensive data coverage, scaled with computation, should produce stronger capabilities.
3. Unified versus Traditional World Models
At the underlying level, a traditionally defined world model is a transition function: given the current state and action, predict the next state. A unified world model is more than a single transition function.
Probabilistically, it models several distributions together: video to video, action to video, video to action, and action to action. These conditional relationships are learned jointly. The underlying formulation therefore differs from one-way dynamics modeling; it is a more general joint-modeling framework.
You can view it as a more complete probabilistic framework that explicitly incorporates relationships in different directions. That aligns with some ideas from our long-standing work on Bayesian modeling.
Many people interpret a world model as a transition function from the current state and action to the next state. Mathematically, it models the conditional distribution of the next state given those inputs. That transition distribution is just one of the possible distributions.
Our unified world model does not stop at that relationship. The transition you mentioned is only one information flow: action → observation.
We model multiple relationships and distributions. Written out more fully, they include video generation, VGM (observation → observation); inverse dynamics, IDM (observation → action); and policy modeling, including self-attention among actions.
A traditional world model is therefore a subset of the larger framework. We bring different prediction relationships together rather than modeling only one transition.
4. Different Routes to a Robot Policy
There are common foundations. Diffusion, for example, is essentially a process of energy modeling or energy minimization, so that underlying level is shared. But unified and traditional world models still differ substantially in their modeling formulations. [Editorial note: this is the interviewee’s conceptual analogy; diffusion sampling is not generally identical to minimizing a single fixed energy function.]
Many world model researchers take an incremental route: first build a good one-way transition model, then extend it—for example, adding an IDM and eventually obtaining a policy. That is a natural step-by-step approach.
We took a more ambitious approach from the outset, trying to build and model these components together in a unified, scalable way. Whether that route is better remains an open question; there is no definite answer yet.
Several information flows are being explored to bring video-model capabilities into policies:
One route goes through an IDM: predict video, then use inverse dynamics to infer actions. Examples include UniPi, RoboDreamer, Vidar, Gen2Act, and LingBot-VA. The route is VGM → IDM → action policy.

Another uses an action-conditioned world model. First learn dynamics, then perform reinforcement learning or planning inside the model to obtain a policy: WM → RL → action policy. I would expect teams working with the original definition of world models to want to use them to strengthen policies.

Our route models video and action together, with information flowing in both directions, and directly learns a joint model: joint modeling → policy.

So people are exploring several paths: from video to an action policy, from a world model to an action policy, or from both sides together. Which will ultimately prove most viable is unsettled. Video generation offers an excellent starting point—the shoulders of giants—and an action policy is our ultimate goal. We are exploring the bidirectional route.
5. The Motus Training Framework
Our initial ideal was a unified model with several experts: one for video generation, one for action, and one for understanding. Joint attention would let all of them interact, unifying both cross-modal interactions and interactions between cognitive functions such as understanding and generation.

Ideally, a single training process would support all tasks—IDM, world modeling, VLA, video generation, and more—and data from different sources would enter through a unified training strategy.
In practice, doing that directly causes problems. Training every module together from the start can damage the capabilities already learned by the video model. We made many engineering tradeoffs in training.
For example, we first train video generation, then freeze the video component while training other modules, such as action and understanding. [Editorial note: this describes staged freezing in the interview. The published final recipe specifies full Motus pretraining in Stage 2 with the pretrained VLM frozen, rather than a permanently frozen video expert.]
For understanding, our implementation took features from a VLM, Qwen. Its image–text features already contain strong semantic information. We feed them into an expert-like module made of multiple Transformer layers, each interacting with the other experts through joint attention to fuse information.
In simple terms, the understanding component adapts a VLM and connects it to the rest of the training system. The architectures did not align directly at the time, so we used a fairly engineering-driven solution. It could be made more elegant in future.
The ideal is to train everything together, but the practical process still needs stages.
At a more abstract level, we had three goals:
First, and most importantly, integrate multimodal, heterogeneous data. We believe data diversity and scale are important sources of intelligence.
Second, build as much as possible on existing work and foundation-model priors: video generation models, VLMs, and even action modules trained by others, rather than starting everything from scratch.
Third, unify model types such as VLAs, world models, and video generators within one framework. We use a UniDiffuser-like formulation: controlling each expert's diffusion timestep lets one model switch between inference modes. That differs in some respects from DreamZero's approach.
From the feedback we received, ours was relatively early in scaling this direction and achieving strong results. Many people subsequently emphasized the unified idea: bring multiple modalities into one framework.
The benefits are those three points again: use data from more sources; draw on pretrained foundation models such as video models and VLMs; and unify different task modes.
The framework can also expand. We could add modalities such as audio and touch by adding expert modules, for example another diffusion Transformer (DiT) or Transformer.
The UniDiffuser idea you mentioned operates at that final unification level. We train in a UniDiffuser-like way, essentially using a DiT architecture and diffusion timesteps to control denoising, unifying and switching among modes.
6. Motus Training Data
We designed a six-level data pyramid. From bottom to top, it contains internet video, egocentric data, simulator data, task-agnostic data, data from multiple robot embodiments, and finally data from the target embodiment.

These six levels enter progressively at different training stages. A unified world model can accommodate the different data types, which makes this possible.
For example, task-agnostic data can be sufficient for training an IDM or a world model. Video generation can use the lower levels of video data. A VLA that ultimately outputs physical actions primarily uses multi-embodiment and target-embodiment data.
We used a few million data entries—roughly two million—which was a fairly large scale at the time. Training used around 20 machines. Each stage took roughly a week, with the three stages completed in a little over two weeks overall.
There were broadly three stages: video-model training, joint training to integrate different model components, then supervised fine-tuning (SFT). The data differed between stages.

Initial video-model training mainly used video, including internet video and some task-related video. Any data fitting the distribution of predicting the future from a current frame and conditions could be useful. We did not introduce the top-level target data at the beginning, to avoid contaminating later testing or fine-tuning.
More task-relevant data was introduced as training progressed.
We initially tried feeding all data and modes into joint training directly, but found it difficult and unstable. In practice, we trained in stages: train one part, freeze it, then progressively bring in other modules, building the system up gradually.
Overall, the process progressively integrates different data and task modes rather than training everything jointly from the start.
We introduced latent actions because much of the data has no action labels, or its action signals are weak and noisy. Action spaces also differ across robot embodiments, creating further inconsistency.
A unified latent representation aligns motion information from these sources and gives the action expert a better prior for initialization.
7. Data Requirements at Different Levels
The amounts of data at different levels vary enormously.
At the bottom, internet video such as YouTube can amount to billions of hours—an enormous scale. Many teams now collect egocentric data and talk about collecting a million hours this year. That level is already substantial, yet remains several orders of magnitude below internet video.
Simulator data can be supplemented through automatic generation, but still has a gap from real data and mainly plays a supporting role.
Task-agnostic data is particularly interesting. It helps the model adapt quickly to a new robot embodiment. Collecting task-specific data manually is expensive, whereas this level can be automated: write scripts that let the robot explore its action space, a little like a baby first learning by crawling and experimenting.
It need not consist only of successes. It can include many failed attempts—touching things or trying different movements—to cover more possible states and actions. This type of data is comparatively easy to obtain.
At the top is data for the target robot and target task. It is usually the most carefully curated and cleanest data, collected through real operations by people.
The available scale generally decreases as we move upward.
Internet video is on the scale of billions of hours; egocentric data is now at roughly millions of hours; multi-embodiment data might be only tens of thousands of hours.
Some teams' total robot data is at that latter scale. On the video-generation side, I have heard of training data approaching 100 million hours.
Egocentric data usually comes from a head-mounted viewpoint. We think first-principles reasoning about people offers a useful reference: humans primarily use head-level vision for perception and decisions, rather than cameras on their wrists.
In practice, many collection systems still add wrist cameras. The reason is straightforward: today's models are not yet capable enough, and a head view alone misses details. Hands can occlude manipulation, or the viewpoint can be too far away for adequate precision.
Wrist cameras provide supplementary views to address occlusion and missing fine-grained information, helping maintain data quality and usability.
Force feedback or tactile information might reduce this dependence on multiple camera views in future. Adding hand sensors or sensorized gloves is certainly one direction.
That does not mean multi-view systems such as wrist cameras must be replaced entirely. From a first-principles perspective, an ideal system might not require so many cameras. But that may be a long-term possibility, contingent on stronger computation, more mature models, and better tactile modalities before auxiliary sensing can gradually be reduced.
In the short term, compatibility is more practical. Our framework supports multimodal input, so multiple cameras, viewpoints, and data collected from different angles can be brought together.
This data level is developing quickly. Many teams can reach millions of hours of egocentric data this year. Combined with billions of hours of internet video, these sources offer substantial scale advantages that should strongly improve model capabilities.
The quality requirements are also demanding. Annotation granularity—atomic actions or whole tasks—how data is segmented, and diversity of scenes, objects, and tasks all affect results.
It is a substantial data-engineering problem. Once the workflow is established, including data organization, annotation standards, and standard operating procedures, subsequent work becomes smoother and model iteration accelerates. Many data providers are collecting egocentric data this year.
8. Post-Training
During fine-tuning, we jointly train all modes: video generation, inverse dynamics, world modeling, and VLA.
Video and action interact bidirectionally during training, conditioning and influencing one another rather than having a one-way dependence. At inference, they can be generated jointly, with synchronized denoising timesteps for video and action.
9. From Next-Token to Next-State Prediction
In language modeling, tokens are discrete, so modeling next-token prediction with cross-entropy is natural and straightforward.
In robotics and world models, however, what we model is action and video, which are essentially continuous. Forcing them into discrete representations causes substantial information loss, especially in precision and quantitative relationships.
I therefore think continuous modeling is more appropriate. Diffusion is a common way to model the distribution of a single frame or state, and that part is fairly straightforward.
Once time is involved, however, I see an autoregressive structure as necessary: roll the sequence forward step by step.
The mainstream approach can therefore be understood as autoregression (AR) along time and diffusion for the continuous distribution within a frame or step. It combines AR and diffusion. I think this has largely become a consensus.
I see autoregression, temporal structure, and causality as fundamentally linked.
The current state is necessarily determined by past states, so there is a natural causal relationship. Once we model a time series, we therefore almost inevitably end up with an autoregressive framework.
The same applies to world models. The mainstream approach combines autoregression and diffusion: advance one frame or one chunk at a time. That is next-step prediction, an autoregressive process.
Within each step, diffusion generates a frame or state. In short, time uses AR and each step uses diffusion. This has basically become a consensus, with work such as Genie 3 following a related idea. [Editorial note: this is Dr. Tan's modeling perspective. Temporal causality does not require every model to generate one frame at a time; models can also jointly generate multi-frame chunks.]
At the broader level, there are roughly three routes:
One is a world action model or our unified approach, jointly modeling video and action with information flowing both ways.
A second begins with an action-conditioned world model: learn dynamics first, then move toward a policy.
A third follows approaches such as UniPi and Vidar, going from a video model through inverse dynamics to infer actions and obtain a policy.
So there are three information flows being explored: model both sides together, move from the world-model side toward policy, or move through the IDM side toward policy.
We began working in this direction in 2024, when many people did not believe in a video-centric route. Only over the last year or two has its importance become more widely recognized. More people now think world action models might gradually replace earlier VLA approaches and become a new paradigm.
10. Why Does China Have So Many World Model Startups?
Video models are developing differently from language models. From what I see, Chinese companies are strong overall in video generation.
Expectations were high when Sora appeared, but its results and subsequent development did not establish an unbridgeable lead. Looking at some rankings and actual products, many leading models now come from Chinese teams.
There are also clearly more startups in China than in the United States combining world models with embodied intelligence.
There are several possible reasons for China's strength in video models.
One explanation at the more immediate level is China's accumulated video-data advantage, whereas video data in the United States faces compliance and related issues.
A broader explanation is that the people have changed. Many core researchers in the language-model generation were born around 1990; many of those driving video models were born after 1995 or even after 2000. I think this younger generation has stronger educational backgrounds and technical capabilities than the preceding one.
I see that improvement as especially pronounced in China: the increase in capabilities from people born in the 1990s to those born after 2000 is larger than the corresponding generational change in the United States. I attribute that to long-term improvements in education and the technical environment.
Of course, the United States also has excellent young researchers, including many with Chinese backgrounds. But I still see differences in the overall numbers and composition.
From the perspective of the talent pool, this shift favors China. Combined with a more active domestic startup environment, I think video models are beginning to show a trend toward overtaking in commercialization and deployment.
11. Comparing the Approach with Interactive World Simulators
There is a broader way to look at this. One approach perfects one route before gradually extending it. At a more abstract level, however, things are interconnected.
Video and action are tightly coupled, rather than entirely independent. Their relationship is bidirectional. If we model only video to action or only action to video, we learn incomplete information and a partial view of the world.
The probabilistic perspective is similar. Modeling just one conditional distribution, such as a traditional transition distribution, covers only part of the system.
A more complete joint distribution, encompassing different marginal and conditional relationships, can in principle capture more information and cover long-tail or complex situations more effectively.
It comes down to a modeling choice: learn only some relationships, or use a unified approach from the beginning and model them together.
At a deeper level, all AI models are essentially modeling probability distributions. Whether it is next-token prediction in language models, video generation, or world modeling, the underlying idea is the same. [Editorial note: this is the speaker’s broad conceptual framing, rather than a literal characterization of every possible AI architecture.]
Another interesting point is what we observed in data scaling.
We also studied conventional scaling curves, including comparisons with π₀.₅. Those results were fairly direct: we observed roughly 2.6× to 13× better data efficiency, a substantial advantage in those experiments.
Multi-task behavior is more interesting. People generally think methods such as VLAs essentially fit an image-to-action mapping within a single task, so their multi-task generalization is limited.
We ran experiments increasing the number of tasks from one to 50. π₀.₅'s average success rate declined as the task count grew, suggesting interference between tasks rather than a positive benefit.
Our result went the other way: more tasks increased average success. This shows that the model is learning shared “world knowledge” across tasks rather than learning each task in isolation.
In approximate numbers, π₀.₅ began around 70% and declined toward 50% as tasks were added. Our model improved from a similar level toward nearly 90%.
Put another way, increasing task count raised average success from roughly 50% to 90%, which is a substantial improvement. [Editorial note: the interview gives differing approximate starting values here and in the preceding paragraph. Both are retained as spoken; they should not be read as one exact, internally consistent numerical curve.]
These evaluations primarily concern policy performance. They were conducted on a simulation-style benchmark with roughly 50 tasks. [Editorial note: these are benchmark-specific results, not a universal success rate across real-world robot deployments.]
Reference
Motus: https://motus-robotics.github.io/motus
A Smart Note Recommendation
Investors and founders working on embodied intelligence have probably been exceptionally busy lately: many projects to assess, many people to speak with. After a week of travel, a weekend by the sea or on the grass with family and children can be a welcome way to relax. I recommend a stress-relieving product that adults and children can enjoy together. It takes about five minutes to get started, and you can play melodies and songs you like. It is suitable for family gatherings or outdoor downtime—playing a little music while letting your mind wander. I was hooked as soon as I tried it, and wanted to recommend it to Smart Note readers.