Back to World Model

World Models · EP01

World Models EP01: Shenyuan Gao on DreamZero, DreamDojo and a New Paradigm for Robot Intelligence

A conversation with Shenyuan Gao about DreamZero, DreamDojo, world action models and the technical approaches to robotics world models.

Zha Dami2026-03-1828 min readWeChat original
Contents

World models went from a term I heard occasionally in 2024 to a very popular topic today. Following our three previous educational series—Robot Data, including EP10: A Summary of the First Season of Robot Data Interviews; Human Data, including Series Summary: Human Data—The “Key” to Robot Data; and Robot Foundation Models—Smart Note will launch a series explaining world models in the first half of 2026.

For the first interview, we are fortunate to welcome Shenyuan Gao, a contributor to both DreamZero and DreamDojo. He is pursuing a PhD at the Hong Kong University of Science and Technology, researching interactive world models and related topics. He is also a member of NVIDIA Research’s GEAR team, working with Yuke Zhu and Jim Fan. [Editor’s note: The original introduction uses an honorific equivalent to “Dr.” while describing him as a PhD student; the translation retains the stated student status.]

Smart Note will continue this world-model series throughout the first half of 2026. I am both a technology blogger and an investor. If you have interesting academic work on world models, please recommend it for an interview. If you are considering a startup in this area, I would also welcome a conversation about entrepreneurship and financing. We hope to support young people with entrepreneurial ideas at the earliest stage. I look forward to meeting you in this new era.

Original article outline

Contents

👇

1. A taxonomy of world-model approaches

2. What is a world action model (WAM)?

3. World models for autonomous driving versus robotics

4. How DreamZero works

5. How DreamDojo works

6. Connecting DreamZero and DreamDojo

7. Perspectives on 3D data

8. How data collection can evolve

9. Combining world models with reinforcement learning

1. A taxonomy of world-model approaches

I think the term “world model,” in the context we are discussing, initially came from reinforcement-learning researchers. In RL, you need a model of the environment, which you can think of as the world. Put simply, a world model is a state-transition function: it takes an action and a state as input and outputs the next state. It complements a policy. You give the world model an action, and it tells you the next state. You give that state to the policy, which produces another action, and repeat this interaction. [Editor’s note: This describes the RL usage; ideas about internal models of the world also have an earlier history in other fields.]

I started working on world models during an internship at Shanghai AI Lab. Tesla had already done some work on them. I initially worked on synthesizing autonomous-driving data, around the time early image-diffusion and video-diffusion papers appeared. While using diffusion to generate driving data, I thought: if diffusion can generate data, why not build a world model directly? We collected a large dataset and started doing that.

At the most abstract level, a world model is an action-conditioned state-transition equation. But actions can be represented in different modalities, and states can take many different forms.

Video-generation models: pixel space

The most mainstream approach today, in my view, is a world model based on video generation. It is the most scalable approach and entirely data-driven, and benefits from advances in video diffusion.

JEPA

Yann LeCun has long advocated approaches such as JEPA. His central argument is that states should be represented in latent space rather than pixel space, because predicting pixels is not an appropriate objective for decision-making. He often gives examples: when you enter a new room, you cannot precisely predict every pixel; when making decisions while walking down the street, you do not predict exactly what everyone’s face looks like. Such low-level details are nearly impossible to predict and irrelevant to the decision. Asking a model to predict them consumes its capacity fitting things that are inherently unpredictable and do not help decisions. In this view, excessively low-level information should not be the prediction target of a world model.

The idea is to construct a more semantic latent space and make predictions there, retaining only information useful for decision-making. You can construct that space through unsupervised learning or supervision from other tasks, making it more useful for decisions while avoiding low-level detail prediction. That is the thinking behind LeCun’s approach.

For the energy-based models (EBMs) mentioned in the JEPA discussion, see our Interview with Yilun Du: Embodied Intelligence Through Energy-Based Models and Video Generation. [Editor’s note: “EMB” in the source is a typographical error for “EBM.”]

Dynamics models

Another approach is the dynamics model. A world model and a forward-dynamics model are essentially consistent in definition. The distinguishing feature of some dynamics-model work pursued by researchers in North America is how states are represented. They use highly abstract structures—keypoints or graphs, for example—to represent an object’s state and its relationship to a hand. This is neither an unstructured latent representation nor video pixels, but a structured abstraction that may include keypoints or physical properties.

Earlier and more traditional robotics research often favored these abstract representations because they facilitate few-shot or zero-shot transfer, allowing human demonstrations to transfer quickly to new tasks. In my view, however, such methods usually make many assumptions, for example about material properties, and introduce strong inductive biases. They are not entirely data-driven. Although useful with small samples, they can struggle to benefit fully from large datasets because they incorporate so many manually specified priors.

Yunzhu Li has explored dynamics models extensively. See our earlier interview, In Conversation with Yunzhu Li: A New Approach to General Manipulation Through Learned Dynamics Models.

3D world models

Another category is the 3D approach represented by Fei-Fei Li’s World Labs. I think it has strengths and weaknesses. It decomposes next-state prediction into stages: first explicitly reconstruct the entire world in 3D; then, after an action or viewpoint changes, generate and render the next observation. In other words, it separates next-state prediction into reconstruction and rendering.

Explicit 3D reconstruction can provide strong spatial consistency. But rendering is handled separately, so this is not a purely end-to-end pipeline. Once you reconstruct the 3D world, dynamic processes—pedestrians walking, vehicles moving or liquids flowing—still need to be simulated separately. That can use rules or learning, but reconstruction and dynamics simulation remain decoupled rather than part of one end-to-end model. Overall, I do not find this way of simulating dynamics especially scalable. [Editor’s note: These categories and their tradeoffs are Gao’s interpretation of the approaches at the time of the interview.]

World models based on physics engines

Another category, pursued by NVIDIA and others, uses physics engines. A physics engine already performs a world model’s function: give it an action, and it simulates the next state. Its advantage is potentially very precise physics—for example, dynamics governed by Newton’s laws. The difficulty is achieving sufficient diversity. Introducing a new object often requires manually designed assets, although this is improving as 3D-generation methods automatically create more assets and scenes. [Editor’s note: A physics engine numerically approximates a specified physical model; this does not mean it reproduces all real-world physics exactly.]

Broadly, its pipeline resembles the 3D approach associated with Fei-Fei Li: reconstruct the world, simulate dynamics in some way, and render the result. It is not a purely end-to-end method.

Genesis also follows this route. See our earlier interview, Introducing Genesis: A New Robotics Physics Engine—Will It Transform Robot Simulation?

2. What is a world action model (WAM)?

That is a very good question. We call it a world action model to emphasize its ability to predict actions as well as the world.

DreamZero and related work can also be understood as world models. Traditionally, a world model is a state-transition function: its inputs are a state and an action, and its output is the next state. Within that framework, the conditioning action can take different forms.

DreamZero combines a world model with an inverse-dynamics model that decodes actions from predicted future states. If you view it as a world model, the conditioning “action” is actually a task instruction—a textual description of the task. Unlike a classical state-transition world model, DreamZero predicts both future states and actions. Its inputs are the state and task instruction, and its outputs are future states and low-level actions. It therefore jointly predicts actions.

That is why we call these world action models: they model the world, meaning states, and actions together. Here, “world” refers to a representation of state. Current implementations typically use video to represent how the visual world changes.

More broadly, the world need not contain only visual information. Audio, touch and other modalities could be added in the future to form a more complete world state. The name “world action model” accommodates modeling such a multimodal state while predicting corresponding actions; it does not imply that every current model already includes these modalities.

3. World models for autonomous driving versus robotics

This distinction resonates strongly with me because I moved from autonomous driving into robotics world models.

In autonomous driving, Huawei, for example, has proposed frameworks similar to world action models. The basic idea is the same: pretrain or co-train an action policy using predictions of the world’s future, improving action-prediction efficiency. Many works take forms similar to ours, including joint action–video prediction.

Implementations differ. Some alternate predictions; some remove the world branch when predicting actions and train only the action branch; others jointly predict states and actions as we do. The concepts are consistent, but training strategies and modeling details differ.

Those differences largely reflect the tasks. Autonomous driving has a relatively uniform structure: reach a destination safely. Robot tasks are closer to an open world and much more varied. The driving action space is also simpler: you can often abstract it as waypoint planning, perhaps with only two degrees of freedom. Robot control has more degrees of freedom, with far more complex control dimensions and action combinations.

At the framework level, similar methods could serve both. For example, some speculate that Tesla might transfer its autonomous-driving world models to Optimus for robotics. That is a possibility rather than an established fact. Autonomous driving is certainly an easier setting in which to begin building a world model.

There are three main reasons. First, it has abundant data—one of the richest data sources among embodied-intelligence tasks. Second, its action space is simpler, essentially path or waypoint planning. Third, and crucially, safety means avoiding collisions. Although the scene is open and dynamic, vehicles should ideally avoid contact with other objects. A world model therefore primarily models their motion rather than the intricate contact interactions and material properties central to manipulation. This comparison concerns collision avoidance, not the absence of vehicle–road contact or other physical effects.

Robotics is substantially harder in that respect. Even picking up a towel involves deformable materials and complex contact modeling. A robot may interact with a huge variety of objects, each with different physical properties and modes of interaction. At the modeling level, that makes a robotics world model much more complex than one for driving.

4. How DreamZero works

There are two common ways to understand VLA, or vision–language–action models. The narrower meaning is a pretrained vision–language model (VLM) adapted into a policy that outputs actions. The broader meaning emphasizes the modalities: a mapping from video or vision plus language inputs to actions. DreamZero takes video and language and produces actions, so it also counts as a VLA in the broader sense.

DreamZero overview: diverse non-repetitive pretraining and joint video–action prediction
DreamZero emphasizes task diversity over repetition. Its world action model takes video, language and proprioception and predicts future video and continuous actions, supporting task and embodiment transfer.

The method is fairly direct. We start with an existing text-conditioned video-prediction model. Given the current frame and text as conditions, it predicts future video.

We do not change its existing video-prediction capability; we add action prediction. Specifically, we introduce two modules, an action encoder and an action decoder, to encode actions into the same latent space as video. The model can then predict future actions while predicting future video. That is its basic function. [Editorial note: “same latent space” describes joint modeling here; the implementation uses modality-specific encoders and decoders with a shared backbone.]

For training, we use a large amount of AgiBot data. The original model predicts only future video. Through supervised training on real data, we train it to predict both future video and future actions, supervising both outputs.

After training, it can be fine-tuned for other embodiments. Using datasets from other robots, you can provide further supervision for the video and action modalities, adapt it to a different robot platform, and make it a policy for another embodied setting.

During data collection, we record video and the corresponding joint-control data while the robot performs tasks: visual observations alongside low-level actions.

What is distinctive about our data collection?

Unlike earlier models, our collection strategy emphasizes diversity. Here, “VLA” means the category adapted from VLMs. Previously, VLA training often involved collecting 50 or 100 trajectories for a fixed task such as pick-and-place. At the task level, those data were highly repetitive.

For DreamZero, we use a different strategy. Contractors operate the robot, move freely through the environment and perform varied actions, many only once. We annotate the entire trajectory with text afterward. Navigation and mistakes are allowed; we do not deliberately constrain the sequence of actions.

Our data are therefore generally long and highly diverse. You can hardly find two identical trajectories or strictly repeated task procedures. The aim is to approximate how people might collect data while actually working. Compared with hiring people to repeat one task, collecting non-repetitive data incidentally during work—housekeeping, for example—makes a scalable collection model easier to establish.

Several factors may explain why world action models learn more effectively from diverse data.

Video models reduce ambiguity relative to VLMs

Earlier VLAs adapted from VLMs essentially learn a mapping from task instructions to actions. Their pretraining is based on language models, with text as a core modality. But text does not adequately express low-level actions; substantial information is lost.

For example, a plan in text space might say “pick up the water bottle.” That description does not specify how to execute it. You do not know the approach angle, grasp pose or force. There is considerable uncertainty between text and a concrete action. Earlier VLAs therefore need many trajectories for an individual task so that they also learn how the action should be performed, before decoding it into low-level control signals.

What they inherit from a VLM is mainly high-level understanding and planning. They have not learned during that pretraining how to perform specific movements or decode semantics into executable actions.

Video planning is different. Take picking up a cup: if the model has already predicted the future video, then, setting aside occlusion and related issues, the execution details are largely specified. Hand pose, approach path and grasp angle appear in the video. Video supplies a specific, unambiguous execution outcome.

When turning the model into a policy, we only need to learn a video-to-action mapping. This mapping is almost deterministic, because the video already contains all the low-level details. The model no longer has to derive concrete movements from text; it only learns to decode an already determined visual plan into control signals. [Editorial note: this is the interviewee’s representation-based intuition. Video does not uniquely identify hidden forces, contact states or every robot command.]

We therefore do not need to repeatedly collect many examples of one task to teach the model how to pick something up. The mapping from text to planning has already been learned through visual modeling in video pretraining. Compared with language alone, video loses less information, expresses the plan more concretely and makes the intended outcome more definite.

With that understanding, we focus on collecting sufficiently diverse data to learn the video-to-action mapping, rather than repeatedly sampling a single task. The priority is covering more variation rather than increasing the number of trajectories for the same task.

WAMs improve data efficiency

Besides video planning reducing ambiguity, we find world action models very efficient at using training data. This is not a brand-new observation: it reflects earlier conclusions of ours that recent results have further supported.

A VLA adapted from a VLM essentially fine-tunes a large VLM into a policy, usually supervised only by actions. A low-dimensional action signal supervises a large model originally centered on video or language, converting it into a policy. That supervision is sparse.

Earlier work also identified problems such as a diffusion-based action objective differing from the original VLM training objective. Many current methods therefore initialize a new action expert and train it separately. But these changes do not remove the underlying issue: actions themselves are a low-dimensional signal and do not make full use of the data.

A world action model’s advantage is supervising video and actions together. The data contain both, and video provides dense supervision. Every frame constrains the model’s prediction of future states, rather than only an action output. In one trajectory, you have action supervision and frame-by-frame video supervision, allowing more efficient use of the data.

WAM performance

Recent benchmarks have supported this observation. AI2 in North America released a new benchmark, and Robot Arena also has one. We outperformed π0.5 on both. DreamZero ranked first last week, without using AgiBot’s proprietary data for that comparison. [Editor’s note: These rankings are the speaker’s account at the March 2026 interview, not a claim about today’s leaderboards. The project uses the name RoboArena.]

We used only the joint data and trained the world action model on the same data, yet exceeded π0.5. Importantly, π0.5 used extensive cross-embodiment data, including bimanual data and datasets collected by several universities in Open X-Embodiment—a large-scale training effort. We reached first place by fine-tuning on just the joint data. [Editor’s note: “Joint data” is the wording in the transcript; it should not be taken as an independently specified dataset name. The paper distinguishes AgiBot and DROID experimental settings.]

This supports the two points discussed earlier. First, traditional VLAs need much repetitive data to resolve ambiguity between language and concrete actions. Second, a video model provides a less lossy, more definite plan representation. Turning it into a policy then mainly requires learning the video-to-action mapping, rather than relearning the language-to-planning process.

Video also provides dense supervision, making use of every frame. Fine-tuning can therefore be considerably more efficient. This is one reason we obtain good results with comparatively less data.

Insights from latent actions

Interestingly, several of us previously worked on latent actions, and I think that line of work offers useful insights. We observe that video models already represent actions well, whereas a VLM processing video alone does not have the same capability, because its representation loses much more information.

We also find that we can extract action representations from video without supervision. Put simply, a latent action is a compact representation extracted from video through self-supervision, with a high proportion of action information relative to noise. We can compress video into a very small dimensional space and extract action information directly from that latent.

This shows two things. First, video and action already have a very deterministic relationship. Second, we can extract action-representing information from video entirely through unsupervised methods. This mapping is therefore very easy to learn. Mapping language embeddings to actions is much harder. [Editorial note: “deterministic” is the interviewee’s intuition about a specified visual movement; the limits of identifying physical commands from video are noted above.]

You can therefore understand a world action model as combining two things: video specifies a fairly definite plan, and the video-to-action mapping is relatively easy to learn. Making both parts easier improves overall data efficiency. I think this also opens up more possibilities for collecting pretraining data.

WAM efficiency challenges

If you ask me about the biggest problem with world action models—or video action models—I would say inference efficiency. Much of the DreamZero paper’s work concerns inference optimization. We sped it up by roughly 30 times. When we first worked on it last year, real-time evaluation was painful: it would execute a movement, stop, execute another movement, then stop again.

The project also gave me a clear realization: pure engineering optimization can reduce model inference costs to a very low level. The prevailing assumption about video models and WAMs was that they were slow and could never compete with VLAs at inference time. But efficiency is critical to closed-loop control. The more frequently a policy executes, the more frequently it can correct errors; a higher execution frequency can itself improve success rates.

Conversely, a stop-and-go execution pattern can reduce success even when the policy loss is low, because corrections are too infrequent.

We initially worried a great deal about inference efficiency and tried many distillation methods to shrink the model and accelerate it. Later, we found engineering optimization alone could produce an overall speedup of roughly 30 times. [Editor’s note: The paper reports a combined 38× speedup from model, system and implementation optimizations; the interview gives a rounded account.]

Many optimizations are relatively granular engineering measures: torch.compile, KV caching, multi-GPU parallel inference and restructuring the overall inference pipeline. Our model is indeed expensive and needs a high-end multi-GPU server, rather than an ordinary single GPU. Parallelism and system optimizations substantially reduce latency.

More advanced system tools can now automate many engineering optimizations. With a well-designed inference structure, system-level changes can resolve many performance problems. Hardware remains a prerequisite, however: high-end GPU resources are necessary, and we cannot avoid that requirement.

The final DreamZero version has 14 billion parameters, so it is large. We subsequently tested smaller models, whose performance declined, although the extent depends on the backbone. Some smaller models with suitable architectures lose relatively little performance.

We also experimented with distillation, but the gains were not substantial. Looking back, we initially pursued video quality intensely, using large models and many denoising steps. Then we realized inference efficiency was the key and gradually placed less emphasis on video quality, reducing the step count. We eventually built a “Flash” version that outputs a plan before the video is fully denoised. The number of steps was already very low—four, or even one.

Distillation often has the same goal: reducing denoising from many steps to four or one. Once our inference design had already reduced steps to approximately that level, distillation offered limited additional benefit. System optimization and inference design delivered more direct gains than algorithmic distillation.

Over the longer term, smaller but more capable models will keep appearing, and engineering optimization will continue to improve. I therefore do not see inference efficiency as a fundamental barrier.

5. How DreamDojo works

DreamDojo is a world-model simulator for low-level actions. Given a robot’s action, it predicts future states. Other models offer similar functions, such as Google DeepMind’s Genie series: you can control a generated environment through a keyboard and operate within that generated world. Its generalization is very strong—for example, you can interact with any generated image scene. [Editorial note: this is the interviewee’s broad comparison. It does not establish that every Genie version supports every arbitrary image or uses robot-style continuous controls.]

DreamDojo world model method overview
DreamDojo pipeline: human-video pretraining using In-lab, EgoDex and DreamDojo-HV; robot post-training; autoregressive distillation; and applications in novel environments, live teleoperation, policy evaluation and model-based planning.

Previously, low-level action world models in robotics generally did not generalize that way. Most were trained on a fixed dataset and evaluated in almost the same setting—for example, an action-conditioned model trained on a tabletop scene in one laboratory from a fixed viewpoint, then used for RL or evaluation in that same setup. They largely worked only within the training scenes and action distribution.

But that is not the real goal of a world model, especially a video-based one. Its potential lies in generalization. At test time, for example, you could use model predictive control (MPC): predict future states under different actions, helping a policy generalize across scenes and action distributions. If training always uses the same scene and action distribution, the model has not learned generalizable world dynamics, and policy generalization will not improve.

Robot data are difficult to make sufficiently diverse because teleoperation is expensive. Moreover, most existing robot data consist of expert demonstrations, with few random behaviors or failures, so the action distribution is narrow.

Improving generalization with human data

We therefore considered another source: human data. You can view a human as a kind of robotic embodiment. Internet video offers large amounts of human behavior with varied object interactions and scene changes that robot data struggle to cover. To collect those interactions with a robot, you would have to buy the objects and actually gather demonstrations. Human video naturally contains them.

Human behavior also has a rich action distribution. People perform all sorts of actions; at sufficient scale, the diversity becomes very high. We therefore built a human-video dataset of approximately 40,000 hours to pretrain an action-conditioned world model.

There is a difficulty: humans and robots have different embodiments, so we need some form of action label. One simple option is a hand-pose estimator that extracts hand poses from video and treats them as actions. But that creates several problems.

First, hand-pose estimation is often inaccurate on highly diverse internet videos. Many estimators assume a stationary camera, such as a fixed third-person view, whereas human videos often come from head-mounted or moving cameras with changing viewpoints. Accuracy then falls. Second, hand pose is a very specific representation, such as a five-fingered hand configuration, which does not transfer naturally to other embodiments such as grippers.

We ultimately chose latent actions. They can be extracted directly from video through self-supervision, without additional annotation or a separate pose-estimation model. We can also control how much information they contain, retaining action-related information without tying it too tightly to one embodiment. A hand-pose representation confines much of its information to hand structure; a latent action can retain more general movement information and transfer better across robot embodiments.

The overall method first pretrains on large-scale human data to improve generalization. By then, the model can predict movements and scene changes well. We subsequently optimize it through distillation.

Distillation

The original model predicts chunks of future sequences—for example, 12 latent frames at once. Even if you only want the first frame, you must wait for all 12, slowing interaction. We add a distillation stage that makes prediction autoregressive and reduces the required denoising steps. The model can output latent frames incrementally, decoding and interacting after, for example, every four predicted frames.

This has two benefits: faster inference and more frequent interaction. In applications such as test-time search, the world model repeatedly simulates future states under different actions to choose the best action. Excessive model latency lowers closed-loop control frequency and harms performance. A faster model is therefore preferable.

Another interesting application is controlling the modeled robot directly with VR. The world model runs on a computer; your VR movements are converted into robot actions, letting you “teleoperate” a virtual robot inside the model. The robot is not physically present, but you can see its behavior through generated video.

If future world models become sufficiently stable and generalizable, anyone with a computer and VR equipment could collect data or operate a robot inside them, without a physical robot. That would effectively replace a traditional simulator with a world model.

For deployment on a real robot, we still need real-world data for calibration. A world model can supply augmentation or co-training data, but completely replacing real data is not currently realistic. As datasets grow and models learn more physics from data, that gap may gradually narrow.

6. Connecting DreamZero and DreamDojo

Both projects perform world-model functions. As discussed, a world model predicts future states conditioned on actions. In that sense, both qualify, but they model actions at different levels.

DreamZero’s conditioning action is comparatively high-level: a text prompt. It takes a textual task instruction and outputs the corresponding world state and low-level actions that the robot can execute directly. It is thus both a world model and a policy. Given an instruction, it generates actions and predicts how the world evolves under that instruction. This lets you plan at the text-prompt level, searching and optimizing in high-level semantic space by comparing predicted futures under different instructions.

DreamDojo has a different motivation. Its input is also an action, but a low-level action—the kind of concrete control signal DreamZero outputs for the robot. Given that action, DreamDojo predicts future states. It resembles a low-level simulator.

From a robotics perspective, you can think of DreamDojo as simulation and DreamZero as a policy. If text itself counts as an action, however, DreamZero can also act as a simulator. Planning may involve high-level task decomposition, not just low-level action planning—for example, a VLM proposing “fold this first, then fold that.” You need a simulator at that semantic level too, and DreamZero provides text-conditioned simulation.

Overall, DreamZero is a text-conditioned world model that also acts as a policy and supports high-level planning. DreamDojo is a low-level action-conditioned world model, closer to a simulator, supporting low-level optimization and search. They model and plan at different levels.

Consider putting something in a refrigerator. At the high level, possible textual steps include opening the door, putting the item inside or closing the door. DreamZero can predict—or “imagine”—how the world state would change after each step. You compare the consequences and choose. At this level it serves both as a policy and as a high-level simulator for search in text-based planning.

DreamDojo instead takes low-level actions. Suppose the high-level step is already “open the refrigerator door.” There are many possible trajectories: the approach angle, the path to the handle, how force is applied, and the subsequent movement. These are low-level differences. Given one concrete action trajectory, DreamDojo predicts future states so you can assess its outcome, including whether it succeeds.

DreamZero can decide what to do next at the text level and predict how the world would change; DreamDojo predicts what would happen if you execute it in a particular way. DreamZero can directly serve as a policy or as a text-level simulator for high-level planning. DreamDojo predicts and searches over concrete control trajectories in low-level action space. This gradually turns semantic decisions into executable controls.

The pipeline just described is a test-time use: at inference, you use DreamZero and DreamDojo for search and optimization, progressively translating high-level decisions into low-level execution.

But a world model is useful beyond test-time planning. It can also serve as a training simulator. With two simulators—DreamZero and DreamDojo—you could use them for training, rather than only searching during inference.

DreamZero’s most basic role is a policy: given the current state and task condition, it outputs actions. Although it also predicts states, using only the action output gives you a conventional policy. That lets you connect DreamZero directly to DreamDojo.

Specifically, DreamZero outputs an action; DreamDojo takes it and predicts the next state; that state feeds back into DreamZero, which outputs the next action. The models form an alternating closed loop.

They could then run entirely within the models, without a real environment, forming a mechanism reminiscent of self-play. Beyond test-time search, they could constitute an internal simulation system for training, evaluation or policy improvement. This is a potential use of the combined system.

7. Perspectives on 3D data

Our collection method requires very little equipment and aligns with the format of existing large-scale human-video data. Essentially, we collect RGB video without extra 3D information.

From our perspective, many video-world-model projects try to align different forms of information with the video modality. Once you have video, you can extract much from it. Does 3D information necessarily need separate collection or explicit modeling? My current personal view is that it may not.

Explicitly introducing a 3D representation often turns an end-to-end pipeline into a multi-stage system. A video-based world model can learn from data much of the knowledge that graphics methods traditionally encode through explicit 3D modeling. For example, long-term temporal memory in video prediction can capture spatial structure and object relationships. Many such capabilities can be learned without explicit 3D representations.

At the data level, we use purely RGB egocentric video. A head-mounted camera records the first-person view and the person’s two-handed manipulation. The resulting data are a first-person RGB video sequence. In training, the model primarily uses the head-camera view.

We crowdsource this collection, recording people manipulating objects in real environments. It scales well because most existing internet video is also in RGB-video form.

We did not mix internet videos into this particular work: all our data were collected by us. We had already obtained a sufficiently large and diverse dataset, so there was no immediate need to add internet data. [Editorial note: the paper’s complete human-data mixture also includes the public EgoDex dataset and in-lab recordings, alongside the much larger, internally crowdsourced DreamDojo-HV dataset. This is distinct from mixing in general internet video.]

It contains approximately 44,000 hours of human manipulation video. I will not go into collection details such as the number of participants or the time span here. Broadly, the approach is efficient and uses simple, portable equipment, making collection much more efficient than methods requiring complex hardware.

Compared with projects that need specialized equipment, such as some robot-data collection setups, this method is faster and has an advantage in scaling the dataset.

8. How data collection can evolve

This is a crucial question. I feel data are the central issue for video-model approaches. The two models we discussed, DreamZero and DreamDojo, actually need different kinds of data.

DreamZero is fundamentally a policy, so it mainly needs expert demonstrations. Behavior should be directed toward performing a task. Some error recovery is acceptable, but the data should not contain large amounts of meaningless random movement. Overall, they should demonstrate the process of completing a task.

DreamZero currently relies mainly on robot data, although human data can readily be added in principle. Training uses two signals: action supervision and video supervision. In some cases, video supervision alone is enough. For example, robot data have three camera views. We simply stitch them into a grid for the model to predict, minimally disrupting the pretrained video model. Single-view human video can be incorporated similarly. Video-only supervision can still improve understanding of task execution. Adding video supervision opens up many forms of co-training.

DreamZero’s future development will mainly emphasize more diverse scenes and objects, helping map its general video-prediction and planning capability correctly into actions.

DreamDojo has different requirements. Its core function is to simulate the future consequences of actions: it is an action-conditioned world model. Such a model needs diverse actions as well as diverse objects and scenes.

To improve DreamDojo, we would ideally make the action distribution as broad and even—and actions as unpredictable—as possible. A simulator’s goal is not to learn an excellent policy; it is to faithfully simulate world changes under varied actions. It can therefore use successful demonstrations, failures and even random trajectories. Any data that help explain the consequences of different actions are valuable.

Training DreamDojo therefore requires increasing action diversity alongside object and scene diversity. In other words, the more uniform and irregular the action distribution is, the better the world model’s controllability. Better controllability means more accurate feedback when simulating different actions, which in turn helps optimize policies afterward. This is the relationship I am describing.

This raises a concern: would covering so many actions require a much larger dataset—perhaps 100 times larger? That concern is real, but the intended use matters. For policy optimization or test-time search, DreamDojo need not cover the entire action distribution perfectly. Accurately simulating actions the policy is likely to produce can already make it useful.

In practice, both models could start from the same dataset, such as one dominated by expert demonstrations. For DreamDojo, however, we would additionally prefer a more even action distribution, improving its modeling of different action outcomes.

9. Combining world models with reinforcement learning

DreamZero serves as a policy; DreamDojo is a world model and simulator. Together, they can form a model-based training framework. Given a state, DreamZero samples a series of actions. DreamDojo simulates their future consequences. You can assign rewards to trajectories based on the resulting state evolution, then use those rewards to optimize DreamZero toward better actions.

DreamDojo itself can also improve. Once DreamZero—or another sufficiently mature policy—has reached a suitable level, you can deploy it in a real environment and collect actual state–action–transition data. Updating DreamDojo with these data improves its modeling and simulation quality.

In other words, DreamDojo can help optimize DreamZero, while real-world data continually improve DreamDojo itself.

References

Shenyuan Gao’s profile: https://github.com/Little-Podi

DreamDojo: https://arxiv.org/abs/2602.06949

DreamZero: https://arxiv.org/abs/2602.15922

Smart Note · World Models · EP01