Back to World Model

World Models · EP02

World Models EP02: Interactive World Simulator

Yixuan Wang on interactive visual world models, diverse play data, long-horizon stability, policy training, and reproducible evaluation.

Zha Dami2026-04-1722 min readWeChat original
Contents

In this series, we interview researchers working on world models, hoping to examine the concept calmly amid the noise. After World Models EP01: Shenyuan Gao on DreamZero & DreamDojo introduced 2D visual world models, our second interview looks at a distinctive recent project: Interactive World Simulator for Robot Policy Training and Evaluation, an action-conditioned visual world model supporting stable, long-horizon interaction.

Our guest is its first author, Yixuan Wang. At the time of this interview, he is a fifth-year PhD student advised by Yunzhu Li at Columbia University, and a research scientist at NVIDIA. His research asks how foundation models such as SAM and DINO can be used more effectively in robotics, and how robot tasks can teach world or predictive models. Longer term, he is interested in more general, foundational world models. We have previously interviewed both Yunzhu and Yixuan; see A Conversation with Yunzhu Li: Learned Dynamics for General Manipulation, and GenDP: Improving Diffusion Policy Generalization with 3D Dynamic Semantics. [Editorial clarification: “fifth-year” refers to his doctoral studies overall. His official biography records a transfer from UIUC to Columbia in September 2024, rather than five years spent at Columbia.]

Original article outline

Contents

👇

Part I · Interactive World Simulator

1. The central idea

2. The method

3. Data

Balancing high quality and “messy” data

Does training require large amounts of data?

4. Why call it a world simulator?

5. Questions about the project

How does it differ from simulators such as Genesis?

How many tasks did you test?

How does interaction compare with reality?

What can it generalize to?

How can others use it?

Can interactions produce unphysical behavior?

Future improvements

The research journey

Unexpected difficulties

Part II · World-Model Concepts and Categories

6. What substantive progress have world models made?

Understanding world models as policy evaluators

7. World models versus dynamics models

8. Main categories of world models

3D world models

2D world models

Where do World Labs and AMI fit?

9. Video-action world models

Comparison with existing video generators

10. Further questions about world models

Are world models more resource-intensive?

Does embodied AI need dedicated world models?

What does the proposed closed loop mean?

Are world models fundamentally autoregressive?

Part I · Interactive World Simulator

1. The Central Idea

There are three central points. First, physical realism: our physical and visual predictions must be realistic enough for immersive interaction. Second, speed: the model must support real-time interaction. Third, stability: in our experiments, people interacted continuously with the model for over ten minutes, roughly 9,000 prediction steps, with stable and consistently realistic results.

Project demonstration of real-time interaction with the generated world simulator.

Demonstration from the paper's project page

In some demos, a person uses a control device to interact with the video model. Everything on screen is generated, not footage from a real camera. Many first-time viewers struggle to tell the difference, suggesting that the output is very close to reality.

The Smart Note author trying the online simulator; the original recording includes network lag.

The Smart Note author trying the simulator on the project page, with some network lag

Our model understands real physics. In the first example, placing a rope into a clip, it accurately predicts the interaction between the rope and clip when the robot inserts it:

Rope-routing example: the model predicts interaction when the rope enters a clip.

When the rope merely swings outside the clip, the appearance is similar but the physical interaction is entirely different. The model can still predict what happens next accurately:

Comparison example: the rope swings outside the clip without being inserted.

A second example is mug grasping. The model accurately predicts the mug slipping out of the robot's grasp:

Mug-grasping example: the model predicts a mug slipping out of the gripper.

It can also predict motions such as the mug handle being pushed aside:

Mug-interaction example: the model predicts the handle being pushed and rotated.

The robot arm's movement follows the input commands closely too, supporting the reliability of the dynamics model. At live demos, many people first suspect a hidden camera and take some time to accept that the images are generated.

Once built, an important application is data generation. Without collecting additional real data, we can train a fairly robust policy using only data generated by the model.

A second application is policy evaluation. Previously, evaluating a trained policy in reality was cumbersome and difficult to reproduce. We evaluate policies inside our world model and analyze the results quantitatively. Policies that perform well in reality also perform well in the model; poor policies do poorly in both. The model can therefore reflect real-world policy performance to some extent.

2. The Method

Interactive World Simulator overview: six interaction tasks, action-conditioned video rollout, speed/realism comparison, data generation and policy evaluation.
Overview: mug grasping, rope routing/collection, T pushing, box packing, and pile sweeping supply interaction data. The simulator predicts video conditioned on actions at 15 FPS over ten minutes. The right panels compare generated/real data mixtures for policy training and correlated simulated/real task scores.

There are several key methodological elements.

The first is data, which is extremely important here. We collected our own robot interaction data rather than using Open X-Embodiment or other large existing datasets. Such datasets mainly target policy training, including Diffusion Policy, and their action distributions are often narrow and biased toward correct execution. [Editor's clarification: the paper also includes a simulated PushT setting; the statement here concerns the real-world robot interaction data.]

A world model needs broader action distributions: successful and failed, fast and slow, and different styles of interaction. That gives it a more complete understanding of dynamics, so we collected data ourselves.

We collect “play data” rather than conventional expert imitation data. Imitation learning usually values clean, smooth, successful demonstrations. We instead want messier, more random interaction to cover a broader distribution and improve generalization.

The second element is model design, with two main parts.

First, representation learning. We train an encoder to map images into a representation space, predict within that space, and decode back to images. This lets us impose constraints on the representation, helping prevent divergence during long predictions.

Second, faster prediction. We incorporate consistency models. Diffusion models require multiple iterative steps, whereas consistency models can predict in fewer steps, substantially increasing speed—a critical property for an interactive system.

Overall, prediction happens in representation space, with many details in the particular design.

An important lesson is that the central idea is neither complicated nor full of flashy new techniques. Many details must be tuned carefully, including architecture and parameters, and those details strongly affect the result. That resembles the experience of work such as Diffusion Policy.

3. Data

How do you define good-quality data? You also want it somewhat “messy,” with diverse behaviors. How do you balance quality against randomness or noise?

That is an excellent question. There is no unified, quantifiable definition of quality here; judgment is largely empirical.

A central principle is diversity. For rope routing into clips, the dataset should include successful insertions, failures, simply waving the rope around, and both slow and fast actions. It must cover the behavior distribution rather than only optimal solutions.

Collection also differs across tasks and must be designed accordingly, making one universal standard difficult.

Ultimately, data must be diverse enough. If it contains only successful actions, the model cannot learn failures or intermediate processes, and cannot predict more general situations. That is the most important point.

We do not simply collect whatever happens, however. The process is systematic.

For each task, I design a simple interface to guide each episode: where objects begin and what interaction to perform, for example placing a rope in a clip, placing it in the middle, or putting it through two clips.

This ensures reasonably even coverage across situations while capturing enough variety.

Data collection is crucial. When Yunzhu presents the work, some people report reproducing our results with our data but failing with their own. That illustrates how strongly data design affects the model.

Does training require large amounts of data?

Not particularly large amounts. We used about six hours per task. In a university, collecting enormous datasets is difficult, and I collected most of this myself.

Six hours for a task is manageable. Your point remains correct, though: the model needs sufficient interaction experience to learn good predictions.

4. Why Call It a World Simulator?

We spent a long time on the name and initially considered “interactive world model.”

Two reasons led us to “simulator.” First, people interpret “world model” very differently, so the term is ambiguous. We chose World Simulator instead.

Second, we really use the model as a simulator. Conventional tools such as MuJoCo or Isaac take actions and produce subsequent states or images. Our model has a similar form.

The uses are similar too: generating data, training policies, and evaluating policies. Our model is learned from real interaction, but afterward can be used as a simulator for data generation and policy testing.

Functionally, it is a learned simulator, not merely a predictor.

“Interactive World Simulator” also helps distinguish it from other projects.

5. Questions About the Project

How does it differ from simulators such as Genesis?

Those are closer to conventional simulators, albeit with different underlying physics engines or modeling approaches.

Our setup is purely 2D, closer in that sense to Dreamer or DreamDojo: input a current image and an action, then predict the next image.

How many tasks did you test?

We ultimately demonstrated six tasks, aiming to cover different settings: deformable objects such as ropes placed in clips, rigid interaction such as pushing a T-shaped object, and object piles, such as sweeping balls into a dustpan.

We also included articulated objects, such as opening and closing a box, to cover varied physical interactions.

How does interaction compare with reality?

I collected the interactions inside the video environment myself too. Personally, they feel very close to real interaction. There is an initial adjustment because you control through video rather than looking directly at a table.

Overall, the predictions are realistic enough to collect very high-quality data.

We also validated this systematically. We collected demonstrations inside the video model and trained imitation policies on them. For rope routing, a policy trained entirely on simulated demonstrations transfers directly to the real robot and performs very well. Even after we scramble the rope, it can put it back successfully. That indicates high generated-data quality.

Another experiment mixed generated and real demonstrations. Policy success remained approximately constant across the mixture, producing a fairly flat curve. From the downstream learning perspective, the two types were almost indistinguishable.

That is a strong result: generated data can approach real-data quality.

What can it generalize to?

Many previous world models had struggled to get even one task close to perfect. We therefore first wanted a single-task model good enough that interaction felt almost indistinguishable from reality, before discussing broader generalization.

We currently have two types of generalization: action variations such as different pushes, pulls, and robot-arm movements; and state variations within a task, such as different initial rope positions.

Stronger generalization—to entirely new environments, robots, or tasks—is not currently supported.

We do not see that as fundamental. Four to six hours can already train a good model for one task. With tens of thousands of hours, much more may become possible.

Our strategy is to make single-task performance nearly perfect first, then investigate large-scale generalization. That depends more on data, compute, and resources. The trajectory resembles early Diffusion Policy: strong single-task results followed by progressively broader generalization.

This work shows only the first small segment of that scaling trajectory. More data, people, and engineering are needed to validate further progress.

How can others use it?

We have fully released the code, and others have already reproduced our results.

There is also an online demo. You can open the webpage, connect to our cloud-hosted model, and try it without local deployment.

For deeper use, you can deploy locally or connect more advanced interaction devices for more complex interaction and data collection. The system is open for people to try and use.

We have deployed several tasks on the platform—about four—for visitors to experience.

The barrier to use feels low. At the time of our conversation, users in China could access it without additional network settings. I even sent it to my parents, who have no robotics background. After a brief explanation of connecting and controlling it, they could play with it.

We deliberately built the cloud service so people could experience it through a browser without complicated configuration.

Can interactions produce unphysical behavior?

Yes. If you give unreasonable actions that the model has never encountered, it struggles to predict sensible outcomes. It learns from data; it does not understand everything.

An autonomous-driving world model might predict sensibly on a normal road but struggle if you drive a car into a building, a situation absent from its data.

A major limitation is the lack of force feedback, because our current robot platform does not provide it. This can be addressed in future. We are considering force-related feedback to help the model understand contact and constraints better. With that, when unreasonable actions occur, the model will be able to recognize “this cannot be done,” rather than continuing to predict penetration or other unphysical behavior. [Editorial clarification: this is a proposed next step, not a demonstrated guarantee that force inputs eliminate all unphysical predictions.]

We are taking a first step centered on vision, and want to add modalities such as force feedback and touch for a more complete understanding of interaction.

Future improvements

There is substantial follow-up work in this first direction.

We can improve the model's stability and accuracy, and further explore downstream control and planning applications.

Data is important too. Our understanding of good data remains empirical. Currently, I largely use experience and world-model performance to judge it.

We need more systematic, rigorous study of what data is useful, ineffective, or even harmful.

There are worthwhile research questions spanning models, applications, and data.

The research journey

This project feels special to me. Even when we last spoke, we were considering a world model as close to perfect as possible. We had not anticipated how difficult it would be.

The project took roughly a year and a half. I began soon after arriving at Columbia and invested a great deal of time and effort.

We are satisfied with the final results and impact. Looking back, we underestimated the difficulty over a year earlier. Many people initially expected hallucinations to prevent a world model from truly working.

We found that, under reasonable actions, the model did not produce many severe hallucinations and could operate stably. That was interesting and an important finding.

Unexpected difficulties

A difficult part was finding the right recipes for both the model and the data at once. Neither is easy.

Data was especially hard. It is difficult to know in advance what is good, difficult to collect, and harder to analyze quantitatively. Models are comparatively easier because parameters and code allow systematic tuning.

Data is more experience-driven: you suspect a type may help and then test it.

We therefore continually iterated on both model and data.

Training also took a long time. A run could take days before we could see results and test an intuition, slowing iteration.

We tried different methods and combined techniques into a coherent system, encountering many pitfalls.

After resolving the details, we still needed extensive experiments to validate the whole system. It took a long time overall.

Did you think it might fail?

There was certainly risk. But I was approaching graduation, and my advisor encouraged a bolder project.

In retrospect, I think that strategy makes sense. Early in a PhD, pursue some lower-risk projects, like building an investment portfolio. I heard this idea during an MIT internship two or three years earlier, in conversations with Professors Leslie Kaelbling and Tomás Lozano-Pérez: research should be managed like a portfolio.

Include low-risk work that supports graduation, steady output, and employment, alongside high-risk work with potentially greater impact. Failure of a risky project should not undermine everything.

I see this as a higher-risk project in my portfolio. Fortunately, we made it work, which makes me happy.

Part II · World-Model Concepts and Categories

6. What Substantive Progress Have World Models Made?

World models are not new historically. Aerospace used similar models before modern robot learning, usually calling them dynamics models: given a state and action, predict a future state. Such models support control, determining actions that take an aircraft or rocket to a target.

Robotics similarly used manually modeled dynamics for control before robot learning became prominent, including model-predictive control. These can be seen as early world models. Learned approaches and multimodal development have renewed interest.

From a robot-learning perspective, I roughly divide current models into 3D models, such as point-cloud models predicting environmental changes in 3D, and 2D or video models, including video predictors and interactive world simulators. Both have seen recent work.

More broadly, some people call any future-predicting model a world model, including video generators such as Sora. In robotics, we generally also require action inputs: how does the world change when I take an action?

I see three main areas of progress in robotics applications.

First, policy evaluation. Robot policies are difficult to assess without standardized, reproducible environments. A world model lets us run a policy inside it and evaluate its performance.

Second, data generation. Slow models previously made interaction difficult. Faster, near-real-time predictions now make collecting training data through a world model feasible.

Third, control and planning: directly making decisions through the model, planning with it, or assisting control-policy learning.

There has been important progress in representation and multimodal modeling, but practical solutions to robotics problems remain early. Different world models suit different applications, and the field is still exploring.

Understanding world models as policy evaluators

After training a robot policy, how do we tell whether it is good? Vision has shared benchmarks for evaluating models on the same dataset; robotics makes that much harder.

Hardware and environments differ across labs. Even the same environment can produce different results at different times. Standardized, reproducible evaluation is difficult.

A world model can serve as an evaluation engine: test policies repeatedly in controlled, reproducible conditions, running many experiments without rebuilding each real scene.

Real evaluation is expensive, time-consuming, and needs manual intervention, such as resetting object positions. World-model evaluation improves efficiency and reproducibility, which explains growing interest in this use.

7. World Models Versus Dynamics Models

There is a substantial distinction. Aerospace often focuses on the system's own dynamics—an aircraft or rocket's motion—which can be modeled numerically or derived physically.

Robotics world models increasingly focus on interaction with the surroundings: pushing a mug or grasping an object.

Those interactions vary greatly across scenes and objects, making a single accurate analytic or numerical model difficult. Large amounts of data and learning are often needed.

8. What Are the Main Categories, and How Do Representations Affect Their Uses?

3D world models

First, particle-based models, including Yunzhu's early work and my first PhD project. They abstract the world into a small number of points: an object or rope becomes a handful of key points. Small state spaces allow lightweight, fast prediction.

That suits model-predictive control (MPC), which repeatedly evaluates many candidate actions. A fast model supports frequent, parallel computation. The disadvantage is losing detail.

Second, point-cloud world models such as PointWorld. Their representations are denser: thousands, hundreds of thousands, or even millions of points in a scene. They are more expressive but harder to model and predict. Better architectures and larger datasets have recently made them practical.

Their uses similarly include MPC and planning, with richer representations supporting more complex and varied tasks.

2D world models

We can classify 2D models by their inputs and outputs.

First, models producing both actions and images, sometimes called video-action models. Inputs may include images, actions, and text, while outputs include future images and actions. They resemble Diffusion Policy but additionally model future visual outcomes, closer to imagining before acting.

Video-action model: input S, A; output S, A. [Editor's clarification: this is schematic notation. Conditioning on past actions is different from conditioning predictions on a proposed future action sequence; implementations differ.]

Second, models producing images only, closer to video generation such as Sora. They can serve as high-level planners: predict how a scene evolves given the current state and task, then let a low-level controller execute.

Video-generation model: input S, text; output S.

Third, models taking the current image and action and predicting the next image. These are closer to classical world or dynamics models: explicitly predicting the consequences of an action, making them well suited to control.

Action-conditioned video-generation model: input S, A; output S.

The main distinctions are state representation and input–output design, which shape suitability for planning, control, generation, or simulation.

Where do World Labs and AMI fit?

I do not know their internal work very well. Based on public work and my understanding, Fei-Fei's projects fit the 3D category, with some differences from robotic world models.

They generate a scene and let you move a camera to see it from different viewpoints. This models changes in what is seen as viewpoint changes, using an underlying 3D representation.

The action is camera motion rather than robot manipulation. It does not change the environment by pushing or grasping objects. It models looking more than acting, so it is a 3D world model oriented toward vision and spatial understanding.

My understanding of the AMI-related approach is closer to 2D or latent-space world models. Rather than predicting pixels directly, it maps images into features, perhaps with something like DINO, and predicts future features.

Representation matters greatly. A more complex representation may contain more information but be harder to predict. Some representations suit vision tasks but not prediction; others suit prediction but retain less information.

The emphasis is on discovering representations appropriate for predicting the future.

Their starting point also seems different from robotics: they may focus on vision or representation learning rather than how to use predictions for control. These are related pursuits with different entry points.

9. Video-Action World Models

Video-action models jointly predict actions and visual changes. Their training resembles diffusion models or diffusion policies more than traditional, purely dynamics-based modeling.

Methods of this kind are not particularly suitable for direct MPC. One main reason is that their inputs are generally not conditioned on actions: the model does not explicitly take actions as input conditions.

Instead, actions are predicted as outputs, making conventional action-conditioned planning or control harder in that setup.

I see them as closer to diffusion policies than traditional model-based control, and somewhat further from directly using a model for control. [Editor's clarification: this concerns the designs discussed here; joint video-action prediction does not universally rule out action conditioning or planning.]

Yet these methods are also called world models?

An interesting development is that embodied AI researchers no longer assume a VLM backbone; a video-generation backbone is another option. That brings the 2D image-based world models we discussed into robot systems.

An important reason is that video models appear to contain considerable implicit world knowledge.

Their outputs can show some understanding of object interactions and changes of viewpoint in 3D space, suggesting implicit knowledge learned internally.

People are exploring how robots can exploit those capabilities: interaction knowledge and understanding of 3D structure. That is the high-level idea, while implementations differ substantially.

Comparison with existing video generators

Current closed-source video generators still produce physical hallucinations. That is understandable: they contain relatively little robot-specific training data, and primarily target cinematic video rather than real interaction and control.

Adapting these powerful models into directly usable robot models is a promising future direction.

Speed is another problem: their inference is relatively slow for interactive tasks. We therefore trained a smaller, more controllable model from scratch rather than using one directly.

Longer term, with more compute and data, fine-tuning existing models into robot-specific versions may become possible. I find that promising.

For the tasks in this project, however, the closed-source large models we compared performed worse than our small model.

10. Further Questions About World Models

Are world models more resource-intensive?

We can consider several angles.

Compared with imitation learning, world models do require more data. Imitation outputs actions, whereas a world model also predicts their consequences across images or states, imposing additional requirements.

I nevertheless think it is the right direction. Humans have world models too: psychological research suggests we predict consequences mentally. When grasping something, I expect it to move with my hand.

That can make a system more interpretable. We can understand an action's intention: the robot grasps because it predicts lifting the object, or pushes because it expects motion in a particular direction.

World models also have greater scaling potential in data. Imitation often requires high-quality expert demonstrations with smooth, successful actions, which are difficult to collect at scale.

World models tolerate—and may prefer—messier, diverse interaction, including failures and random exploration. That makes data easier to obtain and broadens behavior understanding, including how to avoid failure.

From that perspective, I see a more scalable route.

It remains difficult: more data, larger models, and sometimes real-time requirements are practical challenges.

Overall, I am confident in the direction, with many questions still to explore.

Does embodied AI need dedicated world models, or can general ones suffice?

I think dedicated world models are still necessary for embodied AI.

Even strong foundation-level models have limited training coverage. Some use particular robot platforms; some video models have no robot data at all. Applying one to your platform still requires training or adaptation on relevant data.

Methodologically, we are also exploring a closed loop from a world model to policy learning: how to obtain a genuinely effective control policy from the model. I do not yet see a fully convincing answer.

Imitation can often directly produce a good policy, so people ask whether a world model is essential. That remains unresolved.

My judgment is that world models could raise the ceiling. Imitation policies often fall short of very high success rates. If the world-model route works, I think it might produce more stable policies approaching 100% success. That loop is not yet fully realized.

I expect more work to connect it over the next one or two years.

Companies should therefore prepare for this direction rather than wait until its effectiveness is completely established, which could be too late.

Practically, pursue both paths: make good use of imitation learning while building world-model capabilities.

Many current applications—generation and evaluation—still support imitation learning. I see that as a transition. World models will very likely become a central component to policy learning. That is my personal judgment.

What does the proposed closed loop mean?

There is more than one possible route.

I currently see two main possibilities. First, an MPC-like approach: plan through a world model, predict future states, and infer the best current action.

Second, use the world model as an environment in which to train an RL policy, generating data and repeatedly optimizing the policy there.

I think our work makes both planning and RL routes more feasible.

Other approaches may emerge, but I believe at least one of those two has a good chance of working.

Are world models fundamentally autoregressive?

At least the kind we build is autoregressive: predicted outputs become inputs for subsequent predictions further into the future.

From that perspective, most world models have some autoregressive structure. At inference, we need to keep predicting forward; the future unfolds indefinitely, so we must roll it out step by step. [Editorial clarification: this describes iterative rollout; it does not require every world model to generate individual frames or tokens autoregressively. Some predict future chunks jointly.]

References

Yixuan Wang's homepage: https://www.yixuanwang.me/

Interactive World Simulator

https://www.yixuanwang.me/interactive_world_sim/

Shima's Product Recommendation

Investors and founders in embodied AI have probably been unusually busy lately: evaluating projects, meeting people, and traveling. After a hectic week, a weekend by the sea with family and children is especially welcome. I recommend a relaxing product that adults and children can enjoy together. In five minutes, you can begin playing favorite melodies and songs, whether at a family gathering or resting outdoors, making music while letting your mind wander. I was hooked as soon as I tried it, and recommend it to Smart Note readers too.

Smart Note · World Models · EP02