Back to World Model

World Models · EP03

World Models EP03: Robotic World Model

Chenhao Li on autoregressive dynamics models, uncertainty-aware policy optimization, and the goal of continuous online learning for real robots.

Zha Dami2026-04-2122 min readWeChat original
Contents

Humans improve at tasks through repeated attempts, with a strong capacity for self-improvement and online learning. We want robots to keep learning and improving in real environments too, rather than remaining fixed once training is complete.

When I decided to begin a series explaining world models, I noticed ETH's Robotic World Model project. Its central goal is continuous online learning on real robots. Whatever the base policy—basic velocity tracking or more general teleoperation—the method can serve as an additional layer. As long as improvement remains possible, or the environment, dynamics, or even reward changes, online data can keep improving the policy. Unlike our first two installments on 2D visual world models—World Models EP01: Shenyuan Gao on DreamZero & DreamDojo, and World Models EP02: Interactive World Simulator—this interview focuses more on dynamics.

The first author of Robotic World Model and its follow-up, Uncertainty-Aware Robotic World Model, is Chenhao Li. At the time of this interview, he is a third-year PhD student at ETH Zurich, advised by Marco Hutter and Andreas Krause. His research focuses on robotics and online learning, with a growing interest in world models and model-based reinforcement learning.

Original article outline

Contents

👇

1. What is a world model?

2. How should we understand a dynamics model?

3. Robotic World Model (RWM)

4. Uncertainty-Aware Robotic World Model

5. Core ideas and motivations

6. Comparison with simulator-based policy training

7. Reinforcement learning inside world models

8. How can we obtain a good model?

9. Which comes first: world models or embodied intelligence?

10. Why explore world models?

12. Is learning policies on real robots expensive?

13. Where does the greatest benefit come from?

14. The diversity of world-model training data

15. Comparison with existing methods

1. What is a world model?

Today, almost any model that predicts forward into the future can be called a world model. There are many forms, including language-conditioned models, video generation, and action-conditioned models.

In robotics, the term usually has a narrower meaning: a dynamics model that predicts the next state given the current state and action. Full state observability would be ideal, but many practical problems are partially observable. The representation may include images, and the full state may contain privileged information unavailable directly to the robot. In that setting, we still predict the next partially observable state or observation.

In my work, “world model” primarily means this kind of dynamics model. Many current projects instead emphasize photorealism: whether generated video looks realistic or obeys physical laws. For control, we care more about reliability—prediction accuracy and how far ahead the model can predict—which is more important for robot control.

2. How should we understand a dynamics model?

You said that, broadly, any model able to predict future states might be a world model. From your perspective, how should we understand dynamics models relative to video prediction, world models in simulation, 3D-state models, and other representations? Specifically, what are the states and actions, and how are they usually represented?

That is a good summary. We usually understand dynamics models from a control-theory perspective. For a robot dog or another robot, the state may include overall velocity, individual joint velocities and torques, robot position, and other information about the robot itself.

With perception, it can additionally include environmental information such as height maps or LiDAR scans, forming a larger state representation.

Actions differ across control tasks. Locomotion commonly controls joint torques or target joint positions because robots are multijoint systems. Manipulation often uses incremental commands, such as changes in joint positions or end-effector pose.

Given these definitions, a dynamics model predicts the next state: how the robot's state, including joint positions and velocities, changes at the next instant under the current state and action. In control theory, this describes how the system evolves over time—its dynamics.

3. Robotic World Model (RWM)

Let us start with the first paper, the basic Robotic World Model. Its central methodological contribution is a training approach we can call self-forcing or autoregressive training, essentially a loss design.

Robotic World Model demonstration accompanying the explanation of autoregressive imagined rollouts.

Usually, we predict the next state from the current state and action, compute a loss against the true next state, and backpropagate.

We extend that procedure. After predicting the next state, we feed that prediction and the following action back into the model to predict the state after that, and continue. The model rolls out autoregressively over multiple time steps.

Each predicted state is compared with its corresponding ground truth, and all those losses contribute to backpropagation. This self-forcing training substantially improves stability over long prediction horizons.

The method is simple and general enough to use in different tasks. But it does not itself solve distribution shift. During policy learning, such shifts can lead to hallucination, so that the policy updates inside an imagined world that is wrong. Addressing this requires accurate uncertainty estimation.

4. Uncertainty-Aware Robotic World Model

In the second paper, we introduce an uncertainty-aware approach to address differences in model reliability across regions.

Uncertainty-Aware Robotic World Model demonstration accompanying ensemble uncertainty and policy-learning discussion.

We want to quantify uncertainty during autoregressive prediction. The implementation is fairly direct: instead of predicting the next state with just one predictor, we use several predictive members or heads—for example, five—each producing a prediction, as an ensemble.

Disagreement between predictions estimates uncertainty. Large differences suggest that the model has seen less relevant data and is more uncertain; agreement suggests better coverage and reliability. [Editor's clarification: ensemble disagreement is an uncertainty estimate, not a guarantee that agreeing predictions are correct.]

This helps us identify where the world model is accurate and where it is unreliable.

That is crucial during policy learning, which mainly uses imagined data generated by the world model. In unreliable regions, the policy may be misled or even exploit model errors. Uncertainty lets us constrain policy learning and discourage entry into those regions, improving stability and reliability.

5. Core Ideas and Motivations

Our main application goal is continuous online learning on real robots. Whatever the base policy—basic velocity tracking or more general teleoperation—we can add our method as an upper layer. If there is room to improve, or if the environment, dynamics, or reward changes, online data can continue improving the policy.

Ideally, robots would keep learning in real environments instead of becoming fixed after training.

There are two main research motivations.

The first is online learning itself. Real data is very limited, so we introduce a world model to generate almost “unlimited” imagined data for policy training. That creates new problems, including model errors and hallucinations: how do we maintain stability and reliability when using imagined data?

The second addresses gaps in world-model research. Many projects focus on prediction quality or realism, but less on how to use a learned world model for control, or handle accumulated long-horizon errors and uncertainty. We want to first solve these questions in a relatively simple state-based setting.

Longer term, we ask whether a system can adapt to dramatic changes that were absent from training and impossible to cover in simulation. Examples might include moving from walking on flat ground to a trampoline, or, more radically, learning to swim. These are typical out-of-distribution settings.

The biggest challenge to such capabilities is safety: exploring in the real world without dangerous behavior during learning.

Even a model with generally good predictions can hallucinate implausible states or dynamics. Large datasets and strong models do not automatically close the gap between predicting and supporting control.

Our approach is therefore to systematically address these problems in a simpler, more controllable setting: handling uncertainty, stabilizing long-horizon prediction, and using limited data. Data will never be sufficient in practice. Reliable decisions with insufficient data are a fundamental, long-term problem.

6. Comparison with Simulator-Based Policy Training

We began building a robot world model because the established pipeline trains control policies in a physically accurate simulator, usually through reinforcement learning. If sim-to-real transfer works well and the simulator is sufficiently close to reality, those policies can transfer directly to hardware with good results.

Its major limitation is dependence on simulator quality. Simulation works very well for many rigid-body tasks, including robot dancing and obstacle traversal. It is a good source of training data, and policies learned there can transfer successfully.

In more complex settings, however, we often lack a good enough simulator. It may be slow or, more commonly, inaccurate, leaving policies effective only in simulation. Soft ground or fluid environments such as swimming are not simulated nearly as well as many rigid-body tasks. The sim-to-real route therefore does not work in those cases.

Our response is to ask: once a policy is deployed, why not use real-world data to improve it directly? That is particularly advantageous when simulation is poor or unavailable.

The key question is how to improve control effectively with online real data. If a robot initially walks unstably, how can its real failures progressively teach it to become more stable?

A central RL issue is low sample efficiency, particularly in model-free reinforcement learning. Real-robot data is limited, so we must improve policies from small amounts of it.

That is why we introduce a dynamics model. Even an imperfect simulator can pretrain a world model, which supplies imagined data for policy learning. We then fine-tune the world model with real data.

The world model can continually generate almost unlimited training data. This is the standard model-based RL paradigm: learn a world model, then learn a policy through it. Even if both are initially weak, the policy interacts with the environment and gathers new data, which improves the model and then the policy, forming an iterative loop.

The core goal is continuous online self-improvement. The critical issue is how to generate almost unlimited imagined data through a world model when real data is limited.

There are practical challenges too. Locomotion typically has a high control frequency, for example 50 Hz, while manipulation typically has a lower one, around 1 Hz. This makes planning difficult in locomotion, so we generally need to learn a policy directly. [Editorial clarification: these frequencies illustrate the interview’s settings, not universal rates or a fundamental prohibition on planning-based locomotion.]

Learning a policy inside a learned world model creates further problems. An RL policy may exploit model errors or make unreasonable decisions when the model hallucinates.

Our follow-up introduces uncertainty modeling so that the world model can recognize regions with insufficient data. That uncertainty constrains policy learning and improves stability and reliability.

So RWM learns a policy without a simulator or RL, using an imagined world model?

Yes, that understanding is very accurate. Our central idea is to replace reliance on a physics simulator with a learned world model for policy training. [Editor's clarification: the question's “without RL” is misleading. This framework still uses reinforcement learning, including PPO, inside the learned model; it replaces the simulator, not RL.]

Although simulators have become more accurate, inaccuracies remain. Real systems also change: components degrade, motors heat up, and dynamics shift. A fixed simulator cannot continually adapt to those changes.

We want a policy to adapt quickly and update continuously online.

In this classification, training a policy with RL in a conventional simulator is generally model-free because no dynamics model is explicitly learned. Our approach first learns a world model and then trains through it, making the overall method model-based.

7. Reinforcement Learning Inside World Models

In his JEPA blog writing, Yann LeCun said he thinks we should not rely too heavily on RL. Yet world models and dynamics models themselves developed out of RL, for example model-based RL. How do you view this contradiction? [Editorial clarification: this is the interviewer’s framing. Dynamics modeling also has a longer history in classical physics and control.]

If we have an accurate model, ideally we use it directly to make decisions: planning, optimization to obtain a policy, or reinforcement learning within the model. All use the model to solve control problems.

I understand “not using RL” here primarily as avoiding model-free RL: explicitly using dynamics and state transitions rather than bypassing them.

Model-free RL instead learns a mapping from states to actions without modeling dynamics. Without a good model, it is often the only practical approach.

I broadly agree with that perspective. If a sufficiently good model exists, using it should be the first choice. If not, and data efficiency matters, we can learn a model and use it for control or to assist policy learning.

People are questioning RL's limitations while newer methods inherit its modeling ideas. Our method fits that trajectory: drawing on environmental modeling in RL while trying to overcome problems of training stability, efficiency, and generalization.

8. How Can We Obtain a Good Model?

In manipulation, the most prominent pipeline is currently VLA, with world models and world action models emerging too. Fundamentally, insufficient data limits model capabilities.

I divide that shortage into two aspects.

First, insufficient breadth or diversity. Real-world combinations of robots and objects are too varied to cover every situation in training. Models therefore often struggle to generalize to new settings.

Second, insufficient modalities, or limited observability. Most data is visual, but images contain limited information. When a gripper twists a bottle cap, for example, contact and contact force are difficult to observe directly. Vision alone makes it hard to learn a complete, accurate state representation.

Learning high-quality representations is already difficult under those conditions. If even a VLA, a method with relatively lower data requirements, remains imperfect, more complex world action models or world models attempting to simulate the world are harder still.

My rough ordering of data demand is therefore VLA < world action model < world model. Moving along that sequence requires greater quantity and quality. A world model capable of “simulating everything” remains distant under present conditions. [Editor's clarification: this is the interviewee's assessment, not a universal ordering independent of task or architecture.]

There is an interesting issue here: people expect world models to solve problems, yet they themselves require substantial data and compute and may be costly.

Even in our nonvisual robotic world models, with lower-dimensional and more observable states, we see something striking. Given the same data, directly learning a policy often performs better than first learning a world model and then a policy through it.

To achieve the same policy performance, the world-model route often needs more data: learning a sufficiently good model and extracting a comparable policy requires more than direct policy learning.

That is already true in a low-dimensional, more observable setting. With image inputs and poorer observability, the gap could become larger.

9. Which Comes First: World Models or Embodied Intelligence?

I completely agree that world models may take longer to solve practical problems. Despite their growing popularity, we are still some distance from reliably predicting with them and effectively extracting usable control policies.

The direction is nevertheless well worth studying. A world model is not a completely new concept; it is fundamentally a dynamics model, a classic element of model-based methods. Adding modalities such as language and vision has renewed interest—something of a repackaging.

Its appeal lies in a very intuitive ultimate goal: if we could really build a model that simulates the entire world, it would very likely be the ultimate solution.

Current mainstream approaches, including VLAs, are showing bottlenecks. Their capabilities are constrained by available data. Stronger generalization requires explicitly incorporating dynamics and physics, rather than relying only on a data-driven mapping.

Intermediate approaches have emerged, such as world action models. A typical idea replaces a VLA's structure with something resembling video generation, implicitly incorporating physical consistency into prediction. Some work has shown that this does improve understanding of the physical world.

Longer term, I would prefer to guide a policy directly out of a world model with limited fine-tuning, rather than still relying on behavior cloning.

10. As a Team That Pioneered GPU-Based RL for Robot Control, Why Explore World Models?

There are two motivations. First, simulator-based training is bounded by simulator performance. Inaccurate simulation limits the policy. We want a new paradigm of continuous online learning on real robots, adapting after deployment to changes in the environment and robot itself. This could be the next stage of locomotion and robot control.

Second, world-model research mostly focuses on prediction accuracy and realism, with less attention to extracting usable control policies or handling inaccuracies and drift over long prediction horizons.

We start with a relatively simple state-based model and seek unified ways to address these practical problems, providing guidance for more complex future applications.

Those motivations are advancing continuous online learning and filling the methodological gap in using world models for control.

There is also a general question: how do we handle regions that the available data does not cover? It exists with both image- and state-based models, though images may make it harder.

That is why we begin with a simpler state-based setting. Solving the problem effectively even there is already difficult and valuable.

Longer term, the whole field faces the same question: how can a world model yield effective, reliable control policies?

Before working on world models, I focused on control. Early approaches mainly used imitation learning: give a robot one or two trajectories to imitate.

Then attention shifted toward generalization, extending limited demonstrations to broader behavior so that a robot could respond to whatever it saw. The prevalent view is that these methods work very well with enough reference data, and many projects already perform impressively.

We ask a different question: what happens when data is insufficient? In many practical settings, it always is.

Some work therefore introduces inductive biases or structural designs to improve policy generalization. That connects with world models: both ask how to generalize with limited data, or quantify uncertainty.

Uncertainty-Aware Robotic World Model (RWM-U) extends RWM by modeling not only what will happen, but explicitly how reliable the predictions are. [Editorial clarification: continuous online hardware learning is the broader research goal discussed here. The published RWM-U experiments train from fixed offline datasets, then deploy policies on real robots.]

It introduces ensemble-based uncertainty estimation within an autoregressive world model, quantifying epistemic uncertainty arising from limited or biased offline data.

12. Is Learning Policies on Real Robots Expensive?

The model is essentially a neural network deployed on the robot. Initially, both the world model and policy might be poor, with limited real-world performance. Would that make hardware learning very expensive?

That is an excellent question. We have not yet carried out the complete process of training from scratch on real robots. We generally pretrain a world model with simulation data.

For locomotion, the model needs broad coverage of state–action pairs and dynamics variations. Starting from zero in the real world risks unstable or uncontrolled behavior because the model lacks experience. Pretraining is therefore very important.

Existing simulators are currently the most reasonable source of pretraining data. Even if inaccurate, they provide a useful prior. We pretrain the model there, then fine-tune using online real-world data.

Our policy is learned through that model. In the experiment being discussed, we did not validate the complete online training loop directly on hardware; instead, we used a second simulator, such as Gazebo, as a stand-in for the real environment. [Editor's clarification: this distinguishes fully online hardware training from deployment. The published projects also demonstrate learned policies deployed on physical robots.]

Even with pretraining, safety is the main reason not to train directly on hardware. RL requires exploration, which can produce dangerous behavior. A walking robot may need experience of falling to recognize it as undesirable. If the world model has never seen falls, the policy may wrongly believe it always stays stable and attempt dangerous actions.

That motivated uncertainty modeling in the follow-up. When states such as falls are rare in the data, predictions there are unreliable. We penalize policies in high-uncertainty regions to discourage entering them.

Our eventual goal is to deploy the complete training and optimization loop on real robots. We are concentrating on safe exploration—also called safe reinforcement learning—so that exploration can proceed safely.

The central idea is to use available simulation data for substantial pretraining, reducing unnecessary real-world exploration and improving safety and efficiency.

So pretraining still relies on simulation. One detail: you used another simulator, such as Gazebo, to represent the real environment. Was Gazebo also the initial pretraining simulator, or did you use something else, such as MuJoCo?

We used Isaac Lab for pretraining. Fundamentally, world-model pretraining does not depend on which simulator supplies the data.

What matters is sufficient data. Simulation efficiently generates large quantities, so Gazebo, Isaac Lab, or other simulators can all support pretraining. Real data can be used too.

The key is to make full use of whatever data exists. Pretraining is usually substantially better than learning from scratch. It is not mandatory, but generally brings clear practical benefits.

13. Where Does the Greatest Benefit Come From?

Existing methods are mostly simulator-based. If the simulator is sufficiently accurate and deployment dynamics barely differ, they work very well.

Problems arise when simulation is inaccurate, sim-to-real fails, or real dynamics change. Walking on ice and grass, for example, involves clearly different dynamics.

Two common responses are domain randomization, which trains a robust policy for several environments, and adaptation, which infers the current environment and adjusts accordingly.

These approaches still largely depend on in-distribution coverage: simulation needs to include the relevant situations. If a situation was absent in training, handling it well becomes difficult.

We want a more general capability. An intelligent system should not have to enumerate every possible scenario in simulation. Ideally, a learned dynamics model generalizes better than a model-free method because its structure is closer to how the system actually evolves.

A model trained on ice and soil might therefore reasonably generalize to grass, even without seeing grass specifically.

Model-based methods are therefore generally considered to have better adaptation in out-of-distribution settings. This is one of the greatest advantages we see: adapting to unknown or changing environments.

14. Does a World Model Still Need Diverse Data to Predict Well?

In my experience, a world or dynamics model often needs more data than direct policy training. It must cover a broader distribution of states, scenarios, state–action pairs, and dynamics variations.

A highly accurate model that supports long rollouts typically requires more data than a policy.

Ideally, if we learned a completely accurate physical model and the real environment remained unchanged, a policy learned through it could work directly without additional treatment.

But many state transitions are stochastic rather than deterministic. A ball or block's rebound after hitting a wall, for example, can have randomness. A world model must explicitly account for it instead of assuming entirely deterministic transitions. This is distinct from domain randomization; it concerns uncertainty within the modeled system itself. [Editorial clarification: a rebound can be stochastic in a model because of hidden conditions or measurement uncertainty; the example does not establish that all contact dynamics are intrinsically random.]

We can still ask whether world models need randomization. If the model matched reality perfectly and the environment never changed, additional robustness measures or domain randomization at deployment would not be necessary.

In practice, many successful approaches rely on extensive simulation domain randomization: external forces, joint-parameter variations, and other disturbances train a robust policy for transfer.

We can incorporate similar ideas, but differently. Randomization is straightforward in traditional simulation. Applying imagined disturbances inside an autoregressive learned model can accumulate errors and cause long rollouts to diverge.

We propose controlled perturbations in learned dynamics: perturb intermediate states while keeping the rollout stable enough to avoid a complete predictive collapse.

The effect resembles domain randomization. The model accommodates state variations, supporting more robust policies and more stable real-world behavior.

In simulator-based methods, this is comparatively easy: push or pull the robot during training so the policy learns to handle disturbances.

Learned dynamics can be more fragile. Perturbing a state may make predictions rapidly diverge or even explode.

A central experiment shows that, with our training method, imagined trajectories remain stable and reasonable under certain intermediate-state perturbations. Policies trained on that data can still perform well and exhibit robustness.

15. Comparison with Existing Methods

Where does your approach have a higher potential ceiling—for example, generalization, long-horizon decisions, or stability—and what are its bottlenecks, such as representation, accumulated prediction errors, or scalability?

I can start with the distinction from existing approaches in locomotion.

Locomotion and manipulation differ greatly in control frequency. Manipulation usually runs at a lower frequency, allowing planning—for example, sampling candidate actions and selecting the best with the cross-entropy method. Locomotion, however, typically runs at a high frequency such as 50 Hz, leaving insufficient time for planning, so we must learn a policy directly. [Editorial clarification: this refers to the planning setup discussed in the interview; planning-based locomotion also exists.]

Learning a policy raises two issues. First, RL learns a value function and relies on bootstrapping, recursively estimating returns through the Bellman equation. This makes reliable predictions over sufficiently long horizons important.

There is a tension: longer return horizons generally reduce reliance on biased bootstrapped estimates but increase variance. Meanwhile, longer dynamics rollouts accumulate model errors and can diverge. Stabilizing long-horizon prediction is therefore central.

We proposed an autoregressive training method to stabilize long-horizon predictions. It is a general training method applicable to both locomotion and manipulation. The issue is more pronounced in locomotion because we cannot rely on planning and have to rely on a policy.

Second, learned policies can exploit model errors and enter inaccurate, high-uncertainty regions. We must constrain policies using uncertainty to avoid those unreliable regions.

These designs are fairly general, but they are especially necessary in locomotion.

Conventional sim-to-real performance is bounded by simulator quality. With good simulation, it works well. Rigid-body motion and flat-ground walking are already mature examples.

We focus on more complex, dynamic environments: adapting quickly to changes or situations simulation cannot cover. That is where our method has a higher potential ceiling.

Traditionally, a policy is trained in simulation, deployed, and stops learning. Deployment continually produces valuable real data, which we believe should keep improving the policy.

We aim for continuous, online, or lifelong learning after deployment.

For near-term rigid-body tasks on flat ground, the advantage may be modest. For more complex settings, such as soft robots without reliable simulators, world-model-based continuous learning from real data could be more valuable.

Another issue is general-purpose control across robots and states. Robot embodiments change: even the same robot may behave differently today and tomorrow, making a fixed offline policy less reliable over time. Some online learning is needed. Our approach could build on existing strengths in stable control while improving weaker generalization and adaptation to dynamic changes.

For environmental or dynamics changes, current methods broadly follow two approaches.

First, robustness: a sufficiently robust controller still completes tasks despite some changes. Domain randomization introduces training disturbances to make the policy stable under different conditions.

Second, adaptation: identify what changed and adjust. Some model-free methods infer system changes, such as motor degradation, from histories of states and actions, then adapt the policy. Teacher–student training and Rapid Motor Adaptation (RMA) are established examples.

When a robot moves from concrete to ice, the dynamics change. If training covered that transition, the policy can implicitly infer it from observations and act appropriately.

The model-based idea is similar, but more of the knowledge resides in the world model. If pretraining covered similar changes, the model can adapt nearly zero-shot and quickly generate a suitable policy.

For unseen changes, we hope a small amount of real data can rapidly update the world model to the new dynamics and then produce a new policy.

Robustness methods—domain randomization or perturbations in learned models—and adaptation are different but complementary. Longer term, adaptation is more important: robustness cannot cover every environment, and we ultimately need optimal performance in the particular one we face.

Model-free and model-based methods are similar in this respect. The former encode more of these capabilities in the policy, the latter in the world model.

Although our approach is model-based overall, its implementation connects directly with model-free methods.

We learn a world model, but still use a model-free RL algorithm such as PPO for policy training. We do not train the policy by backpropagating through the dynamics model; in that sense, we treat it as a zeroth-order optimization problem. [Editor's clarification: PPO still uses gradients to update policy parameters. “Zeroth-order” here refers to not differentiating through the world model, not to gradient-free policy optimization.]

The framework can therefore be understood as replacing the original simulator with a neural-network dynamics model while leaving much of the policy optimization pipeline unchanged.

References

Chenhao Li's homepage

https://breadli428.github.io/#publications

Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics

https://sites.google.com/view/roboticworldmodel/home

Uncertainty-Aware Robotic World Model

https://sites.google.com/view/uncertainty-aware-rwm/home

Shima's Product Recommendation

In this age of emerging AI capabilities, as the mother of a young child, I can watch human intelligence develop alongside AI's gradual maturation. Last year we encountered an interesting conversational product for children: Kabao. Children can carry it around, use many character cards, and create their own characters. The product has just launched, and I recommend it to Smart Note readers. It is still being iterated on, and feedback is welcome.

Smart Note · World Models · EP03