Back to World Model

World Models · EP06

World Models EP06: VPP

A conversation with Yucheng Hu and Yanjiang Guo on VPP, BagelVLA, and Ctrl-World: video generation, unified models, and robot decision-making.

Zha Dami2026-08-1725 min readWeChat original
Contents

In the World Models series, our first episode covered video-generation-based World Action Models (world models as action policies): World Models EP01: A Conversation with Shenyuan Gao—DreamZero & DreamDojo, a New Paradigm for World Models and Robot Intelligence. The second covered interactive world models based on 2D vision (world models as simulators): World Models EP02: Interactive World Simulator. The third explored dynamics models: World Models EP03: Robotic World Model. The fourth covered unified world models: World Models EP04: Motus. The fifth explored 3D world models: World Models EP05: A Conversation with Wenlong Huang—PointWorld and 3D World Models.

For episode six, we interviewed the two first authors of VPP, one of the earliest works in China to propose the WAM approach: Yucheng Hu and Yanjiang Guo. Both are PhD students advised by Professor Jianyu Chen at Tsinghua University's Institute for Interdisciplinary Information Sciences. VPP was released in 2024, when the VLA approach using VLMs as its backbone was beginning to gain traction and steadily attracting more attention. At that time, Hu and Guo had the foresight to explore a WAM approach with video generation as its backbone. Against the backdrop of claims in the first half of 2026 that “Beijing has 200 world-model startups,” their work feels particularly focused and forward-looking.

During the VPP interview, I also discussed the two researchers' subsequent work building on VPP: BagelVLA and Ctrl-World. To help readers place these approaches, the three works covered here can be compared with earlier episodes as follows:

VPP vs. World Models EP01: A Conversation with Shenyuan Gao—DreamZero & DreamDojo, a New Paradigm for World Models and Robot Intelligence

BagelVLA vs. World Models EP04: Motus

Ctrl-World vs. World Models EP02: Interactive World Simulator

Original article outline

Contents

👇

Part One: World Models

1. Why did you focus on world models so early?

2. Definitions and categories of world models

3. How should we understand the differences between approaches?

4. How might WAMs develop in the future?

Part Two: VPP

5. The core idea behind VPP

6. How VPP works

7. What VPP offers the field

Part Three: BagelVLA

8. The core idea behind BagelVLA

9. How should we understand MoT?

10. BagelVLA's training and data

How the VLM and world model fit together

Does adding VLM understanding improve performance?

11. How BagelVLA's data modules relate to its method

12. Why are you committed to this direction?

Part Four: Ctrl-World

13. Differences between Ctrl-World and VPP

14. How Ctrl-World works

15. Thoughts on world models today and in the future

Part Five: World Models vs. VLA

Part One: On World Models

1. Why did you focus on world models so early?

Yanjiang Guo: I vividly remember Sora's release in early 2024. Everyone was amazed because, for the first time, it modeled changes in the physical world quite well. I was already researching general-purpose VLA policies based on VLMs. After watching Sora's videos, I realized that this kind of video model could be very useful for robot decision-making.

If we start from first principles and compare two problems, which is easier: generating a video of a robot performing a manipulation task, or training a robot policy that can perform that task? Which is more mature? The former is clearly closer to what current technology can achieve. Physical manipulation is itself a harder problem, and large-scale pretrained video models already have strong generalization capabilities. If we cannot even solve the problem of generating videos of general-purpose robot manipulation, I find it hard to imagine solving the problem of general-purpose robot policies.

Robot datasets were actually very limited at the time. The general-purpose VLA policies we trained did not seem to generalize well: they basically learned only what we gave them. In new scenes or tasks, VLAs would often freeze or behave aimlessly, struggling to produce “reasonable” actions. More recently, improvements in data quality and scale have also led VLAs to show generalization to new tasks—for example, PI's recent π0.7 work. That has prompted some new thoughts for me as well.

But in that earlier context, video's generalization capabilities really drew me in. Given the current observation and an instruction, a video model could predict what would happen next. So my thinking was: if I cannot directly train a robot policy that generalizes, could I take a step back, first train a video model that generalizes, and then use it as a foundation for exploring policy generalization?

2. Definitions and categories of world models

Yucheng Hu: At its core, a world model predicts future states based on current and historical state observations.

Today, when people mention world models, they usually mean video prediction or video generation, especially video prediction. In a broader sense, however, methods such as VLAs and VLMs can also be viewed as world models: they too are essentially predicting states, even if they do not explicitly output images.

In practice, we usually predict images or videos together with actions, because images contain a great deal of useful information that helps us model the environment better.

Looking at current developments, world models can broadly be divided into three categories:

The first predicts images or videos together with actions, as in VPP, RT-1, and OpenVLA. These methods use current observations to predict future states or help generate actions. They can currently also be grouped under the increasingly popular world action model category. [This is the interviewee’s broad taxonomy. RT-1 and OpenVLA are action-prediction policies; they do not inherently predict future images or qualify as video world models in the narrower sense.]

The second models the world using 3D representations—for example, the “spatial intelligence” proposed by Fei-Fei Li's team, which understands and models the world by predicting or constructing 3D representations.

The third uses implicit representations, as in JEPA. These methods do not directly predict pixel-level images. Instead, they predict compressed features or latent representations of future states.

Overall, these three approaches differ in the kind of future state they predict: explicit images or videos, 3D structures, or implicit feature representations.

3. How should we understand the differences between approaches?

Yucheng Hu: An analogy may help explain the difference between our approach—World Action Models such as VPP—and Fei-Fei Li's. A 3D world model places more emphasis on rendering quality: future observations should reproduce the world as realistically as possible. Its core objective is to build a high-fidelity environmental model, making it more suitable for simulation, virtual reality, and similar applications.

Methods such as VPP do not place excessive emphasis on the precision of pixel-level prediction. We are more interested in using video modeling to support action generation. You can think of it as an “action decoder”: video modeling brings in large amounts of cross-embodiment data without action labels and acts as a bridge between perception and action within a VLA framework.

Of course, this approach has drawbacks. Pixel-level prediction contains rich information but also redundancy, especially backgrounds and other task-irrelevant content, which are modeled and predicted as well. One disadvantage is therefore the substantial computation needed to reconstruct redundant pixel information, particularly irrelevant backgrounds. On the other hand, its strength is that it preserves all the information. Despite the redundancy, genuinely useful pixel information is retained too.

By contrast, methods such as JEPA compress information. This is more efficient but may lose information. Our pixel-based approach can be viewed as preserving information in a “nearly lossless” way. In some respects, we are actually similar to JEPA: when generating actions, we do not necessarily use all the pixels directly.

The key difference lies in training. During training, we still perform complete reconstruction in pixel space, explicitly predicting future images or videos. JEPA instead mainly predicts in feature or latent space and does not directly generate pixels.

At inference time, we do not feed the complete predicted video into the action module as a condition. We use its intermediate representations. So you could say that we follow a pixel-space modeling approach during training, while at inference time we are closer to using implicit representations to assist action generation.

4. How might WAMs develop in the future?

Yucheng Hu: First, we think current video models such as Wan still have limitations, particularly relatively weak high-level semantic understanding. VLMs, by contrast, have stronger semantic understanding because they have been trained on large amounts of vision-language data.

The first direction is therefore to train a unified model coupling a vision-language model with a world model. This is one of our main current areas of exploration.

The second direction is data scale. VPP and related work still have insufficient post-training data. We need to incorporate more robot data to train stronger world models.

The third direction is to introduce more modalities. We may add touch and other modalities, both as conditioning inputs and as prediction targets. However, tactile hardware is not yet sufficiently mature, so this may be pursued gradually in subsequent work.

Yanjiang Guo: I think the biggest advantage of video-based world models is scalability. An important direction is therefore to scale to larger manipulation datasets. New capabilities are likely to emerge as data scale increases.

Another point is that we should learn pretrained video models specifically for robots. Most world-model research currently relies on video foundation models trained by the video research community. But video-generation researchers' training objectives differ from those in robotics. Video models pay more attention to aesthetic quality, which is different from the goals of robot manipulation. Our current practice is to fine-tune those video foundation models for robot tasks.

In the long run, a more desirable direction is for the robotics community to train its own foundation models: world models designed specifically for manipulation, with optimization objectives that directly serve manipulation tasks rather than generating “better-looking” videos.

Part Two: On VPP

5. The core idea behind VPP

Yanjiang Guo: Professor Yilun Du's UniPi work had the greatest influence on me at the time (see “Interview with Yilun Du: Embodied AI Research Based on EBMs and Video Generation”). It was far ahead of its time: the concept of a “video foundation model” did not even exist yet, and video models were generally quite weak. Yet UniPi had already proposed using video prediction for robot decision-making. As a pioneering work, it also had some limitations. The main one was open-loop control: because video generation was too slow, UniPi first predicted a complete video, then used an inverse dynamics model to extract actions for the entire trajectory and execute them. After reading it, I began thinking about how to turn this into a closed-loop policy, which led me to start VPP. [Control-loop clarification: this is the interviewee’s comparison of expensive video-plan generation with VPP’s fast feedback policy. UniPi’s authors also describe closed-loop execution of inferred actions, so “open-loop” here should not imply that every control layer lacked feedback.]

In implementation, one core objective was to replace the video model with an open-source video foundation model available at the time, and, most importantly, to make it substantially faster. Our straightforward analysis was that video models were slow mainly because they required multiple denoising steps, usually between 10 and 50.

We examined the video-prediction denoising process carefully. Large-scale dynamics were already captured in the first few steps; later steps mostly refined details. We therefore compressed the process as much as possible, using only one denoising step. With a single step, inference is actually not slow. The model is essentially just a transformer with a few billion parameters, with inference speed on the same order as a language model of comparable size.

In experiments, we found that although the details from one-step denoising were inaccurate, the result already contained substantial overall motion information—for example, which direction the robot should move and how it should approach the target object. Based on this observation, we used the video model's intermediate latent representations directly. Since a single denoising step can essentially be viewed as one forward encoding pass over the input, the video model can be treated as a powerful visual representation encoder.

6. How VPP works

Yanjiang Guo: VPP training can be divided into two stages. In the first, we continue training a general-purpose video model on a large amount of robot manipulation data. The overall procedure remains the same as the original video model's; the main change is the data distribution. We substantially increase robot manipulation data while removing content unrelated to robotics or with very different dynamics, such as natural scenery, landscapes, and faces. The goal is to focus the model on manipulation-related dynamics. We started this direction in mid-2024. There were far fewer video foundation models then than there are now. We used Stable Video Diffusion, a relatively early but widely used model with a transformer U-Net architecture.

The second stage trains an implicit inverse dynamics model, or implicit IDM. The core idea is that its input is not two clearly visible images, but a clear current observation together with a somewhat blurry representation of the predicted future. That uncertainty comes from using only one forward prediction step, so future details are incomplete. For this reason, action prediction here is not the same as a traditional IDM that strictly infers actions from the difference between two images. More precisely, it uses future information supplied by the video model to assist action inference, which is why we call this mechanism an implicit IDM.

7. What VPP offers the field

Yanjiang Guo: We made another important observation in VPP: video-prediction quality is strongly positively correlated with final decision-making performance. We emphasized this in our paper. Subsequent work has repeatedly confirmed it, and almost all world action model research mentions it.

Before this, most VLAs and related methods were essentially black boxes: it was hard to understand what the model was doing internally. Video-based VPP gives us a chance to “open the black box” and indirectly observe its decision-making process through predicted videos. That is something I like about it.

Part Three: BagelVLA

8. The core idea behind BagelVLA

Yucheng Hu: I think preserving as much of a VLM's—or an existing foundation model's—capabilities as possible is very valuable for robotics. After VPP, we gradually discovered some of its limitations. Around 2025, particularly toward the end of the year, a class of unified models combining understanding and generation began emerging in the multimodal field, and their capabilities were improving rapidly. We found that these models aligned very well with what robots need. In robotics, a world model not only predicts; it also essentially encodes video data. At the same time, a robot needs understanding capabilities.

We therefore wanted to build a unified model capable of both understanding and prediction, using understanding to improve prediction. With this in mind, we began exploring unified models as a new generation of VLA frameworks. In early 2025, we proposed UPVLA based on an earlier model combining generation and understanding. Although it was still at an early stage, it already had some useful capabilities.

In May 2025, ByteDance released Bagel, a more mature multimodal model combining generation and understanding. We saw its potential in multimodal understanding and editing, and developed BagelVLA on that foundation. The core advantage of this approach is that it combines the high-level semantic understanding of vision-language models with the temporal prediction capabilities of world models, improving understanding and decision-making within a unified framework. [The original says June; BAGEL’s official release was May 20, 2025.]

A unified model can couple different capability modules. In our framework, we do not simply combine prediction and understanding; we also couple text reasoning, future-state prediction, and action prediction in one model. It is an integrated framework with understanding, reasoning, and prediction capabilities. Through this unified framework, we hope to use larger-scale data to learn richer capabilities.

Another example of a unified model is covered in World Models EP04: Motus. An important difference between BagelVLA and Motus is that their vision-language model is completely frozen and used only as an encoder, effectively a feature extractor. We strongly couple the vision-language model and world model through MoT, producing a more integrated architecture.

Their vision-language model therefore does not participate in reasoning and lacks text-reasoning capabilities. Our vision-language model can perform text reasoning and supports both autoregressive and diffusion modes.

By comparison, they mainly use diffusion for video prediction. We model understanding, reasoning, and prediction together in a unified framework. This is a core architectural difference between the two approaches.

9. How should we understand MoT?

Yucheng Hu: Simply put, PI combines a VLM and an action expert through a Mixture of Transformers (MoT) architecture. The central idea is to incorporate action capabilities while preserving as much of the original VLM's capabilities as possible. In some earlier methods, such as GR00T and Motus, joint training can damage VLM capabilities unless the VLM is frozen or handled specially.

PI's MoT architecture keeps the different modules relatively independent, allowing it to learn actions while preserving the VLM's own capabilities. That is one of its main advantages.

Following this idea, we also use a similar Mixture of Transformers framework in Bagel or BagelVLA. Our goal is to build a unified model that preserves three capabilities: the vision-language model's understanding, the world model's future prediction, and the action expert's action generation.

10. BagelVLA's training and data

Yucheng Hu: We train in stages. The first stage does not introduce action-labeled data. We first train a foundation model combining generation and understanding, then learn the action expert in later stages.

BagelVLA Overview

The first stage mainly uses several types of data. First is VQA data, including vision-language data such as Objects365, to preserve VLM understanding. Second is planning data for robotics, primarily task decomposition and planning data that we process and construct ourselves. We also add video data, human-hand manipulation data, and some open robot datasets. We further process these data—for example, segmenting task sequences and labeling intermediate subtasks to understand task structure, while training prediction together with goal images. These are the main data used during pretraining.

VQA data are essentially a common data format for vision-language models: given an image and text as input, the model generates corresponding text. We are not the only ones to use VQA data in training. Many related works adopt similar strategies. π0.5 and ChatVLA, for example, both use VQA data to preserve VLM capabilities.

Our difference is that, in addition to VQA data, we incorporate robotics planning data—task decomposition and planning. Overall, we combine open, general-purpose VQA data with self-built VQA data related to robot tasks to train the unified model, improving both its understanding and its task-planning capabilities.

How the VLM and world model fit together

Yucheng Hu: In the first pretraining stage, we train the world-model component alongside VLM capabilities. The generation module needs to predict future observations, whether images or videos, so world-model data must be included in training.

This stage therefore trains two capabilities at once: understanding based on VQA data, through the VLM, and future prediction based on video data, through the world model. You can think of this as already training the “generation expert,” the module capable of predicting future states.

In this framework, the VLM first “understands” and “thinks about” the current task, then passes that information to the world model for future prediction.

There are two main types of information. The first is explicit, such as understanding and decomposing an instruction. The VLM breaks down the original instruction, effectively predicting “what to do next,” and these high-level semantics condition the world model. The second is implicit. Rather than existing as text, it resides in the VLM's intermediate representations, which contain rich high-level semantic and state information.

Under the MoT (Mixture of Transformers) architecture, the world model can obtain this information directly from the VLM's intermediate representations rather than through explicit text. In other words, the implicit representations are naturally passed through the attention process.

The VLM's input to the world model therefore consists of two parts: explicit task understanding, through instruction decomposition, and implicit high-level semantic representations.

Does adding VLM understanding improve performance?

Yucheng Hu: First, models combining generation and understanding are significantly stronger at image editing and prediction than models without an understanding module—a VLM. For complex-task prediction, it is difficult to make high-quality predictions without an intermediate “thinking process,” or text-level reasoning.

This is analogous to current language models: without intermediate thinking or reasoning steps, they often struggle to reach the correct answer in one step. Our goal is therefore to build a “text-image chain of thought,” using intermediate reasoning to improve prediction for complex, long-horizon tasks.

11. How BagelVLA's data modules relate to its method

Yucheng Hu: An important distinction in BagelVLA's image prediction is that we predict keyframes rather than complete videos.

BagelVLA architecture

Specifically, during data processing, we annotate videos to identify the keyframe corresponding to each subtask. Each subtask has an associated keyframe.

During training, the prediction target is these subtask keyframes rather than a continuous video sequence. The framework therefore contains two core capabilities: linguistic planning, which generates subtasks, and video forecasting, which predicts the corresponding keyframes. Generating actions on top of these two capabilities yields better overall performance.

The effectiveness of this design is reflected in our experimental results, including simulation and real-robot experiments. Intuitively, it is reasonable too: the most useful prediction is actually an incompletely formed image prediction, rather than a fully denoised image.

Introducing subtasks also breaks a long task into simpler ones, helping the world model make more accurate predictions. Because world-model data—especially robot data—are relatively limited, the model is usually better at predicting simple tasks than directly predicting from complex, long instructions.

The “subtask + keyframe” approach reduces prediction difficulty, allowing the model to complete complex tasks step by step.

See Figure 4 in our paper, where we show the predicted subtasks and keyframes.

For somewhat longer tasks, introducing intermediate “text + image” thinking substantially reduces learning difficulty. Compared with direct end-to-end prediction, adding subtasks—textual decomposition—and keyframes—key visual states—helps a VLA learn the structure of long-horizon tasks more easily and improves overall performance.

12. Why are you committed to this direction?

Yucheng Hu: One of our core ideas is to iterate on models around “maximizing data utilization.”

Our original reason for moving from VLA to world action models such as VPP was that video generation is self-supervised and can use large amounts of data without action labels. This is crucial, because robotics is fundamentally an engineering problem in which “there is never enough data.”

Beyond continually increasing data scale, it is even more important to make full use of existing data, including cross-embodiment data. This is why we moved toward world action models: they can exploit these unlabeled data and alleviate the data shortage in VLA.

But world models also have limitations. Even the best current video-generation models cannot produce the strong reasoning capabilities of LLMs.

We therefore proposed a unified framework to use video, robot, and vision-language data together, bringing different data sources into training to maximize utilization. This is the overall thinking behind our work.

Motus can also be viewed as a unified framework, but its limitation is that understanding does not truly help generation. More specifically, Motus's VLM is often frozen and serves mainly as a fixed feature extractor, passing the original, coarse-grained instructions to the generation model.

Architecturally, you can think of three “experts”: an understanding expert—the VLM—for text output and task understanding through subtasks; a generation expert—the world model—for keyframe prediction; and an action expert for action generation. All three exist simultaneously and share attention computation while retaining independent parameters. They interact within the same computation graph but have different responsibilities.

The input contains different token types, such as text tokens, visual tokens associated with the VLM, and VAE tokens associated with generation. These tokens participate together in attention computation to fuse information.

At the output, the VLM produces subtasks through linguistic task decomposition, while the world model predicts keyframes representing key visual states. Both condition the action expert, which generates the final actions. Overall, it can be understood as a “unified large model” that absorbs multimodal information and internally performs understanding, prediction, and decision-making.

Intuitively, this structure is more unified in form and better suited to scaling: it can continually absorb different sources of data and learn from them within a single model.

Part Four: Ctrl-World

13. Differences between Ctrl-World and VPP, or categories of video-based world models

Yanjiang Guo: We can first divide video-based world models into two major categories. The first is video world models as policies, which directly use a video model as a policy. Essentially, they use its predictive capabilities to generate optimal trajectories and help produce actions.

World models as policies can be further divided into several paradigms. The first is two-stage methods, such as UniPi, VPP, Mimic Video, and Large Video Planner. They first predict video, then extract actions from the predictions or their representations. Their advantage is that they preserve pretrained video models' generalization capabilities as much as possible. Their disadvantage is accumulated error: once video prediction goes wrong, subsequent actions usually go wrong too. The second jointly models and predicts actions and video, as in my earlier PAD work, and current methods such as DreamZero and Cosmos Policy. These approaches share model parameters. Their advantage is that actions may still be correct even when video predictions are wrong. Their drawback is that introducing the action modality disrupts the video model's original pretraining.

The second major category is video world models as simulators, which use a video model as an environment simulator. Given the current observation and a robot action, the world simulator predicts future observations, including both successful and unsuccessful outcomes. This is essentially the same as simulation: modeling “what happens after an action is executed,” corresponding to real-world dynamics.

Before Ctrl-World, I had already done VPP, so I wanted to try something different. Having worked on world models as policies, I wanted to see how far world models as simulators could go. I later completed this work with Lucy Shi in Chelsea Finn's group at Stanford.

My intuition was that world models as simulators must be harder because they have to cover a broader distribution. World models as policies essentially need to model relatively “optimal” trajectories, focusing on futures close to success. World models as simulators need to model every possible outcome, reproducing it as faithfully as possible regardless of the action and observation.

14. How Ctrl-World works

Yanjiang Guo: On dataset selection: world-model simulator quality depends heavily on the dataset. I ultimately chose DROID for two reasons. First, it is highly diverse. Chelsea and many other teams collected DROID with Franka arms over two years. There was no “data factory” system at the time. Across university laboratories in North America, PhD students, undergraduates, and others moved Franka robots around and collected demonstrations one by one, producing a very rich distribution. Second, the dataset contains a substantial amount of failure data, which is important for world models as simulators. Evaluation platforms such as RoboArena also use this dataset.

On model architecture: the video model needs adjustments during training. A world model as simulator introduces a new robot-action modality and must predict multiple views to match the best current VLA policies, allowing the two to interact. We train it to predict future observations conditioned on “observation + action.” The output format itself remains unchanged: future video or image sequences.

The result is that you can set up a new DROID platform, and the model can generalize to the new scene or camera viewpoint, although many failure cases remain.

Comparison with World Models EP02: Interactive World Simulator

These two works emphasize different things: accuracy and generalization. Interactive World Simulator mainly collected a very large amount of data for around five or six tasks and fitted those specific tasks thoroughly. It therefore achieves high accuracy on those six tasks and more accurate policy evaluation. Ctrl-World used DROID, which is much more diverse and covers a broader distribution. The resulting model generalizes reasonably well across the DROID platform, but it is not sufficiently precise on individual tasks.

This illustrates a major challenge for world-model simulators: their enormous data requirements. They need not only successful trajectories but also large amounts of failure trajectories to cover the entire state-action distribution.

From another perspective, however, world-model simulators can absorb and use failure data, which seems like a valuable advantage. VLAs and world models as policies can usually train only on optimal data; failure or suboptimal data are hard to use. World models as simulators provide a way to exploit them. [This describes common demonstration-only training setups. Failure or suboptimal data can also be used in some policy-learning methods; π0.7, for example, conditions on metadata to incorporate such data.]

15. Thoughts on world models today and in the future

Yanjiang Guo: The two current uses—world-model policies and world simulators—differ as follows. World-model policies can directly output actions for decision-making and execution. World simulators can serve as evaluation tools to assess policies, and can also support online planning: a world model can simulate different future trajectories, generate multiple candidate sequences, and choose the best one to execute. The two types of models can therefore have different roles in practical systems while also being integrated into the same decision-making framework. Overall, these directions are valuable and worth further exploration.

Beyond video-based models, there are other approaches:

One is based on 3D. Representative work first performs 3D reconstruction, then uses rendering to interact with and model the reconstructed 3D world. I think a core problem for 3D approaches is still insufficient data. Unlike 2D video, for which almost the entire internet offers usable data, 3D data are relatively scarce. Some work has also found that predicting with pure video and then performing 3D reconstruction can outperform models trained directly on 3D data. This largely benefits from video's greater data scale and scalability.

Another approach is LeCun's JEPA, or Joint Embedding Predictive Architecture. Its starting point is that pixel-level reconstruction is inefficient, so it focuses on learning high-level, intrinsic representations. The idea is elegant, but the supervision signal for these representations is relatively weak and their quality difficult to measure, unlike 2D or 3D reconstruction, which has explicit supervision targets.

There are also graph-, particle-, or dynamics-based modeling methods, such as Yunzhu Li's work, which can make good use of inductive biases and be useful when robot data are limited.

Overall, my personal view is that video is currently the most scalable approach and the best suited to large-scale data, which is why it has received relatively more attention recently.

Part Five: Thoughts on World Models and VLA

Yanjiang Guo: Although world models are popular now, they are not necessarily better than VLM-based VLAs. I began world-model research early, but I have also continued researching VLA.

Many people who have worked on VLA for a long time see unresolved problems and then encounter world models, which they may regard as a “better solution” or even a “savior.” My experience suggests that this is not necessarily the case. I have observed that world action models perform better on some tasks, while VLAs have advantages on others. My current impression is that world action models usually generalize slightly better to new tasks, mainly because they better preserve capabilities from video pretraining. But VLA methods often perform better on the most precise, delicate manipulation tasks. After reading the π0.7 paper, I think both might work once there is enough data, while each may have advantages when data are limited.

So I do not think world models are the “one correct” direction. World models and VLA are parallel families of modeling approaches. Before reaching sufficient data scale, it is hard to know the right learning objective for robotics—what form of supervision will be its “next-token prediction.” At the end of the debate between world models and VLA, we may come back to the data issue everyone is discussing. In my view, constructing the dataset with the greatest information entropy is the real barrier to training general-purpose policies today.

References

Yanjiang Guo's website:

https://robert-gyj.github.io/

Yucheng Hu's profile:

https://scholar.google.com/citations?user=WamdPtwAAAAJ&hl=zh-CN

VPP:

https://arxiv.org/abs/2412.14803

Ctrl-World:

https://ctrl-world.github.io/

BagelVLA:

https://cladernyjorn.github.io/BagelVLA.github.io/

Smart Note · World Models · EP06