Back to Robot Learning & Foundation Models

Robot Learning & Foundation Models

Conversation with Yunzhu Li: Learning Dynamics for General-Purpose Manipulation

A full conversation on learned physical models, particle representations, graph networks, MPC, deformable objects and the value of modular robot systems.

Zha Dami2025-01-2017 min readWeChat original
Contents

Some say that solving general-purpose robotic manipulation would effectively mean solving AGI: general manipulation is the ultimate challenge for AI. Yet solving it would also bring a new transformation in productivity, large enough to change how we experience everyday life. However difficult, it is worth pursuing. With that in mind, Smart Note has consistently paid particular attention to research on manipulation. We have published a review, Bringing Robot Hands Closer to Human Hands: A Review of Dexterous Manipulation Research by Chinese Researchers; covered imitation learning in Diffusion Policy: Generating Robot Actions with Diffusion Models; discussed reinforcement learning on real hardware in Conversation with Jianlan Luo: Reinforcement Learning Can Work on Real Robots; and explored differentiable simulation in Episode 8, Conversation with Chuang Gan and Zhou Xian: How RoboGen Uses Generative Models and Differentiable Simulation to Synthesize Robot Demonstrations at Scale, Including a Review of Differentiable Physics Engines.

Today, under the influence of the prominent North American startup Physical Intelligence, embodied AI startups in China seem to have converged on end-to-end imitation learning. Is there too much consensus? Can policies learned inside an entirely black-box framework really be used in people's lives? For practical deployment, should we pay more attention to approaches that are relatively interpretable and modular—or, in the currently popular terminology, hierarchical—to solve general-purpose manipulation?

Today I want to introduce an approach that receives less attention in China than it deserves: solving robotic manipulation with learning-based dynamics models.

I suspect many people are as puzzled by 'learned dynamics + MPC' as I was when I first heard it. That uncertainty is useful: it is a source of curiosity and motivation to learn. To understand this approach better, I invited Professor Yunzhu Li, whom I regard as the leading Chinese researcher working on it, for an interview. I am also grateful to his PhD student Yixuan Wang, who first explained the idea to me. For an interview about Yixuan's two earlier projects, see GenDP: Improving Diffusion Policy Generalization with Dynamic 3D Semantics.

At the time of this January 2025 interview, Yunzhu Li was an assistant professor in Columbia University's Department of Computer Science. Before joining Columbia, he was an assistant professor of computer science at the University of Illinois Urbana-Champaign, or UIUC. He was also a postdoctoral researcher at the Stanford Vision and Learning Lab, or SVL, working with Fei-Fei Li and Jiajun Wu. He earned his PhD at MIT's Computer Science and Artificial Intelligence Laboratory, CSAIL, advised by Antonio Torralba and Russ Tedrake, and his bachelor's degree at Peking University.

I hope this article adds a promising approach to your understanding of general-purpose manipulation, as it has to mine: learned dynamics combined with model predictive control, or MPC.

Original article outline

Contents

↓

1. What are learning-based dynamics models?

How this compares with human learning

Connections with other fields

2. The core of learned dynamics: a predictive model

3. State representations for learning-based dynamics models

4. Modeling deformable objects with particles

Which architectures work best with particles?

5. How does this differ from a differentiable physics engine?

6. Combining learned dynamics with Transformers

7. Combining learned dynamics with reinforcement learning

8. Learned dynamics and VLA

9. Does the model apply to simulation or the real world?

10. How does it compare with current imitation learning approaches?

1. What are learning-based dynamics models?

The broad idea is quite intuitive. It is based on intuitive physics. People have a strong intuitive ability to predict their surroundings: we can anticipate how our actions will affect the environment. When you squeeze a piece of dough, for instance, it may deform. When you push a pile of granular material such as coffee beans, the beans may move or scatter.

People have a strong intuitive understanding of the physical world. We interact with our surroundings through multiple sensory modalities, including vision, touch and hearing, and observe how a scene changes. During those interactions, our brains process sensory data, build a mental model of the world, and predict how the environment will change if we take a particular action. This intuitive understanding does not rely on explicitly applying physics equations, and it works across different kinds of matter, including rigid bodies, deformable objects and fluids.

This predictive ability is highly generalizable. We can anticipate how our actions affect rigid objects, deformable objects, granular materials and flexible objects such as ropes. It is continually present in our minds. When we make decisions, we often use such predictions to plan our behavior.

If we want to push a phone from one point to another, for example, we can roughly predict how pushing will change its state on the table: how it will move or rotate. That predictive ability helps us plan the push so that the phone reaches the desired state.

You can understand this as predicting how the world will change from its current state when an action is taken. We can abstract that ability into a model, and there are several names for it. You might call it a simulator, an intuitive physics model or a world model. Its essence is to predict how the world's state changes with actions.

Developing predictive models that let robots understand the environmental consequences of their actions could therefore greatly help general-purpose manipulation. Like human intuition, those models would support effective plans based on predicted outcomes. Physics-based models can generalize well, but they usually require detailed state information about the environment, which may be difficult or impossible to obtain in complex manipulation tasks. Learning-based dynamics models offer an alternative: they learn predictive models directly from raw sensory observations and can capture factors that are hard to represent in analytical simulation.

Learned dynamics compared with human learning

Our aim is to learn directly from the real world. This closely resembles human learning: people build models through actual interactions with their surroundings. We can call this intuitive physics because it mirrors the way people gradually develop an understanding of physical phenomena, and the ability to predict them, through accumulated experience and interaction in everyday life.

Intuitive physics means an intuitive understanding of the physical world, generally grounded in experience and perception rather than explicit calculations or mathematical derivations. Knowing how to catch a falling object or predict a bouncing ball's trajectory are examples of knowledge acquired through interaction. This learning process is highly scalable: as our interactions increase, our predictive ability can improve.

Connections between learned dynamics and other fields

Computer vision: predictive models intersect with many fields, including computer vision. Vision is fundamentally a process of perception, mapping high-dimensional inputs such as visual or tactile signals into a representation of the environment's state. It might capture changes in an object's six-degree-of-freedom pose, or represent its state through particles and their changing relationships, as DPI-Net does. This lets perceptual models work with predictive models to describe environmental state.

Video demonstration from the original article

DPI-Net, ICLR 2019

Graphics: predictive models also connect with graphics, physics-based simulators and differentiable simulators. We generally train these learned models using neural networks, which are differentiable, so gradients can be used to optimize model performance.

Control and optimization: there are strong connections here too. A predictive model is a forward model: given the current state and an action, it predicts what will happen next. An interesting counterpart is an inverse model, which infers an action from the current and desired states. The distinction is that a forward model predicts a future state from state and action, whereas an inverse model infers the action needed to reach a desired state.

Through these connections, predictive models can help us understand and control complex environments, providing tools for better decisions and action planning.

2. The core of learned dynamics: a predictive model

In the policy-learning setting we are discussing, reinforcement learning learns a policy function. A goal-conditioned policy takes the current state and desired goal as inputs and produces an action. A predictive model works in the other direction: given the current state and an action, it predicts the future state.

Although reinforcement learning and predictive models both involve the current state, their objectives differ. The predictive model asks how an action will affect that state. Policy learning asks what action to take to reach a goal.

In this intuitive comparison, predictive modeling and policy learning are two sides of the same coin: one is forward-looking dynamics prediction, the other maps a situation and goal to an action. Editorial clarification: describing a policy as an 'inverse model' here is an analogy. Reinforcement learning generally optimizes expected reward, and not every RL policy is explicitly an inverse dynamics model or takes a goal state as input.

We learn the predictive model and use it to plan robot behavior. This is similar to the human intuition I mentioned earlier. Growing up, we have countless interactions with the environment and gradually learn to anticipate their effects. People possess this ability, and acquire it through interaction. We want robots to have a similar capability, learned through their own interactions with the world.

As I said, human predictive models work well across different materials: rigid objects, deformable objects, fluids, granular materials, ropes and clothing. Much of our earlier work has focused on collecting data through robot–environment interaction and learning forward predictive models from it. We have worked on ropes, cloth and even dough.

Video demonstration from the original article

RoboCook, CoRL 2023

RoboCook, for example, can be seen as a continuation of the particle-network approach exemplified by DPI-Net. Its tasks are more complex, but it still represents the environment with particles and learns a forward predictive model. We want to predict how different tools affect the shape of dough: how the dough will change when a particular operation is performed. At its heart, this is a predictive model.

The model can be used not only for forward prediction but also to solve inverse problems, including policy learning and trajectory optimization. Our series of earlier projects has focused on learning predictive models for different objects: ropes, clothing, dough and granular collections such as coffee beans, sugar cubes or diced carrots. We aim to use these environmental representations to predict changes under particular operations, and then use those predictions for robot action planning.

3. State representations and architecture choices

A predictive model's core task is to predict the next state from the current state and an input action. The crucial question is how to define that state. Is it an image or a vector in latent space? Is it something between the two, containing 3D information or semantics? The definition largely determines how difficult the model is to learn, how it should be designed, and what it can do when used for planning and control.

We have studied the strengths, weaknesses and capabilities of different state representations. For a rigid object—a solid object that does not deform—a six-degree-of-freedom pose is often appropriate, describing position and orientation. But that representation is insufficient for deformable objects such as ropes or cloth, which require other descriptions.

For ropes and cloth, we have found particles to be an effective representation. They can be obtained from 3D visual information to describe shape, and can support modeling of internal interactions: whether a rope can bend or stretch, for example, or how parts of a piece of cloth influence one another. Particle-based models can describe both an object's form and interactions between different objects, such as a rigid object contacting cloth. In that sense, particle representations are flexible and broadly applicable.

Video demonstration from the original article

GS-Dynamics, CoRL 2024

The representation of world state is therefore crucial to learning a predictive model. We can choose object-based descriptions, particles, pixels or latent vectors, each with its uses. Sparser particle-like representations, such as keypoints, are another option. Choosing an appropriate state representation is an important decision in building a learned predictive model.

Different state representations for learned dynamics and robot planning

Once the state representation is chosen, we define the architecture. Although these models are neural networks, their form can vary with the representation. Images have often been processed by convolutional neural networks, or CNNs. Particle sets call for architectures that flexibly handle sets and relationships, such as graph neural networks, or GNNs, Transformers or PointNet-like structures. For latent vectors without explicit object, spatial or temporal structure, multilayer perceptrons, or MLPs, are a common choice. These are typical design choices in the discussion, rather than restrictions on which architectures can be used.

In other words, the state representation directly influences the model choice and how the learned model supports planning, control and other decisions.

4. Modeling deformable objects with particles

Particle representations use discrete points to describe the surfaces or volumes of rigid and deformable objects, capturing their 3D geometry. This supports more precise interaction modeling, while explicit structural priors can improve sample efficiency and generalization.

Particles have long been used in physics-based simulation techniques, including position-based dynamics, or PBD, and the material point method, or MPM. Such methods underpin many simulators. Their approximations can introduce a sim-to-real gap, requiring complex system identification to address it. A natural alternative is to learn particle dynamics directly from real-world data, potentially reducing that gap substantially.

Which architectures work best with particles?

We have found GNNs to be among the most effective neural architectures for particle structures. Graphs are useful in many applications, especially for relationships between objects. They are a broadly applicable and intuitive representation: many things can be expressed as graphs, including relationships between people in a social network.

Video demonstration from the original article

DynRes, RSS 2023

In a GNN, each node can hold rich information about an object, including material, semantics and geometry. Edges describe interactions between nodes. In physics-based simulation, one often has to specify interaction rules manually—for example, the interaction between a knife and a steak or between two other objects—which can be complicated and laborious. With a graph representation, edges encode those relationships. They are not merely connections: they can carry many kinds of interaction information. This lets the network learn complex interactions without requiring every one to be manually specified.

5. How do learned dynamics differ from differentiable physics engines?

That is a good question. It gets at the distinction between a learned predictive model and a physics-based simulator; see Talking About Sim, Part 1: The Robot Simulator Ecosystem. Both fundamentally implement a predictive function: given the current state and action, they predict the next state. Differentiable physics simulators perform that same simulation task; see Episode 8, Conversation with Chuang Gan and Zhou Xian: How RoboGen Uses Generative Models and Differentiable Simulation to Synthesize Robot Demonstrations at Scale, Including a Review of Differentiable Physics Engines. A physics-based simulator need not learn the function from data, however; it can compute it from specified physical equations.

The difference is in how the model is constructed. A differentiable physics simulator typically uses physical equations. A learning-based predictive model uses a neural network in place of explicitly specifying those equations. This is the distinction being drawn here; hybrid models can also combine learned components and physical structure.

Each has advantages. Physics-based models can generalize well when the equations and assumptions are appropriate. They have relatively few adjustable parameters, such as stiffness or viscosity, with explicit physical meanings. A neural predictive model usually has many more parameters. My expectation is that, without sufficient data or structure, it can generalize less reliably than a well-specified physics-based model; that is a trade-off, rather than a universal ordering of their performance.

Learned approaches have several advantages. The most important is that they can learn a useful predictor without requiring all the environment's information explicitly. Consider fastening a button. To model that process in a differentiable simulator, you need detailed information about the garment: its material, its contact with the body and the local connections between button and fabric. Without an adequate specification of the relevant states and properties, a physics-based simulation of that interaction is difficult to construct.

People do not need to know every physical property or interaction equation. Through countless interactions, we develop an intuitive physics engine in our minds. Experience and observation let us make reasonable predictions about an object's response without knowing all its properties in detail. This ability accumulates through a large amount of data—our interactions with the world. It is what learning-based simulation, intuitive physics and world models seek to achieve.

6. Are Transformers suitable for learning dynamics?

So far, we have found GNNs very effective in relatively small-scale settings. We have not yet scaled them extensively, and increasing the data scale is a next step. In the constrained but complex tabletop settings we have studied, such as making dumplings, GNNs have worked better for us than Transformers.

Earlier research has noted similarities between Transformers and GNNs. One distinction relevant to our models is whether interaction information is explicitly encoded on edges. Our graph networks do this; a standard Transformer does not use those explicit edge features in the same way. That can give GNNs an advantage when such information is useful. Editorial clarification: this is a comparison of the architectures used here, not a claim that all Transformer variants lack relational or edge information; attention itself models relationships, and graph-aware variants can include edge features.

We plan to scale graph networks in two directions. First, can large-scale data produce a general learned simulator that models diverse materials and interactions? At present, we train a model on a dataset and it works well within that distribution, but another dataset or setting may require another model. We want to investigate whether one model can handle many different interactions. Second, we want to scale in space and time: moving from tabletop applications to room-scale environments, and exploring how GNNs can work with robotics methods such as SLAM for larger-scale modeling. These are future research directions with much still to explore. Structurally, graphs are easy to extend by adding nodes and edges, although learning and computation at larger scales still need to be addressed.

7. How can learned dynamics and reinforcement learning work together?

They are not in conflict; they can reinforce one another. The model I began with is a forward predictor. Imitation learning and reinforcement learning, in the policy-learning sense, instead learn a mapping from the current situation to an action. They directly choose actions rather than directly predicting what the world will look like afterward. Calling that an inverse function is the same intuitive analogy used earlier.

These are complementary sides of the same coin. Our direction can be described as model learning, with imitation learning and reinforcement learning built on top of it. Model-based RL, for example, learns a model and then uses it to learn a policy. Many RL systems learn policies inside physics simulators; a learned predictive simulator can take the place of that simulator. We have used the broader idea of learning a policy through a learned model in our robot dumpling-making work. Editorial clarification: RoboCook's published policy-learning procedure is self-supervised; using learned dynamics to train a policy does not by itself make that particular procedure reinforcement learning.

An analogous example appears in autonomous driving. A predictive model can anticipate how other vehicles' trajectories may change in response to your own vehicle's behavior. Those predictions can be used to plan a trajectory. This is the idea behind model predictive control, or MPC: using a predictive model to optimize actions or trajectories, typically repeatedly as new state information arrives.

8. Learned dynamics and VLA

There is also a straightforward connection with imitation learning. Combine the learned model with MPC, optimize a trajectory, and use that trajectory as a supervision target for an imitation policy. The model can therefore be both a simulator for reinforcement learning and a data generator for imitation learning. It can be the simulator for reinforcement learning. It can also be the data generator for imitation learning.

Many current vision-language-action, or VLA, models are large imitation learning models: they receive vision and language inputs and produce actions. An efficient world model can act as a data generator to help train them. Editorial correction: the Chinese original expands VLA as a vision-language pretrained model; in this context the A stands for action.

Besides generating data, a learned model can be an evaluator. When training a large policy, you need to assess its performance. Evaluation in the real world can be expensive and noisy. The same test in the morning and evening, for example, may differ because of lighting. We can use the learned model as a proxy for reality. It cannot completely replace the real world, but it can be more efficient and less costly. Testing in the model can filter out poor policies in advance, substantially reducing the number of real-world tests required. In short, the model can provide both training data and an evaluation tool for a large imitation-trained VLA.

9. Does the model apply to simulation or the real world?

Its training data can come from simulation or the real world. Ideally, if you want it to match reality, you learn the model directly in the real world. Another option is to learn it in simulation first, then adjust or fine-tune it using real-world data. Although the generated data is simulated, it may then resemble real-world data closely, because the generating model has itself been learned and refined using real observations.

In this way, simulated data can better reflect the characteristics of the real world, improving practical performance.

10. How does learned dynamics + MPC compare with end-to-end imitation learning?

With a learned model in a modular system, it can be easier to understand what went wrong when a real robot fails. Was perception inaccurate, producing an incorrect estimate of world state? Was the object's geometry reconstructed poorly? Was the predictive model inaccurate? Or did optimization fail, with MPC unable to find an effective trajectory? Separating these modules makes it easier to locate the source of the problem.

This modular design has practical advantages, especially for large-scale deployment in complex environments. Safety and debuggability become particularly important. You can better identify the potential failures of perception, prediction and planning, and work to fix them. Modular systems in autonomous driving likewise help with testing and debugging under real-world challenges.

By contrast, when an end-to-end imitation policy makes a mistake, tracing its source can be difficult. Imitation learning relies heavily on data. Without useful intermediate models and a modular structure, system failures are harder to analyze and repair effectively.

References

Yunzhu Li's homepage

https://yunzhuli.github.io/

RoboCook

https://hshi74.github.io/robocook/

DPI-Net

http://dpi.csail.mit.edu/

DynRes

https://robopil.github.io/dyn-res-pile-manip/

GS-Dynamics

https://gs-dynamics.github.io/

Smart Note · Robot Learning & Foundation Models