Diffusion Policy has emerged as a highly effective approach to imitation learning in recent years. After interviewing its author Cheng Chi last year for Diffusion Policy: Generating Robot Actions with Diffusion Models, I planned a series on work that builds on it. DP3 and iDP3: Improving Diffusion Policy’s Generalization with 3D Inputs showed how replacing 2D image inputs with 3D representations can substantially improve generalization. GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy explores what happens when we add 3D semantic information as well.
Diffusion policies are excellent at complex robot manipulation, but their lack of explicit geometric and semantic representations often limits generalization to unseen objects and layouts. GenDP introduces an imitation-learning framework that incorporates explicit spatial and semantic information through 3D semantic fields. It uses those semantics to enable category-level generalization, emphasizing the role of semantic understanding.
GenDP’s treatment of 3D semantics builds on D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement, which incorporates dynamic 3D semantic information into robot manipulation representations. In my interview with Yixuan Wang, a lead author of both projects, he also explained another route alongside imitation learning, reinforcement learning and conventional MPC: learned dynamics combined with model predictive control. This direction grew out of work by his advisor, Yunzhu Li. Wang’s projects bring together the control perspective of MIT professor Russ Tedrake’s group and the 3D perspective of Stanford professor Fei-Fei Li’s group, both part of his advisor’s research lineage. That is one of the appealing aspects of intellectual inheritance.
At the time of this January 2025 interview, Wang was a computer science PhD student at Columbia University, advised by Yunzhu Li. Before Columbia, he studied in the PhD program at the University of Illinois Urbana-Champaign and earned undergraduate degrees from the University of Michigan and Shanghai Jiao Tong University. I hope the conversation below helps explain both diffusion policies with 3D semantics and learned dynamics with MPC.
Original article outline
Contents
👇
1. About GenDP
The Core Idea
How Well Does GenDP Work?
Obtaining 3D Semantic Information with D3Fields
2. Learned Dynamics and MPC
The Frameworks Behind D3Fields and GenDP
The MPC Component
The Dynamics Component
How Should We Classify Learned Dynamics with MPC?
3. Spatial Intelligence
Dynamic 3D Semantics and Spatial Intelligence
What Does Spatial Intelligence Include?
3D Spatial Perception and General-Purpose Robots
1. About GenDP
The Core Idea
Before GenDP, we developed D3Fields. It can represent a 3D scene from a limited set of views—for example, just four RGB-D cameras—while also providing semantic information. It does not require training a dedicated scene-specific neural network and runs very quickly. That efficient representation led us to ask whether combining D3Fields with Diffusion Policy could improve the policy’s generalization.
A major limitation of Diffusion Policy is its relatively weak generalization. We hoped that combining 3D and semantic representations could produce a breakthrough. After exploring several approaches, we found that using 3D point clouds as inputs, together with semantic-information channels, could substantially improve generalization.
The motivation came largely from D3Fields’ advantages: fast execution and strong representations, making it well suited to imitation learning. Those properties fit the needs of Diffusion Policy and encouraged us to explore the combination.
How Well Does GenDP Work?
GenDP focuses on performance with unseen objects. Diffusion Policy already works well with familiar environments and objects; we want to improve its behavior on new instances. Our approach uses a 3D semantic representation. If different objects are similar in geometry and semantics under that representation, performance can remain stable, enabling better transfer. Two different cups, for example, may receive similar representations, allowing the policy to transfer smoothly between them.
Geometry alone is insufficient. Suppose we want to stand up an overturned cola bottle. Geometry might indicate a bottle-like shape without adequately distinguishing the opening from the base, leading the arm to grasp the wrong part and spill the liquid. With a knife, we want the robot to grasp the handle rather than the blade. A semantic representation can distinguish those parts and support safer manipulation.
Our work therefore addresses two core questions: how to generalize well to unseen instances, and how semantic information can enrich geometric representations so that a model adapts to new settings while acting more safely and intelligently.
Obtaining 3D Semantic Information with D3Fields
We use the representation from D3Fields. An important advantage is that its feature extraction does not require additional task-specific training. We can collect demonstrations through the usual Diffusion Policy workflow without separately retraining that representation. Semantic extraction relies on DINOv2, a large vision model that maps image pixels to high-dimensional feature vectors. The handle of one cup, for example, can receive features similar to those of other cup handles. These semantic features let us represent scenes and objects more effectively. GenDP’s action policy still needs to be trained on demonstrations; the training-free claim concerns extracting the 3D representation, not learning the robot policy.
Feature fields can find all similar targets in a scene from a selected point in an image. Pointing at the rim of a cup can highlight other cup rims; pointing at a fork’s handle can locate and mark other fork handles. Those feature correspondences help us represent semantics such as what constitutes a handle and use that information in a task.
Some simple annotation is needed, but only once for an object category. For a cup, for example, clicking four points can be enough. The process is straightforward. A single DINOv2 model then handles most of the remaining semantic extraction automatically.
D3Fields was motivated by an observation: foundation models such as Segment Anything and DINOv2 already perform extremely well on certain tasks, without extensive additional training. Their limitation is that they focus primarily on 2D images, whereas robotics requires understanding 3D environments.
Putting shoes on a shoe rack, for example, requires knowing which shelf to use in 3D space and acting in a changing environment. Everyday tasks are dynamic, whether pushing shoes aside or making fried rice. A good representation therefore needs three properties: 3D understanding of the scene’s layout; dynamic understanding that accommodates changes; and semantic understanding, such as distinguishing the toe from the heel to align shoes correctly.
We designed a framework that collects observations from multiple cameras, extracts information with foundation models and fuses the results into a 3D representation. This provides geometric, semantic and dynamic understanding. It is plug-and-play because it uses existing foundation models without additional representation training, and it works well even with limited viewpoints.
Compared with some work from the previous year, our method obtains a high-quality 3D representation without retraining and without a dense camera arrangement. That is the core idea of D3Fields. Once we have this efficient representation, we can use it for tasks such as object inspection and zero-shot object rearrangement without task-specific policy training. It simplifies the process and broadens the range of practical applications.
2. Learned Dynamics and MPC
The Core Frameworks of D3Fields and GenDP: Learned Dynamics with MPC and Imitation Learning
For the downstream control framework demonstrated with D3Fields, we train a dynamics model and plan using it. Compared with imitation learning and reinforcement learning, learned dynamics with MPC has distinctive advantages. A new task can sometimes be specified by changing the objective, with the robot solving an optimization problem to complete it. It is also more interpretable. Given the current observation and an action, the dynamics model predicts future observations and object states. We can try candidate actions, predict their consequences and select the best. Solving an optimization problem yields the desired action plan. This typically combines the dynamics model with a model predictive control framework.
The foundations of this approach can be traced to some of Yunzhu Li’s earlier work, particularly on deformable-object manipulation. GenDP, by contrast, uses a more conventional imitation-learning framework: collect demonstrations and directly train a policy, as in other imitation-learning methods. That is simple and efficient, but the two frameworks may suit different tasks.
I would highlight several common ideas. First, we use foundation models efficiently at the perception level, including DINOv2 and Segment Anything. Second, we do not need densely sampled viewpoints: four RGB-D cameras are enough. Third, the representation does not require additional training. A newer foundation model can replace an older one while the system continues to operate, making updates fast without retraining the representation. Replacing the visual backbone still requires a compatible feature interface; this is not a guarantee that arbitrary backbone weights can replace one another without adaptation.
The MPC Component
MPC comes in many forms. Traditional MPC often uses an explicit mathematical model, where the relationship between input x and output y can be written analytically. But that relationship can also be learned from data: collect examples and train a model to predict y from x. The learned model can then be used within MPC. That is our approach, and it is common in Yunzhu Li’s earlier work: learn the dynamics, then perform MPC with that learned model.
D3Fields follows that approach in its downstream demonstration. We first construct a representation, then show how it can support a task. For control, we chose learned dynamics combined with MPC.
The Dynamics Component
Dynamics can be simple or complex. In simple cases, explicit equations describe the behavior. A vehicle’s dynamics, for example, are relatively intuitive: pressing the accelerator increases speed, braking reduces it, and turning the steering wheel changes direction. A mathematical model can describe those relationships directly.
Many real-world interactions are not easy to express with explicit equations. Push a rope and its motion is highly nonlinear, making it difficult to write a complete model by hand. Similarly, kneading dough changes its shape in ways that are hard to predict with a simple formula.
For these complex systems, we collect data to train a model. We record repeated interactions and consecutive state changes—for example, the dough’s shape in the previous frame, the pinching action and its shape in the next frame. A model trained on those examples can predict how a given dough shape will change under a particular action. That is what we mean by learning dynamics.
Related ideas appear in work from Fei-Fei Li’s group, including RoboCraft and RoboCook: collect data, train a dynamics model, then use it for prediction and planning. The approach has considerable potential and remains relatively uncrowded in academia and industry, leaving substantial room for research.
It has meaningful advantages over imitation learning in some respects. With imitation learning, a changing training loss often does not directly indicate final policy performance. Before testing on the robot, you may not know whether the policy is good or bad. That uncertainty is frustrating. Many workflows repeatedly collect data, train, test, discover problems and collect more data. The cycle is inefficient, particularly for complex problems that may require large datasets without any clear indication of how much data is enough.
Dynamics-based methods are more intuitive and interpretable. You can compare predicted future states with actual outcomes to judge the model’s quality directly, allowing its accuracy to be assessed during training. Planning and control can also be more interpretable: you can understand why an action was selected, rather than receiving an opaque output from a black box.
A dynamics model can also be understood as a world model; world model is simply the more fashionable name. Its central idea is to understand reality by learning how the world changes and responds to interaction. I find this promising both for practical deployment and for research, and it is attracting attention from academia and industry. There is still much to investigate in using dynamics or world models for more efficient and interpretable planning and control.
How Should We Classify Learned Dynamics with MPC?
As used here, this approach is neither reinforcement learning nor imitation learning; it falls within a model-based control approach using MPC. In locomotion, common approaches include MPC and RL. We also use MPC for manipulation, but in a different way. For locomotion, relationships between applied forces, motion angles and velocities can often be modeled precisely with equations. Such analytical models suit conventional MPC well.
In manipulation, we focus more on interactions with the environment. Pinching dough or modeling clay changes its shape through dynamics that are difficult to express analytically. We collect data and train a learned model to capture those complex interactions, then use it within MPC. This illustrates how general MPC is: given a dynamics model, whether analytical or learned, it can support planning and control.
The central requirement for MPC is a model of how the environment or objects evolve. That model can be an analytical one written by hand or a learned one obtained from data. Once it is available, it can support control for either locomotion or manipulation.
3. Spatial Intelligence
Dynamic 3D Semantics and Spatial Intelligence
Spatial intelligence is a broad concept, and our work is certainly related to it. We focus strongly on representing and understanding scenes in 3D. While 2D representations have advantages—they are intuitive and may require less processing—3D representations currently show greater potential for our purposes.
An important value of 3D representations is that they help us reconstruct the world’s structure: shapes, coordinates and relationships between different information and the 3D environment. That is central to spatial intelligence—how to understand, model and represent an environment effectively. A robot needs a good environmental representation before it can perform complex tasks.
This has been a continuing research interest for me. Both D3Fields and GenDP concern high-quality 3D representations. In a recent project, for example, we modeled a scene graph to guide a robot’s exploration. It helps the robot understand the layout, open cabinets and explore where objects are. Those tasks all depend on a reliable 3D representation.
Our research is therefore closely tied to spatial intelligence, from representing an environment in three dimensions to enabling robots to understand and manipulate it through that representation. This remains one of our central directions.
What Does Spatial Intelligence Include?
My understanding centers on several aspects. First is semantics: knowing what an object is and what it can do. A cup handle can be grasped, and a cup can hold water; a knife can remove a potato’s skin. Spatial intelligence also includes location: where an object is now and where it will go if moved. Another aspect is dynamics. A knife passing over a potato separates the skin, while pushing an object moves it. Dynamic understanding predicts how the environment changes after an action.
I therefore think of spatial intelligence in terms of geometry, semantics and dynamics. Geometry concerns shapes and positions, including how to grasp an object appropriately and where it is in space. Semantics concerns objects and their functions: a door handle can be used to open a door, and clothing can be folded. Dynamics predicts the outcomes of interaction: cutting separates an object, and pushing changes its position.
These forms of understanding are hierarchical rather than confined to one level. High-level semantics might support buying a ticket at a station, boarding a train and getting off, requiring recognition of screens and tickets and understanding how to find the right platform. Lower-level semantics might identify a door handle and how to push the door. Dynamics is similarly hierarchical, from predicting action sequences at a high level to the immediate changes caused by a particular interaction.
3D Spatial Perception and General-Purpose Robots
I see two broad directions in robot intelligence today. One builds solutions through spatial intelligence and explicit understanding of 3D environments. The other, exemplified by PI, dispenses with explicit 3D representations and goes directly from images to actions, using an end-to-end foundation model to perform tasks.
In my view, end-to-end approaches have substantial potential and can produce striking demonstrations, but may struggle to solve genuinely complex problems in the near term—say, five to ten years. Better hardware, including GPUs, could improve their capabilities, but explicit understanding of 3D environments is still needed to support complex tasks today.
Suppose a robot enters an unfamiliar home or public space and is asked to fetch a bottle of water. It needs to know where water is likely to be, remember the kitchenette’s location, find it and navigate there. That requires spatial memory and 3D representations for localization and planning. Without such understanding, current observations alone are unlikely to support navigation and localization, much less dynamic tasks.
It also needs memory of changes during the task. When opening cabinets to search for an object, it must remember which cabinets it has opened, which will not open, and what each contains. An end-to-end model alone struggles with this kind of extended dynamic interaction and memory.
End-to-end models are useful in many settings, but assigning all computation to them is neither realistic nor necessary. SLAM—simultaneous localization and mapping—is already mature and can map and localize efficiently. Navigating with existing maps and models can be more efficient than relying entirely on one large end-to-end model.
Large models also have limits when handling extended tasks. GPT-style models can process long language contexts, but visual contexts are much larger. Even a 128-by-128 image contains more than 10,000 pixels. At ten frames per second, a video involves well over 100,000 pixels each second, reaching tens of millions over three minutes. This illustrates the large visual-information burden relative to the context-processing capacity available at the time of this interview. Pixel counts are not equivalent to language-model context-token counts: visual encoders can compress or tokenize images. The example illustrates the memory and computation challenge rather than proving a context-limit violation from raw pixels alone.
I therefore think a combination may be more practical in the near term: use 3D representations and spatial intelligence to understand and localize within the environment, while applying end-to-end models to particular task components. These are my current conjectures, of course, and further developments may change the picture.
References
D3Fields
https://robopil.github.io/d3fields/
GenDP
https://robopil.github.io/GenDP/
Yixuan Wang’s website
https://wangyixuan12.github.io/
Representative work by Yunzhu Li on learned dynamics and MPC
https://hshi74.github.io/robocook/