Over the past six months, I have noticed more and more startups trying Diffusion Policy. Many people have also told me that my earlier interview, Diffusion Policy: Robot Action Generation with Diffusion Models, was helpful. I am delighted to make cutting-edge academic research accessible to a wider audience. A recent RSS 2024 paper, 3D Diffusion Policy, builds on Diffusion Policy by replacing 2D visual inputs with 3D information, substantially improving generalization.
I was fortunate to invite Yanjie Ze, a first author of DP3—3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations—to discuss the project. I also invited the project's adviser, Huazhe Xu, to add his thoughts on sources of 3D data and combining DP3 with large models.
Building on DP3, Yanjie made improvements and deployed the policy on a humanoid, proposing iDP3: Generalizable Humanoid Manipulation with 3D Diffusion Policies. Unlike earlier diffusion policies that needed data from multiple environments to generalize, iDP3 uses data from just one environment to generalize humanoid manipulation to many real-world settings. This article covers both DP3 and iDP3. The evaluated system controls the upper body on a height-adjustable cart; these results do not demonstrate autonomous legged locomotion.
At the time of this interview, Huazhe Xu is an assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences (IIIS). Previously, he was a postdoctoral researcher in Stanford's Vision and Learning Lab (SVL), working with Jiajun Wu. He earned his PhD at Berkeley Artificial Intelligence Research (BAIR), advised by Trevor Darrell. He is also a co-founder of the Chinese embodied AI startup Galaxea AI.
At the time of this interview, Yanjie Ze is a first-year computer science PhD student at Stanford. His research focuses on general-purpose robot intelligence. He earned his undergraduate degree in computer science at Shanghai Jiao Tong University, where he worked with Xiaolong Wang and Huazhe Xu.
Original article outline
Contents
👇
1. Introducing 3D Diffusion Policy
2. How do 2D and 3D encoders differ?
3. Data collection and training efficiency: 3D versus 2D
4. Scaling 3D and 2D data
5. How can internet data help Diffusion Policy?
6. Does DP3 improve object perception?
7. Why use a multi-fingered hand?
8. Simulation and the real world
9. What does iDP3 improve over DP3?
10. Does iDP3 use any simulation?
11. Data collection
12. Additional thoughts from Huazhe Xu
1. Introducing 3D Diffusion Policy
Diffusion Policy works very well, but its inputs are 2D images, which limits generalization when data is scarce. We wondered whether richer visual information, such as 3D, could improve the model's perception and further improve generalization.
We began with a simple thought experiment to test whether 3D data helps. We designed a task in which a robot with a small gripper reaches a target point in space. We supplied five demonstrations of reaching those points, then tested it on 1,000 different targets.
With 3D data, the robot's spatial perception improved significantly, allowing it to reach target points distributed uniformly throughout the space. With 2D data, successful reaches were concentrated on a plane rather than distributed throughout the volume. This demonstrated the advantage of 3D data for spatial understanding.
Based on this experiment, we explored how to design the 3D component. Initially, we tried complex networks and additional loss functions, but eventually found that a simplified model worked well enough. We kept only a simple encoder that processes 3D inputs efficiently. After removing everything unnecessary, the core design was sparse point-cloud input and an MLP with a few linear and nonlinear layers. This simple design performed well across a variety of tasks and generalization experiments.
We named the model 3D Diffusion Policy, and felt it had to live up to that name as a 3D improvement on Diffusion Policy. Complex models can easily overfit a single benchmark. We wanted broad applicability across tasks, as in the original Diffusion Policy. We therefore made the architecture as simple as possible, and it proved broadly applicable to many tasks.
2. How do 2D and 3D encoders differ?
First, we need to understand the difference between 2D and 3D data in visual processing. A 2D image can be viewed as a height-by-width matrix. A volumetric 3D representation instead consists of voxels—small cubes arranged in space, a little like a Rubik's Cube. How best to represent 3D remains an active research topic, with recent approaches including NeRF and Gaussian splatting. In this project, we use an efficient, classic representation: the point cloud. We represent the world as an unordered collection of points, each describing a position in space. The points can appear in any order.
People with a computer vision background may already be familiar with the difference between processing 2D and 3D data. Our focus is an unordered point set, so we cannot directly apply 2D image processing methods or conventional convolution operations. Many projects have studied unordered point sets, including classic methods such as PointNet. We simplify their core design choices and retain only the most essential modules. This makes 3D processing more efficient while preserving good performance, and gives us a model that is simpler to understand and use.
3. Data collection and training efficiency: 3D versus 2D
A high-quality 3D camera is indeed more expensive than a 2D camera, but collecting the same demonstrations takes the same time. If you need data of a robot grasping objects, spending a little more on a better camera can improve its quality. The hardware costs more, but 3D can reduce the amount of data required. A 2D setup might need one or two hundred demonstrations, whereas a 3D setup might need only a dozen to twenty or thirty. From that perspective, 3D collection is more efficient because it requires less data. These demonstration counts are illustrative comparisons from our experience, rather than one uniform count for every reported benchmark task.
Obtaining high-quality 3D information in the real world is notoriously difficult. We used a RealSense L515 in the paper because it provides relatively accurate point clouds. A more common, cheaper D435 gives rough depth information with lower accuracy, resulting in much poorer point clouds. If you run our algorithm with a D435, its real-world performance falls short of expectations. This comparison describes the camera configurations used in our experiments, rather than every possible setting of those sensors.
4. Scaling 3D and 2D data
For scaling through internet datasets, 2D has a clear advantage: images are abundant online, while 3D point clouds are relatively scarce. But that does not mean 3D cannot scale. Vision technology keeps advancing, and many projects use images for depth estimation and point-cloud generation. Some papers at CVPR 2024, for example, showed high-quality 3D generation from only one or two images. As the technology develops, 3D data has considerable potential to expand and eventually scale.
5. How can internet data help Diffusion Policy?
Using online data to support robot learning is an important research topic. A simple and common approach is self-supervised pretraining of the visual encoder. Broadly, Diffusion Policy has two components, as we discuss in our paper: visual perception and decision-making. The perception component can be pretrained on images or other visual data, allowing it to benefit from internet data. Decision-making, however, needs specific action information, which is not available from ordinary online data. That makes improving the decision-making component with internet data difficult.
6. Does DP3 improve object perception?
The real world is complex and constantly changing. Lighting varies, and objects change too. Each time you reshape a dumpling wrapper, for instance, its shape is different. Even a small position change may look like only a slight shift in an image, but the robot may be unable to locate the object precisely. It cannot determine the exact X, Y and Z coordinates from that small image translation alone.
Lighting changes matter as well. If you record demonstrations in the morning and test again in the afternoon, the results may be very different. Such real-world variation makes our approach more advantageous and adaptable. The central benefit is that 3D information gives a clearer perception of the world.
7. Why use a multi-fingered hand?
People have different views. Some think two or three fingers are enough. We chose multiple fingers because we believe a robot should be as flexible as a human hand and perform many different tasks. We therefore look for relatively difficult manipulations. Rolling something with two fingers, for example, is hard to do flexibly. A large object may also be impossible to grasp with only two fingers, making a multi-fingered hand necessary.
My choice of 3D and of multi-fingered manipulation reflects a similar approach to research. I think multi-fingered hands are better in the long run, so I want to study them more. Personally, I prefer pursuing a larger goal and trying to achieve it to simply repeating what is already known to work. Compared with 2D, 3D data has greater future potential. Collection is difficult today and hardware is expensive, but I see broader prospects. Multi-fingered manipulation may currently be less effective than two-fingered manipulation in some cases, but many technologies are already enabling highly dexterous behavior as the field advances.
8. Simulation and the real world
One lesson I took from this project is that the real world is very complex, whereas simulation is very simple. Many academic researchers prefer to test ideas quickly in simulation instead of designing and validating reward mechanisms in the real world. Others also start their work in simulation. There is nothing wrong with that: simulation is convenient and avoids practical challenges in robotics. But for a researcher focused more closely on robotics itself, simulation is overly simplified and cannot fully reflect real-world complexity. Algorithms that are truly useful for robots need extensive real-world validation. That is why we subsequently proposed iDP3, an algorithm tested extensively in reality.
One aspect of DP3 that I am dissatisfied with is its demanding requirement for point-cloud quality. As I mentioned, an L515 produces better point clouds than a D435, and the model performs much better accordingly. This is an important robotics challenge. In a perfect simulation, every condition is set up for you, making the problem relatively straightforward: you only need to think about planning. In most real-world settings, obtaining perception inputs of that quality is extremely difficult. We want similar performance regardless of the 3D capture device. Ideally, even a 2D device that derives 3D information algorithmically would give equally good results.
9. What does iDP3 improve over DP3?
We proposed an improved DP3 model, deployed it on a humanoid and completed manipulation tasks in the laboratory.
iDP3 deployed on a humanoid robot
The results show strong generalization in real-robot manipulation, including across viewpoints, objects and environments.
Viewpoint generalization
Object generalization
Environment generalization
iDP3 makes two main methodological improvements over DP3. The first is to replace world coordinates with a robot-centered frame. DP3 requires coordinate calibration: the camera must be calibrated in the world frame. In iDP3, we move from world coordinates to the camera's own coordinate frame. This makes the representation centered on the robot rather than the environment. Consequently, changes to the environment have much less impact on the system, which is a main reason it works across the different methods and settings we demonstrate.
An egocentric viewpoint
The second improvement addresses the increased difficulty of policy learning after this change. I drew inspiration from When Do We Not Need Larger Vision Models?: we may not need a larger vision model, and can instead improve performance by increasing the input resolution. Sometimes raising image resolution is more effective than making the model more complex. In our design, I therefore focused on improving data quality rather than expanding the visual encoder. In iDP3, scaling up the visual input means using a fuller egocentric point-cloud representation of the scene. The image-resolution example is an analogy, not a switch to a purely 2D policy.
10. Does iDP3 use any simulation?
We used some simulation while initially building the teleoperation system. Data collection, training and testing, however, relied entirely on real environments, with no further simulation. The process is very different from operating in simulation. In particular, real robot movement and interaction are influenced by many factors that are difficult to reproduce fully in a simulator. We therefore tested directly in actual environments to ensure that the data was authentic and reliable.
After building a model, we need to deploy it on the actual robot for testing. Our daily work is a continuous cycle of iteration. We may have gone through hundreds of cycles of collecting data, testing and adjusting based on feedback. Each round helps identify a problem: an algorithmic bug, a pipeline design issue, a problem with the data, or even an issue with the operator or hardware.
11. Data collection
A central difficulty in collecting data is that the robot cannot understand the environment as intuitively as a person. When you wear a Vision Pro, for example, you see the robot's camera feed rather than a view as sharp and immediate as human vision. We use a monocular camera, which gives a single image view. As you try to grasp an object ahead of the robot, it is hard to tell precisely whether the grasp succeeded. This demands considerable skill: the challenge involves both operating the robot and getting accurate visual feedback. It is therefore crucial for the entire teleoperation system to feel natural and fluid to the operator.
12. Additional thoughts from Huazhe Xu
3D versus 2D
3D information is very helpful, but the best form in which to supply it remains an open question. DP3 uses point clouds, while we have recently tried depth maps with good results as well. In many situations, there is no fundamental performance difference between the two. Overall, though, adding 3D information works better than using only 2D data.
In the long run, I think 3D information may be inferred from 2D data. Once we have enough 3D data, we can learn to reconstruct it from 2D. Generative models and related techniques may make this possible, and the potential is exciting.
Sources of 3D data
3D data is indeed scarce—many orders of magnitude smaller in volume than 2D data. LiDAR is an option and its cost is falling, although depth cameras may be more common. Collecting 3D at scale is still very difficult. That is why I think 2D-to-3D conversion may matter more in the long term. Vast amounts of 2D data already exist, including abundant online video. Rather than collecting 3D directly, we could generate it from existing 2D data.
Generating 3D from 2D will introduce some bias or error, but I think a policy can learn to account for it. The generated data is an estimate and contains inaccuracies, yet human visual perception also has errors. For manipulation, this need not be a major problem, provided the policy recognizes the bias and adjusts for it.
How can DP3 be combined with embodied foundation models?
When DP3 was introduced, it was not explicitly integrated with the embodied foundation model paradigm; it is fundamentally an imitation learning method. In the robot foundation model framework proposed by Sergey, the core idea is that directly learning by imitation from a small dataset is not especially data-efficient, whereas pretraining improves both data efficiency and success rates. Under this newer paradigm, a diffusion head may be particularly valuable at the post-training stage. For more on embodied foundation models, see RFM EP01: Pi and the Pi0 Embodied Foundation Model. Looking ahead, building and relying on powerful foundation models remains very necessary.
Understanding the diffusion head
In the Pi model, for example, the foundation model is a VLM based on PaliGemma, an open-source vision-language model. Its output is somewhat like GPT's: it does not directly produce actions. How, then, do we make it generate actions? We add a small module called an action expert after the PaliGemma model.
How does that action expert work? It does not simply insert its own parameters into PaliGemma to form one large model that outputs actions. Instead, it attends to PaliGemma through an attention mechanism. Put informally, the action expert looks at information from PaliGemma and uses it to generate specific actions. More precisely, attention exchanges token representations, not the numerical model weights themselves.
The action expert's training objective uses a diffusion-related method—more specifically, flow matching. There are also other variants of diffusion methods, such as DDIM and DDPM.
Why use a small action expert instead of integrating everything into one large model? I suspect there are several reasons. First, direct integration would make training very slow and greatly increase the parameter count. Second, such a large model might perform worse than a separate action expert. Third, a smaller module is lighter, easier to train and gives better results.
References
DP3
https://3d-diffusion-policy.github.io/
iDP3
https://humanoid-manipulation.github.io/
Yanjie Ze's website
Huazhe Xu's website
PaliGemma
https://arxiv.org/abs/2407.07726