Back to Robot Learning & Foundation Models

Robot Learning & Foundation Models

Diffusion Policy: Generating Robot Actions with Diffusion Models

An interview with Cheng Chi explaining Diffusion Policy, multimodal action distributions, coherent action sequences and the data needed for general-purpose robots.

Zha Dami2023-12-0414 min readWeChat original
Contents

A few days ago, I listened to the CoRL debate Can Large Models Enable General-Purpose Robots? Russ Tedrake, someone strongly associated with model-based methods, was unexpectedly sitting on the learning side and repeatedly mentioning Diffusion Policy. Curious about what it was, I looked up the paper. The robots’ fluid movements, coherent trajectories and ability to handle objects made of different materials immediately caught my attention. My reaction was simple: this is state of the art.

What is Diffusion Policy? First, diffusion is a generative method behind many achievements in image generation, including familiar systems such as Stable Diffusion and Midjourney. A policy, meanwhile, is the component of a machine-learning system that drives a robot. It takes perceptual information—camera images or video, for example, and robot joint positions—as input and outputs the actions the robot should execute. Put simply, Diffusion Policy uses diffusion to generate robot actions.

That description alone may not sound especially remarkable. I do not want to exaggerate its future impact, but I strongly believe this could be an important contribution on the road to general-purpose robots. This article aims to explain the paper differently from a conventional academic introduction or a promotional media story: it is a set of notes that readers without an academic background can understand. To make that possible, I was fortunate to interview Cheng Chi, first author of Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. I am grateful for his clear and vivid explanations.

Set aside about 20 minutes and stay with me to the end as we explore Diffusion Policy and the possibilities it may open up. Enjoy!

Original article outline

Contents

👇

What Diffusion Policy Can Do

The Team Behind Diffusion Policy

What Is Diffusion Policy?

Why Use Diffusion Policy?

Other Questions from the Interview

Some Reflections

1. What Diffusion Policy Can Do

In Bringing Robot Hands Closer to Human Hands: A Review of Dexterous Manipulation Research, we discussed a view that action execution is the weakest link in robot manipulation. Diffusion Policy has demonstrated impressive results and considerable potential in addressing that problem. As someone outside academia, I particularly like watching research videos: they make the results immediately tangible. Before discussing the method, let us look at what it can do.

Spreading pizza sauce: at normal playback speed, the robot spreads the sauce without hesitation or jerky motion. Even when someone deliberately moves the pizza underneath it, the spreading action remains fluid.

Spreading pizza sauce with Diffusion Policy. Watch the video in the original WeChat article.

Flipping a mug: even when the mug is tossed into a different position, the robot can turn it over precisely to the desired position and orientation.

Flipping a mug with Diffusion Policy. Watch the video in the original WeChat article.

Push-T: the robot precisely pushes a T-shaped block into the target outline. Compared with Diffusion Policy, other methods produce less coherent trajectories and can easily get stuck near the target.

Push-T with Diffusion Policy. Watch the video in the original WeChat article.

Push-T action trajectories comparing Diffusion Policy, LSTM-GMM, BET and IBC.
Video demonstration from the original article

In a video, MIT professor and Toyota Research Institute robotics leader Russ Tedrake describes the remarkable results achieved with Diffusion Policy. He says that every morning he looks forward to discovering new skills the robots can perform.

Diffusion Policy at TRI, presented by Russ Tedrake. Watch the video in the original WeChat article.

According to the paper’s abstract, the authors evaluated Diffusion Policy on 12 tasks across four robot manipulation benchmarks. It outperformed existing state-of-the-art robot learning methods, with an average improvement of 46.9%. Toyota Research Institute, or TRI, which supported the research, also praised the approach. It described diffusion-based generative AI as a breakthrough for teaching robots new dexterous skills quickly and confidently, significantly increasing their utility and moving toward large behavior models, or LBMs, analogous to the large language models that had recently transformed conversational AI.

At the time of the announcement, TRI had used the method to teach robots more than 60 challenging dexterous skills, including pouring liquids, using tools and manipulating deformable objects. These results required no new lines of code: the only change was the data given to the robot. Building on that success, TRI set the ambitious goal of teaching hundreds of additional skills by the end of 2023 and 1,000 by the end of 2024.

2. The Team Behind Diffusion Policy

Diffusion Policy was a collaboration between Shuran Song’s group at Columbia University and Toyota Research Institute, with MIT professor Russ Tedrake. At the time of this interview, first author Cheng Chi was a computer science PhD student at Columbia, studying robot manipulation and perception under Song’s supervision.

Chi’s research interests center on interactions between computers and the physical world. He graduated from the University of Michigan in 2019. Besides being first author of Diffusion Policy, he was first author of the RSS 2022 Best Paper, Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. He also contributed to two intriguing manipulation projects: RoboNinja, which cuts avocados, and DextAIRity, which uses airflow to fold clothing. Both appear in Bringing Robot Hands Closer to Human Hands: A Review of Dexterous Manipulation Research. I previously introduced Shuran Song in my Map of Chinese Researchers in AI and Robotics. Other authors of the Diffusion Policy paper include Zhenjia Xu at Columbia, Yilun Du at MIT, and Siyuan Feng, Eric Cousineau and Benjamin Burchfiel at TRI.

3. What Is Diffusion Policy?

Diffusion Policy is a new method for generating robot behavior that represents a visuomotor policy as a conditional denoising diffusion process. That is how the abstract introduces it. For many readers without a specialist background, including me, the individual words may be familiar while the sentence as a whole remains bewildering.

Diffusion Policy: observations condition iterative denoising to produce actions in a closed loop.

Chi offered a helpful explanation: Diffusion Policy addresses the robot’s output. Much previous work has focused on inputs, but eventually the robot must act, and our work focuses on actions and outputs. More specifically, our innovation lies on the action side rather than the input side, where we use fairly ordinary components. Inputs can certainly be improved, but robot learning must also address outputs, where earlier algorithms did not perform well enough. However innovative the input processing, poor outputs prevent the system from realizing its potential. It is like cooking dumplings in a teapot: they are inside, but you cannot get them out through the spout.

Shuran Song also gave a very clear explanation of Diffusion Policy in her RSS 2023 presentation.

Video demonstration from the original article

How Are Diffusion Policy and Diffusion Models Related?

You can understand Diffusion Policy as an application of diffusion models to robotics. In image generation, diffusion models have produced excellent systems such as Stable Diffusion, with results much better than those of many GAN-based generators. Diffusion Policy applies the same family of methods to generating robot actions, particularly actions executed on physical robots.

Comparison of image diffusion and action diffusion through iterative denoising.

4. Why Use Diffusion Policy?

The Core Problems It Addresses

In its simplest form, learning a policy from demonstrations can be formulated as supervised regression: learning to map observations to actions. In practice, however, action prediction brings several challenges, including multimodal distributions, temporal correlations and training stability.

Three challenges: action multimodality, action-space scalability and training stability.

Handling Multimodal Action Distributions

Here, multimodal means that a real-world task may have several valid solutions rather than just one. A conventional neural-network predictor may output only a single solution, making it difficult to handle tasks with multiple valid ways of acting.

Chi explained this clearly: imagine driving toward a tree. Suppose I hire 100 drivers to deal with it. Fifty might turn left and go around the tree on that side, while fifty might go around the right. Both are perfectly reasonable. Combining these demonstrations produces a multimodal distribution: the same observed situation has two distinct valid choices. That is challenging for conventional neural-network prediction, which is often trained with mean squared error, or MSE, and cannot handle this situation effectively.

A conventional deterministic feedforward network is a function: a given input produces one output. But here, one input may correspond to two different outputs. The task we want it to perform conflicts with what that formulation can express. Forcing it to fit the data usually makes it predict an intermediate value that minimizes distance to the demonstrations. That can yield undesirable behavior, such as driving straight into the tree. Modeling a probability distribution instead allows a given observation to correspond to multiple possible outputs. This adds the flexibility to represent different distributions and overcomes the original limitation.

Multimodal Push-T trajectories and mode commitment across four policies.

That flexibility lets a robot choose different actions in the same situation, helping it avoid getting trapped in repetitive loops. In practice, this nondeterministic property can be highly useful because the algorithm need not remain stuck in one unchanging behavior. It does depart from the traditional robotics view that actions should be tightly controlled, with each execution treated as fully deterministic. That is one reason many people have not represented robot action generation as a probability distribution.

Humans also show some nondeterminism in their decisions: we may choose differently in the same situation. With a tree ahead, perhaps you turn right when you are in a good mood and left when you are in a bad one. That variability can be an advantage for humans and robots alike. Suppose a bug causes a robot to circle a traffic cone. With a deterministic algorithm, throwing a cone onto the road might make it circle forever. With a probabilistic prediction, it may sometimes circle and sometimes go straight, giving it a chance to escape the loop and return to normal behavior. This property can substantially improve robustness in practice, preventing endless repetition in unfamiliar situations.

Action-Space Scalability and Temporal Correlation

My simple understanding of action-space scalability and sequential correlation is that a robot’s prediction should not stop at the next one or two actions. It should look farther ahead, potentially predicting dozens of future steps.

Chi explained that there are two broad approaches to numerical prediction: directly output a value, or divide possible values into bins and predict a discrete category. For multimodal actions, researchers often favor discretization, turning a continuous-valued problem into classification. But the computational cost can be very high, especially in high-dimensional spaces. Classification-based control often predicts just the next action, whereas we would prefer to predict several steps together. That introduces the need for consistency across a continuous action sequence. The challenge is to balance computational cost with high-dimensional continuous control.

Single-action prediction versus temporally consistent action-sequence prediction.

Return to the driving example. We can turn left or right, which we can think of as controlling the steering wheel—a single control dimension. Divide that dimension into 100 bins and ask a neural network to choose one. In principle, that works, and classification is manageable for one dimension. But consider a robot manipulator with six degrees of freedom, or seven including the gripper opening. If each dimension is discretized into, say, 1,000 bins, the total number of combinations is the product across all dimensions. The cost becomes enormous and grows exponentially with dimensionality. Technical note: This exponential cost refers to explicitly enumerating joint combinations across dimensions. Factorized or autoregressive classification can avoid that full enumeration; not all discrete-action methods have the same cost.

There is also the earlier consistency problem. A viable driving trajectory goes continuously left around the tree, or continuously right around it. We can predict one step at a time, but that may cause oscillation: after moving a little left, the next prediction might favor right, then left again. This is action inconsistency. Even when we intend to commit to turning left, the network retains some probability of choosing right, making its decisions hesitant and alternating.

Because joint classification over high-dimensional spaces is expensive, such approaches may be limited to predicting one step. Adding more future steps increases the dimensionality further until the method becomes impractical. What we want is a method that can predict whole sequences in high-dimensional continuous control. We can directly predict the next 20 or 100 steps, committing to left or right, rather than predicting a step, executing it, and only then deciding what comes next. In the paper's receding-horizon controller, only part of the predicted sequence is executed before new observations are used to generate another sequence; the robot is not committed to running the entire prediction open-loop.

Training Stability

Compared with policies based on other generative models, a particularly important feature of Diffusion Policy is its very stable training process.

Training stability comparison between Diffusion Policy and implicit behavior cloning.

Chi elaborated: robot action generation broadly uses three types of methods—direct regression, classification and generative models. Regression treats the neural network as a function that takes an image and outputs an action. It is the most common approach and is used by most reinforcement-learning systems, but it has the problems we discussed earlier. Classification generates actions by predicting categories, as described above. Generative models can, in principle, predict continuous multimodal distributions, but many are unstable to train.

Diffusion-based methods have an important advantage: training is very stable. That helps explain their success in image generation compared with GANs at the time. GANs produced some good academic results, but turning them into products was difficult. Training an effective GAN often required extensive hyperparameter tuning before you could obtain a usable generator.

Diffusion methods can achieve performance comparable to GANs while being much more stable to train. Across a fairly wide range of parameter choices, the generator will usually produce results. They may not be optimal, but the method generally works. That stability also helps explain why Stable Diffusion and other image generators can be trained on enormous datasets. With less stable methods at that scale, unusual problems may emerge partway through training and prevent further optimization.

5. Other Questions from the Interview

How Does Diffusion Policy Relate to Reinforcement Learning and Imitation Learning?

Two common approaches to robot manipulation are reinforcement learning and imitation learning. Diffusion Policy does not conflict with either; it can be used in both. It is a policy formulation for mapping inputs such as images to corresponding actions. In our paper, we use imitation learning: humans teleoperate the robot, we collect demonstrations and learn a policy from them. The process begins with people remotely controlling a sequence of actions, which is then transferred to autonomous robot execution. Inputs include images from the past few frames, while outputs predict future actions.

Many reinforcement-learning researchers generate large amounts of data in simulation. To accelerate training, the RL policy often takes low-dimensional underlying state information rather than images as input. Those states may not be available in the real world, so the policy cannot directly drive the deployed robot. Researchers can then use the RL policy’s demonstrations to train an image-based imitation policy, a process called distillation. Here, the expert being imitated is an RL agent rather than a human. That is another possible application.

How Does Training Differ Between Manipulation and Locomotion?

Reinforcement learning tends to work better for locomotion, where the sim-to-real gap is relatively easier to address. For manipulation, a major difficulty is closing that gap. Manipulation requires modeling not only the robot’s own physics, but also complex environments and the objects it handles. Those objects have different centers of mass, friction and dynamics. Accurately simulating all of them is difficult, and available simulations may be imprecise or slow. For locomotion, much of the external environment can be treated as rigid, so some of its physical details are less important. Researchers can devote more effort to an accurate model of the robot itself and more sophisticated physics engines. That helps explain why RL has been more successful in locomotion than in manipulation deployed on physical robots.

What Are the Current Limitations and Future Directions?

The largest problem today is not the policy itself but the data. Training data is crucial for executing a particular task. We have accumulated some teleoperated demonstrations, but deploying a practical application—for example, a home robot that washes dishes—requires a much larger and more diverse dataset, on a scale closer to ChatGPT’s, to ensure robustness across different homes. The biggest challenge is collecting large amounts of diverse data efficiently. That is the key next research step. We hope that collection and training at greater scale will address current problems, while recognizing that new challenges may emerge as the dataset grows.

6. Some Reflections

Direct conversations with researchers are a format I have long wanted to try, hoping to make frontier research and its possibilities understandable to more people. I put it off because I worried that my grasp of the fundamentals was not deep enough and that I would ask very amateur questions. For this first attempt, I am especially grateful to Chi for patiently answering so many of those questions and offering detailed suggestions on the wording of the finished article. I learned a great deal from our conversation.

Recently, I have organized what I have learned about robot learning into notes and shared them through my WeChat account, receiving encouraging responses from readers. Thank you to everyone who reads my work. Writing never came naturally to me: Chinese language was my weak subject at school, and composition was the subject I disliked most. I never imagined that I would write for other people. Some investor friends ask how I can stay so focused on one field. There are many external reasons, but personally I am curious about whether general-purpose robots will arrive, when they will arrive and how. When studying other sectors, I often learned in order to make an investment. With robot learning, the order is reversed: I seek answers in frontier research, and along the way meet excellent people and discover promising startups.

I have recently been reading Wu Jun’s The Road to University. One passage particularly resonates with me: “Education is a lifelong undertaking. We need not worry about losing at the starting line, because most people stop running halfway through, leaving opportunities for those who keep going. A person who knows how to learn will discover their interests as they grow older. If they can still keep learning then, their learning will become purposeful and focused.” One defining feature of investing is the need to keep learning continuously, and that also pushes me to continue.

The enthusiasm around robotics startups and investment has cooled somewhat recently, and may cool further in 2024, the year following this article. But for someone truly interested in the field, how much does the market’s temperature matter?

References

https://pressroom.toyota.com/toyota-research-institute-unveils-breakthrough-in-teaching-robots-new-behaviors/

Diffusion Policy project page:

https://diffusion-policy.cs.columbia.edu/#paper

Cheng Chi’s website:

https://cheng-chi.github.io/

Russ Tedrake, Dexterous Manipulation with Diffusion Policies, on YouTube:

https://www.youtube.com/watch?v=whpK0HDtOJ0&t=482s

Shuran Song’s RSS 2023 presentation:

https://www.youtube.com/watch?v=hEO2-LY5i0o

Smart Note · Robot Learning & Foundation Models