In the first eight episodes of Robot Data, we worked through the robot data pyramid, introducing strong research that learns from human video, motion capture, teleoperation and simulation. These projects often focus on using one kind of data to advance robot learning. Can robots make use of all these modalities together? The answer is yes—and they must. Human learning through interaction with the physical world also combines different sources. A child may mentally imitate a cartoon character's movements, analogous to simulated data; imitate an adult, analogous to human video; or need an adult to guide their hands through a difficult task, analogous to teleoperation. With help from these different kinds of experience, people become better at tasks.
How, then, do we handle the extremely heterogeneous datasets and tasks found in robotics? Data comes from simulation, real-robot operation and human video demonstrations. Today's paper, Policy Composition, or PoCo, uses diffusion models to combine information across domains, modalities and tasks. Its composed policies demonstrate strong scene-level and task-level generalization in tool-use tasks.
I invited the paper's first author, Lirui Wang, to explain PoCo. At the time of this September 2024 interview, Lirui was a PhD student at MIT, advised by Professor Russ Tedrake at MIT CSAIL, and also worked closely with Professor Kaiming He. Before MIT, he earned his bachelor's and master's degrees at the University of Washington, working with Professor Dieter Fox. His interests span robotics and machine learning, particularly algorithms and systems that generalize to complex, unstructured real-world environments, and fleet learning that can handle heterogeneous robot data.
PoCo: Policy Composition From and For Heterogeneous Robot Learning was written by Lirui Wang, Jialiang Zhao, Yilun Du, Edward H. Adelson and Russ Tedrake, and accepted at RSS 2024.
Original article outline
Contents
↓
1. Introducing PoCo
2. PoCo's core idea
3. Understanding data combination across domains
4. What does policy composition mean?
5. Why focus on heterogeneous data?
6. Is PoCo only for tool-use tasks?
7. Why include touch?
8. Other questions about PoCo
9. HPT
10. Fleet learning
Some reflections
1. Introducing PoCo
How do we address robotics' highly heterogeneous datasets and tasks? Robot data comes from domains such as simulation, real-robot operation and human video. We investigate policy composition using diffusion models—PoCo—to meet these challenges.
An introduction to PoCo
We focus on generalizable robotic tool use, which highlights the compositional and contact-rich nature of real-world manipulation. Tool use is important in robotics, and there is a substantial supply of real human video involving tools. To support this work, we developed a benchmark in the Drake simulation environment.
Data
PoCo's core idea is to optimize a composed energy landscape over action trajectories. This composition takes place during inference without retraining, and can apply across tasks, such as hammer and wrench tasks; behaviors, such as a trajectory-smoothness objective; and domains, such as simulation and reality.
An overview of the PoCo method
PoCo generalizes across scenes with changing object and tool poses, distractors, new objects and dynamic conditions. Inspired by human learning, it composes policies across modalities such as RGB, point clouds and touch, and across domains such as simulation, real robots and human video.

Our composed multitask policy demonstrates task-level generalization in tasks involving contact with the environment, deformable objects, forceful manipulation and external disturbances. It also outperforms an unconditional policy trained directly on the four tool-use tasks.
Tool 1: A rice paddle
Tool 2: A wrench
Tool 3: A small hammer
Tool 4: A knife
2. PoCo's core idea
A large part of the difficulty in collecting robot data comes from the diversity of robot embodiments. Different robots represent their sensor modalities differently. Fragmented hardware platforms and diverse data formats are among the reasons we still lack a unified robot model. By multimodal data, we mean not only RGB and text but also point clouds and touch, which are crucial to dexterous manipulation. To distinguish this broader challenge from how 'multimodal' is often used for foundation models, we describe it as heterogeneity.
Beyond modality differences, training domains vary enormously: human video, real-robot data and simulation data. This is a distinctive challenge in robotics and embodied AI. Each domain has strengths and weaknesses. Simulation can provide large quantities of data, but some phenomena are difficult to simulate and a sim-to-real gap remains. Human video is abundant online but may lack action labels, making learning harder. Real-robot data is expensive. Research across these domains is therefore difficult.
PoCo's central question is how to combine this diversity across domains to make robot learning more general. We consider composition at three levels. First, the task level: train for different tasks, such as using different tools, with classifier-free diffusion training, then compose task-related behaviors. Second, the domain level: combine policies learned in domains such as simulation and real robots. Third, the behavior level: specify costs at test time, for example to favor smoother or safer trajectories. We can manually define such objectives and compose them with the policies.
3. Understanding data combination across domains
A real robot may have several sensors: cameras on the hand, external cameras and tactile sensors. During teleoperation, they collect data for training a model. The resulting model may work, yet have limited generalization. We often find that demonstrations lead to overfitting. Consider a tabletop task in which a robot uses a hammer to hit a nail. Move the nail to a position not seen in training, and the policy may fail. Add obstacles or move the target quickly, and its generalization problems become even clearer.
In simulation, those scene variations are easy to design. We can introduce clutter and use a planner to generate demonstrations. A simulation-trained policy can consequently generalize better to such variations, potentially using another modality, such as point clouds.
At test time, suppose we have two already-trained policies, one from simulation and one from real-robot operation. Can we combine them so their strengths complement each other—retaining smooth real-world behavior while benefiting from simulation's generalization? That is the motivation for this paper.
We assume the policies were trained separately, without sharing data. For example, two companies might each train a model. We then combine them and test in a new setting neither has encountered, asking whether the combination generalizes better. In robotics, we can call this compositional generalization. We might know how to hammer a nail and how to turn a screw with a wrench, but want to hammer the nail using the wrench. A directly trained RGB policy may not manage that transfer; policy composition offers a way to combine skills and support such generalization.
4. What does policy composition mean?
Each domain's policy is trained independently, and the trained policies are then composed. At the time of this work, this kind of composition was relatively uncommon in robot learning. We can do it because of diffusion policies' implicit generative structure.
A diffusion policy can be interpreted through an implicit energy landscape. That gives us a way to combine two or more landscapes. Another intuition is gradient-based optimization: different policy models provide different directions, which we combine during inference. The composition happens at test time, not by pooling all the data and retraining a single policy.
In traditional trajectory optimization, we often define several objectives—for instance, making a trajectory smooth and avoiding collisions. Their gradients guide optimization toward satisfying both. Policy composition replaces such objectives, conceptually, with learned policies. We can select different policies like components and compose them to carry out a task.
Simply averaging the outputs of two feedforward policies, such as MLPs or Transformers, can behave strangely. If one chooses left and another right, their mean points down the middle. If an obstacle is in the middle, averaging two individually sensible routes could produce a collision. Diffusion-policy composition is not merely averaging final actions: it is iterative and implicit, which gives a mathematical basis for composing distributions in this framework. That supports forms of composition that a naive action average would not.
This relates to Cheng Chi's point in our earlier interview that Diffusion Policy handles multimodal action distributions well. Here, 'multimodal' means several plausible action modes, rather than several sensor types. For more, see Diffusion Policy: Generating Robot Actions with Diffusion Models.
Diffusion policies are the policy class we use. The key is their connection to energy-based or score-based modeling. During training, a policy learns a denoising prediction. At test time, it starts with a noisy trajectory and repeatedly denoises it, conditioned on the current observation, toward plausible actions represented in its training distribution. With two diffusion policies, each produces a prediction from the current observation and noisy trajectory, and we combine those predictions during the iterative process. Editorial clarification: this energy-landscape explanation is an interpretation of diffusion scores; it does not mean that every diffusion implementation explicitly outputs a scalar energy or that an arbitrary pair of policies can be combined without compatible representations.
Ideally, this would give us an interface with different modules to choose from. You could select a safer trajectory, behavior closer to a simulation-trained policy, or more hammer-like actions. After selection, the modules would be composed at test time, even though each policy was trained separately. That is an envisioned use of the method, rather than a guarantee that a safety preference establishes safety.
The same logic applies to policies trained on different modalities, not only different domains. If their action spaces and representations are compatible, policies using different modalities can be composed. They might come from the same domain with different sensors, or from different domains using the same modality.
We have not yet discussed human video. We built a data pipeline to track hand–tool–object point clouds in human video and use them to train policies. This is a scalable way to collect human data. Such policies can then be composed with policies from the other two domains to improve performance.
In our experiments, composing simulation and real-world policies performed better than using either one alone. The simulation policy used 3D point clouds, while the real-world policy used cameras and touch, so this combination also crossed modalities.
5. Why focus on heterogeneous data?
Earlier work, including classifier-free diffusion methods, contains related composition ideas. We approach the problem from the perspective of data heterogeneity and try to make the method more general. That perspective is central to our contribution.
One way to think about it is to use an objective to bias the sampling process. Suppose we train a diffusion policy for a maze. If the goal is in the top-right corner, sampling is guided toward that corner. This is guided generation.
Similarly, imagine a real-robot policy and a simulation policy. Composing their guidance can pull the real-robot distribution toward simulation-like behavior, and the simulation distribution toward real-robot behavior. The interview uses the intuition of reward or objective guidance here; PoCo performs the combination through its diffusion-policy predictions.
My PhD research focuses on heterogeneous data. It includes my earlier work on Robot Fleet Learning via Policy Merging and Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?, which we refer to here as DEC SSL. PoCo also asks how to use heterogeneous data more efficiently. This is a broader, fundamental issue: outside data-rich fields such as language and vision, many areas have limited data but still want to use machine learning, and therefore encounter heterogeneity.
How can we efficiently extract information from this data? That broad question remains underexplored. Researchers often focus on one robot, one task, one simulator, one real-world problem or a particular algorithm such as imitation learning or reinforcement learning. I think it is valuable to integrate these factors and study them together.
Many people now want to train large models for diverse tasks, which is a meaningful goal. But how do we select data mixtures? Should we use, for example, 30% simulation and 50% real-world data, or 10% simulation and 70% real-world data, alongside other sources? We do not yet understand those choices well; they are often adjusted intuitively. I believe research on heterogeneity will support foundation model training and application more broadly, and is important for the field.
GPT-4o offers an analogy. Earlier approaches often centered on a language model and language outputs. The idea of bringing different modalities into token representations and learning from them together is also behind HPT: different sources can inform a shared representation. Some recent research suggests that representations across modalities may converge because they offer different views of the same world. Editorial clarification: the speaker's description of GPT-4o's internal tokenization and training is an analogy and interpretation, not independently established knowledge of its private training pipeline.
6. Is PoCo only for tool-use tasks?
No. It can apply to other robot tasks. I chose tool use because it is interesting and challenging, with demanding generalization requirements. Making it our main focus does not mean the method is limited to it.
Changing tools is different from using tools with a dexterous hand. Tool changing is common in industry, but if each new tool requires retraining a policy, that is inconvenient. Ideally, you would have a base policy that only needs adaptation when a tool changes—or, in the best case, no adaptation at all.
When choosing manipulation tasks, I felt grasping and pushing had already received considerable attention. I wanted something new, with different skills. Tool use encompasses a great deal. We discuss hammers and wrenches, but cooking also involves tools. Although the category is broad, I believe these interactions can be learned. A person who wants to strike something with a stone has an intuitive idea of how to do it. These basic interactions between objects are learnable. I chose the problem despite its somewhat loose definition and its mechanical complexity because I found it interesting. Adding tool tasks adds heterogeneity, skills and embodiments. It makes learning harder, but perhaps that is precisely how an embodied foundation model needs to be trained.
Why should combining more heterogeneous data, tools and tasks help? How can it work, and how can we train them together? Consider language models: code data can substantially improve their capabilities. At first, one might not expect code to help a model learn English or Chinese. But harder tasks, paired with useful objectives such as next-token prediction, force the model to develop abstract knowledge and reasoning skills that serve different tasks. I think that way of thinking is fundamental.
7. Why include touch?
In tool use, it is important to sense the forces involved when one object contacts another. Tactile sensors provide information vision alone may not reveal. It is also crucial to know the tool's position and orientation within the hand, which is why many tool-use studies include touch. In PoCo, we use GelSight Svelte, a tactile sensor developed by Jialiang Zhao. For more about this sensor family, see Fifteen Years of GelSight: A Brief History of Vision-Based Tactile Sensors. Editorial correction: the original spells the sensor name 'Svelvt'; the published name is Svelte.
8. Other questions about PoCo
What does 'without retraining' mean?
We do not have to gather all the data into one training process, as in the comparison with ChatGPT. Policies can be trained separately and then achieve cross-domain behavior through composition. That is a more challenging setting in one respect: the data remains separate, so we must make the learned policies work together effectively. 'Without retraining' refers to the composition stage, not to eliminating the initial training of the policies.
Where can PoCo improve?
One practical issue is speed, because several policies have to run at test time. More compute and distributed execution could help address this, though the speedup depends on the implementation and communication overhead. There is also plenty of room for algorithmic improvement.
9. HPT
We have a follow-up direction called Heterogeneous Pre-trained Transformers, or HPT. Its idea is simpler and more general than PoCo's: bring together more than 50 datasets, including Open X-Embodiment, simulation and human video, and train with them jointly. This is an extremely heterogeneous collection. We tokenize each embodiment separately. You can imagine tokenizing different modalities, but in robotics an embodiment also brings its own observations and actions, which must be handled appropriately. Embodiment-specific tokenizers map the inputs into a shared Transformer, and task-specific outputs map the representations back into actions. We developed this architecture and studied whether more data or larger models improve performance.
HPT results
We can transfer the representation to a new domain without retraining the central Transformer. By training a small input tokenizer and output network for the new robot, we adapt to its observations and action space while keeping the shared trunk fixed. The results are promising. At the time of this interview, we expect to release the work soon.
HPT adds both data and challenges, pushing the model to learn reusable representations. This resembles the data pyramid: some sources are easier to obtain and others more expensive, with different qualities. We need an effective method for bringing them together.
10. Fleet learning
Fleet learning is an idea we have been developing, one that also interests Russ Tedrake and has been discussed by TRI. As robotics technology and investment expand, hundreds or thousands of robots will appear. When one robot encounters new data or a new situation, must every other robot experience the same thing to learn it? We would like a fleet of, say, 1,000 robots to share what any one of them learns. Ideally, the others could benefit immediately. That is the goal behind fleet learning, rather than a claim that immediate transfer has already been achieved for any robot fleet.
Autonomous driving is an example. Each vehicle produces large amounts of data daily, which conventionally passes through a complex processing pipeline. Fleet learning asks how to learn across the collection. Robots perform different tasks and have different sensors and designs, so their experiences are heterogeneous. How do we handle that efficiently, and learn useful data representations across it? Those are the questions behind the idea.
Let me summarize the four directions: HPT, fleet learning, DEC SSL and PoCo. HPT pools heterogeneous data, tokenizes it and trains a shared large representation. Fleet learning combines policy parameters rather than uploading all the original data. A policy update might be only a few megabytes, compared with terabytes of daily raw data; after a period of local training, such as a month, the policies can be merged. These quantities are illustrative. Editorial clarification: the policy-merging method accounts for neural-network permutation symmetry; it is not simply naive averaging of arbitrary independently trained weights. DEC SSL asks whether unlabeled decentralized data can first support self-supervised representation learning, then combination. Finally, PoCo asks whether already-trained policies can be composed in action-distribution space. They tackle heterogeneity from different angles.
Some reflections
For the past few nights, I have been reading Dinosaur Kingdom at bedtime with my child, and learning about species that once inhabited the same planet we do. Dinosaurs appeared in the Late Triassic, more than 200 million years ago, and non-avian dinosaurs disappeared around 66 million years ago after a catastrophe—a span of roughly 160 million years. From Diplodocus reaching around 30 meters in length to the much smaller Compsognathus, described in my original account as only a few tens of centimeters long, I feel that more than a hundred million years of evolution gave dinosaurs far greater survival abilities than humans. Their long history makes me reflect on how fragile our own tenure is. Many of those creatures now survive for us only as fossils underground. Even five thousand years of Chinese civilization is tiny beside an age measured in tens of millions of years. We, living in 2024, are smaller still. Every ordinary moment of health, peace and freedom from disaster is worth cherishing. The fall of that dinosaur world makes me wonder whether human civilization could face a similar end. I suddenly understand a little more of the connection Elon Musk draws between settling Mars and continuing civilization. If I were someday standing at the end of human civilization, I would be grateful for the relentless work people are doing today on brain–computer interfaces, spacecraft, general-purpose robots, new energy and artificial intelligence. Human civilization is too rich and extraordinary to survive across generations only as fossils buried underground. It must continue in some form. Editorial clarification: the original suggests dinosaurs and humans shared an era and calls the extinction an unknown catastrophe. Humans did not coexist with non-avian dinosaurs; the end-Cretaceous extinction is strongly associated with an asteroid impact, and birds are surviving dinosaurs. The reflection above concerns our shared planet across different ages. The original size comparison is a rough description, and the survival comparison is my reflection, not a biological measurement.
The final research interview in Robot Data Season 1 is complete. I sincerely thank the scholars who accepted my invitations and everyone who has continued following the series. The diagram below summarizes the robot data pyramid explored across all nine interviews. General-purpose robots represent a far-reaching ambition for productivity and a formidable application challenge for frontier technology. That future may take longer than people expect, but I believe it will come. Along the way, perhaps these notes will offer a small spark of insight to an entrepreneur, researcher or practitioner: someone looking up at the same stars while doing grounded work in a different field.

Robot Data Season 1, Episodes 1–9
EP1, Conversation with Yang Gao and Chuan Wen: Training Robot Foundation Models with Internet Human Video—ATM, an RSS 2024 Paper Described in the Interview as Receiving Perfect Review Scores
EP2, Conversation with Cheng Chi and Zhenjia Xu: UMI, a Physical API for Robot Training
EP3, Conversation with Xuxin Cheng: Immersive Robot Teleoperation with Open-TeleVision
EP4, Conversation with Chen Wang: MimicPlay & DexCap—Learning from Human Video and Motion Capture
EP8, Conversation with Chuang Gan and Zhou Xian: How RoboGen Uses Generative Models and Differentiable Simulation to Synthesize Robot Demonstrations at Scale, Including a Review of Differentiable Physics Engines
References
PoCo
https://liruiw.github.io/policycomp/
GenSim
Fleet learning
https://arxiv.org/abs/2310.01362
DEC SSL
https://arxiv.org/abs/2210.10947