All nine interviews in Robot Data Season 1 have now been published. My sincere thanks to the researchers who accepted my invitations and to everyone who has followed the series. The interviews revolve around a robot-data pyramid. At its base are internet and synthetic data: abundant and inexpensive, but less directly aligned with a particular robot's training needs. At the top are demonstrations collected by a person teleoperating the target robot itself: highly relevant, but difficult to obtain and relatively scarce. In the middle is human motion-capture data, bridging synthetic data and robot-specific demonstrations. Robots have diverse embodiments, limited data for any one embodiment, and many sensor modalities, making frameworks that combine different sources essential. Every Season 1 interview explores a level of this pyramid or how to bring those levels together. The pyramid's 'quality' ranking refers to direct usefulness for a target robot, not an absolute ranking of all datasets.

Robot Data Season 1, EP01–EP09
To bring the nine interviews together, I will revisit their explanations of each data route and the core ideas behind the individual projects. The earlier episodes are linked below. The first-person passages that follow retain the researchers' voices from those interviews.
EP2: Cheng Chi and Zhenjia Xu on UMI — A Physical API for Robot Training
EP3: Xuxin Cheng on Open-TeleVision — Immersive Robot Teleoperation
EP4: Chen Wang on MimicPlay and DexCap — Learning from Human Videos and Motion Capture
EP5: Zipeng Fu on HumanPlus and Mobile ALOHA — Synthetic Data, Real Data and Their Combination
EP9: Lirui Wang on Heterogeneous Robot Datasets and Tasks
Original article outline
Contents
↓
1. Internet videos of human activity
EP1: ATM
EP4: MimicPlay
2. Synthetic data
EP5: HumanPlus
EP6: Eureka and DrEureka
EP7: Robot Parkour
EP8: RoboGen
3. Motion-capture data
EP2: UMI
EP4: DexCap
EP5: HumanPlus
4. Robot teleoperation data
EP3: Open-TeleVision
EP5: Mobile ALOHA
5. Combining robot data
EP9: PoCo and HPT
1. Internet videos of human activity
EP1: ATM
Why must we use complicated teleoperation systems to obtain the data needed for imitation learning? Human videos—from YouTube or Douyin, for example—are an important, abundant source. They contain physical information and patterns of movement, making them valuable prior knowledge. But using them effectively is difficult, because they usually lack action labels and cannot directly serve as conventional action-supervised imitation data. Even where action labels exist, research groups may define and collect them differently. Some represent end-effector motion; others represent the rotation of each joint. Those incompatible definitions make combining demonstrations difficult. ATM studies how to extract useful experiential knowledge from video.
What is our main idea in ATM? We do not want to predict entire video frames. For our control objective, we care about where objects move, not how beautiful the future image looks. Motion information is what matters. We drew inspiration from earlier ways of modeling movement. Game and simulation engines sometimes represent an object as a collection of particles, modeling their collisions, compression and relative motion to describe the scene. This is especially useful for complex materials such as fluids, deformable objects and sand. The original says future video appearance has 'no meaning' for control; here that expresses our choice of motion-focused representation, not a universal claim that video prediction can never help a robot.
ATM asks whether, instead of modeling video pixel by pixel, we can represent it as a set of points and model their movement to capture information about a scene's physical behavior for control. The analogy is to particle-based modeling, while ATM itself predicts image-point trajectories rather than simulating literal physical particles. EP1: Yang Gao and Chuan Wen on Learning from Internet Human Videos — ATM at RSS 2024 (reported perfect review scores)
EP4: MimicPlay
MimicPlay aims to learn robot policies and control from human behavior. Directly obtaining a robot's motor actions from ordinary human video is difficult: a camera cannot easily recover exact hand positions in three-dimensional space, and the robot moves differently from a human. Our idea was therefore to take a step back, learning a higher-level strategy rather than motor commands directly. Consider a long task: put food in a bowl, place it in the microwave, heat it and take it out. Human and robot hand trajectories differ, but the sequence and strategy—where to find a handle or pull out a tray—can be shared. We call this high-level planning. A low-level controller then uses that plan to control the arm's end effector and carry out the movements.
Separating the two levels has a major advantage: the high-level planner can learn directly from human videos. This relatively inexpensive source is called play data—open-ended interaction, not necessarily playing games. You need not label every action as task A or task B. Give it a goal, such as the dish you want to make, and the high-level planner should produce an appropriate strategy. Imagine a camera in your kitchen recording your daily activity: opening the refrigerator, taking out ingredients and cutting them on a chopping board. A third-person camera can readily capture those behaviors.
With more human data supporting the high-level planner, the low-level controller needs less robot data. It can focus on executing, say, twenty planned steps, without also learning the overall task plan. Its data requirements may therefore be modest: perhaps learning a grasping or movement skill. Some movements just require moving a few centimeters in a direction, and may not need to be learned at all. The twenty steps are an illustrative explanation, not a fixed horizon imposed by MimicPlay.
High-level planning learned from human activity can substantially reduce the system's need for robot demonstrations. Separating planning from control was our central idea. Generalization in the human-data planner supports the rest of the system, making it more efficient and reducing data requirements. That remains our core approach. In the published system, the high-level guidance takes the form of learned latent plans rather than simply replaying human trajectories as robot motor commands. EP4: Chen Wang on MimicPlay and DexCap — Learning from Human Videos and Motion Capture
2. Synthetic data
EP5: HumanPlus
With HumanPlus, we tried to combine the strengths of synthetic data and robot teleoperation data. Simulation can produce huge quantities, though its data may not match reality closely enough; real-world data is highly relevant but slow to collect and difficult to scale. HumanPlus asks whether we can collect high-quality data in the real world while also benefiting from abundant simulation experience.
HumanPlus combines robot simulation with human motion. We cannot simply copy human motion onto a robot, because their masses and joint structures differ; direct copying may make it lose balance and fall. For the lowest-level control, we therefore still use reinforcement learning and large amounts of simulation experience. When the robot receives human joint-position targets, it actively controls its motion to imitate them while respecting its own physical limitations and maintaining balance. This is designed to make imitation stable; it is not an unconditional guarantee that the robot never falls.
Through imitation, we can turn human movement patterns into robot policies. It feels as though a neural network is controlling a person's movements, with its output mapped onto a humanoid. Because humanoids resemble people, that mapping can be relatively intuitive, allowing us to use human data more effectively for human-like movements and tasks. Editorial clarification: HumanPlus does not directly behavior-clone arbitrary internet videos into motor actions. Its simulation-trained low-level policy enables real-time shadowing of human motion; demonstrations collected on the robot through shadowing then train an autonomous skill policy. EP5: Zipeng Fu on HumanPlus and Mobile ALOHA — Synthetic Data, Real Data and Their Combination
EP6: Eureka and DrEureka
In Eureka, we asked whether an LLM could automate parts of simulation design so that we could learn new skills quickly and transfer them to reality. Eureka uses a large language model to write the reward functions for simulated robot tasks. A reward function is executable code, like other code in the simulation, and a model such as GPT-4 is good at writing code. We therefore use an LLM to generate reward code—but not just once. We also ask it to improve that code iteratively.
For sim-to-real reinforcement learning, designing a reward is not enough; the policy must transfer successfully. Simulation and reality differ in many ways, especially in their physical models. A simulator cannot perfectly reproduce all relevant gravitational, frictional and other physical effects. A popular response is domain randomization: vary the physical parameters during training. But people normally have to design the parameter ranges manually. In DrEureka, we automate that design as well as reward generation.
With automatic reward and domain-randomization design, we can select policies that perform best under the generated rewards and randomized physics settings. To our delight, we found that these policies could transfer directly to the real world in our evaluated tasks. This gives us an automated sim-to-real design pipeline, avoiding iterative manual intervention in reward and randomization tuning. Editorial clarification: 'fully automated' describes this algorithmic design loop. Researchers still supply the task simulator, task and safety instructions, hardware setup and physical deployment; the LLM does not build and operate the entire experiment unaided.
This demonstrates the potential of LLMs to automate reinforcement-learning design, from rewards and domain randomization to policies suitable for deployment, reducing dependence on repeated expert tuning. That improves efficiency and opens possibilities for more complex real-world problems. The reported automation and transfer results are within the experimental tasks and assumptions, rather than proof of a universal autonomous deployment system. EP6: Yecheng Jason Ma on Eureka and DrEureka — Using LLMs to Address Key Challenges in Simulation and RL
EP7: Robot Parkour
For the parkour work, I see two central questions: how to design and tune an effective training strategy, and how to overcome the sim-to-real gap. Designing the training strategy is complicated, involving reinforcement learning with different dynamics constraints and the distillation of skill policies.
Many earlier quadruped systems used proprioception without vision. Here we introduce visual perception, particularly a stereo depth camera, which makes the sim-to-real gap more important. We try to simulate depth sensing as realistically as possible and process real-world data to resemble the cleaner, idealized images in simulation. Bringing the two distributions closer helps narrow the gap.
We tried several ways to narrow that gap, including filling holes and removing noise. Depth images work reasonably well for this purpose. The parkour challenge is comparatively focused: traversing terrain, for which understanding its geometry and depth is crucial. But future practical robots should not rely only on depth. They will also need to use RGB images.
Multitask learning is another longstanding challenge. When one model learns several tasks at once, their backpropagated gradients may conflict and each skill can lose performance. Tuning one skill is relatively easy; tuning several together is hard. After trying different approaches, Ziwen found a useful strategy: train each skill separately as a teacher, then train a shared student from them. This minimizes performance degradation. The result is a systematic solution and a broad capability improvement, rather than one isolated innovation. EP7: Hang Zhao and Ziwen Zhuang on Robot Parkour and Autonomous Driving — Where Robot Data Comes From
EP8: RoboGen
The idea behind RoboGen is to extract relevant information and use it to build a robot-training environment. That includes the skills a robot should learn, the setting for each skill, the environment's appearance and how the robot interacts with it. We also need clear completion criteria, usually expressed as rewards or losses. Large models, such as language models, can generate the code for those functions automatically. We want to automate every preparation step and connect the pipeline—from defining rewards to generating code—so the training process becomes highly automated from beginning to end. This is the goal and core idea behind the fully automated training pipeline we first proposed.
Reinforcement learning works well for many rigid-body interactions, both locomotion and manipulation. Deformable objects complicate matters, making differentiable components important. Fluids and clothing differ from rigid objects, and RL needs an effective representation of their state. If we move beyond rigid-body interaction, the complexity of that state space becomes a challenge. Simulators often use particle systems; a piece of cloth might contain hundreds of thousands of particles. Writing a conventional reward function that assesses those particle states can be difficult and expensive. First, the particle count makes describing the state and checking goal completion costly. Second, it is hard to write an explicit function for a deformation such as folding clothing and its corresponding particle configuration. These difficulties motivate differentiable components, which can make learning soft-object interaction more efficient. This explains their role in our approach, rather than establishing that every deformable-object task requires differentiable simulation.
Gradient information can accelerate policy learning and optimization compared with relying on reinforcement learning alone. Combining the approaches effectively is still an active research topic, without a mature framework that resolves all these issues. Gradients are not always ideal. Contact can make the optimization landscape nonsmooth or discontinuous, introducing noise and harming training efficiency. Gradients also convey local information, which may be insufficient for global optimization in complicated nonconvex problems. These are challenges the research community is still addressing. EP8: Chuang Gan and Zhou Xian on RoboGen — Generating Robot Demonstrations with Generative Models and Differentiable Simulation, with a Differentiable Physics Review
3. Motion-capture data
EP2: UMI
At the heart of UMI is a physical API. We found a physical interface with a form sufficiently similar to a robot's, yet not restricted to one particular robot embodiment, and much cheaper than a robot. I can explain it from three perspectives.
First, portability. Suppose a robot cost only RMB1,000–2,000. With the weight of today's robots, collecting restaurant data would still involve loading it onto a truck, driving there, unloading, installing and calibrating it. Half a day could disappear. Every new environment incurs that setup cost, making diverse-environment data expensive. UMI's handheld device addresses this. Its gripper is still somewhat heavy, around 800 grams, but fits in a bag. Once we arrive somewhere—for example, sit down at a restaurant—we can begin recording within two minutes. That greatly simplifies collecting diverse real-world data. The robot price here is a hypothetical example, not the price of the UMI system.
Second, similarity of physical form. Many research platforms combine an industrial robot arm with a parallel-jaw gripper, and UMI was designed around that common form. It records the gripper's six-degree-of-freedom trajectory in space. If a robot can reach a required pose, the particular joint angles used to get there are less important; in principle, even another platform such as a drone could reach a position. In that sense, UMI is relatively independent of the robot's embodiment: the brand matters less than its ability to execute the demonstrated poses. Editorial clarification: having six degrees of freedom alone does not guarantee compatibility. Reachability, gripper geometry, actuation, timing and other constraints still matter; the drone is a conceptual example, not a demonstrated universal transfer result.
Third, intuitive data collection. We tried a dynamic throwing task with round and square objects and two corresponding bins: throw each object into the bin of the matching shape. This fast movement is difficult to demonstrate through robot teleoperation. In our earlier tests, fifteen minutes of teleoperation did not successfully throw a single block into the bin. With UMI, people have a strong intuition for using the gripper, and collection ran at about half the speed of using bare hands. That sounds slow, but it is a major improvement over obtaining no successful teleoperated demonstration. In another task, turning a cup, UMI again collected data at roughly half bare-hand speed, while robot teleoperation was about three times slower than UMI. These examples show its collection advantages. EP2: Cheng Chi and Zhenjia Xu on UMI — A Physical API for Robot Training
EP4: DexCap
DexCap's wrist cameras track the scene using SLAM. In the interview, I described the combined spatial tracking as having roughly centimeter or subcentimeter accuracy, and called DexCap a 'nuclear weapon,' metaphorically for obtaining higher-precision data than third-person MimicPlay videos. The wrist cameras are for tracking rather than primarily collecting policy-training images. Human and robot hands look different, creating a domain gap in those images. We tried learning directly from human wrist-camera footage, and it did not work well. Editorial clarification: this overall tracking figure is distinct from the millimeter-level relative fingertip measurement supplied by the electromagnetic glove below.
DexCap uses motion-capture gloves with an electromagnetic sensor on each fingertip. A wrist-mounted camera tracks the hand's six-degree-of-freedom pose. An electromagnetic transmitter sits below it, with receivers at the fingertips. The electromagnetic measurements give millimeter-level estimates of fingertip positions relative to the hand. Combined with the wrist camera's hand pose relative to the chest, this lets us recover fingertip positions in the camera's coordinate frame. The chest camera, wrist cameras and gloves work together to provide the overall measurement.
The chest camera tracks the person's pose in world coordinates; the wrist cameras track the hands relative to the chest; and the gloves track the fingertips relative to the hands. As you walk wearing the system, combining those measurements gives absolute fingertip positions in space. Those high-quality trajectories can then guide a robot through inverse kinematics. Editorial clarification: inverse kinematics is one part of retargeting, not a guarantee that any robot can replay every human motion without accounting for its embodiment and physical constraints. EP4: Chen Wang on MimicPlay and DexCap — Learning from Human Videos and Motion Capture
4. Robot teleoperation data
EP3: Open-TeleVision
Open-TeleVision proposes a new teleoperation method suited to humanoids and today's collection challenges. An operator cannot simply stand behind a humanoid, because its torso blocks the view. Standing beside it and craning one's neck is also unintuitive and tiring. A humanoid is mobile, so installing many fixed cameras is impractical. The most intuitive approach is therefore immersive control—as if you had become the robot. This inspired Open-TeleVision: operators need not stand beside or behind the robot or control it from a third-person view. They wear a headset and operate it immersively.
There are three main features. First, immersion: the operator sees stereoscopic video rather than a flat 2D view, like a 3D film. Different images reach the left and right eyes, giving natural depth cues that help estimate distances and perform fine tasks. Second, our system has an active neck so its head can move. When people act, they focus using the small foveal region at the center of the retina. With a fixed camera, looking elsewhere may require sideways glances or peripheral vision. The active neck makes control more intuitive: turn your head and the view follows, without awkwardly shifting your eyes. Third, it can reduce computational cost. The operator can focus on a smaller central view rather than processing an extremely high-resolution wide-angle image containing much irrelevant information. Turning the head brings the desired area into view. In our setup, this supported around 80 Hz, compared with about 40 Hz when processing larger images. A higher frame rate helps the experience feel more immediate and intuitive. These rates describe the interview's setup, not a guaranteed rate on all hardware. EP3: Xuxin Cheng on Open-TeleVision — Immersive Robot Teleoperation
EP5: Mobile ALOHA
With Mobile ALOHA, we used real-world data rather than simulation. People collected demonstrations through teleoperation, and efficient imitation learning allowed training from a relatively small amount of data. That provides realistic visual signals, such as RGB images, and information about the task's semantics.
One challenge is that real-world data is slow to collect. Mobile ALOHA combines an accessible collection system with an efficient imitation-learning algorithm: around fifty human demonstrations per task can teach a range of skills. Compared with earlier collection methods using VR equipment, a 3D mouse or keyboard, this interface can be quicker and easier to learn. Its value is helping people rapidly learn to operate robots and collect demonstrations for many real-world problems. Editorial clarification: the paper reports selected tasks, including co-training with static ALOHA data. The original's phrase 'almost all problems in the real world' is an aspirational description, not an experimentally established capability. EP5: Zipeng Fu on HumanPlus and Mobile ALOHA — Synthetic Data, Real Data and Their Combination
5. Combining robot data
EP9: PoCo and HPT
A major reason robot data is difficult to collect is the diversity of embodiments. Different robots may represent the same sensor modality differently. Fragmented platforms and varied data formats are a major obstacle to a comprehensive shared model. By multimodal data, we mean not only RGB images and text but also point clouds, touch and other modalities crucial for dexterous manipulation. To distinguish these broader differences from the usual image-and-language discussion of multimodal foundation models, we use the term heterogeneity.
There are also substantial differences between data domains: human videos, real robot demonstrations and simulation, for example. That is a distinctive challenge in robotics and embodied intelligence. Each domain has strengths and weaknesses. Simulation supplies abundant data but cannot perfectly reproduce everything. Human videos are plentiful online but often lack action labels, making training difficult. Real robot data is expensive. Cross-domain research is therefore challenging.
PoCo's core idea is to combine heterogeneous policies across domains to improve robot learning's generality. We consider three levels. At the task level, policies trained for different tasks, such as using different tools, can be combined through diffusion training and classifier-free guidance. At the domain level, we can compose policies trained from sources such as simulation and real robot data. At the behavior level, we can specify costs at test time—for example, to favor smoother or safer behavior—and combine those objectives with the learned policies. The composition operates through compatible policy distributions and their scores or energy representations, rather than simply averaging unrelated action vectors.
My PhD thesis mainly concerns heterogeneous data, including my earlier work Robot Fleet Learning via Policy Merging and Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision? PoCo also asks how to process and use such data more efficiently. It is a broader, foundational problem: outside vision and language, many fields have too little data but still want to apply machine learning. They therefore face heterogeneity as well.
How do we extract information efficiently from these datasets? This larger question remains underexplored. Researchers often focus on one robot, task, simulator, real-world setting or algorithm, such as imitation learning or reinforcement learning. I think studying these elements together, rather than only in isolation, is highly valuable.
People are now focused on large models, and training one model to support diverse tasks is a meaningful challenge. For example, should we use 30% simulation data and 50% real-world data, or 10% simulation and 70% real-world data? We still have limited understanding of how to choose such mixtures, and often adjust them intuitively. Those example pairs are partial mixtures rather than complete recipes summing to 100%. I believe research on heterogeneous data will broadly support the training and application of large models and be important to the field's development.
GPT-4o provides a useful analogy. Earlier systems were often language-centered, with a language model in the middle and language as the only output. We can instead think about tokenizing different data types and pretraining them jointly, which is an idea behind HPT. That offers the possibility of integrating information across modalities. Some recent work also suggests that representations from different modalities may converge, because they describe the same world from different perspectives. Editorial clarification: this comparison expresses the interviewee's reasoning, not verified knowledge of GPT-4o's private training pipeline. HPT uses modality-specific stems, a shared Transformer trunk and downstream policy heads. EP9: Lirui Wang on Heterogeneous Robot Datasets and Tasks
References
EP1: Yang Gao and Chuan Wen on Learning from Internet Human Videos — ATM at RSS 2024 (reported perfect review scores)
EP2: Cheng Chi and Zhenjia Xu on UMI — A Physical API for Robot Training
EP3: Xuxin Cheng on Open-TeleVision — Immersive Robot Teleoperation
EP4: Chen Wang on MimicPlay and DexCap — Learning from Human Videos and Motion Capture
EP5: Zipeng Fu on HumanPlus and Mobile ALOHA — Synthetic Data, Real Data and Their Combination