Back to Robot Data

Robot Data · EP04

Robot Data, Episode 4: Chen Wang on MimicPlay, DexCap and Learning from Human Motion

The full interview on hierarchical planning, human play, wearable motion capture, retargeting, dexterous control and the differences between DexCap and UMI.

Zha Dami2024-07-1417 min readWeChat original
Contents

Robot training data can be pictured as a pyramid. At the base are internet data and synthetic data: abundant and inexpensive, but often less directly suited to a particular robot task. At the top are demonstrations generated by a person teleoperating the target robot. These can be highly relevant and accurate, but are difficult to collect in large quantities. In between sits human motion capture, bridging inexpensive human observations and robot-specific demonstrations. Today's two projects, MimicPlay: Long-Horizon Imitation Learning by Watching Human Play and DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation, use relatively inexpensive human play data and wearable motion-capture data, respectively, to support robot learning.

Let me say a little more about the recognition these projects received. In November 2023, MimicPlay was named a finalist for CoRL's Best Paper, Best Student Paper and Best Systems Paper awards—a remarkable sweep of nominations. These were finalist selections, not wins in all three categories. In May 2024, DexCap was accepted at RSS 2024. MimicPlay's authors are Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Fei-Fei Li, Danfei Xu, Yuke Zhu and Anima Anandkumar. DexCap's authors are Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Fei-Fei Li and C. Karen Liu.

We invited Chen Wang, the first author of both projects, to discuss them. At the time of this July 2024 interview, Chen was a PhD student in Stanford's Computer Science Department, advised by Professors Fei-Fei Li and Karen Liu. Before Stanford, he earned his bachelor's degree in computer science at Shanghai Jiao Tong University in 2020. His goal is to build robots with human-like dexterity and the ability to handle everyday tasks. His research focuses on learning for dexterous manipulation, imitation learning from human motion, and long-horizon planning and control.

Original article outline

Contents

↓

1. Ways to collect embodied AI data

2. MimicPlay: Long-horizon imitation learning from human play

MimicPlay's core idea

The MimicPlay method

How does its planning differ from an LLM's?

Learning trajectories from human video versus trajectory optimization

Learning 3D trajectories from 2D video

Addressing the embodiment gap

Achieving millimeter-level precision

3. DexCap: Motion capture for dexterous manipulation

The DexCap system

Why do demonstrators sometimes move more slowly?

What is inverse kinematics?

Mapping human hand data to robot hands

How well do the learned policies work?

What challenges remain for DexCap?

4. Additional questions

Combining MimicPlay and DexCap

DexCap versus UMI

5. Some thoughts

1. Ways to collect embodied AI data

Think of data as a pyramid. The bottom contains the cheapest, most abundant sources, such as internet data and generated data. They are plentiful, but may not accurately reproduce the physical world. Virtual data can struggle with physical interactions such as collisions and friction, and simulation introduces a sim-to-real gap. These remain research challenges. In this comparison, data quality means usefulness for the target manipulation task, not that all internet or synthetic data is inherently poor.

The next layer is human motion capture. A person may need to wear equipment, but it provides relatively precise data, sufficient for many tasks such as pick-and-place. It does not solve everything: the embodiment gap means a robot cannot necessarily move as a person does. The data costs somewhat more because equipment is required, and it is less abundant than internet or synthetic data. It is collected in reality, so it avoids a virtual-to-real gap in the observations themselves, although transferring human motion to robot execution still presents challenges.

At the top is teleoperation data, collected by manually operating the robot and showing it what to do, such as grasping an object. It is less abundant because operation is slower and requires dedicated people. You need the robot itself and personnel to operate it, which raises cost. The benefit is that demonstrations come directly from the target robot and are closely aligned with its usable actions. These three approaches each have their characteristics and advantages.

2. MimicPlay: Long-horizon imitation learning by watching human play

MimicPlay's core idea

MimicPlay learns robot control and policies from human behavior. Directly turning ordinary human video into robot motor actions is not feasible in the approach we study: cameras may not recover sufficiently precise 3D hand positions, and robots move differently from people. The data therefore cannot simply be used as robot action labels.

We take a step back and learn guidance rather than directly learning motor commands. Consider a long task: put food into a bowl, heat it in a microwave, and take it out. A person demonstrates useful structure—where the hand should go, how to approach the handle, open the microwave, pull out the tray, place the bowl on it and close the door. Human and robot trajectories differ, but the sequence and spatial guidance can be shared. We call that high-level planning. A low-level controller then uses the plan to produce robot-specific movements.

An important benefit of separating the levels is that the high-level planner can learn from inexpensive human video, which we call play data: people freely interacting with their environment, not necessarily playing a game. You do not need to label each action as task A or B. Given a goal, such as a finished dish, we want the planner to provide suitable guidance. Imagine a camera in your kitchen capturing daily activities: opening the refrigerator, taking out an ingredient and cutting it on a board. An external camera can readily collect this kind of behavior.

With more human data supporting the high-level planner, the low-level controller needs less robot data. It follows the supplied guidance rather than learning the entire long-horizon plan. For an illustrative 20-step task, it may only need to learn local grasping or movement skills. Some movements are simply shifts of a few centimeters in a direction; in certain cases, they may not require learning at all.

Human-derived high-level guidance can substantially reduce the overall demand for robot demonstrations. Separating planning from control was our idea: let human behavior data support generalization in planning, while robot data teaches execution. This makes the system more efficient and reduces robot-specific data requirements. It has been our central approach.

The MimicPlay method

MimicPlay is a hierarchical imitation learning algorithm. It learns high-level plans from inexpensive human demonstrations and low-level control from a small amount of multitask teleoperated robot data.

Video demonstration from the original article

There are three broad phases. First, human play video trains a goal-conditioned trajectory-generation model, providing high-level guidance for different goals. Second, a small amount of teleoperation data trains a low-level robot controller. Third, at test time, a video prompt for a long-horizon task—either human motion or robot teleoperation—guides MimicPlay's generation of plans and the controller's execution. More precisely, the controller is conditioned on learned latent plans, rather than merely replaying the human hand trajectory.

MimicPlay learns high-level plans from human play and robot control from teleoperation

How does MimicPlay's planning differ from an LLM's?

There is an important difference from a language-only plan. An LLM might give you steps numbered 1, 2 and 3 without specifying exactly how to execute them. The high-level guidance I describe contains 3D spatial information: approaching a handle along a particular path, grasping it at an appropriate angle and opening it. It goes beyond listing the steps.

Those trajectory and movement details are difficult to express precisely in natural language. Our planner's advantage is that it learns spatial guidance and passes it to the low-level controller. This comparison concerns a plain language-level plan, not a claim that an LLM integrated with spatial tools can never support such details.

The plan is therefore not just an instruction manual or a verbal description. It encodes concrete 3D spatial guidance, including target locations and grasping angles, about where the robot should move and how the interaction should unfold. That is grounding. If an LLM says, 'First open the refrigerator,' that instruction still has to be grounded: locate the handle, move toward it, grasp it and follow a suitable opening trajectory. Those concrete operations turn a semantic instruction into action.

Refrigerators differ; some doors open in the opposite direction. Grounding must adapt to those details. A language plan alone may struggle with that. MimicPlay's high-level representation already contains learned guidance about implementation, helping the low-level controller carry out the task.

Is learning motion trajectories from human video a form of trajectory optimization?

Not in the sense of finding an optimal trajectory. Human behavior is not necessarily optimal. We are learning a workable solution to a task. If an LLM gives the semantic goal 'open the refrigerator,' we still need a 3D approach and grasping configuration that accomplishes it. A learned solution can be feasible without being optimal. Training of the network does involve optimization; the distinction here is about optimizing the executed path against a task cost.

It may not be the shortest path. People generally do not explicitly minimize that, although their movements are usually reasonable: they do not deliberately wander around touching unrelated things before opening a refrigerator. We seek a workable behavior, not necessarily the mathematically optimal one.

Most internet video is 2D. How do you learn a 3D trajectory from it?

For MimicPlay, we used multiple RGB cameras—two views—to recover 3D human hand trajectories. Ordinary internet videos cannot be used directly in that pipeline because they do not provide the required 3D information. But 2D trajectories could still be useful. Even in an image plane, watching a hand move tells you where it goes next. You can see which handle it approaches and the handle's pixel location. That is informative guidance too.

One could modify MimicPlay to use 2D rather than 3D guidance, potentially opening up more internet video datasets. We did not validate that in the released work. Another difficulty is that internet cameras often move. Camera movement changes the apparent image coordinates, so the model must understand how those relate to the world. Learning directly from YouTube video still poses substantial research challenges.

How do you address the human–robot embodiment gap?

Human hands and robot grippers interact differently, and grippers themselves vary in size and design. Pulling a tray from a microwave, for example, may require different approaches with large and small grippers. The data demand for this robot-specific part may not be enormous once the plan specifies what to grasp, where to reach and where to move. The low-level controller can specialize to the robot's embodiment. A few teleoperated pick-and-place demonstrations teach it to translate high-level guidance into executable robot commands. That translation is reusable: a new plan can be executed through the same controller. Here, low-level describes the control hierarchy, not necessarily a low-dimensional action space.

Human video lacks millimeter-level accuracy. How do you handle that?

That is why the high-level plan is approximate. It might place the handle within a roughly five-centimeter region. Precise control is delegated to the low-level controller, which uses the robot's own observations, including an eye-in-hand camera mounted on or near the gripper. The camera supports alignment and grasping. Such local control can also generalize: after learning to grasp three or four handles precisely, it may handle other colors or shapes. The high-level guidance says where to go; the local controller sees the handle, aligns the gripper, approaches and grasps it.

DexCap goes further by using motion-capture equipment for tracking at approximately sub-centimeter precision, including fingertips rather than only wrists. This gives more directly useful motion data. With MimicPlay's video, human trajectories may be accurate only to roughly three to five centimeters; the robot's learned low-level controller must supply the finer adjustment. DexCap instead improves capture precision through wearable equipment, allowing the robot to learn more directly from retargeted human demonstrations.

3. DexCap: Motion capture for dexterous manipulation

The DexCap system

The hand-mounted tracking cameras in DexCap support SLAM-based localization. Combined with the gloves, the system tracks the hands and fingers at roughly sub-centimeter accuracy. I describe it as a 'nuclear weapon,' metaphorically because it gives more precise data than MimicPlay's third-person video. The hand cameras' primary purpose here is tracking, not supplying robot-policy images. A human hand looks different from a robot hand, creating a visual domain gap. We tried directly learning a robot policy from the human wrist-camera images, and it did not work well.

Video demonstration from the original article

DexCap uses motion-capture gloves with electromagnetic sensing for the fingers. The hand-mounted camera localizes the palm in six-degree-of-freedom space. An electromagnetic transmitter sits below the camera, with receivers at the fingertips. The field gives millimeter-scale localization of fingertips relative to the palm. Combined with palm tracking and the chest-mounted system, we can place those fingertip positions in the observation frame. The chest and wrist tracking equipment and the gloves jointly establish the captured pose; millimeter-scale relative finger sensing should not be confused with identical accuracy of the entire global tracking chain.

The chest tracking system localizes the wearer in a world frame. Wrist tracking establishes the palms' locations, and the gloves locate fingertips relative to the palms. In effect, as you walk with the system, we can recover where you are, where your palms are and where your fingers are, composing those measurements into world-space fingertip positions. Those accurate trajectories can then be retargeted to a robot using inverse kinematics, provided the robot can reach and execute them.

What is inverse kinematics?

Forward kinematics computes a fingertip or end-effector's 3D position from the robot's joint angles. Inverse kinematics works in the other direction: given a desired fingertip or end-effector position, it solves for joint angles that reach it. There can be multiple solutions, constraints or no feasible solution, depending on the robot.

Once we capture human fingertip trajectories at centimeter-scale accuracy, we can attempt to retarget them to different robots, including humanoids and arms with dexterous hands. Position the robot appropriately, and inverse kinematics translates target fingertip locations into joint configurations. For opening a door, the captured demonstration places fingers on the handle, closes the grasp and presses down; the goal is for the robot to reproduce a compatible version of that trajectory. Editorial clarification: the original presents this as applicable to any robot by IK alone. Actual transfer still depends on hand morphology, reachability, dynamics, contact and control; position targets do not guarantee success on every embodiment.

Why do people wearing the gloves sometimes move more slowly than usual?

Not every human demonstration can be directly executed by a robot. People can change direction and stop their fingers very quickly—for example, tracing a short arc in about 0.1 seconds. The robot hardware we use may overshoot or respond late when its motors are commanded to stop abruptly. That reflects differences between motor control and human muscular control. Editorial clarification: the original says the human motion has no inertia. Human hands do have inertia; the practical point is the difference in controllability and response, not an absence of physical inertia.

Video demonstration from the original article

Sometimes we therefore need to collect the demonstrations again. A dataset can look complete, yet include motions the robot cannot learn or reproduce well because it is not as fast or agile. I deliberately slow down some demonstrations to improve their suitability and the eventual success rate, making the transfer to robot actions more effective.

Think again of a hand rapidly changing direction or stopping after a short arc within 0.1 seconds. With our robot, abrupt braking can produce overshoot: it passes the target and comes back. This is another form of embodiment gap, involving the difference between motors and muscles. We cannot assume that the current robot will reproduce every natural human movement. DexCap demonstrations may need to be recollected despite looking good, so I sometimes slow them down deliberately. That improves data usability and robot success. As above, this is about differing dynamics and control, not human motion being free of inertia.

Robot hardware and motors still have considerable room to develop. We cannot yet reproduce the full capabilities of human muscle and its control. That limitation is one disadvantage of this data-collection approach.

How is human hand data mapped to robot hands?

After DexCap captures human manipulation, observation retargeting and action retargeting transfer the data to the robot. For observation retargeting, the robot uses the same RGB-D observation camera as the human collection system. We construct 3D point clouds and transform them into the robot's workspace. For action retargeting, we transfer human finger motion to the LEAP Hand using fingertip inverse kinematics to compute its 16 joint positions, and use IK to obtain the robot arm's joint configuration.

DexCap observation and action retargeting from human motion to robot hands

Because robot hands do not exactly match human hands, an embodiment gap remains. To reduce the visual difference, we use forward kinematics to generate a point-cloud representation of the robot hand and add it to the scene observations. We then train a point-cloud-based diffusion policy, related to the approach discussed in Diffusion Policy: Generating Robot Actions with Diffusion Models. It takes point clouds as input and predicts a sequence of future target positions as robot actions.

How well do the learned policies work?

A point-cloud diffusion policy trained on retargeted human motion-capture data can control both hands in a 46-dimensional action space for tasks such as collecting tennis balls and packing objects. The demonstrated policies use 30 minutes of human motion-capture data without teleoperation. These are results for the evaluated tasks, not a universal data requirement for all dexterous skills.

Video demonstration from the original article

0:00–0:11: Collecting bimanual human motion-capture data

0:11–0:28: Fully autonomous policy execution

Position data alone is not always sufficient for tasks requiring appropriate force and contact. We therefore add human-in-the-loop correction during policy execution. With up to 30 corrections, the demonstrated system can prepare tea and use scissors. Editorial clarification: the project page reports these fine-tuned examples using one hour of human motion-capture data plus 30 corrections, rather than the 30-minute data setting of the preceding examples.

Video demonstration from the original article

Autonomous bimanual tea preparation—including opening a bottle cap

Video demonstration from the original article

Cutting paper

DexCap supports two correction modes. First, residual correction measures changes in the human wrist's 3D position and adds them as residual adjustments to the robot's wrist motion. This requires relatively little human movement but careful control. Second, teleoperation translates the full human hand motion into robot end-effector actions through IK. It gives more complete control, at the cost of more movement and effort.

Video demonstration from the original article

What challenges remain for DexCap?

There is plenty to improve. First, wearable equipment could become much cheaper. Second, I wonder whether the chest camera could be replaced with something less obtrusive, such as a small camera integrated into glasses. Gloves also impose limits: they may be fine for many activities, but you cannot wash your hands or dishes if the equipment cannot get wet. Could a future wrist-worn or ring-like device infer finger motion, allowing gesture interaction without gloves—for example, summoning a menu with a gesture, as with smartwatch interactions? The interview compares this aspiration to smartwatch gestures and muscle-related sensing; it is a proposed direction, not a verified description of Apple Watch sensing technology. Glasses with cameras plus unobtrusive finger tracking might fit into daily life so well that others barely notice data is being collected. More affordable, less intrusive wearables are an important future direction.

Another challenge is helping new users understand which demonstrations a robot can execute. Could the system say, 'I recorded that movement, but your robot cannot perform it. Give me the model of the robot you use, and I can check feasibility: its motors cannot move quickly enough for this trajectory'? Such feedback could help improve demonstration collection and the algorithms supporting it.

4. Additional questions

Combining MimicPlay and DexCap

They can be combined. Motion-capture data could train MimicPlay's high-level planner, potentially giving more accurate guidance because DexCap data is more precise. It could also provide pretraining for low-level control. Applying the MimicPlay framework to DexCap data is therefore a promising way to combine efficient collection with stronger robot execution. This is a proposed combination, rather than a reported combined-system benchmark in the interview.

DexCap versus UMI

DexCap and UMI both use portable devices to collect human manipulation without requiring the robot itself; see Conversation with Cheng Chi and Zhenjia Xu: UMI, a Physical API for Robot Training. UMI uses handheld grippers, whereas DexCap uses gloves. Accurate 3D trajectories are difficult to recover without precise tracking, so both use SLAM. DexCap emphasizes collecting behavior through wearables while allowing people to continue their normal activities. A supermarket cashier, for instance, could wear gloves rather than hold two grippers. Grippers would interfere with some actions, such as handling coins.

The appeal of gloves is that they can fit into more activities. I can imagine wearing them while programming or playing the piano all day, which would be hard while holding two grippers. I want collection to extend into more people's everyday lives, much as driving produces large quantities of data. Tesla's driving data is abundant because driving is already part of daily life. Operating robots solely to collect demonstrations is not. Making collection easier and less disruptive is crucial: supermarket employees, quality inspectors and assembly-line workers could continue their work while wearing devices, gradually building a larger dataset. This is the ambition, not a claim that the present gloves impose no limits.

UMI-style approaches add a tool and can change the way people work, potentially requiring training and additional expense. I lean toward collecting existing human behavior and making robots more human-like. Although the two approaches can look similar, I see their starting points as different. UMI starts from the robot: let people collect immediately robot-relevant data without teleoperation. DexCap starts from the human: if future robot hands become more compatible with human hands, natural human demonstrations can scale up. That is the difference I want to emphasize.

5. Some thoughts

This is Episode 4 of Robot Data. At the time of writing, I plan a ten-episode first season of interviews about robot data, updated weekly over roughly two months. The early episodes focus on human video, motion capture and teleoperation; later ones will explore synthetic data, processing and datasets. Readers are welcome to recommend related work or suggest authors they would like me to interview.

References

https://mimic-play.github.io/

https://dex-cap.github.io/

Smart Note · Robot Data · EP04