Among the biggest breakthroughs in robot imitation learning over the past year, Diffusion Policy certainly ranks near the top. Six months ago, I had the pleasure of interviewing its author, Cheng Chi, for Diffusion Policy: Generating Robot Actions with Diffusion Models. In that interview, Chi explained that Diffusion Policy improves as more data becomes available. To make collecting that data easier, his team was already developing hardware for convenient manipulation-data collection anytime and anywhere. Not long after our interview, the paper Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots, or UMI, was released and accepted at Robotics: Science and Systems, RSS 2024.
In March 2024, Yihao invited me to interview UMI’s two co-first authors, Cheng Chi and Zhenjia Xu. With my robot training data series now underway, I have reorganized that conversation as its second installment for interested readers.
Chi and Xu pursued doctoral research at Columbia University, studying robot manipulation and perception in Shuran Song’s group. The Chinese introduction calls both of them PhD graduates; Chi’s February 2024 CV still identifies him as a PhD candidate. Let us explore how they enable people to collect the end-effector action data needed for robot training directly, without using a robot during collection.
Original article outline
Contents
👇
Introducing UMI
UMI’s Hardware Design
What Problem Does UMI Solve?
Is UMI a Teleoperation Device?
How Do You Use UMI?
How Well Do Policies Trained on UMI Data Perform?
How Does UMI Handle Manipulation That Requires Movement?
How Robust Are Policies Trained on UMI Data?
What Needs to Improve?
What Other Work Is Similar to UMI?
Additional Questions
Some Reflections
1. Introducing UMI
UMI’s Hardware Design
UMI’s data-collection hardware is a handheld parallel-jaw gripper equipped with a GoPro camera ①. To collect observations suitable for deployable policies, it needs sufficient visual context to infer actions ②, provided by a wide-angle fisheye camera, and depth information ③, provided by side mirrors for implicit stereo. To collect action data for deployable policies, it must accurately capture motion even during rapid human movements ④, using an IMU; record fine adjustments to grasp width ⑤, through continuously adjustable jaws; and automatically check whether each demonstration is valid under the target robot’s kinematic constraints ⑥, through kinematics-based data filtering.

What Problem Does UMI Solve?
There are two extremes in human-in-the-loop data collection for robotics. At one end is using YouTube videos, which requires no robot-related data. In principle, there is an enormous amount of such data online. The problem is that robot hardware is still immature, and robots used in research differ substantially from humans in their physical form. Even if we could extract human actions accurately from video, mapping them onto a robot would remain a difficult research problem without a particularly good solution today. Extracting accurate human actions from video is itself difficult. We can recover motions that look plausible, but real manipulation often requires millimeter-level precision. Internet videos contain many unknowns, so obtaining robot actions at that level of accuracy remains an unsolved problem.
That extreme ignores the robot’s embodiment. The other extreme is collecting data with exactly the robot you ultimately want to use. Its advantage is that there is no embodiment gap: the data you collect has the same inputs and outputs as the system at test time. The largest drawback is the robot’s high cost, which creates a chicken-and-egg problem. Without a good dataset, it is hard to build a good robot product to deploy in real environments. Without deployed robots, it is hard to obtain a good dataset. I think this problem has been around for many years.
UMI aims to find a useful balance between those extremes. At its core is a physical API: an interface whose physical form is sufficiently similar to a robot’s, yet is not tied to one particular embodiment and costs much less than a robot. I can explain this in three ways.
First is portability. Suppose robots were inexpensive—even just RMB 1,000 to 2,000 each. Given their current weight, collecting data in a restaurant would still mean loading the robot onto a truck, driving it there, unloading it, and installing and configuring it. Half a day could easily disappear. Every new environment brings this setup cost, making diverse environment data expensive to obtain. UMI’s handheld design addresses that problem well. Its gripper is still somewhat heavy at 800 grams, but it fits in a bag. Once you arrive somewhere—for example, when you sit down to eat in a restaurant—you can start collecting data within two minutes. This greatly simplifies collecting diverse real-world data.
Second is similarity of embodiment. Research robots in academia are often industrial arms fitted with parallel-jaw grippers, and UMI was designed with that common configuration in mind. It records the gripper’s six-degree-of-freedom trajectory through space. As long as the robot can reach the required pose, its specific joint angles do not matter; in principle, even a drone could move the end effector there. This provides a degree of independence from the robot’s embodiment. Whatever brand of robot a lab uses, it can work with the interface if it has the necessary six degrees of freedom. Technical clarification: six degrees of freedom alone do not ensure compatibility. Deployment also requires matching the reachable workspace, gripper, execution speed and sensing/control latency.
Third is intuitive data collection. One task we tried was tossing objects. There were round and square objects, and two bins in front of the robot, one round and one square. The task was to throw square objects into the square bin and round objects into the round bin. This is a fast, dynamic action that is very difficult to perform through teleoperation. In our earlier experiment, a teleoperated robot failed to land even one square object in 15 minutes. With UMI, people can use the gripper intuitively, collecting data at about half the speed of their bare hands. That may sound slow, but it is a substantial improvement over collecting no successful demonstrations at all. In another task, turning a cup, UMI collected data at half the speed of a human hand, while teleoperation was about three times slower than UMI. That gives UMI a significant data-collection advantage.
Is UMI a Teleoperation Device?
UMI is not a teleoperation device. Teleoperation requires a person to control a robot performing the task. UMI is not a robot: it has no motors and is entirely passive hardware. You can think of it as something like a glove, or simply a handheld data-collection device.
How Do You Use UMI?
Once assembled, UMI is straightforward to use: hold it in your hand and perform everyday manipulation tasks. Chi also has YouTube videos introducing the hardware; see the references at the end.
How to use UMI.
Collecting demonstrations for different tasks with UMI.
For the cup-arrangement task, the UMI gripper collects demonstrations more than three times faster than teleoperation, reaching 48% of human hand speed.
Comparing data-collection efficiency: UMI and teleoperation.
How Well Do Policies Trained on UMI Data Perform?
First, let me explain the UMI workflow. In our project, all the robot execution demonstrations shown in the videos are driven directly by the trained models and are fully autonomous. Human operators provide no intervention or guidance during execution. We train a separate model for each task. For dishwashing, for example, the model is dedicated to washing dishes: it knows how to perform that task, rather than other operations. The following videos show robots running policies trained on data collected with UMI.
Arranging cups.
Tossing objects.
Washing dishes.
Folding clothes.
Other projects running alongside ours explore multitask learning. They may collect demonstrations for multiple tasks with spoken labels, then use language instructions or images as labels to guide execution. For our current work, it is useful to distinguish two stages: training, and testing or inference.
During training, we need a dataset collected by human operators. Typically, we collect a few hundred demonstrations and use them to train a model. At inference time, when the robot performs the task, it is controlled entirely by that trained model. A human sets up the robot, places it appropriately and starts the task. After that, the operator does not intervene until the task ends or needs to be stopped, at which point they stop the robot.
Single-task training is how our current project works, but it is not a fundamental limitation. In principle, the framework supports multiple tasks. With limited academic resources, however, handling several tasks simultaneously is expensive, so we chose to focus on individual tasks. Our industrial collaborators, such as Toyota Research Institute, are developing models trained on multiple tasks together, with quite promising results. Industry generally has more funding and resources and can hire many people to collect data, which helps it make progress in multitask learning.
How Does UMI Handle Manipulation That Requires Movement?
If our objective is to complete a manipulation task and the robot’s base simply needs to move to make that possible, I think these data are sufficient. The objective is to keep the two hands at particular positions in space, from which we can work backward to determine where the base needs to be. We need to know the surrounding environment, move the base somewhere that lets both hands reach their targets, and avoid collisions. Classical methods can infer the required base position; this is an inverse-kinematics problem. Technical clarification: inverse kinematics addresses reachability; collision avoidance and safe base-motion planning remain additional requirements.
How Robust Are Policies Trained on UMI Data?
First, we use color augmentation during training. For the same data, we randomly change brightness, color temperature and saturation. The resulting model therefore has some robustness to changes in lighting.
Second, we found something interesting. All our observations come from images captured by the wrist-mounted camera. Those images are almost never static: whenever the hand moves, they change. Models trained this way turn out to be more robust to those disturbances. They focus on important information, such as the cup’s position, and ignore irrelevant details. I think this happens because the camera is moving throughout data collection, so the background differs from frame to frame. That movement itself acts as natural augmentation.
Our visual robustness therefore comes mainly from two sources: explicitly introduced color augmentation during training, and natural background variation during data collection.
What Needs to Improve?
First, we need to make the device easier to use. It is still quite heavy, and its center of gravity is not especially well balanced, so holding it during data collection is tiring. After about two hours, for example, you can feel very fatigued, making prolonged use difficult. The design could be more ergonomic.
Second, tracking still has shortcomings. We currently combine GoPro video with IMU data and SLAM-based tracking, but accuracy is not high in every setting. This places some demands on the person collecting data: they need to understand which behaviors help tracking remain accurate. If you simply give the device to someone and ask them to do anything at home, tracking may sometimes be unreliable.
There are several possible improvements. One is to improve SLAM accuracy overall. SLAM is a relatively mature field, but we are not specialists and cannot optimize it particularly well ourselves. Help from students or experts with deeper knowledge would be ideal. There are simpler possibilities too, such as switching to an iPhone. Some iPhone models have three cameras and even LiDAR, and ARKit tracking is very robust. That is another direction worth exploring.
A larger change would be supporting different end effectors. At present, the interface is limited to a two-finger parallel-jaw gripper. People often ask what happens if they use a five-finger dexterous hand, or a three-finger hand. The current device does not support those. Extending this idea to other end effectors would be meaningful, though also more difficult.
Further ahead is collecting multitask datasets. Right now, we collect data for one task and train one model. A genuinely general-purpose robot policy will certainly need a multitask dataset, with improvements in both data quality and diversity. Often, assembling such a dataset is beyond the resources available in academia.
2. Additional Questions
What Are the Main Routes to Collecting Robot Data?
Broadly, there are three routes. The first is collecting data in simulation. The second is letting robots collect their own data in the real world. The third is human teleoperation or manual data collection.
Simulation has a substantial hidden cost: building the simulated environment itself is expensive. In our earlier work, most of our time went into that construction. Roughly 80% of a project could be spent building the environment, leaving only 20% for algorithm research. A major part of that cost comes from physical accuracy and asset creation. There is no single physics model or fixed set of parameters that precisely captures every object’s physical properties, so extensive manual tuning is required.
The second route, robots collecting their own real-world data, includes self-supervised learning and reinforcement learning. Both involve robots exploring their environment and learning skills themselves. But both have large hidden costs. We may want a robot to learn independently, yet doing so requires an autonomous reset system. Imagine a robot turning a cup or washing dishes. If it accidentally breaks a plate, the system must be able to restore the task’s initial state automatically. Building that reset system is expensive, can be harder than the task being learned, and must be extremely reliable. Another hidden cost is reward engineering: using a mathematical formula to tell the robot which states are good and which are bad. Aligning mathematical rewards with human preferences is very difficult and time-consuming. Autonomous resets and reward engineering are therefore major obstacles to robots collecting their own real-world data. Technical clarification: reward engineering applies to reward-driven reinforcement learning here; self-supervised learning does not necessarily require a task reward.
People initially assumed that human teleoperation or manual data collection would be expensive. But once you account for the hidden costs of those other two routes, human data collection no longer looks so costly.
Complex Tasks
I do not think the current system can solve every task. Its two parallel-jaw grippers are a significant constraint for some operations. Humans have five fingers and can grasp certain tools better—a screwdriver, for example. Even with two parallel-jaw grippers, though, I think at least 70% to 80% of tasks could be accomplished. Task complexity therefore should not be a major near-term bottleneck. We may not have achieved even 5% of tasks yet, so the existing gripper technology can meet our needs for a long time to come. These percentages are my rough estimates, rather than measurements over a defined benchmark of all tasks.
Generalization
Whether a policy generalizes to other objects and settings depends largely on how its data was collected. Data from only one scene and one object produces limited generalization. With diverse collection—different instances of an object category in different places, such as red cups, blue cups, large cups and small cups—our experiments show that generalization can be quite strong. But that depends on collecting sufficiently diverse data.
Success Rates
In our experiments at the time, success rates of 80% to 90% are quite achievable. With enough data—say, 200 to 300 demonstrations for one setting and one object—we can reach that range. Multiple tasks and settings, with generalization to other objects, require more data. For our cup-turning task, for example, we collected more than 1,000 demonstrations and achieved a success rate above 80%. Insufficient data will of course lower the success rate.
This raises the question of whether robotics has something like a scaling law. Well-established scaling laws already exist for large models in natural-language processing. Given a model’s size and the amount of data, we can predict its capabilities without training it. Conversely, given a target capability, we can estimate the required compute, data and model size. Technical clarification: such laws primarily describe aggregate trends in prediction loss and resource requirements; they do not predict every specific capability exactly.
At the time of this 2024 interview, robotics research had not yet established such a scaling law. We can only measure a particular task’s success rate by running real experiments. We could not then predict success rates or required dataset sizes without doing those experiments. That is an area robotics research needs to explore further.
From our current perspective, the resources, capital and data required to study scaling laws in robotics may exceed the scope of academic research. Establishing laws comparable to those in natural-language processing would require substantial resources and data, which may be difficult to obtain within today’s academic environment.
Teleoperation
I want to clarify teleoperation because there are two very different uses: collecting data and controlling a robot in a demonstration. Our earlier discussion referred mainly to collecting data through teleoperation and then training a model. In the final demonstration, the robot executes autonomously, showing what the model can do.
In the other case, someone directly teleoperates the robot throughout the demonstration. That primarily reflects the hardware’s capabilities, rather than a model’s capabilities. There is no learned model executing the task in that case.
Can we tell from a video whether a robot is teleoperated or controlled by a model? Honestly, if the model was trained well through imitation learning or a similar method, it is very difficult to tell just by watching. We have to rely on the person releasing the video to be honest about whether the operation was teleoperated or model-generated.
Imitation Learning
Imitation learning is a broad concept, and behavior cloning is one subset. Behavior cloning is among its simplest methods. Given a dataset of paired observations and actions, we train a model to learn the conditional distribution of actions given observations, maximizing the conditional probability of the demonstrated actions. That is one form of imitation learning.
There are many other methods besides behavior cloning. DAgger, or Dataset Aggregation, is another widely used imitation-learning method that incorporates expert guidance. Its learning process is not static: it is not simply collecting a dataset, training a model and deploying it. Instead, it is an online process in which human experts provide additional guidance and data while the robot operates. This can improve learning performance and adaptability. Imitation learning therefore encompasses many methods, with behavior cloning being just one representative example.
How Can High-Level Planning Work with Diffusion Policy?
One approach is to train a language-conditioned diffusion policy. For example, while collecting data with the UMI gripper, I could use the GoPro to record audio and describe what I am doing as I collect images and actions. The data would then contain language descriptions as well. We could train a model that takes the current image and the description of what I want to do, and outputs the corresponding actions. This lets us combine it with a large language model at a higher level. If I want to make scrambled eggs with tomatoes, an LLM could break that request into subtasks: take eggs out of the refrigerator, crack them, then beat them. It could pass each subtask description to the language-conditioned diffusion policy for execution.
A second approach is a behavior library. We could train a separate model for each task: first open the refrigerator, then take out the eggs, crack them, and beat them. A language model or another model could schedule these behaviors, selecting which action model to execute. I think both approaches are feasible.
What Other Work Is Similar to UMI?
There have been some earlier handheld data-collection devices, though not many. Their main problem was insufficient motion accuracy. Earlier structured-light approaches could indicate the direction of motion but could not precisely recover absolute 3D pose: position and orientation. That involves six degrees of freedom—three translational and three rotational. Earlier approaches could not accurately obtain all six. Our combination of a GoPro camera and SLAM provides more precise six-degree-of-freedom tracking, which is one advantage over those earlier systems. This comparison concerns the earlier devices discussed in the interview, rather than a general limitation of all structured-light systems.
There are disadvantages too. For example, Cewu Lu’s group at Shanghai Jiao Tong University uses exoskeletons for data collection. Our approach does not account for the influence of the robot’s own structure, which can be either an advantage or a disadvantage. On the positive side, our data can be used with many kinds of robot arms rather than one particular design. On the negative side, it may not suit every application. If an elbow might collide with a table, for example, end-effector pose alone may not tell us whether that collision will occur. A data-collection device with a structure resembling the actual robot arm could help avoid that problem.
3. Some Reflections
I began following embodied AI in June 2023, exactly a year ago. Back then, I struggled to find my bearings and wanted to know who was actually working in the field. That led me to compile my widely circulated Map of Chinese Researchers in AI and Robotics. I then followed the work of the groups on that map, gradually developing a better understanding of their research directions and approaches. I have continued writing academically oriented articles over the past year partly because I am genuinely curious about how far the latest research has progressed. I also hope that accessible explanations help more people understand the field’s newest work, making a small contribution toward aligning the expectations and understanding of industry, investors and academia.
But honestly, the more deeply I explore the research frontier, the less certain I become about how to invest. I wonder whether fellow investors feel the same way!
[Emoji]
References
https://umi-gripper.github.io/
https://www.youtube.com/watch?v=EJmAg1Bnp-k&t=179s
https://www.youtube.com/watch?v=x3ko0v_xwpg