Retargeting is a crucial step in using human data. Earlier interviews introduced several approaches, including PHC in Interview with Zhengyi Luo: Inside PHC, a General-Purpose Humanoid Controller, and GMR in Human Data EP01—MoCap: TWIST.
Retargeting is the process of converting the movements of one subject—usually a person—into movements that another subject, usually a robot or virtual character, can execute. Put simply, it transfers human movement to a robot.
This bonus installment of the Human Data series introduces an excellent project on humanoid motion retargeting: OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh. First, take a look at the results. All robot videos run in real time. The corresponding RL policies are trained with the same five reward terms and four domain randomization terms across tasks, and rely entirely on proprioceptive inputs.
Wall Flip
Flagship Demo
Traditional retargeting can produce unnatural robot movements and often causes geometric interpenetration. OmniRetarget's results look more natural overall, preserve interactions well, and explicitly enforce hard constraints. It also supports data augmentation, reducing dependence on the number of original motion capture demonstrations—something earlier retargeting approaches struggled to do.
This interview is with Lujie Yang, one of OmniRetarget's co-first authors. She earned her PhD at MIT under Professor Russ Tedrake and her bachelor's degree at UC Berkeley, where she worked with Professor Francesco Borrelli. Her research combines machine learning, optimization, and control theory for robot planning and control. She develops methods that draw on both data-driven and model-based optimization to improve performance and reliability in tasks such as dexterous manipulation and complex contact.
If you are interested in humanoid whole-body control and retargeting, join us to explore the thinking behind OmniRetarget.
Original article outline
Contents
👇
1. The Core Idea of OmniRetarget
2. Understanding the Interaction Mesh
3. What Is Wrong with Earlier Retargeting Approaches?
4. How OmniRetarget Works
5. Data Sources
6. Model-Based and Learning-Based Methods
7. The Model-Based Principles behind OmniRetarget
8. Improvements over Traditional Retargeting
9. Why Traditional Retargeting Struggles with Data Augmentation
10. Which Movements Cannot Be Obtained through Augmentation?
11. Results and Limitations
12. Advancing Whole-Body Control with Richer Interactions
13. Does the Motion Capture Performer Affect Retargeting?
14. An Ideal Robot Foundation Model
15. PhysicsGen
16. A Research Vision
17. Preferences among Data Types
1. The Core Idea of OmniRetarget
I think it helps to begin with how I started the project. Initially, I wanted to extend an earlier project, PhysicsGen, which generated data for dexterous manipulation. A natural question was whether the same approach could generate data for humanoids.
The first step was kinematic retargeting: leave dynamics aside initially and convert human data into robot data at the kinematic level.
I then found many problems in earlier retargeting methods. Feet could pass through the ground or slide; the resulting movements were sometimes unnatural or insufficiently smooth; and it was difficult to impose hard constraints. I realized this was a substantial problem. But people working on RL were not resolving it at its source. Instead, they kept adding reward engineering to compensate for problems introduced at the retargeting stage.
I felt this was an important part of the pipeline that needed a genuinely good retargeter. That was essentially the motivation for the project.
We also found that a common retargeting approach was global keypoint matching. If a person's hand is at a particular position in world coordinates, the robot moves its hand to the corresponding position. This causes problems when a humanoid's morphology differs substantially from a person's. Robots are often shorter and have shorter arms. We cannot simply scale down all human positions or use their global positions to tell the robot where to move.
We wanted a more principled approach that preserved relative spatial relationships and contact information. We found an idea from computer graphics: an interaction mesh. We sample keypoints on the human and robot body surfaces—arms, wrists, heels, head, shoulders, and so on—and on the object. Retargeting then preserves their relative spatial relationships.
For example, if the person's hand is close to the object, we want the robot's hand to remain close to it too. But the robot's joint angles might be entirely different from the person's. That is one of the innovations in our approach.
2. Understanding the Interaction Mesh
You can think of an interaction mesh as a net. I care not only about the position of an individual point, whether on the robot or the object, but about the structure of the net: the edges connecting those points. They represent relative spatial relationships, including whether points lie in front of one another, occlude one another, or span across something, as illustrated below.

I need to make sure the green point remains in front of the yellow point, and that the relative spatial relationships are as consistent as possible between the human and robot. Since the robot is much shorter, the mesh stretches. Those relative relationships are therefore scaled or compressed in adapting to the new body.

The sampled points on the human and robot are not especially dense. In the animation, blue points are sampled on their bodies and green points on the object. Human body points include the left and right shoulders, elbows, palms, several points around the pelvis, knees, heels, and toes.
If we care more about the correspondence of certain points or body regions, we can sample those regions more densely. Their relative positions are then easier to preserve during retargeting.
3. What Is Wrong with Earlier Retargeting Approaches?
The retargeting steps used with PHC, discussed in Interview with Zhengyi Luo: Inside PHC, a General-Purpose Humanoid Controller, or GMR, discussed in Human Data EP01—MoCap: TWIST, are intuitive: broadly speaking, they establish point-to-point correspondences using positions in an absolute coordinate frame. Technical note: This comparison concerns interaction preservation. GMR also uses orientation constraints; it is not limited to position-only point matching.
These traditional approaches work somewhat better online, where a person closes the feedback loop between the robot and object. Offline, problems become more apparent. A demonstrator might be 1.8 meters tall, while a robot such as G1 is around 1.3 meters. Keeping the retargeted demonstration reasonable despite that substantial height difference is challenging.
An obvious approach is to scale the person's movement from a 1.8-meter body to a 1.3-meter body. But if the human hand touches the edge of a box, scaling its position can place the hand inside the box. The relative relationship among the hand, wrist, and object is lost.
The usual remedy is to keep adding reward terms during subsequent RL training to “hack” around the problem. It is not so much fully correcting the issue as using rewards to cover up or compensate for artifacts left by retargeting.
I feel retargeting has often been treated a little casually, and the quality has not been very high. For locomotion, reward engineering or reward hacking can patch things enough to make the robot walk and jump despite imperfect retargeting.
But more complex, difficult tasks expose the problems. Some policies produce jittery or unstable behavior, or strange movements. These are signs that the issue needs to be solved at its source, rather than concealed by downstream rewards.
4. How OmniRetarget Works

We begin with human demonstrations and retarget them into a robot trajectory that satisfies kinematic constraints.
From there, we can augment the data by changing an object's size or pose, or substituting another robot model. Retargeting generates the corresponding data and yields many kinematically feasible trajectories. In most cases, however, these trajectories are not yet dynamically feasible: they cannot necessarily be executed under the robot's actual dynamics.
Next, we need to turn them into dynamically feasible trajectories, or train policies from them. We use imitation-guided reinforcement learning, a method widely used in locomotion. A reference trajectory guides imitation while the physics simulator enforces the dynamics during RL training.
DeepMimic helped establish this paradigm, which has become especially common in recent locomotion work. My collaborator Xiaoyu Huang is one of BeyondMimic's first authors, so we adopted BeyondMimic's complete RL pipeline and extended it with additional object and environment interaction modeling.
We then deploy the policies directly on physical robots. No real robot data or teleoperation data is used in training.
5. Data Sources
Our main sources are two motion capture datasets, LAFAN1 and OMOMO, plus motion capture data we collected ourselves. One collaborator also recorded many demonstrations personally, such as climbing onto boxes of different sizes. Those recordings form part of our dataset too.
Open-source datasets contain relatively few demonstrations of complex environmental interactions. In our final example, the robot picks up a chair, steps onto it, uses it as support to climb onto a higher platform, then jumps down. Data combining object and environment interaction in this way is very scarce in public datasets.
6. Model-Based and Learning-Based Methods
I think both have strengths and weaknesses. Model-based methods rely on explicit models and mathematical principles. They can solve some problems more efficiently and with better quality than learning methods—for example, inverse kinematics or optimization with equality constraints, what I refer to here as an EC formulation.
Their limitations are also clear: they require accurate models and good state estimation. Some systems, such as fluids, deformable objects, and clothing, are difficult to describe accurately in mathematical form.
Learning-based methods, especially deep learning, are comparatively flexible and powerful. They can operate end to end, potentially producing joint torques directly from images without a separate explicit state estimation module.
But they have two limitations: they can use computation inefficiently, and they require enormous amounts of data.
For me, the better approach is to combine the strengths of both. Use model-based methods to generate data, as in my retargeting and data generation work, then use learning to turn that data into deployable policies that are more flexible and respond faster.

7. The Model-Based Principles behind OmniRetarget
OmniRetarget's approach benefits from a background in model-based methods and optimization. When I add constraints, I want them genuinely satisfied: hard constraints should be explicitly enforced. I also want the optimization problem to be clearly defined and cleanly structured, ideally with some guarantee of optimality or convergence.
In practice, that is difficult because the problem is nonconvex, so strict guarantees of optimality or convergence are hard to provide. We use a convex approximation. Across the cases we encountered, solving it was stable and robust, and the process itself was more principled. This lets us enforce equality and inequality constraints as hard constraints.
Examples include preventing foot sliding, respecting each joint's upper and lower limits, and preventing collisions. A common learning-based approach instead converts these requirements into soft penalties in the overall cost function, then optimizes with methods such as stochastic gradient descent.
I prefer to handle them as actual hard constraints and approximate the nonconvex problem with a convex one. First, that avoids tuning many penalty coefficients. Second, it makes solving more controllable: the solution must respect the imposed constraints. Often those constraints are essential.
Here, we are not focusing on the object's dynamics, but on the robot's own kinematic rules. One of the equalities in the final formulation says that the robot's stance foot must not slide.

The left side is the foot's position at time t, and the right side its position at time t−1. We enforce equality so that the position remains unchanged while the foot is in stance.
A learning approach might simply put the squared difference, (P_t − P_(t−1))², into the cost as a soft constraint. That simplifies optimization, but does not guarantee P_t = P_(t−1), Equation (3e). This property matters because it determines whether the reference stance foot slides during walking.
Our method treats that equality as a genuine hard constraint and enforces it, preventing foot skating in the retargeted stance phase. This is a constraint on the kinematic reference trajectory, rather than an unconditional guarantee against slipping on real hardware.
The cost function becomes even more useful in interaction scenarios. We construct an interaction mesh—a network of relationships—and seek to keep the mesh formed by the human and object close to the one formed by the robot and object. The cost expresses that correspondence.
But problems can arise easily, such as a hand passing through an object. One example is the result of directly applying global keypoint matching. Human keypoints are compressed to the robot's size, and the hand can end up inside the object.
Ideally, we want no interpenetration between robot body parts or between the robot and object. We therefore add hard nonpenetration constraints covering both. [Editorial note: the interview cites Equation (3a), but the constraint appears as (3b) in the displayed formulation; (3a) is the optimization objective.]
8. Improvements over Traditional Retargeting
The paper includes quantitative analysis, and the qualitative difference is also easy to see. The example on the left shows the retargeting used with PHC. It is not a carefully cherry-picked case: unnatural movements and interpenetration are common. Our result on the right looks more natural, preserves interactions, and explicitly respects the hard constraints.

Comparison of OmniRetarget and the retargeting used in GMR
Comparison of OmniRetarget and the retargeting used with PHC
Our method also supports data augmentation, which earlier approaches struggled to do. From one human demonstration, OmniRetarget systematically generates variations in (a) terrain height, (b) initial object pose, and (c) object shape. The motions optimized in simulation, shown above, transfer consistently to the physical hardware, shown below.

9. Why Traditional Retargeting Struggles with Data Augmentation
Earlier retargeting methods do not preserve the relative relationship between the human and object, which creates problems. In this example, the first sequence is the original demonstration and the others are augmented results. Our method preserves the relationship between the robot's upper body and the object, so the retargeted movement includes twisting left to hold it.
If only absolute positions are used, the robot still reaches forward to embrace the box even after the box's starting position changes. It does not adjust its posture to actually pick it up, and may be unable to lift it at all.
In online use of methods such as GMR and TWIST, a person closes the interaction loop, deciding how the robot should interact with objects and the environment. That can enable some loco-manipulation. Offline retargeting must account for more factors and is more complex. [Editorial note: the original also lists “visual Mimic” as an online method. If this refers to VideoMimic, that project instead reconstructs human motion and scenes from recorded monocular video; it is not live human teleoperation.]
10. Which Movements Cannot Be Obtained through Augmentation?
Very different movement styles remain difficult. To cross a platform, one person might support themselves with their hands, another step straight over, and another briefly place a knee on it. When the movement patterns differ that much, we still need some original demonstrations of those styles.
Variation within an object category is usually manageable. Cups might be wider or narrower, or have slightly different handles. Such within-category generalization is comparatively easy. Replacing a cup with a rectangular box, however, is a change across categories and is more challenging.
11. Results and Limitations
I think the results are already surprisingly good. Colleagues have directly applied the method to challenging objects, such as lamps, and generally found it usable without major changes. OmniRetarget's core contributions are, first, preserving environmental interactions and producing high-quality retargeted trajectories—for example, without foot skating or interpenetration—and, second, enabling effective data augmentation.
There are still limitations. If an augmented object's position or dimensions differ too much from the original, artifacts can appear. For example, enlarging an object from around 50 centimeters to a box roughly one meter on each side can sometimes make the robot appear to float. The system tries to preserve its relative relationship to the now much larger object, but cannot simultaneously preserve the robot's distance to the ground well enough.
Data augmentation cannot therefore be extended without limit. The method is not fully reliable under arbitrary changes in scale.
We plan to open-source the system. At the time of this interview, the official release was scheduled for December 1. I also intend to release my own implementation later. There will be two versions because the official team reimplemented my method in MuJoCo, whereas I developed it in Drake. Their main difference is the underlying computational engine, including the server-side implementation and the engine used for calculations such as Jacobians and other dynamics-related quantities.
12. Advancing Whole-Body Control with Richer Interactions
First, it can make data collection much faster. One demonstration in a scene can generate trajectories with different sizes, shapes, heights, and orientations. We do not need to record each variation separately, which would be time-consuming. Data generation can run in parallel across many CPUs and is therefore highly efficient.
Second, better reference trajectories make downstream RL converge faster and substantially improve success rates. The figure here illustrates the retargeting quality issues we discussed.

We measure interpenetration, foot skating, and how well relative positions are preserved. The blue results correspond to OmniRetarget. It has the least penetration, the least foot skating, and the best preservation of relative relationships among the methods shown. During RL training, we found that these three metrics strongly affected final performance. Our downstream RL tasks consequently achieved higher success rates.
13. Does the Motion Capture Performer Affect Retargeting?
Essentially, I only need to know the demonstrator’s height. That is all the information I need. Whether they are leaner or heavier, and their body proportions, do not otherwise affect the retargeting result in our setup.
14. An Ideal Robot Foundation Model
Let me describe the data generation system I would ideally like to build. My goal is a fairly general framework that supports manipulation, locomotion, and loco-manipulation alike.

Ideally, it should support every kind of upstream data: teleoperation, motion capture, and video. In terms of quality, mocap and teleoperation are generally more reliable than internet video. Reconstructing a three-dimensional person and object from one two-dimensional image, or a video with two spatial dimensions plus time, is itself very challenging.
Internet video, however, is plentiful, widely distributed, and diverse, and often carries richer semantic information. Tutorials, for example, show step by step how to fold boxes, fold clothing, and organize objects. The content is very rich, so in the long term we certainly want to use all these data sources.
I want the downstream system to be open-ended. It could support behavior cloning, letting a robot imitate things it has “experienced,” or use RL in a sufficiently capable simulator to learn how to exploit the data. Ultimately, it should yield a general robot foundation model supporting many tasks.
15. PhysicsGen
PhysicsGen is a data generation method I developed earlier for robotic arm manipulation. Its overall idea is close to OmniRetarget: start by collecting demonstrations.
The example shows VR data I collected while wearing Vision Pro. I simulated a small block in VR and demonstrated movements through hand gestures, then retargeted those demonstrations to two robotic arms. Interestingly, I manipulated the virtual block with two fingers, yet could retarget that movement to two much larger arms. The VR block was about 6 centimeters, while the physical box might be 30 centimeters, so the difference in scale was substantial.
We can then augment various physical parameters and test robots with different morphologies.
Downstream, I trained a policy using behavior cloning. The arms' final movements are much larger than the original demonstrations I recorded in VR. At that time, I was still using fairly basic global keypoint matching, so the method was simpler.
The key is that, before trajectory optimization, I added dynamics augmentation as well as kinematic retargeting. Trajectory optimization could then explicitly handle changes in box friction, dimensions, and mass. For example, an original demonstration with a light, 100-gram object could be augmented to generate a policy that successfully carried a 500-gram box.
16. A Research Vision
Let me outline my broader vision. Much of my earlier research focused on lower-level control—dexterity and agility. Today, attention often centers on general robotics, cognition, stronger generalization, and higher-level task composition and understanding.

In my view, there is still considerable room to improve low-level control. To reach the next generation of robot foundation models, we need both low-level dexterity and agility and high-level task understanding and cognition.
Both must be included, and they must interact and work together. Only then can robots be dexterous and agile while also being intelligent and stable.
In other words, we need to work on cognition and dexterity/agility together, letting intelligence flow naturally between those levels.
17. Preferences among Data Types
I have some personal reservations about teleoperation. Of course, real physical data is necessary, but teleoperation still has problems: latency, differences in human and robot morphology, and missing tactile feedback. The operator cannot simultaneously feel what the robot feels, so a gap remains between them. The operator's own movements can also be shaky, producing jitter in recorded trajectories and in the resulting policy.
Adding optimization to those teleoperation trajectories might improve their quality. Another limitation is that some tasks are not practical to demonstrate by teleoperation—directly teleoperating a robot through a backflip, for example. If we want certain superhuman capabilities, we need methods beyond teleoperation.
Personally, I prefer human data. There is a great deal of it, and human movements tend to be more natural.
Consider autonomous driving as a comparison. Cars were designed to be driven by people, so human driving data naturally fits driving models. Robots, by contrast, are not primarily designed to be operated by people; they are designed to do tasks people perform in everyday life. If people rarely do something, the practical value of dedicating a robot to it may also be limited.
For tasks people can perform skillfully, human demonstration data is therefore one of the most valuable resources for robots.
References
Lujie Yang's website
OmniRetarget
https://omniretarget.github.io/
PhysicsGen
https://lujieyang.github.io/physicsgen/