Welcome to the third episode of Human Data. We continue with ExBody2, another project that uses human motion capture data for whole-body robot control. Its joint first authors are Mazeyu Ji and Xuanbin Peng. Our interviewee is Mazeyu Ji, who earned his master's degree at UC San Diego under Xiaolong Wang and completed his undergraduate studies at Tongji University. He is also a joint first author of GMT, the project covered in our previous episode, Human Data EP02—Motion Capture: GMT.
My original aim for Human Data was to interview the authors of several papers on humanoid whole-body control using human motion data, and understand their views on the data available today. Beyond the methods and principal contributions of each paper, the details of these interviews deserve close attention: future trends may be hiding in the points they share. My conversation with Mazeyu Ji was full of such thought-provoking details. Let's take a look.
Original article outline
Contents
👇
1. The ExBody2 method
2. Retargeting
3. Motion capture training data
4. Policy fine-tuning
5. The sim-to-real gap
6. Future improvements and iterations
7. Enabling robots to perform tasks autonomously
1. The ExBody2 method
The biggest difference between ExBody2 and ExBody1 is that ExBody1 separates upper- and lower-body control. The upper body mimics motion, while the lower body does not track a specific motion; it follows simple commands, such as moving forward or backward. ExBody2, by contrast, coordinates the entire body through whole-body motion tracking.
A major reason ExBody1 did not perform whole-body imitation was that training it directly in a single stage was too difficult. ExBody2 improves on this with a teacher–student mechanism. The central idea is that a model can access the robot's complete state in simulation, while the same information is often difficult to obtain in the real world.
For example, a simulated robot can directly read key motion information such as its absolute root velocity. Obtaining that information in reality is very difficult. It requires additional sensors or a vision system, together with estimates from an IMU, and there is still no mature, reliably stable solution. We therefore use a two-stage training strategy to bridge this sim-to-real difference. Scope note: the difficulty described here is meeting this project’s deployment requirements; state estimation is an established robotics field, and the statement is not a claim that reliable estimators never exist.
Stage one: training the teacher
At this stage, the teacher can access the robot's full state in simulation, including information that cannot be directly observed in reality. Reinforcement learning trains it from reward signals to track motion stably and accurately in simulation. With complete information, the teacher learns a very precise policy. But that policy cannot be deployed directly on a real robot, which does not have the same inputs.
Stage two: training the student
The student is trained with supervised learning rather than directly with reinforcement learning. The teacher supplies an ideal action for each state as ground truth. During training, the student adjusts its outputs against these targets so that its behavior comes as close as possible to the teacher's.
Because the student cannot access all the information available to the teacher, we typically give it a longer history of observations to compensate. Through its neural network, the student learns to infer hidden variables from changes in its own historical state. This allows the student policy to be deployed directly on a real robot for stable whole-body control.
This two-stage teacher–student approach is very common in locomotion research. Both vision-based tasks and motion control tasks use it widely to reduce the gap between simulation and reality.

2. Retargeting
For whole-body motion tracking, most of the data I use comes from open-source motion capture datasets. Because it was recorded from human bodies, a robot cannot use it directly. We need to process it, and the most important step is mapping human motion onto the robot's body structure. This is called motion retargeting.
Retargeting involves many technical details. Even after retargeting, the robot may not reproduce a human motion perfectly because its dimensions, joint configuration and hardware capabilities differ substantially from a person's. If its hardware performance is limited, it may fail to execute a motion even when the retargeted motion is reasonable.
When training for real robots, I initially tried teaching all the motions at once. For example, we could run thousands of simulated robots simultaneously, assign a different motion to each, collect their data and train jointly, hoping to quickly learn any motion. In practice, this all-at-once approach was extremely difficult, and the model struggled to converge.
In ExBody2, we later adopted a more systematic approach: a progression of motion difficulty. The idea was to teach simple movements first, then gradually increase complexity. This easy-to-hard training path helped the robot learn motion control more stably. But we also found a problem: as the model learned harder movements, it tended to forget the simpler ones it had already learned. This is catastrophic forgetting.
To address that, we refined the training strategy. We first used a preliminary policy to rank all the motions by difficulty and divide them into complexity levels. During training, we then selected only motions within a certain difficulty range. Experiments showed that this improved performance within the chosen range while preserving some generalization to harder movements. Overall tracking accuracy even exceeded the results of directly training on all motions mixed together.
To further improve complex motion execution, we proposed a hierarchical strategy. First, relatively simple motions train a general policy. Then, harder motions are divided into smaller subsets, each receiving specialized training. This maintains good control across motions of different types and difficulties. When the subsets are sufficiently fine-grained, the robot can track and reproduce almost every individual motion stably.
3. Motion capture training data
We currently use AMASS. In my view, this dataset affects our work primarily in two ways: motion quality and the balance of the motion distribution.
Data quality is the most basic prerequisite. AMASS performs very well in simulation. As Zhengyi Luo and his collaborators demonstrated in earlier work such as PHC and UHC, almost all AMASS motions can be reproduced accurately in a simulator. In other words, purely in terms of motion accuracy, AMASS is not a major problem. Its skeletal annotations, keypoint precision and motion smoothness are all of a high standard.
The central issue affecting training is imbalance between motion types. AMASS has a very uneven distribution. It contains a great deal of walking: in a collection of more than ten thousand sequences, perhaps eight thousand are walking or similar basic movements, while difficult or diverse motions are much rarer. During training, the model consequently spends more time repeatedly learning simple patterns such as walking and running. The share devoted to complex movements is greatly reduced, limiting generalization.
I think balancing motion types is one of the keys to successful training. Even with high-quality data, an overly concentrated category distribution introduces bias. PHC, for example, pays close attention to the proportions of motion samples during training. A carefully designed data sampling strategy gives the robot enough opportunity to learn each category, allowing it to master nearly the full range of motions.
In the longer term, we still lack sufficiently diverse motion capture data. Most existing data describes movements of the person alone. To give humanoids intelligent behavior, we also need data on human–object and human–scene interaction. This would let models understand the intention and goal of a movement rather than merely imitate its trajectory. Richer motion capture datasets with more semantic structure should be a key to improving humanoid control further.
4. Policy fine-tuning
In specialist policy fine-tuning, we take the general policy and adapt it specifically to smaller groups of motions.
The basic reasoning is that restricting the model to a smaller motion subset generally produces better results than training directly on a large mixture of complex, diverse motions, regardless of exactly how the subset is defined. Narrowing the scope and increasing motion similarity makes training more stable and efficient.
In our implementation, we put similar or related movements into the same group. One group might contain different kinds of walking, turning and running; another might contain harder motions such as jumping, rolling and spinning. The more similar the dynamics and coordination patterns within a group, the better the resulting policy. Mixing very different actions, such as walking and somersaults, creates greater gradient conflicts and interference during optimization.
Our early grouping method was straightforward. We manually labeled broad motion categories, placing running and walking in one group and jumping and spinning in another, for example. We then fine-tuned separately for each group. Despite its simplicity, this gave clear improvements: each specialist policy achieved better tracking accuracy and stability in its own motion category.
We are also exploring more automated grouping strategies, hoping to use clustering or feature similarity to identify dynamical patterns and divide motions into groups automatically. But for ExBody2, manual annotation and direct category assignment already noticeably improved control for specific motions.
5. The sim-to-real gap
The sim-to-real gap depends entirely on the robot's condition, especially its motors. As motors age, the gap grows, and that is unavoidable. A motor's performance declines with long use. It may even behave well when the robot first starts, then heat up during operation and change its characteristics: speed response, friction and damping can all become different. These changes cannot be completely optimized away in simulation.
Essentially, today's gap is a motor gap. Adaptively adjusting motor parameters while the robot operates is almost impossible. Without very specialized sensors that monitor the motor's state in real time, dynamic correction cannot be achieved. These comments describe the interviewee’s experience with the tested hardware. Sim-to-real gaps can also arise from contact, friction, latency, sensing and other modeling differences; the feasibility of online adaptation depends on the system.
When optimizing sim-to-real transfer, researchers generally model motor parameters. Some use system identification to fit the real parameters; others use neural networks to learn nonlinear motor characteristics. This reproduces the motor's real behavior in simulation as closely as possible and narrows the gap, but the difference can never be eliminated completely.
Sometimes sim-to-real performance involves a little luck. But a sufficiently good policy greatly reduces the uncertainty. In our experience, the largest gaps occur at the ankles, followed by the waist, with the ankle problem being especially pronounced. Ankle errors are large because the loading is complex and directly affects balance and landing. If a policy depends heavily on ankle movements, changes in motor characteristics or friction can make the robot unstable. A robust policy, by contrast, makes landings smoother and distributes forces more evenly, rather than relying on small ankle adjustments for balance. Then the robot can remain stable even with some ankle error.
Good real-world performance ultimately depends on several factors together: a robust policy, reliable motors and other hardware, and adequate modeling of the dynamical differences between simulation and reality. These must work well together for successful sim-to-real transfer.
6. Future improvements and iterations
Having completed the project, several directions seem particularly important for further iteration and improvement.
First, our ExBody2 experiments showed that balance is crucial to robot motion control. GMT, covered in Human Data EP02—Motion Capture: GMT, directly builds on ExBody2 and addresses this with a better sampling method. Balancing data across categories during training lets the robot cover many motion types more stably. In this way, it can perform a wide range of movements in most situations without separately fine-tuning each group, provided the situation is not particularly extreme.
Second, we want a single policy to adapt automatically to different motion groups. In its final stage, ExBody2 divides the data into groups, each of which trains well, but we did not yet have a mechanism to merge them. Some new research now tries to let a single network switch automatically among groups without explicitly distinguishing motion categories.
The idea is to supervise a new unified policy with the training results from each group. The network learns to transition naturally between motions, without manually specified switching logic. A single policy could then handle motion tracking across the full range.
Many new projects are pursuing this direction, using a network to adaptively combine control policies for multiple motion types and further improve generalization and stability.
The next question is what the robot can do after learning to track all these motions. The field is moving toward demonstrating stable tracking across different motion types, beyond purely imitative tasks such as making a robot dance.
More importantly, the next research focus is how the robot can use these movements autonomously in different situations. Once it has a complete set of basic motor skills, the question is no longer whether it can perform a movement, but which movement it should perform in a given context. We need to explore how to prompt it to independently select and combine motions at the right time and in the right setting.
This will be a central future direction: enabling robots not only to imitate movements, but to understand their intent and act according to goals and environments in real task settings.
7. Enabling robots to perform tasks autonomously
By activating skills, we mean teaching a robot that already has many motor skills to call on them in appropriate situations to achieve a goal or complete a task. The ideal end state is similar to manipulation. Imagine an embodied agent with a vision system—a camera observing its surroundings. When I tell it to pick up a cup on the table, walk three steps forward or perform a dance, it understands the instruction and executes it with the most appropriate body movements.
There are broadly two approaches to activating these capabilities.
The first is to generate motions directly with a generative model, such as a diffusion model.
This is straightforward: supply text or a task description, such as walking three steps forward and then taking two steps to the right, or performing a dance, and the model directly generates a continuous whole-body motion sequence. Since whole-body tracking already works, the robot can follow the generated movements naturally. Many projects are exploring this. Systems such as GMT include examples of generating robot motion trajectories directly from language.
This has clear limitations, however. It is better suited to tasks without complex interaction, such as dancing, navigation and movement demonstrations. For interacting with objects and completing specific tasks, motion generation alone is insufficient because it lacks environmental feedback and task constraints.
The second approach is latent skill activation.
Rather than directly generating an entire motion sequence, this approach encodes all of the robot's skills in a latent space. Each vector represents a skill, such as walking, crouching, grasping, carrying or putting something down. To make the robot perform a task, we sample or select the corresponding skill vectors from that space and activate them in conjunction with the task input.
Suppose the task is to move an object from one place to another. The robot already has the necessary skills: approach it, crouch, lift it with both hands, stand up, walk and put it down. Activating these vectors in the latent space and coordinating them with a high-level model lets the robot compose a complete motion sequence for the carrying task.
During training, task rewards can also be incorporated, for example through reinforcement learning. The robot can then do more than activate existing skills: it can dynamically adjust its movements according to the task goal. It reuses its existing control capabilities while learning higher-level behavioral planning through rewards.
Put simply, the first approach goes from text to motion; the second goes from skills to tasks. The former is closer to being told how to do something. In the latter, the robot is told what to achieve and decides how to do it. That is a direction the whole field is now exploring.