Smart Note has been updating more slowly over the past few months, and many readers have been asking for new articles. With only two months left in 2025, I will be staying up after work to keep writing. Following the first installment, Human Data EP01—MoCap: TWIST, we continue exploring human motion capture data and humanoid whole-body control with GMT: General Motion Tracking for Humanoid Whole-Body Control. GMT is a general motion tracking framework that trains a single policy on large-scale, unstructured human motion data to achieve high-fidelity motion tracking on humanoids.
GMT’s co-first authors are Zixuan Chen and Mazeyu Ji. Our guest is Zixuan Chen, a PhD student in Xiaolong Wang’s group at UC San Diego who earned his bachelor’s degree at Fudan University. He is also a co-first author of TWIST (Human Data EP01—MoCap: TWIST). This interview contains many illuminating details. Let’s take a closer look at the work.
Original article outline
Contents
👇
1. GMT’s Core Idea
2. The GMT Method
3. Understanding Motion Tracking
4. Motion Capture Datasets
5. Motion Capture Data vs. Video Data
6. GMT vs. DeepMimic
7. Mixture of Experts
8. Comparing Whole-Body Control Methods
9. Motion Retargeting
1. GMT’s Core Idea
A central question for GMT is how to achieve something genuinely general: a single policy that can produce many different movements and execute them on a robot.
Many companies demonstrate impressive movements such as dancing, somersaults, and jumping. Their usual approach, however, is to train a separate policy for each motion. That works for individual movements, but it is difficult to scale when we want to build a general-purpose controller.
GMT’s idea is that if one policy can cover and understand different motions, we can generate arbitrary motion sequences during task execution and have the policy track them, allowing the robot to exhibit different behaviors. It can then serve as a foundational controller for a wide range of tasks.
2. The GMT Method
GMT builds on DeepMimic, introduced by Jason Peng in 2018. At its core, it still uses reinforcement learning: we define rewards for completing motion tracking. GMT mainly addresses two categories of problems—those arising from the data and those arising from the model architecture.

The first data issue is training data quality. Training such a controller requires a large amount of data, typically from large human motion capture datasets such as AMASS and LAFAN. These datasets contain considerable noise. Some recordings are incomplete; others involve highly dynamic movements that require forces a robot cannot produce. We need to filter these out. We therefore curate the data, removing overly dynamic movements and low-quality recordings to obtain a better training dataset and improve policy performance.
The second issue is sampling. Some movements, such as walking, are relatively easy; others, such as dancing and jumping, are more complex and nonperiodic. In datasets such as AMASS or LAFAN, 60–70% of the recordings involve walking, while the remaining 30% contain more challenging movements. We want the policy to learn both easy and difficult movements, so we use adaptive sampling. Specifically, we assign probabilities to different motions, sampling complex movements more often and simple movements less often to balance training. I think this is an important point.

As for architecture, many previous approaches, especially in locomotion, use simple multilayer perceptron networks. But we need one policy to handle many different motions over long time horizons, and an MLP often is not sufficient. We therefore switched to a mixture-of-experts architecture with multiple experts, borrowing the idea from language models. We found that MoE substantially increased the model’s representational capacity and improved the results.
We also made changes to the motion inputs. People anticipate what they are about to do. We may know several seconds in advance which movement comes next, and adjust our bodies accordingly. Inspired by this, rather than inputting a single frame, we compress the motion targets for the next two seconds into a vector and feed it into the model. Compression, however, loses information and can reduce tracking accuracy. To address this, we explicitly provide the nearest target frame alongside the compressed future vector. Combining the two lets the model perceive the most important immediate tracking target while also understanding the future trend. The resulting policy performs better. GMT’s main improvements therefore concern both data and model design.
3. Understanding Motion Tracking
Motion tracking is simple to understand: we give the robot a sequence of human motion data, typically based on kinematics, and ask it to reproduce the movement. It can be online, tracking and reproducing a person’s movements in real time, as in TWIST. Or it can be offline, taking a recorded sequence and reproducing the whole movement.
For offline motion tracking, we give the robot a prerecorded motion sequence and input the entire recording at once. The main difference is that online tracking collects data in real time, so only the next frame is available to feed into the robot. Offline, the whole sequence is provided beforehand, allowing access to motion information at any future time. We can, for example, input a window of future movements, giving the robot a longer temporal horizon and a more comprehensive understanding of the sequence. Terminology note: in live teleoperation, the “next frame” is the latest received reference target; an unobserved future human movement is not available in advance.
4. Motion Capture Datasets
AMASS is large, with recordings totaling several dozen hours, but it contains many low-quality segments, including feet penetrating the ground or floating above it. Optimizing details such as ground contact could improve the overall results. For GMT, however, I think it is more important to filter out motions the robot cannot perform and train only on executable movements. That produces a more substantial improvement.
LAFAN’s movements are generally more dynamic. Some are difficult, but overall they are executable. Each motion is also relatively high quality. The creators manually adjusted some data during preprocessing, so its overall quality is quite good.
Among the open datasets available today, AMASS is the largest. We chose it because it covers a wide range of skills, although its quality may have some shortcomings. The idea we are pursuing is a foundation model at the behavioral level: a model that can execute all kinds of movements, including movements produced by generative models, and track them within the model. Achieving this may require a very large training dataset that covers as many kinds of movement as possible. A new, high-quality dataset collected from humans would, I think, substantially improve the overall results. Here “the largest” reflects the interviewee’s comparison of the motion-capture datasets under discussion, rather than a ranking of all human-data datasets.
5. Motion Capture Data vs. Video Data
I am not particularly inclined to rely exclusively on video data at this stage. With video alone, we need a video-to-motion model to convert it into usable motion data, because robots can directly use only the motion data. This conversion is difficult and imposes demanding requirements on camera angles and other conditions. Better camera angles may produce better converted results. In terms of accuracy, however, video data certainly falls short of motion capture. If the ultimate application is robotics, I think motion capture is preferable: it converts directly into data the robot can use and offers higher quality.
6. GMT vs. DeepMimic
I think DeepMimic’s most important contribution was formally framing motion tracking as a reinforcement learning problem. At each step, it provides a motion target and computes the difference between the robot’s current state and that target. This produces a reward that defines the reinforcement learning task. To me, that is DeepMimic’s greatest contribution.
Almost all current models, including PHC, MaskedMimic, ASAP, and the more recent GMT and TWIST, use the paradigm introduced by DeepMimic and build innovations on top of it.
Another DeepMimic innovation is random state initialization, or RSI. When the robot falls while performing a motion during training, we do not always restart from the beginning. Instead, we choose an arbitrary frame from the motion as the initial state. The robot can thus learn different stages of a movement without having to master the beginning before learning later parts. This significantly improves training. I think it is one of DeepMimic’s most influential innovations. Terminology note: DeepMimic calls this reference state initialization; the reference frame is sampled randomly.
7. Mixture of Experts
The MoE we use is actually a soft MoE. We train on all the data together, rather than using what I describe as the traditional MoE approach in large language models. In a traditional MoE, for example in a language model, the idea is typically to train many different experts, each responsible for a particular skill, and then combine them. Technical note: sparse MoE language models commonly train their experts and router jointly as well. The separate-training description above is an intuitive comparison in the interview, rather than a defining property of LLM MoE; the distinction here is the dense weighted combination used by GMT.
Soft MoE takes a different approach by adding a gating network. Imagine that we have six experts but do not know in advance what each one will do. The gating network outputs six probabilities representing their weights. We then compute a probability-weighted sum of the six experts’ outputs to obtain the final action.
The advantage is that we do not have to manually plan or predefine skills. Instead, the model learns for itself, discovering, categorizing, and combining different skills to complete a movement.
One intuitive inspiration came from ExBody2. Its authors found that grouping similar motions for training—for example, putting walking movements in one category and dancing or kicking in another—produced better policies. This suggested that different motion categories need different experts to truly master their respective skills.
Building on that idea, we ultimately chose soft MoE. It preserves end-to-end training while allowing different experts inside the model to automatically divide the work, specialize in different skills, and combine their capabilities to execute complex movements.
One feature of soft MoE is that even when it emphasizes one skill at a particular moment, it does not drive every other expert’s output to zero. Some experts might be better at kicking, while others are better at walking. When the robot kicks, the kicking expert receives a higher weight, but the other experts still contribute. Adding MoE produced a clear improvement in our experiments. It offers relatively little improvement for simple motions, but substantially improves overall performance on difficult movements.
8. Comparing Whole-Body Control Methods
First, it is useful to discuss PHC and MaskedMimic, which are oriented toward computer graphics. I think graphics and robotics differ fundamentally. Graphics relies heavily on global information and uses simpler simulation conditions, without the range of domain randomization used in robotics. As a result, these methods tend to depend on global information and privileged information, such as the robot’s linear velocity.
Choosing a motion tracking controller for graphics is simpler and less demanding. But the conditions and assumptions in graphics cannot be transferred directly to robots. I see our work as distinct from graphics work. Graphics more often validates algorithmic feasibility, while actual robot deployment requires additional engineering and design.
Among robotics projects, HumanPlus and OmniH2O train on the entire dataset, as both papers report. These were relatively early projects using Unitree H1. H1’s body differs substantially from a human body, leading to considerable quality differences after motion retargeting. Its configuration is also quite different from a person’s. These projects focus more on teleoperation than on developing a broadly compatible motion tracking controller. That is another difference from our work.
ExBody tracks the upper body while relying on a locomotion policy for the lower body, so it does not perform complete whole-body tracking. Although its policy controls the entire body, it does not actually track the whole body. ExBody2 was the first work to support a variety of genuinely whole-body motions, but it uses a smaller dataset. It trains separate policies for different motion categories to achieve good results, rather than covering every motion with one policy. ASAP demonstrates extreme skills such as jumping and tumbling, but each motion has its own policy, limiting its ability to scale. Its dataset is manually curated from internet data and open datasets.
GMT, by contrast, centers on a general policy that tracks many different movements. The motion data is curated, and the architecture uses MoE. Progress in whole-body control has been rapid. Recent breakthroughs include dynamic movements we might never have expected robots to perform, such as somersaults or getting up after falling. Many recent projects have done excellent work. These advances benefit from smaller robot bodies that can reproduce more movements, helping academic progress. But functionality remains insufficiently explored: demonstrating athletic ability does not directly translate into useful capabilities. Household or factory tasks require a general policy that supports multiple skills.
9. Motion Retargeting
GMT’s retargeting mainly builds on the method introduced in PHC. We focus on key points in the robot’s movements, such as the hands and feet, and have the corresponding robot body parts match those points as closely as possible. Solving an optimization problem then maps human motion onto the robot.
An important question is how to match a skill. Human skills are influenced by factors such as height and weight, so different people perform movements differently. We set coefficients for different joints to transfer a person’s skill onto the robot. This is a crucial part of the process.
Retargeting quality also depends on the robot’s configuration. Open datasets and motion capture recordings are based on people, so in theory, the more humanlike the robot, the better the retargeting should work. We clearly observed this in practice: retargeting works much better on G1 than on H1, because G1 is more humanlike.
References
https://gmt-humanoid.github.io/