Back to Human Data

Human Data · EP06

Human Data EP06—Motion Capture and Video: GAE

Yuefan Wang discusses General Action Expert, scaling human motion data, supporting different control inputs and building a path from teleoperation to autonomy.

Zha Dami2025-11-0713 min readWeChat original
Contents

Before China's National Day holiday in 2025, the embodied AI community was impressed by a video of a person teleoperating a humanoid while wearing Noitom's inertial motion capture equipment. The latency was low, the movements diverse and the execution impressive. Following three interviews on motion capture—Human Data EP01—Motion Capture: TWIST, Human Data EP02—Motion Capture: GMT and Human Data EP03—Motion Capture: ExBody2—and two on video—Human Data EP04—Video: VideoMimic and Human Data EP05—Video: HUB—we also interviewed the team behind GAE, General Action Expert, a system for giving a person a robotic avatar.

Video demonstration from the original article

GAE demonstration, September 2025

A main contributor to GAE is Yuefan Wang, a PhD student in Donglin Wang's group at Westlake University. He completed his undergraduate degree at Harbin Institute of Technology. Let's hear what he learned while working on GAE.

Original article outline

Contents

👇

1. GAE's core idea

2. The GAE method

3. Data sources

4. Video data

5. The balance between data modalities

6. Which data sources dominate different training stages?

7. Different teleoperation interfaces

8. Is GAE a general-purpose whole-body controller?

9. Fine-grained tweaking or scaling?

10. The scale of our own motion capture data

11. Success rates

12. The timeline from teleoperation to autonomy

13. Future directions

1. GAE's core idea

From the outset, we wanted to make humanoid motion control general-purpose. When the project began in 2024, our aim was to build something like a robot's cerebellum: a single model that controls the robot and learns and generates a diverse range of movements within one unified system.

In March 2025, we released an intermediate result showing several robots dancing and performing different actions. The model's zero-shot ability was not yet strong enough, but we found it could perform new movements very well after quick fine-tuning, so we decided to share the results. These demonstrations showed the potential for a foundation model that efficiently generates many kinds of robot movement.

Video demonstration from the original article

Results released in March 2025

Over the six months from March to National Day, we kept improving the model. By the holiday, it could take arbitrary motion inputs and produce complex robot movements with smooth, coherent, natural transitions, without fine-tuning. This was a key step from attempting a general model to achieving a robot foundation model with general motion capabilities. The video in this article shows the results from just before National Day.

The principle is to feed in a human motion sequence and infer the corresponding actions for the robot. The March video needed editing because different movements still required some fine-tuning. In the six-minute video released in September, the model could directly accept arbitrary human movement inputs and execute them naturally on the robot, without extra adjustments or editing. The overall performance was more coherent and fluid.

2. The GAE method

Methodologically, we did not adopt a radically distinctive approach, although there are improvements and innovations. In robotics, the final result is often determined by several stages working together rather than by a single algorithm.

A very good overall result requires doing every stage as well as possible. Our training algorithm, for example, may not differ fundamentally from earlier humanoid work, but we made small improvements throughout the technical details. We invested considerable effort in algorithm design, implementation and practical engineering. Those incremental improvements add up to better overall performance.

The overall architecture is largely unchanged from earlier humanoid projects. Reinforcement learning takes a human reference motion trajectory and the robot's proprioceptive information as input and outputs control actions. The basic framework remains the same.

The main change between March and September was scaling the human motion tracking paradigm: we increased training scale, dataset size and model size. The September results show the outcome of scaling, with significantly better humanoid performance than before.

3. Data sources

Our March model also used open-source data, but we found a persistent ceiling no matter how we adjusted it. We eventually realized that insufficient diversity was the main problem. A lack of diversity creates generalization difficulties. Suppose I have only two trajectories: running at 1 m/s and running at 3 m/s. I then need methodological changes to make the model generalize to 2 m/s. But if I have rich data, with trajectories at every 0.1 m/s between 1 and 3 m/s, generalization naturally improves—at least within that distribution. We no longer need to separately work out how to learn the 2 m/s case.

We came to see increasing diversity as the most important step in training. We also changed how we expanded the data. Motion capture is precise, but equipment is expensive and recording requires considerable human labor and time. We therefore introduced other sources, such as human movement estimated from video. That data is easier to obtain but less precise. On balance, we considered all these sources valuable. We did substantial work to balance sampling, but all the sources need to be used. Video's volume and diversity make it important for generalization.

We also thought about how the model would eventually be used. If the goal is a general humanoid cerebellum that anyone can use to control a robot, control cannot depend solely on motion capture. The equipment is costly, cumbersome to wear and requires calibration. Ordinary users are unlikely to control robots that way.

Our model therefore needs compatibility: it must execute both precise motion capture trajectories and human reference motions from other methods. These may come from video estimation or other AI models, such as text-to-motion systems. Although their trajectories are lower-quality than motion capture, the model must adapt to them. We deliberately included data from various sources during training. This helps both general deployment and training-data diversity.

4. Video data

Our dataset is very large compared with earlier work. AMASS has on the order of ten thousand motion sequences; ours reaches the million-trajectory scale, totaling over ten thousand hours. Sources include motion capture, video and other collection methods. We process this data and further expand it with augmentation. The resulting large, diverse dataset provides a strong basis for training.

Our March model was trained on roughly ten thousand AMASS sequences. It performed well during training and fit different motions well in simulation. But real-time tests on the physical robot were noticeably worse, especially for unseen movements. Even movements resembling the training data had subtle differences that the model could not generalize to well. We realized that it had learned the limited data effectively but had essentially memorized the training distribution, rather than learned to generalize across distributions.

We concluded that inadequate coverage was the root problem. Richer data covering natural movements from everyday activities would let the deployed model handle more situations without constraining the person's movements. We therefore began systematically expanding the dataset through different sources and greater diversity.

We also found a clear domain gap when training only on motion capture and testing with trajectories estimated from video. Training must therefore include multiple sources—not only more types of movement, but also different capture methods. That is necessary for stable performance across practical deployment settings.

5. The balance between data modalities

Although we collected a great deal of data, the sampling proportions during training are what really determine the final result. We adjust the sampling strategy dynamically. High-precision motion capture provides more reliable learning targets and helps the model converge faster.

Video-estimated data is less precise, producing noisier and less stable training signals. We therefore use an approach resembling curriculum learning, dynamically changing the proportions. The model first learns stable control patterns from precise motion capture, then gradually incorporates video and other sources. Across parallel simulation environments, we flexibly assign different sources—motion capture in some, video in others, other data elsewhere—to balance accuracy and generalization.

We do not simply pile all the data together. Although the total volume is large, it is organized into separate datasets by source. During training, we sample from those datasets and adjust the proportions to control how much each contributes. This preserves the stabilizing benefit of motion capture while improving generalization with other sources.

6. Which data sources dominate different training stages?

Early in training, we mainly rely on motion capture so that the model learns basic control quickly. Its precise, stable signals help establish foundational imitation capabilities.

Later, as capability and performance stabilize, we gradually introduce lower-quality data such as video-estimated trajectories. Their rewards are noisier and less precise, but the model already has some imitation capability, and its outputs remain near the target trajectory. Noise therefore does not pull it too far away. Throughout training, we adjust the sampling proportions according to learning progress.

The March model needed fine-tuning—an additional adjustment for execution. The current one does not, and its zero-shot performance is very good. Our general model now accounts for more deployment needs, including different control methods, so its training data must cover different sources.

We also account for diversity in the inputs. Motion capture can provide full-body trajectories, but an input may cover only part of the body: an upper-body motion capture setup or exoskeleton, for example, may supply fewer points, perhaps only the upper body or just two hands. VR control may supply only three points: the head and both hands. We handle this during training, with some environments using three points and others using upper-body points. The resulting model needs no additional fine-tuning. It generalizes across both movements and control configurations.

7. Different teleoperation interfaces

We support different ways to teleoperate humanoids. The model can take several joint points as a reference trajectory, such as the upper body alone or just both hands, and these can come from different sources. It is compatible with all these input forms.

The video uses Noitom inertial motion capture because it provides the most accurate inputs and the best-looking results. We mainly released it on short-video platforms rather than as a paper demonstration, so we wanted to show the strongest performance. Our paper will present more input types: motion capture, video estimation with an ordinary RGB camera, and trajectories generated by a text-to-motion model. Our model already supports these methods.

8. Is GAE a general-purpose whole-body controller?

Yes—it is a general controller, which is why we call it General Action Expert. We want capabilities that are truly general-purpose. Earlier work often remains restricted to one control method or a limited motion range. Rather than continually tweaking training algorithms or architectures—for example, whether to use an MoE, which may ultimately change performance by only 1–2%—we focus on other important stages, such as dataset diversity and scale, and retargeting quality. These can yield gains of 10–20% or more. We prioritize factors with a deeper effect on generality.

I recall that compatibility across control methods was an important motivation in similar work such as OmniH2O. HOVER also supports different input forms, including joint angles, positions and velocities. All address input compatibility. But the central question is how to truly solve it. If training remains limited to small motion capture datasets, changing the training paradigm, algorithm or architecture alone is unlikely to deliver the general model people expect. We start from data scale and diversity, using a broader training distribution to support general capabilities.

9. Fine-grained tweaking or scaling?

Changes to the network itself have relatively little impact; data and other stages matter more. Ultimately, people see videos and demonstrations, and judge the result. Our video was well received, and data had a major influence on training. But other factors affect the experience too, such as latency. We optimized deployment to minimize latency, working through many detailed stages. Good performance requires integrating the whole system well.

An academic paper often proposes one innovative idea, shows the difference with and without it, and is then complete. We spent more than a year optimizing this project across the entire pipeline: collecting data, retargeting it to the humanoid used in training, and addressing training issues introduced by scaling. Some were uncommon in earlier humanoid work but resemble problems encountered in large language and vision models. After training, we worked on real-robot deployment and the user experience, including reducing latency. We invested heavily in every stage, which is why the final video feels different.

10. The scale of our own motion capture data

I have not specifically counted the motion capture data we collected ourselves, but it comes from two main sources. We record some movements missing from AMASS, filling gaps in poses or movement types, though that portion is not large. The other source is long-term accumulation. Throughout more than a year of robot testing and debugging, we often wore motion capture equipment. We recorded those sessions and ultimately included the data in training.

Early on, we mainly relied on AMASS. Over time, our own collection grew. Each robot test added new motion capture data, and the accumulated amount eventually became substantial.

During tests, we wear the motion capture suit and have the robot perform different movements and tasks. The trajectories vary somewhat each time, and they are all recorded automatically.

Our system supports several motion capture setups. We initially used relatively inexpensive, less precise equipment, then moved to a more accurate optical system. We now have multiple setups available. Equipment does affect the presentation: when a person appears beside the robot, capture precision influences how closely their movements match, and lower precision can create slight discrepancies. But if the video shows only the robot, the model's output remains smooth and coherent, largely independent of capture precision. In other words, the equipment mainly affects the visual impression of synchronization, rather than the model's own control performance.

11. Success rates

The success rate is very high. The six-minute-plus video we posted on Bilibili was filmed in one continuous take, with no failures. As of this interview, the model has never fallen during our real-robot testing. Large movements sometimes cause slight shaking or stumbling, but never a fall. By contrast, the March model made mistakes more easily, and the robot in that video still showed many scars. The current model is highly robust and can handle various disturbances.

On autonomy, I was struck by something Sergey Levine said in an interview: the important question is not exactly when humanoids reach AGI, but when they can perform a limited set of real-world tasks. Only by doing work in open settings can they accumulate data through execution. Data is currently the biggest obstacle to autonomy. Laboratory and factory datasets have too few types of activity and lack open-world diversity.

The logic resembles autonomous driving. Cars cannot be trained only in a laboratory; they need data from complex conditions on open roads to keep improving. Robots likewise need to perform tasks in open environments and collect real interaction data through practice.

A robot starts by completing tasks in limited settings, accumulates data, and uses it to improve the model so that it can do more. This creates a positive cycle of human–robot collaboration, data accumulation, model iteration and increasing autonomy.

The first systems that actually work in practice will therefore involve human–robot collaboration. Fully autonomous robots are extremely difficult even in limited settings. If combining a person and a robot improves productivity or the user experience, that already has practical value.

That is our goal too: first, let the cerebellum model support high-quality collaboration through whole-body teleoperation with motion capture, cameras or VR. A sufficiently mature system has practical value in itself. As people control robots, the recorded images, movements and interactions can help train the robot's higher-level brain. Human control today can become autonomous robot control tomorrow. We believe the experience accumulated at the cerebellum stage will be an important driver of the robot brain's development.

12. The timeline from teleoperation to autonomy

It is very difficult to estimate. Open-world situations are enormously complex and diverse, so we cannot easily predict how long it will take robots to adapt to all kinds of environments. But for our current whole-body teleoperation approach, I think people will see convenient, effective use very soon. That stage will arrive quickly.

The pace of increasing autonomy depends largely on how widely the cerebellum is used. As more people use these teleoperation systems, data accumulates from the robot's tasks. The rate of accumulation determines how quickly the robot brain can be trained and evolve.

If the system remains confined to a few laboratories, research institutions or closed settings such as performances, accumulation will be slow. If ordinary users and developers can widely control robots through different interfaces, data could grow exponentially and autonomy could improve faster.

Many external factors also affect the overall pace, including hardware reliability, cost and commercialization. In general, however, we consider collecting data through the cerebellum a viable path toward truly general intelligence in the robot brain. At present, the data available to that higher-level brain is still far from sufficient for genuine generality.

13. Future directions

The earlier videos mainly show movements on flat ground. We have some generalization across terrain, but it is not yet fully developed. Next, we will focus on interaction with terrain, such as adapting to different surfaces, and interaction with objects. These are key directions for further improving the model and its generality.

Smart Note · Human Data · EP06