The Human Data series, which began taking shape in June 2025, has finally reached its concluding article after six months. Drawing on our 2024 Robot Data EP10: Season One Interview Summary and the 2025 Human Data series, I have put together a “Human Data pyramid.”

At the top of the pyramid is high-precision human motion data captured with optical and inertial motion capture systems (Human Data EP01—MoCap: TWIST; Interview with Zhengyi Luo: Decoding PHC—A General-Purpose Controller for Humanoid Robots). Accuracy can reach the submillimeter level. This data is difficult and expensive to collect, but it is the highest-quality source for training base models for humanoid whole-body control. Precision note: the submillimeter figure can apply to calibrated optical marker measurements; it is not a general accuracy specification for inertial whole-body capture, whose position estimates and drift depend on the system and conditions.
The middle of the pyramid consists of data from lower-precision capture devices: VR controllers plus ankle trackers (Human Data EP07—MoCap: Twist2); UMI, which captures human manipulation through a gripper interface (EP02: A Conversation with Cheng Chi and Zhenjia Xu—UMI, a Physical API for Robot Training); and DexCap, which records hand movements using five-finger gloves (EP04: A Conversation with Chen Wang—MimicPlay & DexCap: Training Robots with Human Video and Motion Capture Data). This data is easier to collect and can scale, typically with centimeter-level accuracy. The centimeter-scale figure is an approximate category-level comparison in this article, rather than a uniform specification for all three interfaces; their sensors and measured quantities differ.
At the bottom of the pyramid are vast amounts of video and synthetic data (Human Data EP04—Video: VideoMimic; Human Data EP05—Video: HUB; Human Data EP08: OmniRetarget). These sources offer large volumes, but they also face limitations: low precision, missing 3D information in video, and the sim-to-real gap in synthetic data.
Across these three categories of human motion data, more advanced retargeting methods, such as OmniRetarget, can address many limitations of traditional retargeting and substantially improve the quality and efficiency of transferring interactions with objects and environments to robots.
By combining human motion data from every level of the pyramid, as in GAE, and using efficient retargeting methods such as OmniRetarget, the abundance of human data could help overcome the shortage of robot data. This concluding article collects the central views on choosing data sources from our Human Data interviews. Together, they form a selection of perspectives on the data challenges along the whole-body control route.
Original article outline
Contents
👇
1. Human Data EP01—MoCap: TWIST
3. Human Data EP03—MoCap: ExBody2
4. Human Data EP04—Video: VideoMimic
6. Human Data EP06—MoCap + Video: GAE
1. Human Data EP01—MoCap: TWIST
There are roughly three main ways to obtain human data. The first uses camera-based video or images, such as internet videos or pose estimation from images. Its greatest advantages are portability, low cost, and accessibility, but its shortcomings are equally clear: the accuracy is poor, and the data quality is the lowest of the three. The second is VR. It costs more than a camera, and most people would not ordinarily buy the equipment, but it provides higher-quality upper-body data. The limitation is that it lacks lower-body data. Some studies partly address this by having the robot compensate for the missing information or adding inputs, such as a remote controller that specifies lower-body displacement. The third is motion capture equipment. It provides the highest accuracy and best data quality, but it is extremely expensive and lacks portability. These comparisons refer to the systems and experimental requirements discussed in the original interview; VR systems with additional body trackers can provide lower-body estimates.
There is a clear trade-off: the more portable the equipment, the poorer the data quality; the less portable it is, the better the data. Everyone wants more and better data, because scaling up is one of the secrets behind the success of large models. But scaling requires portable, inexpensive systems. Otherwise, nobody can afford equipment costing hundreds of thousands or even millions.
I have therefore tried both VR and monocular estimation methods, but in practice neither seems sufficient for high-quality whole-body motion capture. Root tracking is a particular problem. (Root tracking means a person’s overall position and movement in world coordinates—for example, walking from point A to point B.) With only one or two cameras, it is difficult to accurately capture the whole body’s trajectory through space in real time. At best, you can capture upper-body movements while the person remains stationary.
That almost rules out camera-based approaches, leaving VR and motion capture. Quite a few recent studies have tried VR-based whole-body control. Vision Pro applications, for example, provide only hand and head tracking and ask the policy to learn the missing lower-body movements. The results show that this is feasible. The problem is that the lower body may become unstable or keep adjusting. The robot might already have reached its destination, yet its lower body continues making small adjustments because it lacks actual lower-body input.
Some studies add a remote controller to specify base movement; others cleverly work around the problem by aligning the head’s global tracking. Either approach shows that VR is usable, but precise lower-body movements—such as squatting, stepping over obstacles, or holding a horse stance—are almost impossible.
By contrast, we currently insist on developing with complete whole-body data. The advantage is that we can do everything, but the disadvantage is obvious: we have to rely on motion capture. Motion capture is neither portable nor inexpensive. Even if a laboratory has the equipment, it is less convenient to use than VR. I therefore think the key going forward is to make these systems more portable.
2. Human Data EP02—MoCap: GMT
Among the open datasets available today, AMASS is the largest. We chose it because it covers a wide range of skills, although its quality may have some shortcomings. The idea we are pursuing is a foundation model at the behavioral level: a model that can execute all kinds of movements, including movements produced by generative models, and track them within the model. Achieving this may require a very large training dataset that covers as many kinds of movement as possible. A new, high-quality dataset collected from humans would, I think, substantially improve the overall results. Dataset note: “the largest” is the interviewee’s description of the motion-capture datasets under discussion, rather than a ranking of every human-data collection. The original AMASS release unified 15 optical marker-based datasets and contained over 40 hours of motion.
3. Human Data EP03—MoCap: ExBody2
Looking further ahead, I think we still lack sufficiently diverse motion capture data. Most existing data captures “the person’s own movements.” Giving humanoid robots genuinely intelligent behavior also requires data on human–object and human–environment interactions. This data allows a model to understand the intention and goal behind a movement, rather than merely imitate its trajectory. Richer motion capture datasets with deeper semantic structure should be a key driver of further progress in humanoid control.
4. Human Data EP04—Video: VideoMimic
We pretrain on high-quality human motion data recorded through motion capture. Although this data is limited in scale, its quality is high. We therefore first train a base policy on this small but carefully curated dataset, giving it strong initial locomotion capabilities.
5. Human Data EP05—Video: HUB
Whether video can completely replace motion capture data remains a challenge. The errors in extracting motion from video are still somewhat larger than those of motion capture equipment. In the long run, however, video has enormous potential. The internet contains vast amounts of human motion video, including footage of fitness creators and gymnasts. As the methods improve, this can supplement traditional motion capture data.
6. Human Data EP06—MoCap + Video: GAE
Our model therefore has to be compatible with different inputs: it must execute both trajectories from high-precision motion capture and human motion reference trajectories obtained through other methods.
We also found a substantial domain gap when we trained exclusively on motion capture data but tested on trajectories estimated from video. We therefore realized that training must include multiple data sources—not only a wider range of movement types, but also different collection methods.
7. Human Data EP07—MoCap: Twist2
In theory, it is indeed possible to train or operate directly with low-precision VR motion data, without high-precision motion capture data. It is “possible.”
In practice, however, we take a hybrid approach. We still primarily use high-precision motion capture data to train our models and controllers. About 99% of the data comes from high-precision motion capture systems, including data collected for Twist1. This data provides very stable spatiotemporal consistency and accurate skeletal poses. It forms the core foundation for learning movement patterns and control strategies.
At the same time, we use around 1% low-precision data to help the model adapt to noisier inputs, especially in body regions where VR systems are less accurate, such as the legs and waist. Drift is more pronounced in these areas. We therefore mix a small amount of low-precision data into the high-precision data, helping the model learn some robustness during training so that it can adapt automatically when deployed with lower-precision systems.
In other words, this is a mixed training strategy that primarily uses high-precision data and supplements it with low-precision data. It preserves the reliability of high-precision motion capture while giving the model the adaptability to operate with lower-precision systems.
8. Human Data EP08: OmniRetarget
Our data mainly comes from two motion capture datasets, LAFAN and OMOMO, as well as motion capture data we collected ourselves. One of our collaborators also personally recorded many demonstrations, such as climbing onto boxes of different sizes. These recordings form part of our data as well.
Existing open datasets contain relatively few demonstrations of complex interactions with the environment. Consider our final example: the robot must pick up a chair, step onto it, use it as support to climb onto a higher platform, and then jump down. Data for movements like this, involving both object interaction and environment interaction, is extremely scarce in existing public datasets.
I have some personal views on teleoperation. Real physical data is certainly necessary, but teleoperation still has quite a few problems: latency, differences between human and robot bodies, and the lack of tactile feedback. When a person operates a robot, they cannot simultaneously perceive the tactile sensations the robot “feels,” so a gap remains between them. The operator’s own movements may also be shaky, causing jitter in both the recorded trajectories and the resulting policy.
If we apply additional optimization to these teleoperated trajectories and make them better, we may be able to further improve data quality. Another limitation is that some tasks are impractical to carry out through teleoperation. Asking an operator to directly teleoperate a robot through a backflip, for example, is clearly unrealistic. If we want certain superhuman capabilities, we must rely on methods beyond teleoperation.
Personally, I prefer human data. It is abundant, and human movements are generally more natural.
Compare autonomous driving with robotics. Cars were designed from the outset to be driven by people, so human driving data is naturally suited to training driving models. Robots, however, were not designed to be “operated by people.” They are designed to perform tasks that people do in everyday life. If people themselves rarely do something, having a robot specialize in it may not offer much practical value.
For tasks that humans perform proficiently, human demonstration data is therefore one of the most valuable resources available to robots.