Human Data explores data about the human body. In our first four episodes, we interviewed the authors of four papers on humanoid whole-body control using motion capture or video: motion-capture teleoperation in Human Data EP01—Motion Capture: TWIST; whole-body control in Human Data EP02—Motion Capture: GMT and Human Data EP03—Motion Capture: ExBody2; and video-based whole-body control in Human Data EP04—Video: VideoMimic.
For episode five, we interviewed the authors of a particularly interesting project, HuB: Learning Extreme Humanoid Balance. It explores how far current humanoid hardware can push balance capabilities. Unlike the preceding four projects, HuB relies entirely on human videos from the internet to extract SMPL representations as the starting point for motion control training.
HuB's joint first authors are Tong Zhang and Boyuan Zheng. At the time of this interview, Tong is a PhD student at Tsinghua University's Institute for Interdisciplinary Information Sciences, advised by Yang Gao. He completed his undergraduate degree in electronic engineering at Tsinghua and was a visiting researcher at UC Berkeley, working with Koushil Sreenath. Boyuan is a first-year PhD student at Tsinghua's IIIS, also advised by Yang Gao, and earned his undergraduate degree in computer science at Tongji University. We have previously interviewed their adviser twice: Yang Gao on the ViLa and CoPa Embodied Model Framework, and Robot Data EP01: Yang Gao and Chuan Wen on Learning Robot Policies from Internet Videos with ATM.
As Tong and Boyuan explain in the interview, the core of the paradigm is to use human motion as a reference.
The policy tracks that motion. The reference serves both as the robot's target and as a powerful prior. Human Data shows how abundant, diverse human data can expand the training resources needed for general-purpose robots and make scaling possible.
Original article outline
Contents
👇
1. What makes HuB distinctive?
2. The HuB method
Part one: reference motion refinement
Part two: balance-aware policy learning
Part three: sim-to-real transfer
3. HuB's contribution to the community
1. What makes HuB distinctive?
Tong Zhang: Our work differs somewhat from earlier research emphasizing dynamic movements. First, we focus on extreme balance. We designed a movement in which the robot stands on one leg, lifts and extends the other, spreads its arms into a T shape and keeps its torso horizontal. I estimate that the vast majority of people—perhaps more than 99%—cannot do this. It approaches the limits of human balance, and even exceeds the ability of some ordinary people.
We wanted to test whether such extreme balance was possible on a robot. Before the experiment, we were unsure whether current humanoid hardware could support movements this demanding. It turned out that our hardware and algorithm could achieve them, showing that existing humanoid hardware can, at least to some extent, approach human motor capabilities.
Second, these balancing movements differ from dynamic actions. Jumping or shooting a basketball may require balance for a brief moment, with different whole-body control demands from the sustained extreme balance we seek. Our robot not only holds a stable pose but also maintains balance under external disturbances. We tested hitting it with a ball, for example, and it remained balanced. This sustained, difficult balance is something many earlier dynamic-motion studies did not emphasize, and a particular focus of our work.
HuB therefore addresses two central questions: it demonstrates that today's hardware can support extreme balancing movements, and it provides a method for sustaining difficult balance over time.
Boyuan Zheng: From a human perspective, balance is a complex systems problem. Besides muscles and joints, the vestibular organs in the inner ear and the visual system provide feedback. In our own experiments, for instance, standing on one leg with our eyes closed made it easy to fall and difficult to remain upright for long.
For the robot, however, the balance-sensing input in this project is relatively limited: just an IMU. That makes balancing harder. Our movements also involve extreme poses, such as single-leg stands and difficult Bruce Lee-like balancing movements, rather than ordinary everyday actions. Here the IMU refers to the additional balance-sensing channel being contrasted with human vestibular and visual feedback; the control policy also uses robot proprioceptive observations.
Even so, studying these extremes is meaningful. It helps us understand what balance capabilities humanoids can achieve. Practical tasks also involve basic but essential balance challenges. A robot reaching for an object far across a table may need to lean forward or support itself on one leg; it also needs balance while crouching to pick something up. Balance is indispensable in planning and executing such tasks, making this research valuable.
2. The HuB method
Our method differs from other approaches in three main stages. First is reference motion refinement, emphasizing how to improve human reference movements. Second is balance-aware policy learning, involving the policy's learning design. Third is sim-to-real robustness training, which increases stability by simulating real sensor noise and other disturbances.

The overall framework follows the original DeepMimic idea: obtain human reference motions, retarget them to the humanoid, then train a tracking policy in simulation. This pipeline is now common in humanoid motion control. Most humanoid dancing demonstrations, for example, use it. Our key differences are improvements in those three stages, especially to increase stability in extreme balancing poses.
Part one: reference motion refinement
Part (a) emphasizes the importance of data. Ordinary tasks such as walking and running are relatively tolerant of reference motion errors: small deviations need not undermine overall stability. Extreme balance requires much higher data quality. We improve the data processing pipeline to reduce errors and make movements both more accurate and more physically realistic. This is our first level of improvement.
We do not use motion capture data here, although it could be used. We see several limitations in motion capture. First, you need the equipment, and frankly we did not have it at the time. Second, you must record people yourself in an indoor setting. For balancing movements that 99% of people find difficult, it is hard to find an ordinary person nearby who can perform them; you might need a martial arts expert. We therefore used videos downloaded from the internet, showing gymnasts and fitness creators performing very difficult movements.
Reference motion errors come from two sources: extracting motion from video and the retargeting process, whose iterative solution is not guaranteed to converge accurately or remain temporally continuous. We introduced four modules. First, we modified the retargeting method used by PHC. PHC uses gradient descent to minimize the distance between corresponding human and robot keypoints. We found direct iteration could fail to converge for some movements, especially squatting, producing large errors. We initialize from the corresponding human joint angles so that optimization starts near a good solution, making it faster and more accurate. Second, grounded foot correction fixes foot sliding caused by video extraction errors. Third, center-of-mass filtering removes inaccurate motions whose center of mass is far from the support foot. Finally, deployment can produce a large initial shake, so we add a standing interval before the motion to let the robot stabilize.
Reference motions generally come from motion capture or video; we use video. Computer vision algorithms extract human SMPL representations from footage. AMASS and similar motion capture datasets obtain poses directly from equipment, whereas we extract movements from video and map them into SMPL through algorithms.
SMPL, the Skinned Multi-Person Linear Model, is a parametric human body model introduced in 2015 by researchers including Michael J. Black at the Max Planck Institute for Intelligent Systems. A small number of pose and shape parameters realistically reconstruct and represent the geometry and posture of a 3D human body. Its output is a 3D body mesh with fixed topology, and its parameters are divided into shape and pose parameters.
Replacing motion capture entirely with video still poses challenges. Video extraction has somewhat greater errors than capture equipment. In the long term, however, video has enormous potential because the internet contains vast amounts of human movement, including fitness creators and gymnasts. As methods improve, it can supplement conventional motion capture.
Most HuB movements are relatively static, so much of our online footage shows a person making limited movements from a fixed camera view. Current video-based motion capture becomes less accurate with moving cameras or large body movements, and may need additional correction, such as enforcing ground contact. Using abundant internet video is promising, but there are tradeoffs: collecting our own data gives high quality but takes time and covers fewer motion types; online video is abundant but less precise. We focus on minimizing errors through reference motion refinement so the robot can learn the resulting movements and balance successfully.
To expand on retargeting, methods such as PHC first define positional correspondences: robot wrist to human wrist, shoulder to shoulder, foot to foot and knee to knee. With those correspondences established, the problem is essentially optimization by gradient descent.
Earlier methods often initialize all joint angles at zero. For movements such as squats, that can yield unreasonable poses, with the legs incorrectly configured. We initialize the robot's joints with the corresponding SMPL human joint angles before optimization. Starting nearer the optimum makes optimization easier and produces more reasonable movements.
Another approach, used in HumanPlus, directly copies human joint angles as targets. But human joints and robot degrees of freedom do not match perfectly, so direct copying may create large errors in hand end-effector positions. We combine the approaches: initialize from human joint angles, then refine with gradient descent for better retargeting.
Part two: balance-aware policy learning
Tong Zhang: Our starting point for balance-aware policy learning is that, although human and humanoid forms are similar, they still differ in aspects such as mass distribution. These differences are especially important in extreme single-leg poses, where the region in which balance is possible is very narrow and easy to leave.
First, we found that strict motion tracking could leave the robot unbalanced even when it held the same pose as a person. We therefore relaxed the tracking requirements. This increases the tolerated error and allows the policy to search a wider space for a balanced pose appropriate to the robot itself.
Second, a larger exploration space creates opportunities for reward hacking. Reinforcement learning may discover a policy that scores well without learning the intended movement. In a single-leg stand with the other leg raised, for example, a large falling penalty may cause the robot to settle for simple, low-risk actions unless additional constraints are imposed.
We therefore add shaping rewards, including a center-of-mass reward, a foot-contact mismatch penalty and a close-feet penalty. These discourage shortcuts and ensure that the robot learns movements that meet the task requirements and are feasible in reality.
Boyuan Zheng: To add to that, we design specifically for balance-intensive movements so the robot can better accommodate balance differences. There are two main reasons: the data contains errors, and human and robot morphologies differ, including their center-of-mass distributions.
Strictly following a human pose, as illustrated by the red example on the left of panel (b), can give an unsuitable center of mass and cause a fall. We therefore loosen the tracking constraints and add rewards to regulate behavior. We want the target movement, such as lifting a foot, but also a more stable center of mass. The center-of-mass reward encourages stable balance, while the close-feet penalty constrains unreasonable poses and helps avoid self-collision.
Overall, relaxed tracking constraints and designed rewards help the robot adapt to balance-intensive tasks during policy learning.
Tong Zhang: Let me explain the scope and scalability of our policy learning. We currently focus mainly on individual movements, with a separate policy for each. The framework is compatible with a general policy that performs multiple movements in principle. But integrating very different movements into one unified policy is another research problem.
Our present focus is how far balance can be pushed, so a single-motion policy is the simplest, most direct approach. The framework can nevertheless train on multiple kinds of data and integrate them into one policy. I have briefly tried combining movements by distilling multiple teachers into a student, and it is relatively straightforward to implement.
A separate policy for each movement is currently the simplest and most reliable option. The framework is scalable: multiple teachers can be distilled into a student to combine movements in one unified policy.
We filmed most movements for our demonstrations. The only one we did not achieve was an Ip Man-style one-legged squat with the other leg raised. The issue was not the policy's learning capacity, but the data: current extraction from heavily occluded internet videos cannot provide accurate 3D training poses for this movement. Apart from that exception, we achieved all the movements.
We omitted it because accurate data was difficult to obtain from internet footage, rather than because the policy could not perform it. Visual motion capture has limited ability with such heavy occlusion. With more precise motion capture equipment or data, our approach could achieve it quite easily.
Part three: sim-to-real transfer
For sim-to-real, we include conventional domain randomization but emphasize several distinctive choices. First is localized reference tracking. OmniH2O uses visual-inertial odometry to obtain absolute displacement, such as measuring with a depth camera how many meters the robot has moved. People do not themselves perceive displacement that precisely, and real sensors are noisy; large movements can increase the noise further. We concluded that the robot does not need such precise displacement and removed odometry inputs. We train with a teacher–student architecture. The teacher learns through RL in simulation with precise information. The student lacks global position but learns an implicit sense of movement from the teacher's supervision, much as people roughly sense how far they have traveled.
Second is IMU noise modeling. Earlier methods may add independent uniform or Gaussian noise to policy observations, but IMU noise is temporally correlated rather than independent at every step. We add temporally correlated noise to the IMU, then derive other policy inputs from it, including projected gravity and angular velocity in the body frame. Adding independent noise would miss the coupling between these observations. Deriving their noise from the IMU makes the policy's inputs closer to real sensor behavior.
Third, we apply frequent pushes in simulation so the robot experiences instability and learns to recover quickly. This is a form of domain randomization, helping the policy adjust automatically to shaking in reality instead of amplifying it.
3. HuB's contribution to the community
Our original aim was to explore the limits of balance: to take one subfield as far as possible and bring the robot to its extremes at this stage. The experiments demonstrated current humanoid balance capabilities. Frankly, humanoids are advancing extremely quickly; their capabilities today are completely different from a year ago. We released our work in April and May 2025, showing extreme balance capabilities at that time, and further capabilities have appeared since. We principally tested the limits of balance then.
Longer term, we keep thinking about combining humanoid locomotion and manipulation. Many humanoids are less stable than wheeled robots today, but have greater potential in versatility and whole-body control. We want to improve their stability and balance so that they can do more than stand securely: they should execute complex movements and realize their broader locomotion and manipulation potential.
Several factors seem to drive this rapid progress. One is the DeepMimic paradigm, which provides powerful expressive capabilities: given a human movement, a humanoid can theoretically perform something similar. Another is hardware progress. Robots are more robust, move better than earlier systems and have a smaller sim-to-real gap, providing the physical basis for difficult movements.
The core of the paradigm is using human movement as a reference for the policy to track. The reference is both a target and a strong prior. Previously, teaching a humanoid to walk without this approach meant designing complex rewards for forward movement, lifting the feet, avoiding ground scraping and so on. Those relatively sparse rewards leave the robot to explore walking in a very large search space.
Reference-based methods obtain priors directly from human movement and provide more reasonable initial states. For walking, for example, they show that the left and right feet should lift alternately. This gives a clear target and structured reward guidance, shrinking the exploration space and making learning more efficient.