Back to Human Data

Human Data · EP01

Human Data, Episode 1: Yanjie Ze on TWIST and Whole-Body Humanoid Teleoperation

Yanjie Ze discusses human motion capture, real-time retargeting, RL plus behavior cloning, GMR and the practical design of whole-body humanoid teleoperation.

Zha Dami2025-10-1720 min readWeChat original
Contents

Attentive readers will have noticed that, since the start of 2025, Smart Note's interviews have largely focused on whole-body motion control for humanoid robots. This includes three consecutive interviews with researchers in the field—Interview with Xue Bin (Jason) Peng: Exploring a General Controller for Whole-Body Humanoid Control; Interview with Zhengyi Luo: Inside PHC, a General Humanoid Controller; and Conversation with Zhongyu Li: Building a Legged Robot's Cerebellum—as well as an interview about open-source humanoid hardware, OSH, Episode 1: ToddlerBot, an Open-Source Miniature Humanoid from Stanford.

Our new Human Data series focuses on where the data for whole-body humanoid control comes from. Data has always been a challenge for learning-based robotics. In 2024, our Robot Data series covered more than ten representative projects collecting data at different levels of the robot-learning data pyramid. Human Data will focus on how human data can support robot learning. We plan to interview the authors of eight or nine papers that use human motion-capture data or human video data for humanoid learning and control.

For the first episode of Human Data, we interviewed the team behind TWIST, the Teleoperated Whole-Body Imitation System. TWIST is a humanoid teleoperation system based on whole-body motion imitation. It retargets human motion-capture data, or MoCap, to a robot to generate reference motion sequences, then combines reinforcement learning with behavior cloning to train a robust, adaptive and responsive whole-body controller. Yanjie Ze and Zixuan Chen are among the paper's co-first authors. Our guest for this interview is Yanjie Ze, who, at the time of the interview, was a second-year PhD student in Stanford University's Department of Computer Science, advised by Jiajun Wu and C. Karen Liu.

If you are interested in using human motion capture to teleoperate humanoid robots, read on for Yanjie's thinking. TWIST has been open-sourced; the repository link appears in the references at the end of this article.

Original article outline

Contents

↓

1. The Background to TWIST

2. How TWIST Differs from Other MoCap-Driven Controllers

3. Options for Collecting Human Motion Data

4. The TWIST Method

5. Retargeting

6. GMR

7. Practical Considerations for MoCap Teleoperation

1. The Background to TWIST

Our two earlier projects, DP3 and iDP3—see DP3 and iDP3: Improving Diffusion Policy's Generalization with 3D Inputs—mainly focused on getting a robot's upper body to perform movements resembling human actions. While working on iDP3, I began looking more deeply into whole-body motion control. Controlling only the upper body limits a robot's workspace. For iDP3, for example, we designed a system that could control the waist and neck, giving the robot a larger workspace than a conventional dual-arm robot. It could bend over to touch objects in front of it, or remain upright to perform operations nearby.

At that point, our algorithm could already make the robot perform certain actions in a particular environment. It worked quite well in tests at a reception desk, in a living room or in a kitchen. But the robot could not move around autonomously: someone had to push it. I then realized that it should be able to move from one point to another on its own, like a person, and perform tasks. We started thinking about how to give it that ability. As a first step, we would let a person operate it and get it to move like a human. That became the starting point for our subsequent research.

We surveyed earlier work and found that people had already attempted to build systems that let an entire humanoid robot move like a person, using exoskeletons, learning-based control or model predictive control. TWIST uses human motion-capture equipment as its interface. We acquire high-quality human motion data in real time and use it to drive the robot. This synchronizes the robot's whole body—including its feet, waist, knees and arms—with the person's movements, rather than controlling just the upper body or the end effectors.

Once we had built this system, the robot could perform genuinely human-like actions. Imagine it is in a kitchen, carrying something in its hands, with a door blocking its path. It can kick the door open with its foot and continue walking through. This example shows why it matters for a humanoid to move like a human: such movement is the foundation for performing human functions. The first step is for industry to build the hardware platform. The second is for those of us working on algorithms to design systems that allow the robot to actually move.

Video demonstration from the original article

TWIST uses motion-capture equipment to teleoperate humanoid robots.

2. How TWIST Differs from Other MoCap-Driven Controllers

From the perspective of overall system design, I think TWIST does share similarities with a lot of earlier work. Around five to ten years ago, there were some early attempts in Japan to acquire human data through whole-body motion capture or exoskeleton interfaces and use it to drive humanoid robots. Their data-acquisition platforms may have been similar in spirit to ours, but their methods for controlling the robot were very different. For example, after acquiring human data, they would solve optimization problems to match the human motion to the robot. If the motion did not match, they would add another model-based layer, such as model predictive control, to sample candidate motions. Those robots could indeed move like people, but the systems lacked learning-based control. They could therefore perform only certain actions in restricted environments, such as walking slowly or crouching. They did not achieve a broader range of functional abilities, and those lines of work did not continue developing.

There has been progress in recent years. First, control methods have improved. In my view, reinforcement learning has become capable of replacing traditional control methods across much of this work, and almost any robot-control task can be tackled with RL. Second, hardware capabilities have improved. Many things the hardware could not do five years ago are possible now. Hardware is no longer the limiting factor in the same way, so we can start exploring the limits of the algorithms. That is also why people have recently returned to this area. Compared with five years ago, every module has improved substantially. Over the longer term, I see this as an upgrade of the earlier work.

Recent work from the past two years, such as OmniH2O and HumanPlus, is similar to TWIST, but there are still differences. HumanPlus, for example, roughly follows three steps. First, it acquires whole-body human data. Second, it performs real-time retargeting, converting human data into robot data through joint mapping to achieve human-to-robot transfer. Third, it uses RL to make the physical robot execute those motions.

Although the overall idea is similar, TWIST improves each part of the system. First, acquiring high-quality human data is essential. We used what we consider the highest-quality human motion-capture data available, ensuring accuracy at the foundation. Second, the conversion from human data to robot data is crucial. We spent a great deal of time tuning the real-time retargeting algorithm so that it preserves the structure of human motion while avoiding broken or erroneous results. That was a major improvement over the simpler mapping methods available at the time. Third, we need a strong RL controller to execute the converted motions in a physically feasible way. Directly retargeted data does not necessarily satisfy physical constraints, such as ground contact and mechanical balance. We therefore designed a two-stage algorithm. In the first stage, we use extensive privileged information to generate teacher data. In the second, we perform teacher–student distillation and introduce an RL-plus-BC framework that allows the controller to track high-quality motions more effectively.

Overall, compared with work from a year ago or five years ago, TWIST upgrades the system in four respects: high-quality data acquisition, real-time high-fidelity retargeting, multi-stage RL control and stronger hardware support. Together, these improvements produce impressive whole-body movements. The upper and lower body can move freely and in coordination, almost like a real person. That is the main difference from the earlier work.

3. Options for Collecting Human Motion Data

There are roughly three main ways to acquire human motion data. The first is camera-based video or images: for example, internet videos or pose estimation from images. Its biggest advantages are portability, low cost and accessibility. Its drawback is equally clear: accuracy is poor, and it provides the lowest data quality of these options. The second is VR. It costs more than cameras, and most ordinary consumers do not own it, but it provides better upper-body data. The limitation is that it does not capture the lower body. Some research partly addresses this by having the robot compensate for the missing information, or by adding other inputs, such as a remote controller that supplies lower-body displacement commands. The third option is motion-capture equipment. It provides the greatest accuracy and highest quality, but it is extremely expensive and lacks portability. This comparison concerns the setups evaluated for the original project; adding body trackers changes the information available from a VR system.

There is an obvious trade-off here: greater portability tends to mean lower data quality, while less portable equipment tends to provide better data. Everyone wants more and better data, because scaling up is one of the secrets behind the success of large models. But scaling requires portable, inexpensive systems. Equipment costing hundreds of thousands or even millions is beyond most people's reach.

I tried VR and monocular pose-estimation methods, but in practice I felt neither provided sufficiently high-quality whole-body motion capture. Root tracking was a particular problem. Root tracking describes a person's overall position and movement in world coordinates—for example, walking from point A to point B. With just one or two cameras, it is difficult to accurately capture the whole body's trajectory through space in real time. At best, in our experiments, we could obtain upper-body movements while the person stayed in place.

For our requirements, that largely ruled out camera-based approaches, leaving VR and motion capture. Several recent projects have tried using VR for whole-body control. For example, applications using Vision Pro provide only hand and head tracking and ask the policy to learn the missing lower-body motions. This can work, but the lower body can be unstable or keep adjusting. The robot may already have reached its target, yet its lower body continues making small corrections because it has no actual lower-body input.

Some projects use an additional remote controller to provide base-motion commands. Others work around the problem by aligning global head tracking. Either way, VR is usable, but fine-grained lower-body movements—crouching, stepping over something or adopting a wide horse stance—are very difficult to achieve.

By contrast, we currently insist on developing with complete whole-body data. The benefit is that we can cover all these movements. The downside is obvious: we depend on motion capture. It is expensive and not portable. Even if a lab has the equipment, using it is less convenient than using VR. I think the key question for the future is how to make these systems more portable. I have seen vendors working on this. For instance, one European company can provide high-quality global whole-body tracking with just one or two cameras. If commercial solutions like that continue to mature, whole-body motion capture may become more portable.

Overall, I do not think we should give up on high-precision control simply because some data is missing. Instead, we should acquire as much good data as possible, establish a solid data foundation and make the whole system stronger. That is one of my central ideas at the moment.

4. The TWIST Method

Our method can be divided into two stages: data preparation and controller training. For data preparation, the training data mainly comes from two established sources. One is datasets released in earlier papers; the other is AMASS. Both provide mature whole-body human motion data and are widely used to train motion trackers. We also collected a small amount of in-house motion-capture data. We added this because our target is real-time control—in other words, real-time motion tracking.

Illustration accompanying the TWIST data preparation and controller training discussion

There had already been many studies of motion tracking, such as dancing or imitating movements. Essentially, they provide a human motion and ask the robot to follow it. But we found that a controller trained entirely on public data performed poorly for our task. This could be due to the quality of the motions themselves or to the particular demands of real-time control. In our experiments, adding even a small amount of our own in-house data led to a clear improvement. We did not need much: a little was sufficient, and collecting more made relatively little difference. The main role of this internal data is therefore to narrow the distribution gap between training and testing. That was a key lesson from our data selection and preparation.

The second part is controller training, which combines RL and BC, or behavior cloning. The overall process is fairly standard. We modified an open-source framework from ETH and used PPO to carry out extensive online interaction in simulation, collecting rewards to train the policy. Compared with earlier walking-control work, the main differences are the definition of the reward and how the training data is used as reference motion.

At each time step, the robot has a target reference frame and a corresponding motion to track. If, for example, the target for its left hand is a particular point in space, we turn the error between the hand's current position and that target into a reward for the PPO policy update. Designing motion-tracking rewards requires care. They can include spatial positions and rotations, or points and rotations expressed in local coordinates. One important finding is that without global information, precise motion tracking is difficult and controller performance drops noticeably. At the same time, expressing motion-tracking targets in a local coordinate system lets each step focus on the local action. Even if the preceding step deviates, the next can still execute normally, improving the overall result.

On that basis, we use a teacher–student mechanism. The teacher can access extra information unavailable to the student, such as global position, to provide better guidance for root and rotation tracking. The student learns through RL plus BC. RL uses reward signals to let it adjust its behavior, while BC supplies auxiliary supervision that helps it imitate the teacher's actions. It does not force the student to reproduce every action regardless of its own state, thereby avoiding the instability or falls that can arise from mismatched behavior.

We also give the teacher information about future motion. The student cannot access that information, but offline datasets can supply it to the teacher and improve its guidance. Overall, the RL-plus-BC framework provides adaptability and robustness in online control. Even if there is an unexpected deviation in the environment or in the motion, the robot can still behave reasonably.

The third step, operating the robot, involves some small real-world design choices. The inference pipeline first acquires human data from the motion-capture system, then retargets it into whole-body humanoid motion, and finally uses the controller to turn the commanded motion into the robot's actual movement. That is the core workflow, but some details need to be handled on the physical robot.

The first is how to let the robot stand safely or enter a safe state. When wearing a motion-capture suit, you may want to stop controlling it temporarily—for example, to go to the bathroom. The robot cannot remain in active teleoperation all the time, so it needs an exit mechanism. The crudest option is to shut it down or manually suspend it, but that means restarting it the next time, which is inconvenient.

We therefore added an auxiliary device: a handheld controller that the operator holds while wearing the suit, to start and stop the system. If I want to end or pause control and leave the robot standing, I press a button. The robot enters its default standing posture and stops accepting motion-capture signals, receiving only the default control commands. It can then stand safely while I release the controls or do something else.

I think this design is very important. Many people working on robot control focus only on the motion in the middle of the process and overlook the complete user experience: starting, pausing, resuming and finishing. Designs like this are common in mature systems. For example, systems such as ALOHA, and similar systems being built by companies in China, allow users to pause operation whenever they need to and resume data collection seamlessly.

5. Retargeting

Retargeting public motion-capture datasets and my own recordings raises many issues. It can require extensive calibration; different people wearing the suit may need recalibration and produce different results. Accuracy and the mapping process can cause problems too. When retargeting both public datasets and data from the suit I wear, my first step is to use essentially the same representation for both. We treat the body as points in 3D space, each with a position and an orientation. This representation can be extracted from different types of motion-capture data. A human body consists of many bones and joints, and each part can be approximated as a rigid body with a spatial position and rotation. Together, these descriptions approximately reconstruct a person. Whether the data is ours or comes from public sources online, it can ultimately be converted into this common format.

Once we have that representation, what comes next? In a typical offline workflow, we manually establish correspondences between the human body's parts and the robot's body parts. We then use the robot's kinematics or dynamics to satisfy constraints on human keypoints, such as the position and rotation of an arm in space. The result is a robot movement that resembles the human movement. That is essentially retargeting: map the human body's twenty or thirty segments, then use the robot's model to produce a similar-looking motion.

There are three main differences between offline and online retargeting. The first is the optimization objective. Offline, we usually optimize spatial position constraints on keypoints without directly constraining rotations, because sufficiently many position constraints also determine rotations indirectly. Online, I use more rotation constraints. The second difference is smoothness. Offline, we can add smoothness constraints across a trajectory. If we solve one hundred frames independently, the result may jitter badly, but smoothing makes the whole sequence fluid. Online, we could smooth using past frames, but that slows the response: the motion may only follow to a level of 0.7 or 0.8, as I put it, introducing undesirable lag. That is why I omit this step online. The third difference is the numerical method. Offline, I can repeatedly sample and optimize using gradients through forward kinematics, or FK, because speed is less important. Online, I use an inverse-kinematics, or IK, solver to obtain a result fast enough for real-time use. These are related ways of solving the retargeting problem, with different implementations suited to offline optimization and real-time inference.

Overall, the core of retargeting is to unify the data representation, establish a mapping between human and robot bodies, and then choose optimization objectives and methods that suit either offline or online requirements.

The first difference is mainly the optimization objective. Online, we use more rotation constraints rather than focusing primarily on position constraints as we do offline. People wearing the suit differ in height, body shape and the dimensions of individual body parts. Using only position constraints can produce very strange results. Suppose one person wears the suit today and we map their arm positions using an existing configuration. Tomorrow, another person has a different build or wears the suit slightly differently, and the resulting motion changes. It may look quite unlike the movement we expected. With rotation constraints, this problem is less pronounced.

Rotations are more stable than positions and less sensitive to differences between people or changes in configuration. They have their own problems—for example, alignment can be difficult and other errors can arise. On balance, though, rotation constraints reduce online errors caused by body shape and the way the suit is worn. We therefore rely more on rotation constraints and less on strong position constraints for online retargeting. This also addresses the need to recalibrate every time a different person puts on the suit.

I do not know enough about other similar work to say whether it also uses rotation constraints. One of our recent projects on cross-embodiment retargeting, GMR, does use them. Much of the older work, however, explored how to make character animation look more realistic, rather than how to satisfy real-time constraints online. Broadly speaking, the goal is always to make a motion reach its intended target. Online scenarios simply introduce more real-world noise that we need to consider.

The second difference concerns information at the trajectory level. The offline component supplies trajectory-level information that makes the movement look smooth. You can think of offline processing as a kind of preprocessing or preparation, while the online component responds and controls motion in real time. Strictly speaking, the offline part is not pretraining in the conventional sense. It belongs to the same training framework as the online part, but can provide a smoother initial motion reference.

In practice, different data sources also play different roles. AMASS data may be more useful for extracting one kind of information, while our own motion-capture data may emphasize another. IK and FK are also two approaches used in retargeting. IK calculates how the robot should move to reach a target position, potentially iterating to reduce error. An FK-based optimization approach samples configurations and uses differentiable forward kinematics to calculate the resulting state. Both are common solutions.

Retargeting is fundamentally about mapping and controlling multiple keypoints. A robot arm typically has one end effector; a human body can be treated as having twenty or thirty such endpoints, including its joints and bones. The algorithm must account for all their constraints at once. In other words, retargeting involves two things: first, assign the body parts being mapped to their target positions; second, calculate how the robot needs to move to reach those targets.

6. GMR

GMR, or General Motion Retargeting, does not release raw data. It open-sources the process of mapping human data to robot data—in other words, the retargeting method. That is useful to the whole community. Other researchers can use the GMR pipeline directly to retarget motions to their own robots.

GMR is a fairly general pipeline, and we have added support for several common robots on the market. Many people email me to ask whether it can support a particular robot, and I have helped add them. I see this as an important advance for cross-embodiment applications. Cross-embodiment mapping first needs a unified data representation, so that data can be integrated and applied to different robots. The human body is a very good intermediary. If we can map a human body to the target robot, we can use human motion as a data source. That is GMR's greatest significance.

The mapping process in GMR includes both the offline and online parts of the first stage we discussed earlier. It provides a complete algorithmic framework that others can build on and engineer further. That matters for a wider range of robot-hardware companies. Releasing TWIST lets people reproduce human-like whole-body control on their own robots and helps advance the use of data in embodied intelligence and humanoid robotics.

The field has not yet reached a consensus, especially around human–robot interaction and motion control. Many methods and workflows remain unstandardized. Using VR to control robots, for example, is a worthwhile exploration, but there is still no unified approach or consensus on how to teleoperate them in practice. It remains an area under active exploration.

7. Practical Considerations for MoCap Teleoperation

During operation, certain points in the operator's movement matter, but we have not used these operator recordings to train autonomous behavior. Our focus is more on demonstrating the method. As an operator, you might expect to need to slow down or adapt to the robot's structure. In practice, you do not need to adjust very much. The robot is easier to operate than you might imagine because its control closely matches human movement.

Here is one example. We tried a system for beginners that used a dual-arm exoskeleton and buttons to control the robot's waist, forward and backward movement, grippers and other actions. There were five to eight buttons in total. When I tried it myself, I found the overall control much easier than expected. When human data becomes the direct interface for controlling a robot, the natural mapping is intuitive: you lift your leg, and the robot lifts its leg. Beginners can learn quickly.

With traditional separate controllers—for example, one for the waist and another for the lower body—you need to switch controllers after each action or have an assistant help. That interrupts the movement and requires substantial practice to operate smoothly. Whole-body teleoperation is different: it has a lower learning barrier and feels more natural and continuous.

Earlier, in May 2025, we still had some problems. Retargeting and data transmission introduced latency, making operation feel less smooth. Those problems have since improved. The system now provides more immediate feedback, making it more comfortable and natural to use.

My main takeaway so far is that building this system is about more than getting a robot to move. Making it move is a relatively simple initial goal. The next step is to let everyone collect data efficiently. I want the system to become a tool that users can employ directly, without having to make manual adjustments every time. We have only taken the first step. There is still some distance to go before it becomes the easy-to-use, efficient data-collection tool we ultimately want.

References

TWIST Project Page

https://yanjieze.com/TWIST/

Yanjie Ze's Website

https://yanjieze.com/

Open-Source Project

https://github.com/YanjieZe/TWIST

Smart Note · Human Data · EP01