Back to Human Data

Human Data · EP07

Human Data EP07 — Motion Capture: TWIST2

An interview on TWIST2, low-fidelity motion capture, wearable VR devices, and large-scale humanoid data collection.

Zha Dami2025-11-148 min readWeChat original
Contents

In the first installment of Human Data, we interviewed the team behind TWIST (Human Data EP01—MoCap: TWIST), a project that uses motion capture equipment as an interface for humanoid teleoperation. We conclude the paper interviews in this series with Twist2. Compared with Twist1, Twist2 uses less precise teleoperation hardware: VR, controllers, and leg trackers replace optical motion capture equipment for recording human movements. It also introduces a VR visual feedback system. Yanjie Ze, Twist2’s first author, is a familiar guest from our interviews. Let’s take a closer look at this work.

TWIST2: A System for Large-Scale Humanoid Data Collection. The work was carried out by researchers from Amazon FAR, Stanford, UC Berkeley, CMU, and USC. Large-scale humanoid data can be collected within 15 minutes. Website: https://yanjieze.com/TWIST2 Watch the demonstration videos on the project page: https://yanjieze.com/TWIST2/ Scope note: the 15-minute result refers to demonstrations of the evaluated pick-and-place tasks, rather than an arbitrary quantity of data for every humanoid task.

Original article outline

Contents

👇

1. The Core Idea Behind Twist2

2. Accuracy of VR and Wearable Devices

3. Does Low-Precision Motion Capture Need High-Precision Training Data?

4. The Twist2 Method

5. The Hardware Behind Twist2

1. The Core Idea Behind Twist2

After finishing Twist1, my strongest impression was that it still was not convenient enough for everyone to use. Twist1 aimed to enable high-quality whole-body data collection. But once we finished it, we found that although the system worked, we had not really gotten data collection going. That was a central pain point. Meanwhile, most collection approaches in the industry serve simple control or imitation tasks, and their quality is generally low. They usually produce disconnected fragments—“walk a few steps, stop, do something”—rather than continuous, high-quality whole-body data.

Twist2 therefore set out to resolve this tension: preserve high-quality whole-body data like Twist1, while achieving greater collection efficiency. These goals may seem difficult to reconcile, but we wanted to achieve both.

Twist2 system illustration

The first major change was replacing traditional whole-body motion capture equipment with an entirely VR-based approach. VR does not require external cameras or lengthy calibration. Unlike a motion capture suit, it is not complicated to put on and does not require long preparation. This makes the system lighter and lowers the barrier to using it.

The second change was streaming the robot’s viewpoint into VR in real time so that the operator can directly see what the robot sees. This is crucial. Even with a whole-body VR system, data quality remains poor if the operator cannot see from the robot’s perspective. Relying only on a third-person view prevents continuous, precise operation. We therefore designed an active vision module on the robot and streamed its visual input into VR in real time, substantially improving both operation efficiency and data quality.

The third change was refining the whole setup into a complete system that runs continuously. It is more than making individual functions work: we integrated continuous control across the entire workflow. This includes starting and stopping the system with the controllers, pausing and resuming data collection, and reliably recording long interactions. These seemingly small details substantially improved the stability and reliability of data collection.

Finally, we actually put the collected data to use. We connected it to different policies and deployed them in the human–robot system. This enabled what we describe as the first academic experiments training manipulation and locomotion with a vision-based collection system that treats the whole body jointly, rather than separating the upper and lower body. This is an important breakthrough and marks the fundamental advance of Twist2 over Twist1.

2. Accuracy of VR and Wearable Devices

Twist2’s motion capture approach uses VR and wearable devices. Its human pose estimation is indeed less accurate than high-precision motion capture solutions such as Noitom or Vicon.

We considered this during system design and used several methods to bridge the accuracy gap. First, from a reinforcement learning or imitation learning perspective, even if human pose estimates have some error in the global coordinate system, the data remains usable as long as it is consistent in a local reference frame. In other words, the relative structure of movement is coherent within the VR system’s local coordinates. There may be global inaccuracies, but these errors are acceptable for a model learning control policies.

We also designed a local interface to improve stability. In this design, interaction between the human and robot is based on local correspondences, rather than a global coordinate system. In comparison, systems such as Peking University’s CLONE use global registration. They require highly accurate pose estimates for both the person and the robot, then pair them through a strict global mapping.

Our system has no such requirement. This allows longer, more continuous task execution. Even when drift occurs, the person is teleoperating in real time. Active human control can continually correct deviations during operation, dynamically compensating for these errors through interaction.

The advantage is that this approach addresses inaccurate pose estimation on both the human and robot sides.

For the robot, extremely accurate pose estimation requires a state estimation module, such as LIO or VIO algorithms. For the human operator, an accurate global pose requires an expensive, complex VR tracking system. Our approach avoids these complications. Local alignment and real-time correction simplify the overall system while still enabling stable, continuous operation.

3. Does Low-Precision Motion Capture Need High-Precision Training Data?

In theory, it is indeed possible to train or operate directly with low-precision VR motion data, without high-precision motion capture data. It is “possible.”

In practice, however, we take a hybrid approach. We still primarily use high-precision motion capture data to train our models and controllers. About 99% of the data comes from high-precision motion capture systems, including data collected for Twist1. This data provides very stable spatiotemporal consistency and accurate skeletal poses. It forms the core foundation for learning movement patterns and control strategies.

At the same time, we use around 1% low-precision data to help the model adapt to noisier inputs, especially in body regions where VR systems are less accurate, such as the legs and waist. Drift is more pronounced in these areas. We therefore mix a small amount of low-precision data into the high-precision data, helping the model learn some robustness during training so that it can adapt automatically when deployed with lower-precision systems.

In other words, this is a mixed training strategy that primarily uses high-precision data and supplements it with low-precision data. It preserves the reliability of high-precision motion capture while giving the model the adaptability to operate with lower-precision systems.

4. The Twist2 Method

The system has two main parts: system design and the learning system.

Twist2 system illustration

The overall system design follows much the same logic as Twist1: we take motion capture data and process it with GMR. Twist2 uses the complete version of GMR, whereas Twist1 used only an earlier or reduced version. Twist2 also modifies the controller and adds processing for VR and vision.

Twist2 system illustration

The learning system has a hierarchical structure. We first use a generative model to generate human movements. It produces an intermediate representation, which we feed into a controller that imitates the human teleoperation process. A person first looks at the scene, obtains an observation, and produces an action. The controller then converts that action into movements the robot can execute.

This design lets the robot perform relatively continuous upper-body movements while also carrying out lower-body actions such as kicking.

5. The Hardware Behind Twist2

We use PICO’s VR system. It was the only equipment that met our requirements, with matching leg trackers, controllers, and a complete ecosystem. Our system needs both visual transmission and whole-body pose estimation, and this equipment supports all these functions within a single ecosystem. This comparison reflects the project’s hardware and integrated software requirements at the time of the interview, rather than a permanent limitation of every other VR platform.

Vision Pro and Quest, by contrast, could not meet those requirements. Although they offer some capabilities in vision or hand recognition, they lack complete interaction support and cannot implement the entire workflow within the same application ecosystem.

VR controllers provide very good tracking accuracy. Our tracking was previously not stable enough, making particularly complex interactions difficult in Twist1. With Twist2, we have better wrist tracking: tracking of both the hands and wrists is more precise.

This is crucial for manipulation, because hand tracking determines the accuracy and fluidity of object interactions. The current system is flexible enough for operators to naturally rotate their wrists, adjust their posture, and perform a range of dexterous movements. It also enables more interesting tasks, such as the complex interactions involved in folding clothes.

The equipment still has limitations in whole-body human motion capture. The main issue is inaccuracy in the global coordinate system. Its absolute position in space can be off, with drift becoming more pronounced during long operations or movement over large areas.

We currently address this with local coordinate systems. We align the person and robot within their respective local frames, rather than forcing them to rely on a global reference frame. Global pose estimation has long been a difficult problem. This is not unique to this equipment; many other systems have not fully solved it either.

If global coordinate accuracy is essential, one option is engineering optimization: improving hardware synchronization, optimizing algorithms, and implementing better state estimation. Another is the approach we use—working with local coordinate systems.

The reasoning is that people do not actually rely on global coordinates when they act. When walking, for example, you do not think about how many meters you have moved “globally.” You use vision to judge how many steps remain to the door, walk over, and stop. Human behavior is based on visual perception and relative distance estimation, rather than absolute coordinates.

If the goal is more general intelligence, strong reliance on a global coordinate system is therefore unnecessary. A system only needs to understand its environment visually, determine whether it has reached its goal, and recognize whether it has interacted with an object to complete the task.

Traditional robot systems, such as navigation systems, still use global localization, path planning, and related modules to maintain global consistency. This remains effective in certain fixed settings. For embodied AI or more flexible teleoperation systems, however, local perception and visual judgment may be a more natural and practical route.

References

Twist2:

https://yanjieze.com/TWIST2/

Smart Note · Human Data · EP07