Back to Robot Data

Robot Data · EP03

Robot Data: Xuxin Cheng on Immersive Robot Teleoperation with Open-TeleVision

Xuxin Cheng discusses stereo vision, active head movement, motion retargeting and learning robot skills from immersive teleoperation.

Zha Dami2024-07-0713 min readWeChat original
Contents

In The Three-Body Problem, Wang Miao puts on VR and motion capture equipment to enter the Three Body game and explore its world. If Elon Musk establishes a Mars colony, would you board Starship yourself, or first rent an Optimus avatar to experience life on Mars for you? I would certainly rent the robot first, using VR and motion capture equipment like those in the novel. But how would you control an Optimus all the way on Mars? Bringing the idea closer to reality, researchers at UC San Diego and MIT released Open-TeleVision in 2024, a system pointing toward that possibility. The Mars scenario is speculative: the demonstrated system does not remove the substantial communication delay between Earth and Mars.

Open-TeleVision is a robot teleoperation system based on a VR headset. It supports different headset platforms and, with suitable mapping, can connect to different robots. It combines human-like stereo vision with a movable neck, resembling the way we observe our everyday surroundings. An operator only needs the headset, without other wearable equipment, to control a robot remotely.

We interviewed Xuxin Cheng, a first author of Open-TeleVision: Teleoperation with Immersive Active Visual Feedback. At the time of this interview, he was a PhD student at UC San Diego, advised by Xiaolong Wang, having previously completed a master's degree in Deepak Pathak's group at CMU. He has produced strong work in quadruped and biped locomotion, including Expressive Whole-Body Control for Humanoid Robots and Extreme Parkour with Legged Robots, released in 2024 and 2023 respectively (the original introduction groups both under the previous year). Let's take an immersive look at how Open-TeleVision lets people teleoperate robots.

Original article outline

Contents

👇

1. Introducing Open-TeleVision

Open-TeleVision's core idea

How are the robot's two eye cameras configured?

Were you the first to use Vision Pro for humanoid teleoperation?

Matching human and humanoid degrees of freedom

What does mapping involve?

How well does learning from this teleoperation data work?

Does the system require Vision Pro?

How can remote hand control be made more precise?

What limitations does Open-TeleVision have?

2. Training humanoid body control

Why not train humanoids with RL alone, like quadrupeds?

Motion capture datasets

How does RL with motion capture differ from imitation learning?

What approaches are used to train humanoids today?

Different training methods for the upper and lower body

Why does locomotion work well with RL rather than IL?

What makes Extreme Parkour distinctive?

3. Some personal reflections

1. Introducing Open-TeleVision

Open-TeleVision's core idea

Teleoperation is not new. Many earlier projects collected data through teleoperation and then used imitation learning, or learning from demonstrations. The most direct way to deploy learned skills on a robot is to collect on-robot data: recordings of that robot actually performing tasks. Earlier work such as ALOHA explored controlling robots through joint-copy teleoperation.

These approaches have several limitations. First, the operator must stand beside the robot, and the difference between their viewpoints makes control less intuitive. Second, a teleoperation interface must have hardware similar to the task-performing robot, requiring two matching setups and increasing cost. Third, close-proximity operation creates a perception mismatch. A person has a broad field of view, whereas the robot sees through cameras fixed at particular positions—above, in front, behind or on the gripper. The operator may make decisions using information the robot cannot see, which creates a mismatch when training a policy. Mobile ALOHA uses four cameras to cover a larger workspace and lets the operator stand behind the system, making it relatively intuitive.

We propose a new approach addressing these issues and the characteristics of humanoids. Standing behind a humanoid does not work well because its torso blocks the view; standing beside it and constantly turning your neck is also awkward. A humanoid is mobile, so installing many fixed cameras is impractical. The most intuitive solution is immersive control, as though the operator becomes the robot. That inspired Open-TeleVision. Instead of standing beside or behind it and operating from a third-person perspective, the person puts on a headset and controls it immersively.

This form of teleoperation has several features.

First is immersion. The operator sees stereo video rather than a flat 2D feed, much like watching a 3D film. Each eye receives a different image, creating natural depth perception. This enables precise distance judgments for delicate tasks, such as packing a transparent box or handling small items. Without stereo vision, judging distance accurately is difficult, and a first attempt may fail.

Video demonstration from the original article

Accurately aligning transparent plastic tubes

Video demonstration from the original article

Precisely handling transparent plastic boxes and small earplugs

Second, the robot has an active neck so its head can move. People focus on tasks through a small central retinal region, or foveal vision. With a fixed camera, looking elsewhere requires shifting the gaze or relying on peripheral vision, which is awkward. The active neck makes control more immersive and intuitive. The operator simply turns their head toward the desired area rather than looking sideways or through peripheral vision.

Robot Data: Xuxin Cheng on Immersive Robot Teleoperation with Open-TeleVision · Original article illustration

With the active neck, inverse kinematics and retargeting, the operator simply looks toward a point of interest and the robot follows the same head movement.

Video demonstration from the original article

Following head movements

Third, the active neck reduces computation. The operator only needs a small central camera view rather than a high-resolution wide-angle image. Large images contain considerable irrelevant visual information and demand more processing. Focusing on a smaller view and turning the head to see what is needed improves efficiency substantially. Our system achieves roughly 80 Hz, compared with 40 Hz when processing larger images. The higher frame rate makes the system much more intuitive.

Those are the visual components. Translating the operator's movements to the robot is equally important. We use a good inverse kinematics algorithm to map the information provided by the Vision Pro API onto the robot. This is mainly engineering implementation using relatively mature techniques, which can reach the desired performance through system-level optimization.

The perception and active-neck components are built on a web platform. We use Vuer, a framework developed by Ge Yang at MIT, which is essentially similar to a web server and uses existing network technologies for remote data transmission. Think of video conferencing or video calls, with the low latency and responsiveness needed for an online game. The system is easily accessible from a web browser on devices including Vision Pro, Quest, Mac, iPad and iPhone.

Robot Data: Xuxin Cheng on Immersive Robot Teleoperation with Open-TeleVision · Original article illustration

Overall, the system provides low-latency, efficient teleoperation so that the operator can control the robot quickly and accurately in real time.

Video demonstration from the original article

Good responsiveness and precision

How are the robot's two eye cameras configured?

To provide realistic stereo vision, the system must approximate the spacing of human eyes. The two RGB cameras should be separated by roughly that distance, neither too far apart nor too close, to create a natural view and avoid dizziness. Two webcams placed appropriately can produce the same effect. The cameras we use have hardware synchronization and good APIs, so we directly use those existing features.

Were you the first to use Vision Pro for humanoid teleoperation?

An MIT team is doing related work with a somewhat different implementation. VisionProTeleop from Pulkit's group includes a visionOS application and Python library, streaming head, wrist and hand tracking from the Vision Pro API over Wi-Fi using gRPC. Any robot on the same Wi-Fi network can subscribe to that data. But the operator still has to stand beside the robot and look across at it. Recent projects such as OmniH2O and HumanPlus also explore projecting human whole-body motion onto humanoids.

Our innovation is the human-like configuration of the two cameras. Wearing Vision Pro puts the operator into the robot's perspective, making control more natural and intuitive. Web-based transmission also enables synchronization over long distances.

Matching human and humanoid degrees of freedom

A mismatch in degrees of freedom does not greatly affect our approach as long as the humanoid has fewer than a person. Our end effectors, for example, are not limited to five-fingered hands; our website also shows grippers. It depends on the mapping. Suitable mappings can control almost any robot whose degrees of freedom fall within the human range. A robot with three arms or six fingers would pose different challenges. The video shows that both Fourier's GR-1 and Unitree's H1 work, with grippers or five-fingered hands. In practice, compatibility also depends on kinematics, joint limits and suitable control interfaces; a degree-of-freedom count alone is not sufficient.

Video demonstration from the original article

Compatible with different robots

What does mapping involve?

Mapping generally uses retargeting or inverse kinematics to infer joint configurations from end-effector movements. When the robot's degrees of freedom are below the human baseline, control can be implemented effectively. More degrees of freedom introduce redundancy and require more complex treatment. The robot hands available to us at the time have far fewer degrees of freedom than human hands: a human hand may have more than twenty, while the robot hands we use typically have only six. This prevents certain complex movements, such as spreading four fingers or moving the thumb flexibly.

How well does learning from this teleoperation data work?

It works very well for teaching autonomous task policies, showing that the collected data is effective. In particular, we validated the efficiency of combining neck movement with two RGB cameras for robot training.

Video demonstration from the original article

Collecting robot data through Vision Pro teleoperation

Our tasks emphasize long horizons and some generalization. In can placement, for example, the robot can grasp the can from different starting positions. Some earlier demonstrations might succeed only once after several attempts, whereas our system succeeded twelve times in a row with the can inserted at a different position each time. This demonstrates reliability in both decision-making and execution.

Video demonstration from the original article

Training autonomous tasks with teleoperation data

Does the system require Vision Pro?

We use Apple's API directly rather than building our own recognition system from open-source components. Vision Pro's visual tracking is highly accurate, especially for hands. But other headsets such as Meta Quest also work with our system and cost much less.

Video demonstration from the original article

Compatible with different headsets

How can remote hand control be made more precise?

We could add tactile or other feedback, and there are several directions to explore. One is a glove that reproduces sensations transmitted from the robot. A simpler option is for the operator to hold a similar object when the robot picks one up remotely.

That simple approach conveys limited information: if the robot drops its object while the operator still holds theirs, the sensations clearly diverge. Tactile feedback needs further exploration. We have not added it yet, but we find that vision alone sometimes lets people infer tactile information. People are good at adapting to and interpreting these signals. By observing how a hand releases a can during placement, for example, we can roughly infer the force relationship between hand and object.

What limitations does Open-TeleVision have?

The main issue is that mapping is less direct than joint copying. With a six-degree-of-freedom arm, joint copying simply copies each motor position to the other device. We need inverse kinematics for the end effector, which can produce unnatural movements. Early adjustments may be imperfect and cause strange motions.

Another issue is latency. Viewing through a camera introduces some delay. Compared with ALOHA, where the person sees directly and immediately teleoperates the robot, our system inevitably has additional latency. Tactile perception is another potential improvement, although current technology has not fully solved it.

2. Training humanoid body control

Cheng has also produced strong work on quadruped and humanoid control, including Expressive Whole-Body Control for Humanoid Robots (RSS 2024) and the earlier Extreme Parkour with Legged Robots (ICRA 2024). The original dates the first release to before the previous Spring Festival; the paper’s first arXiv submission is dated February 26, 2024. We used these projects as a starting point to discuss control learning for humanoids and quadrupeds.

Why not train humanoids with RL alone, like quadrupeds?

Earlier quadruped work mainly used simulation data, with rewards designed to make the gait look natural. Humanoids have more degrees of freedom, making it hard to generate good movements through reward functions alone. We cannot design an appropriate reward function for every movement we want.

We therefore use human motion capture data alongside simulation to guide RL training. It constrains those additional degrees of freedom and enables diverse movements. The motion source can vary, including videos found on YouTube. After processing into a format the robot accepts, the movements can be executed even if the robot has not seen them before.

Motion capture datasets

We use a motion dataset during training. Once training is complete, new movements need no additional training and can be passed directly to the robot zero-shot. We use CMU MoCap, an old and classic motion capture dataset. This zero-shot claim refers to new reference motions within the controller’s supported motion range, rather than arbitrary physically infeasible motions.

How does learning from human motion capture differ from imitation learning?

It is difficult to draw an exact boundary. This is not purely supervised learning, but imitating with guidance signals, such as instructions, can still count as imitation learning. The closer your movement is to the human data, the higher the reward. It is a kind of indirect imitation that also optimizes other factors, implemented through reinforcement learning.

What approaches are used to train humanoids today?

My understanding is that the broad idea is similar: obtain human motion somehow, retarget it to suitable robot motion and use an RL policy to track it. Locomotion does not require especially elaborate methods; it is mainly teacher–student training followed by sim-to-real transfer.

How do you handle different training strategies for the upper and lower body?

Some projects use whole-body RL, which may improve coordination but lack precise upper-body control. Future work may unify imitation learning for upper-body manipulation and reinforcement learning for lower-body locomotion within a teacher–student framework. Locomotion training already uses a privileged RL teacher in simulation and an imitation-trained student, somewhat like manipulation training. Conventional manipulation imitation learning uses a human teacher directly rather than obtaining one through RL. People cannot directly demonstrate a robot's walking control, so locomotion teachers are trained with RL in simulation.

Why does locomotion work well with RL rather than IL?

The key is the limitation of copying human movements. Video-derived and retargeted motions may be kinematically feasible in a computer model but not dynamically or physically feasible. They do not necessarily obey real physical constraints. Training with RL in simulation teaches the robot to respect the engine's modeled dynamics and kinematics, helping it handle the complex mechanical requirements of the real world.

Teachers and students have different roles. A teacher usually receives more privileged information, whether the task is parkour or walking without vision. Insufficient exploration makes optimal behavior difficult to learn, so we give the teacher as much information as possible. Simulation provides all kinds of information that can be supplied directly. For a quadruped, for example, we can tell it exactly how much weight it is carrying, helping the teacher learn better and faster.

The student must work in reality, so its training cannot depend on privileged simulation information. It needs observations available in the real world. The overall idea is to train a working teacher as quickly and effectively as possible in simulation, then distill it into something deployable in reality.

What makes Extreme Parkour distinctive?

The broad idea is the same, but we add visual feedback rather than having the robot move blindly. Within teacher–student learning, it must execute foot movements precisely while adjusting direction in real time. Simultaneous teacher–student learning can be challenging, so we use mixed teacher–student learning to improve data quality and ultimately train the system.

3. Some personal reflections

Since switching in 2023 from watching variety shows every day to reading papers, I have often wondered why some people know what they want to do before university. Many robotics learning papers are written by exceptional students: undergraduate degrees from Tsinghua, Peking University, Shanghai Jiao Tong or Zhejiang University, often in the Yao, Turing, ACM or Chu Kochen programs, followed by PhDs at MIT, Stanford, Berkeley or CMU. Looking at their paths from the outside, it seems they wasted no time choosing a field. Some knew in primary school, middle school or high school that they liked robotics or computing and wanted to work in robotics or AI. I admire that clarity about their interests and abilities. I still do not know exactly what I like, so I work in investing—knowing a little about everything without mastering any one thing.

Returning to investing, it was only after studying embodied AI deeply in 2023 that I understood why certain leading dollar-denominated funds can reach the strongest founders: their ability and commitment to finding outstanding people are exceptional, far beyond what an ordinary small fund can match. Since June 2023, I have focused only on robotics—more precisely, robot learning, and more precisely still, general-purpose manipulation through learning. After a year, I have a good sense of the people in the field, but mostly through an internet connection, with few chances to spend time face to face. Some investors have already known these researchers for years. Seeing the gap shows where there is room to improve. Keep going.

References

https://github.com/Improbable-AI/VisionProTeleop/blob/main/assets/short_paper_new.pdf

https://github.com/Improbable-AI/VisionProTeleop

https://extreme-parkour.github.io/

https://expressive-humanoid.github.io/

https://chengxuxin.github.io/

https://robot-tv.github.io/

Smart Note · Robot Data · EP03