In the Robot Data series, we interview authors of representative research papers about data for training robots. Among these projects, Mobile ALOHA from Chelsea Finn's group at Stanford has been one of the best-known contributions to imitation learning and teleoperation since 2023. The group released ALOHA and ACT in the first half of 2023, Mobile ALOHA toward the end of that year, and HumanPlus in the first half of 2024. These projects, which use teleoperation to train robots, have attracted considerable attention from academia, industry and technology media. For the fifth episode of Robot Data, we are delighted to welcome Zipeng Fu, a first author of Mobile ALOHA and HumanPlus, to discuss the data stories behind both projects.
At the time of this interview, Zipeng Fu is a computer science PhD student at the Stanford Artificial Intelligence Laboratory, advised by Chelsea Finn. He previously earned a master's degree at Carnegie Mellon's Robotics Institute, working with Deepak Pathak and Jitendra Malik. He completed his undergraduate studies in computer science and applied mathematics at UCLA, where he was advised by Song-Chun Zhu. His research lies at the intersection of robotics, machine learning and computer vision, with the aim of achieving robust robot performance and deployable robotic systems in unstructured, open environments. Biographical note: Fu's official biography specifies a master's degree in CMU's Machine Learning Department and research work in its Robotics Institute.
Earlier in his research career, Fu produced a number of impressive projects on quadruped locomotion. His recent work focuses on solving mobile manipulation through teleoperation and human video data. He is an unusually versatile researcher, working across simulation data, teleoperation data and human video, as well as across locomotion and manipulation. In this article, we use HumanPlus as a starting point to explore his insights into robot training and data.
Original article outline
Contents
👇
1. From synthetic data to real data, and combining the two
Training quadrupeds with synthetic data
Mobile ALOHA: real-world teleoperation data
HumanPlus: combining synthetic and teleoperation data
2. HumanPlus: making a humanoid move like a human's shadow
Where HumanPlus gets its data
The HumanPlus architecture
3. Questions and discussion
What does it mean for a humanoid to shadow a person?
Comparing approaches to humanoid control
Realism in simulation
What happens when the hardware changes?
Why use a single RGB camera?
Hierarchical versus end-to-end approaches
4. Additional notes
1. From synthetic data to real data, and combining the two
Training quadrupeds with synthetic data
My own robotics research began with training quadruped robots. We used simulation software to generate large amounts of synthetic data and trained the robots through reinforcement learning. This approach works very well in certain settings: teaching a quadruped to walk, coordinating a robotic arm mounted on a quadruped, or teaching a quadruped parkour, for example. Simulation can generate data at scale. With reinforcement learning, we can tell the robot to move forward, avoid collisions while doing so, and minimize effort and energy consumption; it can then learn these behaviors for itself. We do not need real-world data—or, in a sense, there is no suitable real-world data available—because I cannot train a whole pack of pet dogs to collect it.
But combining simulation data with reinforcement learning also reaches a bottleneck. It is particularly good at movements such as locomotion—moving the limbs—or in-hand object reorientation. OpenAI, for example, also used generated data to train a dexterous hand. When a task requires higher-level decisions and complex planning, however, the robot needs more visual information. Current simulation software cannot provide this real-world visual information particularly well. It is better at supplying physical collisions and structured information, such as depth data. Rendering realistic color information is comparatively difficult.
Mobile ALOHA: real-world teleoperation data
With Mobile ALOHA, we moved to a setting that uses no information from simulation software at all. The robot learns entirely from real-world data. People collect this data by teleoperating the robot in the real world, and we use efficient imitation learning to train it with a small dataset. This gives us better, more realistic visual signals, including RGB data and semantic information.
One challenge is that collecting real-world data is slow. Mobile ALOHA therefore combines its data collection system with an efficient imitation learning algorithm: roughly 50 human demonstrations per task can enable a robot to learn and perform a range of tasks. The system helps people collect data efficiently. Compared with earlier tools such as VR devices, 3D mice and keyboards, it is both faster and easier to learn. A central contribution of Mobile ALOHA is helping people quickly learn how to operate a robot so that they can address almost any problem they might encounter in the real world. Experimental note: The reported Mobile ALOHA results also use co-training with existing static ALOHA data; this is not a claim of general-purpose competence from 50 demonstrations in total.
Mobile ALOHA
HumanPlus: combining synthetic and teleoperation data
With HumanPlus, we tried to combine the strengths of synthetic data and robot teleoperation data. On one side, simulation gives us enormous quantities of data, but the quality is limited. On the other, real-world data is high-quality, but collection is slow and cannot reach the same scale. The main question behind HumanPlus is whether we can combine the advantages of both: collect high-quality data in the real world while also making use of large amounts of synthetic data.
HumanPlus
2. HumanPlus: making a humanoid move like a human's shadow
Where HumanPlus gets its data
HumanPlus combines two sources of data: robot simulation data and human motion data.
For the simulation component, we cannot simply copy human motion onto a robot. A robot's weight and joint structure differ from a person's, so direct copying can cause it to lose its balance and fall. At the lowest level of control, we therefore still use reinforcement learning in simulation to generate large amounts of data and train the robot. When the robot receives signals specifying human joint positions, it actively controls its movements to imitate those signals while accounting for its own physical constraints and maintaining balance. This is intended to let it imitate human motion while avoiding instability and falls.
As for human motion data, there are not many robots in the real world, but there is a great deal of data about people. Many videos, for instance, show people dancing or cooking. This data already exists in enormous quantities. We are thinking about how to use it well and transfer it effectively to humanoid robots. A major advantage of the humanoid form is its potential to use abundant human data. Our work uses this data to give humanoids fine-grained control.
In most cases, we need to imitate the human movements shown in a video. These videos are typically recordings of everyday life made with a camera placed nearby or worn on the person's head, such as an action camera. We focus on making good use of the data obtainable within the limitations of a single RGB camera. From its images, we capture skeletal signals, including joint data for the body, hands, arms and legs. These signals control the robot in real time so that it can perform different tasks.
We also combine this with efficient imitation learning, similar to Mobile ALOHA, so that the robot can perform tasks autonomously. It obtains visual information from an RGB camera on its head and acts on that information. It can now autonomously perform sequences of tasks: sit on a chair, put a shoe on its foot, tie the laces, stand up and walk. These capabilities come from training with human data. Verification note: The project page documents putting on a shoe, standing up and walking. Tying the laces is retained here as an interview claim and is not separately established by that page.
By learning to imitate human video data, we can turn human movement patterns into robot control policies. It is rather like having a neural network control a person's movements, with the network's final outputs mapped directly onto a humanoid robot. Because a humanoid resembles a person in its physical form, this mapping is relatively straightforward and intuitive. That allows us to use human data more effectively to train and control robots that perform human-like movements and tasks. Method note: In the demonstrated autonomous-skill pipeline, human shadowing is used to collect robot demonstrations, and behavior cloning learns from those robot observations and actions. This is distinct from directly training an autonomous skill policy on arbitrary internet videos.
The HumanPlus architecture

The HumanPlus architecture has two parts. First, we train a Humanoid Shadowing Transformer in simulation: reinforcement learning produces a low-level controller that lets the robot follow a person's movements like a shadow while respecting its own physical properties and keeping its balance, as shown on the left of the diagram. Second, we train through imitation learning with human data using the Humanoid Imitation Transformer, or HIT. The low-level controller trained in the first stage is deployed zero-shot on a real humanoid for real-time shadowing.
In the shadowing stage, a human operator stands near the humanoid and projects their whole-body movements onto it in real time, as if the robot were their shadow. The operator observes the environment and the robot's behavior directly, ensuring a responsive and flexible system for efficient teleoperation. The low-level controller takes humanoid proprioception and target poses as its inputs. Proprioception includes the base state—roll, pitch and base angular velocity—as well as joint positions, joint velocities and the previous action. These observations come only from onboard sensors: an IMU and joint encoders. Target poses include desired forward and lateral velocities, roll and pitch, yaw velocity and joint angles. We use a single RGB camera to estimate body and hand poses in real time, then retarget the human pose to the humanoid's target pose. The controller outputs 19-dimensional body joint position setpoints, which are subsequently converted into torque signals.
In the autonomous-skills stage, the robot learns human skills through imitation learning and performs the corresponding tasks independently. The humanoid imitation learning framework, HIT, requires more than simple imitation: the robot must shadow human movement and also learn from its own observations to develop skills. It needs proprioceptive information and must process RGB images from its left and right eyes. Its inputs are images from two egocentric RGB cameras, proprioception and fixed positional embeddings. Its outputs are predicted target positions, which are sent to the low-level Humanoid Shadowing Transformer; predicted future image-feature tokens are discarded. In effect, it learns to produce the inputs for the low-level controller shown on the left—that is, the target poses we want the robot to reach.
3. Questions and discussion
What does it mean for a humanoid to shadow a person?
Making a humanoid shadow a person's movements is not merely a matter of matching them in zero gravity. Without gravity, the robot does not need to learn how to cope with it. A robot floating in space, for instance, can move freely without gravitational constraints and can more easily match human movements. On the ground, gravity is a challenge. A person can stand on one leg, but a robot trying to copy that pose may lose its balance or fall.
Comparing approaches to humanoid control
Speaking about the hardware landscape at the time of this July 2024 interview, at present, most robots on the market can only perform basic forward and backward walking, and some even struggle to stand. They lack a diverse movement repertoire: they cannot move their arms like a person or crouch to pick up an object near the ground. Most existing robots cannot perform these actions. One feature of HumanPlus is that its upper and lower limbs are trained through the same neural network, giving it good coordination.
Realism in simulation
However realistic an object in simulation looks, the human eye can generally tell whether it is real or synthetic. Neural networks can usually detect the differences that people can see, too. They are particularly good at noticing subtle distinctions, even very small details. Current simulators may handle variations in color and weight, but simulating softness, deformation and complex visual scenes becomes difficult.
What happens when the hardware changes?
Although the training process is similar, changing humanoid hardware requires retraining the algorithm. Different robots have different limb lengths and weights, which affect their motion control and balance.
Why use a single RGB camera?
The idea is to use a single camera to better match the way existing videos are recorded. VR equipment, full-body exoskeletons and motion capture systems are expensive, and they also do not match existing internet data particularly well. Our first step is to make sure the entire data processing pipeline works smoothly, rather than immediately relying on internet data for learning. The second step is to consider how to use internet data for further learning.
Hierarchical versus end-to-end approaches
Mobile ALOHA is an end-to-end model. It has no planning stage in the traditional sense, where a detailed plan is created and then executed. Our approach goes directly from perception to action. Planning is effectively integrated into the neural network's weights. I prefer the end-to-end route to a hierarchical approach to embodied intelligence. If the data comes from the same source, there is no need to divide the system into layers. Human data is the one case where a hierarchy may be useful, because some data comes from simulation and some comes from real-world robots and people. But this is different from separating perception, planning and execution: it is more a hierarchy for making use of different data sources.
4. Additional notes
The authors of HumanPlus: Humanoid Shadowing and Imitation from Humans are Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein and Chelsea Finn. Zipeng Fu, Qingqing Zhao and Qi Wu are joint first authors. The authors of Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation are Zipeng Fu, Tony Z. Zhao and Chelsea Finn; Zipeng Fu and Tony Z. Zhao are joint first authors.
Robot Data has now reached its fifth episode. The first four covered EP01, ATM: how robots learn from human video, in Yang Gao and Chuan Wen on Learning Robot Foundation Models from Internet Videos; EP02, UMI: a simple gripper-shaped motion capture device, in Cheng Chi and Zhenjia Xu on UMI, a Physical API for Robot Training; EP03, Open-TeleVision: teleoperating robots with a headset, in Xuxin Cheng on Immersive Robot Teleoperation with Open-TeleVision; and EP04, MimicPlay and DexCap: combining human video and motion capture to train robots, in Chen Wang on Training Robots with Human Video and Motion Capture. Readers interested in these topics can explore the earlier episodes.
References
https://humanoid-ai.github.io/
https://mobile-aloha.github.io/