The first six episodes of Robot Data focused on research involving manipulation data. This episode explores locomotion for the first time. Among approaches to robot learning, massively parallel simulation combined with reinforcement learning is perhaps the most widely accepted route for locomotion at the time of this 2024 interview. From around 2020 onward, work on quadruped locomotion with RL and NVIDIA Isaac Gym attracted sustained attention. By late 2023, advances in hardware were also bringing greater interest to RL for bipedal humanoid movement.
For Episode 7, I invited Hang Zhao and Ziwen Zhuang, the advisor and first author behind two representative projects, Robot Parkour Learning and Humanoid Parkour Learning, to discuss locomotion. Given Professor Zhao's extensive autonomous-driving experience, we also explored lessons about data from that earlier application of embodied intelligence.
Robot Parkour Learning: Ziwen Zhuang*, Zipeng Fu*, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn and Hang Zhao. *Equal first-author contribution.
Humanoid Parkour Learning: Ziwen Zhuang, Shenzhe Yao and Hang Zhao.
At the time of this interview, Hang Zhao was an assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences and leads the MARS Lab. His interests include multimodal machine learning, autonomous driving and robotics. He previously worked as a research scientist at Waymo, which grew out of Google's self-driving project. He earned his PhD at MIT under Antonio Torralba and his bachelor's degree at Zhejiang University's Chu Kochen Honors College. He is also a co-founder of the embodied AI startup Galaxea.
At the time of this August 2024 interview, Ziwen Zhuang was a first-year PhD student at Tsinghua's Institute for Interdisciplinary Information Sciences, working on motor intelligence, embodied AI and robotics. He was also a research assistant at the Shanghai Qi Zhi Institute, advised by Hang Zhao. Previously, he worked in the MARS Lab at ShanghaiTech University with Sören Schwertfeger.
Thank you to both researchers for their detailed explanations. I hope the conversation offers new insight into how large-scale simulation and RL address local locomotion, and what robotics can learn from autonomous driving's experience with data. Enjoy!
Original article outline
Contents
↓
Part 1, with Ziwen Zhuang: Robot parkour through large-scale simulation
1. The Robot Parkour Learning method
2. Simulation and RL for locomotion
3. The sim-to-real gap
4. Embodied AI datasets
5. Control theory versus reinforcement learning
Part 2, with Hang Zhao: Robot parkour, autonomous driving and the sources of robot data
1. The core challenges in Robot Parkour
2. RGB vision and future robot perception
3. Lessons about data from autonomous driving
4. How does Tesla approach end-to-end imitation learning?
5. Perception simulation versus behavior simulation
6. Behavior simulation for training or testing
7. Perception simulation and real-to-sim
8. Approaches to robot learning
Some reflections
Part 1, with Ziwen Zhuang: Robot parkour through large-scale simulation
1. The Robot Parkour Learning method
The parkour learning approach can be understood through three stages: pretraining, fine-tuning and distillation. The discussion begins with the quadruped framework and relates it to humanoid locomotion.
Stage 1: RL pretraining with soft dynamics constraints
We introduce soft dynamics constraints during pretraining. Rather than immediately enforcing a hard collision with every obstacle, we allow the robot to penetrate obstacle regions and penalize that penetration through the reward. We know the obstacles' geometry in code, but initially do not enforce the corresponding physical collision in the simulator. As the robot enters an obstacle, its reward decreases. This makes it easier to learn a behavior that progressively avoids penetration. 'Soft dynamics' here describes the learning curriculum, not a claim that all conventional simulators are incapable of compliant contact.
Soft dynamics
Stage 2: RL fine-tuning with hard dynamics constraints
During fine-tuning, we enforce the full dynamics constraints and refine the pretrained behavior under realistic simulated contact. Now the obstacle is physically present. A small penetration that was tolerated during the relaxed stage cannot occur in reality, so additional training is needed to adapt the behavior to proper collision constraints before deployment.
Hard dynamics
Stage 3: Distilling a vision-based parkour policy
Once the individual parkour skills have been learned, we use DAgger (Dataset Aggregation) to distill them into a single vision-based policy. It can run on a quadruped using only onboard perception and computation.

Distillation
One motivation for distillation was the rendering bottleneck in earlier Isaac Gym workflows. In the setup we used with an RTX 3090, an RL run might take three hours without rendering but around three days with rendering. Those figures illustrate that particular setup. We could first give a teacher privileged terrain information—a virtual terrain sensor—and then distill the policy into a student that learns to interpret deployable sensor observations. If the problem and compute budget permit direct visual training, such as with an RTX 4090, distillation is not inherently required.
Another benefit is easier sensor replacement. A policy might initially use a depth camera, then transfer through distillation to a different sensor such as lidar if the depth camera proves unsuitable. That can reduce the need to relearn all the motor skills from scratch.
2. Simulation and RL for locomotion
Before around 2020, many RL and locomotion researchers focused on algorithmic efficiency. CPU-based workflows using simulators such as MuJoCo and PyBullet offered much less parallelism in the setups being discussed—for example, roughly 256 environments on a server. Around that time, a member of Marco Hutter's Robotic Systems Lab at ETH Zurich explored scaling data through an internship at NVIDIA and large Isaac Gym experiments. GPU parallelism made thousands of environments, such as 4,096, practical, producing an important breakthrough through scale. In my account, this shifted attention strongly toward parallel data generation. Editorial clarification: these counts are setup examples, not universal CPU or GPU limits. A representative published milestone is Rudin and colleagues' Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning, first released in 2021. Scaling did not eliminate the research problem of RL sample efficiency.
3. The sim-to-real gap
The sim-to-real gap has been addressed effectively for many locomotion settings, but manipulation still presents substantial challenges. Manipulation models can simplify physics too aggressively. To lift an object, for instance, sufficient force must be applied, yet an abstract action representation may ignore its mass or force limits. Such omitted factors can make a simulated solution fail on real hardware. This is progress in particular locomotion applications, not a claim that sim-to-real transfer has been solved universally.
Manipulation also involves complex contact geometry. Finger–object interaction can create many contact surfaces, and the number of contacts can be orders of magnitude greater than in the locomotion examples we discuss. This makes simulation harder. General simulators often use iterative numerical methods to compute successive states. Complex contacts introduce accuracy challenges and computational cost, which can slow simulation substantially.
4. Embodied AI datasets
Researchers coming from computer vision are accustomed to widely accepted datasets and metrics, and naturally want unified training and evaluation benchmarks in robotics. ImageNet, for example, offered an intuitive accuracy metric; later visualization methods also helped analyze models. In robotics, an equally broad and widely accepted standard had not emerged at the time of this discussion. We still lack a dataset covering the range of situations and supporting meaningful evaluation even for groups without real hardware.
In the difficult manipulation settings I have in mind, success rates can be below 10%. Differences such as 3% versus 4% may be difficult to interpret without adequate evaluation. This makes performance assessment challenging. To improve success, systems are often tested in simplified scenes, but those tests may not show how they handle real-world complexity. Demo videos may also be sped up two, four or eight times. Many earlier manipulation formulations use quasi-static dynamics, simplifying the problem by neglecting some dynamic effects. These observations concern particular research settings and demos, not all robotics work; small percentage changes can be meaningful when supported by sufficient trials.
Quasi-static assumptions exclude many dynamic events, such as an object suddenly falling, or fast motions involved in shaking clothing or spreading a bedsheet. ALOHA helped draw attention to real-time execution and control in this context. Earlier work such as SwingBot, associated with research on dynamic manipulation, also emphasized the importance of high-speed environmental motion and inertia. Editorial clarification: the interview attributes this early line broadly to Shuran Song's group, but SwingBot's published authors include Chen Wang, Shaoxiong Wang, Branden Romero, Filipe Veiga and Edward Adelson; it should not be treated as a paper authored by Shuran Song.
For humanoids, directly mapping human motion data to a robot does not ensure physical feasibility. Early graphics models could generate plausible-looking human movement that violated contact or dynamics—for example, feet sliding over the ground as though doing a moonwalk. A robot needs a layer that converts human reference motions into executable behavior consistent with its own physics.
5. Control theory versus reinforcement learning
Mathematically, RL and control theory have many connections. In the comparison I am drawing, RL often handles uncertainty through probabilistic formulations, including analyses of prior and posterior distributions, whereas many model-based planning and control implementations use deterministic dynamics. Editorial clarification: this is a simplified comparison of common implementations. Control theory also includes stochastic and robust control, and RL is not defined by Bayesian prior/posterior analysis or excluded from deterministic settings.
An advantage of learned control is the possibility of responding robustly when a situation goes outside the planner's expected conditions. Consider the comparison with Deep Tracking Control. If a jumping platform is displaced by 0.1 meters, a model-based planner might adjust the motion slightly. If it is displaced by 0.5 meters, beyond the robot's reachable jump, the original plan may become infeasible and require a protective response. A trained policy can learn recovery behavior for such deviations, although that ability depends on its training. Editorial clarification: these distances and the Atlas example illustrate the interview's reasoning, not reported benchmark measurements from DTC. DTC itself combines model-based planning with an RL-trained tracking controller; it is not evidence that RL always succeeds where MPC fails.
Part 2, with Hang Zhao: Robot parkour, autonomous driving and the sources of robot data
1. The core challenges in Robot Parkour
Robot Parkour Learning was a finalist for the CoRL 2023 Best Systems Paper award. I see that as recognition of a complete system, rather than only one isolated technical advance. The two main challenges were designing an effective training strategy and overcoming the sim-to-real gap. Training required RL under different dynamics constraints followed by distillation of the skills.
Earlier quadruped systems often relied mainly on proprioception, without vision. We add visual perception, particularly stereo depth, making the perceptual sim-to-real gap important. We try to simulate depth observations realistically and process real observations to resemble the simulator's idealized images, bringing the distributions closer together.
We tried approaches such as hole filling and denoising. Depth images work reasonably well here because parkour primarily requires understanding terrain geometry. But future practical robots should handle RGB images as well, rather than rely only on depth.
Multitask learning is also a longstanding challenge. Gradients from different skills can conflict, degrading performance. Debugging one skill is easier than debugging several simultaneously. After exploring alternatives, Ziwen found a useful strategy: train each skill separately as a teacher, then train a shared student. This reduces performance loss and brings the components together into a systematic solution.
2. RGB vision and future robot perception
At the time of this 2024 interview, the domain gap remains difficult for RL policies using RGB observations. I expect researchers to work hard on this over 2024 and 2025: robots perceiving through RGB rather than relying exclusively on depth or point clouds. RGB contains rich information about materials, surfaces and context—for example, a sign marking a restricted area. Those semantics provide context that geometry-only depth observations do not convey.
A clarification: RGB images contain red, green and blue color channels, capturing the visual appearance of a scene. Each pixel records color, producing the images people see. Much of the internet's image and video data is RGB. A point cloud is a set of 3D points, each with coordinates such as X, Y and Z, sometimes accompanied by intensity, color or other attributes. Point clouds are commonly produced by lidar or depth cameras.
RGB does not necessarily mean only 2D perception: stereo RGB can recover depth. My longer-term expectation is that monocular or stereo RGB will dominate many robot perception systems, rather than lidar or depth-camera point clouds. That is not just a cost argument. RGB has a major advantage in available data: enormous image and video collections can support perceptual foundation models and generalization. This is my forecast, not a claim that other sensors cannot generalize or will disappear.
Look at how much text was needed for language foundation models. Images and video offer a similar opportunity for perceptual models. I am more skeptical about basing such models on lidar, depth, force or touch, because their datasets and collection pipelines are less unified and less abundant. The original emphasizes a lack of universally adopted lidar formats, transmission and compression conventions. That should be understood as a concern about fragmentation: it is not literally the absence of all standards, nor proof that foundation models for these modalities are impossible.
Visual data benefits from a mature camera industry and established internet compression and transmission pipelines. I therefore expect RGB to play a central role in robot perception foundation models. Multisensor reconstruction is useful today. In the longer term, IMU and lidar data might remain outside some foundation-model inputs while supporting ground-truth generation or evaluation benchmarks. That is an expectation about one possible data pipeline, not a necessary limitation of multimodal models.
3. Lessons about data from autonomous driving
Early autonomous driving did not frame its central challenge as data shortage or scaling in the way robotics does today. It relied heavily on traditional methods: maps for static surroundings, laser sensing for dynamic obstacles, and rules combined with search and optimization to plan trajectories. Navigation and locomotion were largely addressed through those approaches, and data's role was less central in the discussion.
The Google self-driving project, later Waymo, began in 2009. It built on the progress demonstrated in DARPA's earlier vehicle challenges. University teams had shown the feasibility of modular robotics: detection and tracking, mapping and localization, motion prediction and trajectory planning. Google then invested in a related technical approach. In my account, it continued along the broad modular route established by the 2007 Urban Challenge. Editorial correction: the original calls both the 2004 and 2007 events Urban Challenges. The 2004 event was the Grand Challenge; the Urban Challenge took place in 2007, after the 2005 Grand Challenge. The 2004 race had no finishers, while later events demonstrated major advances. 'Waymo in 2009' refers to the Google project that preceded the Waymo name.
When autonomous driving became especially popular around 2016, many companies still pursued a broadly Waymo-like modular route; Baidu's work also followed that general pattern. In my view, Tesla's growing emphasis on AI around 2017–2018 changed how people thought about the problem and later encouraged interest in end-to-end driving. I see that as an important influence on confidence in imitation learning for robots. Editorial clarification: this is the speaker's interpretation of the industry's turning point, not a priority claim that Tesla invented learning-based or end-to-end driving, or that its whole driving stack became end-to-end in 2017–2018.
Initially, people expected superior algorithms alone to create a decisive advantage in autonomous driving. My assessment now is that automakers, including Tesla and newer Chinese car companies, can move quickly through strong data loops, even if they do not always have the strongest standalone algorithms. They collect large amounts of data, select and automatically annotate useful examples, then retrain. I see that loop as central to their progress and driving experience. It does not mean data alone resolves every engineering or deployment problem.
4. How does Tesla approach end-to-end imitation learning?
Public information was limited at the time of this interview. A common interpretation was a model taking images as input and producing trajectories or control signals, potentially with intermediate outputs such as occupancy, lane markings and object detections. In that account, a multi-head network is trained jointly, with planning or control as the main output. This is the interview's architectural interpretation, not a verified specification of every Tesla model version.
The data comes from cars driven by customers, which is a clever collection mechanism. From the robot-data perspective, I compare this to mechanically matched teleoperation, like Mobile ALOHA: a person drives the same vehicle embodiment that later executes the learned behavior. The resulting observations and controls support training. I see Tesla as a powerful example of this route's potential and, in the interview, describe it as approaching Level 4 while acknowledging that it had not reached it. Editorial clarification: that proximity is the speaker's 2024 assessment, not an autonomy certification. Tesla describes customer Full Self-Driving (Supervised) as requiring active driver supervision and not making the vehicle autonomous.
5. Perception simulation versus behavior simulation
Simulation has several aspects. Perception simulation generates sensor observations, such as lidar returns or camera images. Behavior simulation models other vehicles' movement. Robots may additionally require detailed dynamics and physical parameters for contact with objects. Autonomous driving places particular emphasis on perception and behavior simulation. Waymo has done extensive work on the latter. The interview recalls public discussion of tens of millions of real miles and simulated mileage thousands or tens of thousands of times larger; these are recalled scale comparisons, rather than a precisely sourced count for a specified date.
Behavior simulation can vary an event observed on the road. If an intoxicated pedestrian crosses at one meter per second, for example, the simulator can try two meters per second or half a meter per second. For an unprotected U-turn, it can vary whether an approaching vehicle yields. This generates alternative interaction scenarios.
Companies emphasizing image-based end-to-end systems also have reason to simulate perception. Tesla and NVIDIA have used game-engine rendering to create new scenes. With very large amounts of real data, a company may depend less on synthetic perception data. In the interview, I speculate that Tesla might now use less of it, or no longer need some previous simulation workflows. That is a hypothesis, not confirmation that Tesla has stopped all simulation; plentiful real data does not remove every testing or synthetic-data use case.
6. Behavior simulation for training or testing
Prediction models other vehicles' behavior; planning selects your own vehicle's behavior. Together, these components can support a behavior simulator. In the workflows I am discussing, such simulation is widely used for testing but less for training. A distribution mismatch can bias a model if simulated interactions differ substantially from reality. This is a practical concern, not a claim that behavior simulation is never used to train driving policies.
Simulation remains valuable for testing corner cases. Even a scenario that is uncommon or not perfectly realistic can probe the limits of a model. Testing these cases helps assess its behavior beyond expected conditions, including whether it responds safely. Such tests provide evidence, rather than a guarantee of safety in all unseen situations.
7. Perception simulation and real-to-sim
I see two major recent advances in computer vision: video generation and differentiable rendering, including NeRF and 3D Gaussian Splatting, or 3D GS. The latter is especially interesting. Earlier perception simulation relied on hand-built assets and game-engine rendering, but reproducing the diversity of real-world objects is difficult. Differentiable rendering offers another route: reconstruct a real scene, then render new views to create data. From a real driving video, for example, we could ask what the camera would have seen from an adjacent lane. We call this real-to-sim.
A real-to-sim-to-real pipeline collects real robot data, augments it through a reconstructed or simulated environment, trains on the resulting data, and deploys again in reality. Standard 3D Gaussian Splatting primarily supports visual reconstruction and novel-view rendering. It does not, by itself, supply a physical dynamics model for the objects.
8. Approaches to robot learning
At the time of the interview, a common division is imitation learning for manipulation and reinforcement learning for locomotion. RL gets experience from highly parallel simulators. Imitation learning obtains real-robot demonstrations through methods such as VR and mechanically matched teleoperation, with augmentation potentially improving generalization. For researchers, the task is to keep improving data efficiency. For industry, it is to find appropriate applications and close the data loop. This describes prevalent routes, not a prohibition on RL for manipulation or imitation learning for locomotion.
Future data use may form a pyramid of real-robot data, simulation and video. I expect more useful real-world experience to strengthen models. The contribution from simulation depends on its fidelity: higher-quality simulated experience should have greater value, while poorer simulation may contribute less. These are conceptual weights, not measured universal coefficients. Learning from open-world video is a long-term ambition requiring further research.
Some reflections
Recently, robotics startups in China and abroad have been releasing new progress at regular intervals, sustaining excitement. More people continue entering the field. As someone working at an early-stage investment firm and following robotics from within, I encounter surprises every day, alongside plenty of regrets. At the time of writing in August 2024, this wave of embodied AI enthusiasm has lasted around a year and a half. Looking ahead six months, I expect evolving hardware demonstrations to keep it going.
Amid the excitement and dazzling demos, I am glad to record conversations with these excellent researchers in writing. These discussions of representative papers preserve today's uncertainties and breakthroughs. Perhaps in three, five or ten years, we will look back and realize that the route considered mainstream was not the one that ultimately worked, and appreciate the people who persisted in other directions and helped make general-purpose robots possible.
References
Robot Parkour Learning
Robot Parkour Learning: Ziwen Zhuang*, Zipeng Fu*, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn and Hang Zhao. *Equal contribution.
https://robot-parkour.github.io/
Humanoid Parkour Learning
Humanoid Parkour Learning: Ziwen Zhuang, Shenzhe Yao and Hang Zhao.
https://humanoid4parkour.github.io/