Back to Human Data

Human Data · EP04

Human Data, Episode 4—Video: Junyi Zhang on VideoMimic

Reconstructing human motion and scene geometry from casual videos, training contextual humanoid control, and balancing scalable video data with precise motion capture.

Zha Dami2025-11-0418 min readWeChat original
Contents

The first three episodes of Human Data focused on using human motion-capture data as a source for humanoid control: Human Data, Episode 1—MoCap: TWIST; Human Data, Episode 2—MoCap: GMT; and Human Data, Episode 3—MoCap: ExBody2. This time, we turn to VideoMimic, which uses video data as a source for humanoid motion control.

VideoMimic lets humanoid robots learn movements from casually recorded smartphone videos of people. One of its co-first authors is Junyi Zhang. At the time of this interview, he was a PhD student at UC Berkeley working with Professor Trevor Darrell, having completed his undergraduate studies at Shanghai Jiao Tong University. This conversation covers reconstructing dynamic 3D human point clouds from 2D video, thinking about a data pyramid of human video and high-precision motion capture, and choosing between reinforcement learning and imitation learning for whole-body humanoid control. I found it insightful and recommend reading it.

Original article outline

Contents

↓

1. The Core Idea of VideoMimic

2. The VideoMimic Method

Part One: Reconstructing Human Motion in 3D

Human Motion Reconstruction

Human Motion Reconstruction, Continued

Video Data

How MonST3R Reconstructs Dynamic Point Clouds from Ordinary Video

Part Two: Training Humanoid Motion

3. Using Motion-Capture Data and Video Data

4. A Data Pyramid of Video and Motion Capture

5. Comparing Other Uses of Human Video

6. The Strengths and Limitations of Video and Motion Capture

1. The Core Idea of VideoMimic

The core problem we want to solve is building a whole-body motion-control system that generalizes. There are two aspects to this. One is generality in the method: the network design, training procedure and reward design should be broadly applicable. The other is generality across tasks: the robot should move stably across different terrains and environments without a separate design for every task.

Video demonstration from the original article

Our specific task is humanoid locomotion, particularly motion control on complex terrain. We began with ideas from DeepMimic and related work: use reinforcement learning to imitate human motion in simulation. But this approach has several clear problems.

First, robustness to terrain is insufficient. In real or complex environments—such as slopes or stairs—the behavior is often unstable and requires adjustments for each situation.

Second, scalability across tasks is limited. If you want a robot to go upstairs or downstairs, you need to redesign the corresponding reward function and tune parameters manually. That makes the method less generalizable.

Third, the approach depends heavily on motion data. In the DeepMimic paradigm, different movements often need different source motions, and those reference sequences are not easy to obtain.

These issues led us to seek improvements in two directions.

From the task perspective, we wanted to obtain human motion directly from casual videos, reconstructing the corresponding movement without depending on a fixed set of motion-capture recordings.

From the method perspective, we wanted to minimize hand-designed rewards and let the network learn more automatically, making the training process broadly applicable.

Overall, VideoMimic aims both to extract motion information from casual videos and to make the method more general by reducing manual intervention. For example, if the training data includes only climbing stairs, not descending them, earlier models often fail to go downstairs at test time. In other words, each movement needs a corresponding human source motion; otherwise the model cannot learn that skill.

We found that our method can partly alleviate this lack of generalization. We broadened the coverage of training, including several typical scenarios such as going upstairs, going downstairs and sitting on a chair. At test time, the model still showed strong generalization in settings it had not seen during training: stepping over a threshold, stepping up onto a raised surface, walking over a soft earthen slope or traversing uneven ground. By covering diverse movement patterns in training, the model can learn a more general body-coordination strategy and adapt to different terrain in new environments.

2. The VideoMimic Method

Part One: Reconstructing Human Motion in 3D

Human Motion Reconstruction

Our method is a complete system, covering the entire process from an input video to motion that can be used in simulation. The input is very simple: use a phone to record a person, perhaps climbing stairs, sitting on a chair or lying on a bed. It can be a casual recording.

VideoMimic workflow for reconstructing human motion and scene geometry from monocular video

First, we convert the video into 3D information. This builds on a project I worked on October 2024, called MonST3R.

MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. It concerns dynamic 3D vision and can turn videos—including casual phone recordings, livestream footage or YouTube videos—into 3D point clouds. This gives us a point cloud of the scene and dynamic 3D points associated with the person in the video.

Second, we combine this with existing methods for estimating 3D human motion. Given an image or video, these methods estimate the person's pose. Traditionally, however, the predicted motion is expressed relative to the camera, which does not by itself recover a person's world-space trajectory when the camera moves. By combining human estimation with dynamic scene reconstruction, we obtain 3D human motion in world coordinates. Whether or not the camera is moving, we can recover the person's pose and trajectory in that world frame.

For the human component, we combine the result of human-motion estimation with the dynamic 3D points associated with the person that MonST3R produces. Together, they yield 3D human motion in world coordinates.

Third, we reconstruct the environment. Our point cloud contains both the person and the surroundings. We segment the environment, remove the person's points, aggregate the remaining scene points and generate a mesh. This is needed for simulation: the environment and the human motion must have correctly aligned contacts, so we first construct the environment mesh.

Finally, we align the reconstruction with gravity, placing the reconstructed person and environment in a common gravity-aligned coordinate frame. We can then put the results directly into a simulator.

The complete process takes a casual video as input and outputs an environment mesh plus the person's 3D motion in world coordinates.

Video Data

The videos do not necessarily need to be recorded specifically for this project. Some videos found online can also be used. One of the real-to-sim examples in the paper was simply downloaded from the internet: a person doing parkour-like movements, with an Instagram mark in the lower-left corner. There are still requirements, though. The person cannot be too heavily occluded, and the environment needs to be sufficiently clear.

Our system uses monocular video: ordinary 2D video, not stereo imagery or RGB-D data. A normal phone video is enough; it does not need depth measurements. We emphasize this because many earlier approaches collected data with stereo cameras or RGB-D cameras, whereas ours requires only a 2D video input.

In implementation, there are two steps. First, we generate dynamic point clouds from the video, mainly using MonST3R or the more recent MegaSaM. This gives us time-varying 3D point-cloud information.

Second, we process those points along two branches. One produces the person's 3D motion in world coordinates, and the other produces the environment mesh. In this interview's description of the human-reconstruction component, we use HMR2, or Human Mesh Recovery v2. For the environment mesh, we use NKSR for surface reconstruction. Editorial note: the published VideoMimic pipeline also describes VIMO for per-frame human-pose recovery; the discussion here introduces the related HMR2 approach rather than treating the component name as identical across every implementation.

HMR2 can recover a 3D human skeleton and mesh, represented by SMPL parameters, from 2D images or video. It builds on the earlier HMR method introduced by Kanazawa and colleagues at CVPR 2018. HMR was a pioneering framework combining an end-to-end neural network with the SMPL parametric body model to regress a 3D human mesh directly from an image.

NKSR, Neural Kernel Surface Reconstruction, is a 3D surface-reconstruction method based on learned kernels. It can rapidly reconstruct accurate, continuous surfaces from sparse or noisy point clouds. Its efficiency, speed and accuracy make it useful for robot perception, 3D scanning and environment modeling in embodied intelligence.

SMPL stands for Skinned Multi-Person Linear model. It is a widely used parametric 3D human-body model. A small number of parameters represents body pose and shape, producing a 3D body mesh that can be animated.

These human-pose-estimation methods are relatively mature. The crucial step is actually the first one: reconstructing dynamic point clouds from casually recorded videos of moving scenes. Major progress on that step arrived toward the end of 2024.

How MonST3R Reconstructs Dynamic Point Clouds from Ordinary Video

Reconstructing dynamic point clouds from casual videos has a long research history in computer vision. Static 3D reconstruction has developed over roughly forty or fifty years, while dynamic reconstruction has been studied for at least twenty years. A 2022 paper called Particle Structure from Motion could already produce results resembling MonST3R's. The main problem with that class of method was its reliance on optimization, making it inefficient. Processing one or two seconds of video might take half an hour. That is impractical for large-scale data-processing pipelines.

MonST3R extends the direction established by DUSt3R. DUSt3R is an open research contribution to static geometry estimation that demonstrated very strong results using a data-driven approach to 3D reconstruction. Before that, most geometry estimation still relied on traditional Structure from Motion, or SfM. MonST3R develops this data-driven paradigm for dynamic scenes, allowing reconstruction to move beyond a purely optimization-based workflow. A deep-learning model learns spatiotemporal geometry directly. This is a methodological breakthrough and opens the possibility of learning dynamic structure from large-scale data.

DUSt3R, presented in the paper DUSt3R: Geometric 3D Vision Made Easy, reconstructs 3D structure from multiple images. Its two-stream neural architecture extracts image features and predicts 3D point maps, from which camera poses, depth and point clouds can be recovered. The approach offers accuracy, speed and generalization useful for robot vision and scene reconstruction. Editorial correction: the original article expands the name as “Dual-Stream 3D Reconstruction”; we use the project's official name here rather than treating that expansion as authoritative.

Later work, including MegaSaM and approaches based on VGGT, pushed this direction further. MegaSaM, for example, combines traditional bundle adjustment, or BA, with data-driven learning to improve reconstruction performance.

MegaSaM is a visual system for recovering 3D structure and camera motion from video. It efficiently recovers camera poses and depth in complex, dynamic scenes, with speed and robustness advantages over traditional SfM or SLAM methods in the settings discussed here.

VGGT, the Visual Geometry Grounded Transformer, is a Transformer-based visual model that predicts camera poses, depth maps and 3D structure directly from one or more images. It learns 3D geometry end to end, without requiring a traditional feature-matching and SfM pipeline, and can be used for robot perception and scene reconstruction.

SfM stands for Structure from Motion. It is a computer-vision technique that reconstructs a scene's 3D structure and estimates camera motion from 2D images taken from different viewpoints.

MonST3R's main contribution is better reconstruction quality and much greater efficiency than optimization-based methods from two years earlier. Where one or two seconds of video previously took thirty minutes to process, MonST3R can finish in one or two minutes in the comparison discussed here. That is still not extremely fast, but it is roughly an order-of-magnitude improvement. It makes large-scale data generation through automated scheduling feasible, which the earlier optimization-based methods could not support.

The key change behind this progress from video to dynamic point clouds is not more computing power or better hardware, but a shift in research paradigm. For a long time, 3D reconstruction—especially geometry estimation in dynamic scenes—mainly depended on hand-designed or optimization-based methods, such as COLMAP and SfM pipelines. These were often insufficiently robust and inefficient, making large-scale use difficult.

The real breakthrough came when researchers began taking a data-driven approach. Starting from this new paradigm, deep learning models and estimates dynamic 3D structure with much better scalability and generalization. Instead of relying on cumbersome manual parameter tuning, reconstruction learns underlying geometric regularities from substantial amounts of data. This makes dynamic 3D reconstruction from casual videos practical.

Part Two: Training Humanoid Motion

So far, we have discussed human motion. After obtaining it, we still need retargeting to convert the human movement into humanoid motion the robot can execute in simulation. To make the entire system usable, we must retarget human motion to the humanoid before proceeding to training.

In the first training stage, we pretrain on high-quality human motion-capture data. The dataset is small but accurate, so we use this limited, carefully selected data to train a base policy with strong initial motion capabilities.

In the second stage, we train on our own video-derived humanoid motions and environment meshes, working with individual sequences. We call this environment-aware policy training. We use both the human motion and environmental information, sampling observations from the reconstructed environment. The main observation is a height map, although visual feedback could also be incorporated. RL trains the policy to imitate human movements in different environments. This resembles DeepMimic, but we additionally provide height maps from the reconstructed surroundings as input.

The third stage is distillation. In the second stage, we obtain teacher policies for the reference motions, but we do not want to maintain multiple models at deployment. We therefore distill these motion-tracking teachers into one unified student policy. During distillation, we no longer condition control on a detailed reference joint pose. Instead, the motion command specifies only the root direction. The policy still observes the robot's state and the environment, as described above.

At that point, one policy can perform multiple skills. A simple handheld controller supplies forward, backward, left and right direction commands, and the robot chooses an appropriate action based on its surroundings. If I command it to move forward on flat ground, it walks. If there are stairs ahead, it goes up or down them. If there is a stool behind it and I command it to move backward, it recognizes the seat and sits down.

Finally, we add another round of RL fine-tuning to the root-command policy, strengthening it further to improve final performance and stability. This is also an important part of the sim-to-real process.

3. Using Motion-Capture Data and Video Data

For the pretraining stage, we use only a few dozen high-quality motion-capture sequences. There are not many, but they are accurate and clean, helping us quickly build a strong base model early in training. This is somewhat analogous to using a carefully selected supervised fine-tuning, or SFT, stage when training a large model: we first establish basic motion understanding and control abilities. Small-scale, high-quality pretraining gives the model an initial understanding of human movement—roughly, “If I were a person, how should I move?”

The second aspect is combining motion-capture and video data. Motion capture mainly initializes and aligns the model, helping it first learn a well-defined human motion space. Video-derived data then provides richer pose variation, environmental interaction and information from actual scenes. Motion capture offers precise motion templates; video expands diversity and generalization. We first pretrain a base policy with motion capture, then train and align it further using video-derived human motion and environment meshes. This helps it behave more naturally and realistically in complex environments.

We do not directly process raw motion-capture data ourselves. Instead, we use the data and simulation resources supplied by the robot manufacturer. When Unitree released resources for the G1, it included some retargeted motion-capture data. Human motions, mainly from part of the AMASS dataset, had already been mapped to its humanoid model, producing robot-motion sequences ready for simulation.

We use those released results to pretrain the policy. Unitree supplies the hardware model, simulation model and retargeted human motions, and we build our algorithmic research and training on that foundation. In effect, the manufacturer performs the first conversion from motion-capture data to robot motion, and we construct and optimize our policy on top of the converted data.

Unitree provided only a subset of processed data. We first used it to train a base model, then developed the teacher policy from that starting point. This initial pretraining on retargeted data is the beginning of our overall training pipeline.

When training the teacher, the conditioning targets are joint positions—the reference configuration of each joint. In simulation and training, the reference joint angles and poses are known precisely, so they can guide the policy's motion tracking.

At deployment, we cannot supply a detailed reference joint-pose trajectory in real time. Instead, the available command is coarser: a handheld controller supplies directions such as forward, backward, turn left or turn right. These are higher-level control instructions. Editorial clarification: this concerns unavailable reference-motion targets, not the robot's measured joint positions, which remain part of its proprioceptive observations.

We therefore add distillation between the teacher and the deployed model. Its aim is to turn a policy that depends on reference joint-angle information into one commanded by overall direction. In other words, it moves from following an explicit joint-by-joint motion reference to autonomously generating appropriate body movements from a direction command.

4. A Data Pyramid of Video and Motion Capture

The current method still has room for improvement. During development, I added different cost functions, including contact-related terms and basic collision-avoidance losses, to stabilize the constraints. This area has a research history, originally centered on character animation in graphics. Retargeting has continued to develop over the past year or two. I expect the gap to narrow gradually, but it cannot disappear entirely.

Data converted from human video introduces some noise into the final deployed policy. Even if the retargeting gap narrows, perfectly reproducing human movements is difficult because the bodies differ. The advantage is scalability: videos from YouTube and other sources can be turned into training data. Motion capture requires wearable equipment and is restricted by the capture space. It is harder to cover all the tasks people perform in the real world, and the recording process is less natural.

I favor a data hierarchy. At the bottom is a huge volume of noisy human video; above that is ordinary human motion capture; at the top is a small amount of exceptionally high-quality motion-capture data. The training strategy would use large quantities of lower-precision video to cover many tasks, then use small, carefully selected high-quality datasets for post-training—for example, SFT or RLHF—to combine broad coverage with precision. This resembles the trend in language models. Pretraining data cannot always be extremely clean, but curated data in later SFT or RLHF stages improves reasoning and accuracy. Robot control can follow a similar logic: first obtain a locomotion policy that generalizes broadly, then use high-quality fine-tuning to obtain sufficient precision. This is a proposed direction, rather than a claim that VideoMimic already implements that complete sequence.

5. Comparing Other Uses of Human Video

Human video is used in several different ways in current robotics research, and the processing approach depends on the task.

We previously worked on another project at Berkeley called ARM4R. It also uses dynamic 3D vision and learns from human videos, but focuses on manipulation tasks. We wanted to understand which downstream training procedure works best when starting with a video-based pretrained model.

Around 2021 or 2022, earlier Berkeley work—the direction referred to as “IFT” in this interview—explored pretraining models on video using masked autoencoders, or MAEs. Parts of video frames are masked, and the model predicts the missing content, learning general visual and spatiotemporal representations. The pretrained model can then be used for downstream action-control tasks. Editorial note: the original transcript's “IFT” reference does not identify a specific paper unambiguously, so it is retained without an invented expansion.

Our recent work also follows this direction of video training, but differs in some respects. It is closer to ATM—see Robot Data, Episode 1: Yang Gao and Chuan Wen on Training Robot Foundation Models from Internet Human Videos (ATM, Perfect Review Scores at RSS 2024)—except that we operate in dynamic 3D space. We preprocess the video into dynamic 3D trajectories. During pretraining, the model is conditioned on text and predicts future 3D motion trajectories. During fine-tuning, the same model predicts robot joint trajectories rather than future 3D point trajectories.

This can reduce the domain gap of a video-trained model and the changes required during fine-tuning, letting us use video data more efficiently for robot control. The original conversation describes this as locomotion control, while the ARM4R example under discussion primarily concerns manipulation; the broader point is transfer from human video to robot actions.

There is also a clear limitation: a model pretrained this way cannot simply be deployed as it is. It must be fine-tuned to work well on a particular task. That usually requires additional data collection—for example, robot trajectories used for imitation learning to train the control policy.

For manipulation tasks, such as robot-arm control, this is relatively straightforward. You can manually teleoperate an arm and collect more than a hundred trajectories in one evening. Whole-body control is much harder. Getting a humanoid to imitate human movement and collecting the corresponding robot data is far more costly and difficult.

As a result, most methods for whole-body humanoid control currently rely on reinforcement learning rather than direct imitation learning from demonstrated robot trajectories. There is not enough high-quality trajectory data to support the latter. Even during fine-tuning, data collection remains the biggest bottleneck.

Overall, the way human video is used in robotics depends on the task. Manipulation can obtain more action data, enabling video pretraining followed by imitation learning. Whole-body control has scarcer data, so we rely more heavily on RL to optimize the policy. That is the main difference at present.

6. The Strengths and Limitations of Video and Motion Capture

I mainly think about this in terms of scalability. VideoMimic's biggest advantage is that its data-source requirements are relatively flexible. Recording a phone video is much easier than putting on a motion-capture suit and collecting a dataset. If I want a robot to learn parkour, for instance, I can search YouTube for suitable parkour videos, convert a few into the data I need and use them to train the model.

The downside is noisier training data. Some people who saw our final demo said the robot looked a little drunk while walking. That comes from data that is not entirely clean. The model's movements are not as perfect as a person's, and there can be small deviations in both walking and stair climbing.

Methods such as PHC and MaskedMimic share some ideas with ours: initial pretraining, teacher-policy training resembling DeepMimic, and a later distillation stage. Their data is very high quality, but harder to obtain. High-quality data can produce very precise performance on particular tasks, yet is harder to scale. A new task may require new data collection and retraining. That creates a trade-off between task-level scalability and precision at deployment.

References

https://www.junyi42.com/

https://www.videomimic.net/

https://monst3r-project.github.io/

Smart Note · Human Data · EP04