What is the biggest gap between us and general-purpose robots? My answer is training data—whether teleoperated robot data, high-quality synthetic data or something else. Following the success of today's large language models, robotics researchers and entrepreneurs find themselves like cooks without ingredients. Everyone is looking for rice; some have even started clearing land to grow it.
For about half a year, I had wanted to write about the training data needed for robot foundation models. The subject is too important not to spend time exploring carefully. For various reasons, I never started. Recently, for other reasons, the urge to write about robot data became impossible to resist. So much of life happens through coincidence. Thanks to those coincidences, Smart Note is beginning a series on training data for robot foundation models. Stay tuned.
This is the first article in that series. We interviewed Yang Gao and his student Chuan Wen about Any-point Trajectory Modeling for Policy Learning, or ATM, a paper described in this interview as receiving perfect review scores at RSS 2024. Let's explore how they use internet videos of human behavior to train robots.
Original article outline
Contents
👇
1. Introducing our interviewees
2. ATM: learning robot action policies from 2D human videos
How ATM began
Common approaches to robot learning from human videos
ATM's core idea
The ATM method
ATM's perception module
ATM's results
Cross-embodiment transfer
Video viewpoints
3. Additional questions
4. Some personal reflections
1. Introducing our interviewees
At the time of this interview, Yang Gao is an assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences. He completed his undergraduate degree in computer science at Tsinghua and earned his PhD at UC Berkeley, advised by Trevor Darrell, a leading researcher in computer vision. During his PhD, he began exploring reinforcement learning with Sergey Levine, then continued that work as a postdoctoral researcher with Pieter Abbeel. His collaborators and advisers are leading figures in reinforcement learning and robotics. His research combines computer vision and robotics, teaching robots to manipulate the things around them by seeing.
Chuan Wen is a PhD student at Tsinghua's Institute for Interdisciplinary Information Sciences, advised by Yang Gao. He completed his undergraduate degree in electronic engineering at Shanghai Jiao Tong University. He was a visiting researcher at UC Berkeley's BAIR, working with Pieter Abbeel and Xingyu Lin. His interests center on computer vision and policy learning, particularly their intersection in visuomotor control. He studies how to make visuomotor control more efficient and robust in practical settings, especially autonomous driving and robotic manipulation.
Section 2 draws mainly on our interview with Chuan Wen, while Section 3 is edited from our conversation with Yang Gao.
2. ATM: learning robot action policies from 2D human videos
How ATM began
In the GPT era, we know that scaling compute and data can produce remarkable results. Robotics is also becoming increasingly aware of the importance of data volume. Whether you use reinforcement learning or imitation learning, enough data can produce very good performance.
But there is a major problem: robot data is nowhere near as abundant as the text required by large language models. A robot foundation model may take observations of the environment, such as images, as input and output robot actions. These can be motor torques, changes in the coordinates of an arm's end effector, or rotation angles for individual motors—very fine-grained outputs. Such data is difficult to obtain from the internet.
How much robot data do prominent robotics papers currently use? At most, around 25,000 trajectories—far below the scale of language-model data. Collecting it is difficult. It requires teleoperation devices such as GELLO or Mobile ALOHA, motion capture gloves such as those in DexCap, or specialized approaches such as UMI, which extracts control information using a GoPro. OmniH2O, for example, uses human keypoint detection to map human movements onto humanoids. Their fundamental purpose is to record observations and actions together, then train an imitation learning model to control the robot. Data-scale correction: The original interview gives 25,000 as a maximum, but this is not a global upper bound for robot datasets. Larger datasets such as Open X-Embodiment already existed; the figure should not be used as a comprehensive dataset census.
But why must we use complex teleoperation systems to obtain imitation learning data? Human videos are an important and abundant source. YouTube or Douyin videos contain rich information about physics and behavior, making them valuable prior knowledge. We cannot yet use them effectively because they usually lack action labels and cannot directly support imitation learning. Even when labels exist, different groups define and collect them differently. One may describe actions as end-effector displacement, another as rotation of each joint. Inconsistent labels are also difficult to use for imitation learning. ATM studies how to extract useful experience from video.
Common approaches to robot learning from human videos
Many methods now pretrain on video, extracting knowledge in different ways from large datasets. One major approach is video prediction. A model trained on large-scale video receives a robot arm's current observation, such as an image, and predicts its future movement and changes to the object being manipulated. This provides a prediction of what happens next. Learning those intermediate movements and changes makes it easier for the robot to acquire practical manipulation skills.
If a generative model such as Sora could provide perfect video generation, we could use it directly in robotics. For example, I give it the current image and specify that in the next image my arm moves a certain number of centimeters to the left. If it can predict the future correctly, very little data is needed to determine the action that moves the robot from the current state to the next. That simplifies the problem.
The difficulty is that video prediction models are very hard to build. Suppose an image calls for picking up an ear of corn, but the prediction shows only half of it being lifted. That is physically incorrect. Sora's videos look beautiful, but some physical details are wrong. In one example, puppies are playing in the snow and an extra dog's head suddenly appears. Training such models requires enormous data and compute. Sora, for instance, might require thousands of GPUs running for a month, which is impractical for a robotics task. The GPU count and duration here are an illustrative estimate from the interview, not a verified disclosure of Sora's training compute.
ATM's core idea
The main idea behind ATM is that we do not want to predict video images. What future frames look like is not what matters for robot control. We only need to know how objects move, rather than how beautiful the image is. We care about motion. We took inspiration from earlier ways of modeling movement. In games and simulation engines, an object may be represented as many particles, with the scene modeled through collisions, compression and relative movement between them. This is especially effective for fluids, deformable objects and sand.
If pixel-level video modeling is not a good representation, could we borrow that idea and turn a video into a collection of particles? Modeling each particle's movement could capture the scene's physical regularities and support a control model.
Consider an arm picking up a bag and placing it on a table. A conventional representation models pixels changing over time, with values between 0 and 255. Instead, we can model how individual points move. We select points on the bag and on the arm, then track their coordinates through the video. Over time, the bag's points and the arm's points move differently, while points on the same object follow similar movement patterns.
How did you arrive at this approach?
I have been working on extracting control signals from video. After extensive thought and experimentation, we found that changes in pixels, lighting and texture are irrelevant to the arm's control task. We care about how objects move in space. The best approach is to explicitly separate motion from other visual information, such as color and texture.
This is somewhat like a simulation engine: first model movement, then render color and texture. We do not need the rendering step, though. Knowing the motion is enough to control the arm.
We focus only on the moving component of video. We break it down into coordinate changes, ignoring color changes. Existing computer vision tracking models can recover how those points change over time. In 2023, these models were already performing very well, with tracking errors below one pixel. We can use their outputs directly as training labels. The subpixel-error statement describes the performance being cited, not a guarantee for every point in an arbitrary video.
Our representation has several advantages. First, it respects physical continuity rather than producing implausible video events such as objects suddenly appearing or disappearing. Every object's movement in our data is continuous, so the learned motion model naturally has that continuity, and the generated motion data remains continuous as well. Another advantage is explicitly separating motion from lighting and texture. We focus on movement without interference from those other factors. Technical note: Point trajectories encourage object permanence and continuous motion, but the representation alone does not guarantee physically correct predictions or eliminate tracking and prediction errors.
The ATM method

This diagram shows our architecture. Video data enters at the upper left. A tracking model extracts the future trajectory of each point in each frame. The red, blue and green points indicate their trajectories across the video. With these trajectories, we train what we call a track transformer, a multimodal transformer model. It takes the current image, a language description and the coordinates of query points as input, and learns to predict those points' future trajectories.
The track transformer's purpose is simple. Give it the instruction to open a drawer and a current image showing that the arm has grasped the handle and pulled it halfway. It predicts that points on the arm should continue moving right, and points on the drawer should move right with them. The function is simple but effective.
The predicted trajectories are supplied to our control model. Instead of taking only an image and producing an action, as earlier policies did, the control model receives both the image and predicted tracks. This additional input makes control more accurate and effective.
ATM's perception module
ATM's perception module is a ViT operating on 2D images; we do not use 3D perception here. 3D perception is important, but it need not take the form of a point cloud. With two cameras, for example, a model can learn three-dimensional information from the images. Point-cloud approaches often perform less well in practice. A simulated point cloud may be perfect, but real depth cameras frequently produce missing or incomplete points, which hurts model performance. High-quality point clouds require more expensive capture hardware, such as LiDAR.
Our group has a project with a similar idea but 3D inputs, called General Flow as Foundation Affordance for Scalable Robot Learning. Once we know a point's 3D coordinates, we can predict its future trajectory. Opening a safe door, for example, can be learned through these future point tracks, and the same applies to new objects. One advantage of 3D points is that they allow direct zero-shot generalization.
Zero-shot generalization means we do not need extra training examples to train a new control model. We already know the end effector's movement in 3D and can predict its future path. Solving the arm's kinematic equations gives the movement of each joint. This is more efficient and can work well in unseen settings. Avoiding additional data for a new model improves both generalization and efficiency.
ATM's results
ATM primarily addresses planning, but unlike a typical planner, it operates at an intermediate rather than a very high level. I usually distinguish high-, middle- and low-level planning. ViLa is an example of high-level planning; see Yang Gao on the ViLa and CoPa Embodied Model Framework. ATM is middle-level planning. Diffusion Policy is an example of low-level planning; see Diffusion Policy: Robot Action Generation with Diffusion Models.
We tested on 130 language-conditioned simulation tasks, a larger evaluation than in many papers. ATM is a multitask model: we did not train 130 separate models, which would defeat the point. We used five models: four handled ten tasks each, and the fifth handled ninety. A multitask model needs clear instructions for each task. Language was commonly used before, but it is too coarse. Our innovation uses trajectories to specify the task explicitly, making the problem more concrete and precise.
For the real-robot experiments, we built a system in an actual environment. Within the same setting, I described five different tasks with five language instructions, and the robot needed to complete all five.
Cross-embodiment transfer
Complex tasks still need more data. Human video provides prior knowledge about the broad steps and movements involved. But mapping that knowledge into a robot arm's action space requires additional methods and data. Our subsequent ideas include learning from human data and ultimately deploying on a robot. The final training step uses teleoperation, requiring only a very small amount of robot data.

ATM performs well on cloth manipulation

ATM also performs well on long-horizon manipulation

ATM can also reason about tool use

Video viewpoints
Our paper suggests that third-person views may be more useful. In first-person footage, such as videos of extreme sports filmed with a head-mounted GoPro, the person's head moves involuntarily. This introduces substantial noise. Much of that head movement is random and contributes little to robot control. Third-person views may therefore be more important, and they also account for the largest share of human video online.
Another important point is that we support multiple views. We can produce good trajectory predictions from third-person videos, including reasonable predictions when the camera is mounted outside the robot arm.
3. Additional questions
Other representative approaches to learning from human video
Before our work, there were two main approaches. The first directly predicts video. UniPi, from Pieter Abbeel's group at Berkeley, is a representative example. Given an initial frame and a task such as pouring water, it predicts how the video will change during the task. An inverse dynamics model then infers the arm's actions from those predicted frames. This is very end-to-end: videos train a video predictor, whose outputs train the model that recovers robot actions. It can use large amounts of data, but it is computationally expensive. Generating a three- to five-second video may take ten or twenty minutes, half an hour or even longer. Attribution note: UniPi is a collaboration involving Google Research and academic researchers, including Sherry Yang and Yilun Du; it is not solely a Berkeley lab project. The generation times cited above describe the interview's comparison and depend on implementation and hardware.
The other approach is closer to ours: extract an intermediate representation of the action. We use keypoint trajectories. Another family extracts the contact point on an object and its instantaneous movement direction. To open a drawer, for example, it identifies where to grasp the handle and that the drawer should be pulled outward along its surface normal. It simplifies motion into a contact point and direction. This is compact and concise, but cannot handle tasks that depend on the entire movement process, such as pouring water or folding a towel, where the required rotation and direction vary along the trajectory. It is better suited to articulated objects such as refrigerator doors and drawers, where you grasp a handle and pull in one direction. Some work from Deepak Pathak's group at CMU represents this approach.
What do you mean by a task that does not depend on the movement process?
A simple example is warehouse sorting: grasp an object in one location and place it in another. Work such as GraspNet cares about the grasp location and target pose but does not specify how to reach that pose, assuming a conventional motion planner can handle the path. Pouring water is different. You must hold the water and gradually tilt it at the cup's opening in appropriate directions. An ordinary motion planner might spill it.
Is ATM fundamentally a form of imitation learning?
It can be understood as imitation learning across embodiments, because human and robot bodies differ. Fundamentally, it still learns to imitate human actions so that objects follow particular motion trajectories. But because of those physical differences, a model trained on human data still needs a small amount of robot teleoperation data before deployment to bridge the domain gap.
How can 3D trajectories be extracted from 2D human videos?
Some neural networks estimate depth from RGB images, and this technology has been developing quickly. We can use it to add 3D labels to RGB videos and then extract 3D trajectories from that data.
4. Some personal reflections
Thank you to Yang Gao and Chuan Wen for taking the time to explain the paper, and to all the researchers and entrepreneurs I have previously interviewed or spoken with. These conversations have gradually shaped my understanding. I have worked in venture capital for eight years. From the beginning until now, my growth has been improvised and uneven, and my outlook has changed considerably. Compared with entrepreneurs who enter the currents of their time and bravely take on risk and responsibility, most of us called investors are simply workers filtering, selecting, connecting, integrating and communicating information.
Whether a deal closes, a company succeeds or a fund makes money depends on many combinations of chance circumstances. So many external factors are involved. Can we control the outcome? Probably not. Then why be anxious about it? At this moment, perhaps these words are the one thing I can control. Stay humble, keep learning and move on.
References
ATM
https://xingyu-lin.github.io/atm/
General Flow as Foundation Affordance for Scalable Robot Learning
https://general-flow.github.io/
UniPi
https://research.google/blog/unipi-learning-universal-policies-via-text-guided-video-generation/