In 2025, we spent more than half a year introducing and explaining frontier research on human data. The collection is available in the series wrap-up, Human Data—The 'Key' to Robot Data. Last year’s projects mainly addressed whole-body control. Entering 2026, releases such as EgoScale showed us the potential of human data for manipulation. Around EgoScale’s release, a blog post titled To Summon a Sensorimotor Ghost caught my attention, and I was honored to invite its author, Danfei Xu, for this interview. Danfei was also among the researchers I followed in A Map of Chinese-Origin Researchers in AI + Robotics. He has produced excellent AI + robotics research, particularly in using human data over the last two years, including EgoMimic, EgoBridge, InMimic, EMMA and EgoScale. The introduction’s reference to a 2025 map is retained as the author’s recollection; the linked researcher-map article was originally published in 2023.
Danfei Xu is an assistant professor at Georgia Tech and also contributes to research at NVIDIA Research. He received his computer-science PhD from Stanford in 2021, advised by Fei-Fei Li and Silvio Savarese, after an undergraduate degree at Columbia’s School of Engineering and Applied Science. His interests range from traditional robotics problems such as manipulation and motion planning to imitation learning, learning-based planning and robot foundation models. This interview focuses on a question attracting increasing attention: can robots learn from human data, and can we extract knowledge useful to robots from it?
This is a long article—about 16,000 Chinese characters in the original. Save it to read at your own pace, or listen to the interview on Xiaoyuzhou.
Original article outline
Contents
👇
Part One: Why Focus on Human Data?
1. Categories of Human-Data-Based Learning
Part Two: Modeling and Collecting Human Data
2. What Human Data Do Different Technical Approaches Need?
3. How Can We Collect Multimodal Human Data?
4. Should Visual Data Include Depth?
5. Can AI-Glasses Companies Replace Aria?
6. What Would Widespread Wearable Glasses Make Possible?
7. Which Is More Useful: First-Person or Third-Person Data?
8. Can First-Person Data Be RGB-Only?
9. How Should We Collect Human Data That Truly Scales Robot Learning?
10. Must Robot Data Collection Wait for Widespread Wearables?
11. How Do Local and Whole-Body Data Work Together?
12. Can One Device Collect Both Whole-Body and Local Motion?
Part Three: Modeling Human Behavior
13. How Can We Model Human Behavior?
14. What Approaches Are Being Tried?
15. What About Collecting Data on Real Robots?
16. Recent Projects
17. What Are the Main Lessons in Using Human Data?
18. How Do You View System 1 and System 2 for Manipulation?
19. How Can System 2 Organize Behavior Around Long-Term Goals?
20. Could World Models Provide the Foundation for System 2?
Part Four: Human-to-Robot Transfer
21. How Can We Solve Human-to-Robot Transfer?
22. Are Truly General-Purpose Robots Still Far Away?
23. When Might an Early General-Purpose Robot Appear?
Part Five: To Summon a Sensorimotor Ghost
Part One: Why Focus on Human Data?
Human data is a data source, not an algorithm with a one-to-one correspondence to machine learning or robot learning. Broadly, today’s methods for learning robotics from human data fall into two categories.
1. Categories of Human-Data-Based Learning
The first is imitation learning. We can regard a person as a special kind of robot, learn from that robot’s behavioral data and transfer the resulting policy to a physical robot system.
The other is the recently popular world-model direction. In a sense, I see it as a broader form of imitation learning, with richer modeling of future human behavior. A person’s next task may be generated as video, for example. We then plan from that video, turn higher-level representations into concrete actions and have the robot execute them.
Part Two: Modeling and Collecting Human Data
2. What Human Data Do Different Technical Approaches Rely On?
People understand human data in many different ways. Let me outline some common forms.
First is data for whole-body control, such as humanoid motion control. Motion capture is most common here: essentially motion alone. You can think of placing sensors on different body parts to record trajectories. Those data serve as a reference or reward signal in RL. We put them into a simulator, train an RL policy and transfer it to a physical robot through sim-to-real. Many dancing and parkour demonstrations use this approach. The person mainly supplies a motion standard, telling the robot which movements we want.
Second is video of humans performing tasks, from first- or third-person perspectives. Researchers use large video models to learn and model behavior, generate or predict future behavior and apply that information to planning and control.
Third is the type we focus on. I do not believe there is one best representation of human data. I judge value by two criteria. First, is collection scalable and portable? A very expensive system confined to a particular lab is hard to scale. Second, does it capture human decision-making? If we view a person as a system, inputs include vision, hearing and touch, while outputs are actions or other behaviors. Recording both means treating the person as a perceiving, decision-making agent and recording that process to train models.
We therefore prefer to understand human data as recording the input–output process of an individual capable of perception and decisions, then learning a model from it. That is the form we find compelling.
3. How Can We Collect Multimodal Human Data?
We use wearable devices, but not simply because those data happen to be available. The more fundamental reason is that a model interacting with the natural world like a human should have inputs and outputs as close to human ones as possible.
Human vision is first-person rather than a third-person video. Likewise, touch is not about how an object is observed externally but how we ourselves sense it. These signals are fundamentally egocentric.
To record human interaction with the environment faithfully, we need those person-centered perceptual and behavioral data. Wearable sensors let us record first-person vision, touch and motion, producing data closer to the actual human input–output process.
4. Should Visual Data Include Depth? How Does Project Aria Differ from Other Devices?
Project Aria is a research prototype developed by Meta Reality Labs, Meta’s AR/VR research organization. It is a research platform rather than a consumer product. At this interview checkpoint, selected universities and companies could apply, and approved teams could receive devices from Meta free of charge.
Its main advantage is extremely high-quality egocentric perception capture—among the strongest globally, in my assessment. It supports stereo and multiple cameras and handles underlying optical issues such as motion blur and imaging correction very carefully.
The device discussed here integrates five cameras. For multiview 3D reconstruction, camera-relative positions must be known precisely, so the system performs continuing online calibration and optical compensation. It uses disparity for reconstruction and visual-inertial odometry (VIO) to estimate its position in space. Editor’s note: Camera counts depend on generation and whether inward-facing eye-tracking cameras are included. Aria Gen 2’s official specifications list four computer-vision cameras, one RGB camera and two eye-tracking cameras—five outward-facing cameras, seven cameras overall.
It also estimates hand state through hand tracking. The latest, second-generation device can obtain very high-quality depth information. Overall, its underlying capabilities for egocentric visual perception are technically very strong.
In conversations with the team, I also learned that its SLAM performance is strong. On some public benchmarks, they described accuracy reaching 96%, a striking difference. That engineering advantage comes from more than a decade of continuous development. Editor’s note: The interview does not identify the benchmark or define this percentage. It should not be read as a universal SLAM accuracy figure.
We use outputs such as depth and estimated hand states to reconstruct the input–output signals of human–environment interaction and train robot models. The clear drawback is scalability: not everyone has access to the devices.
5. Can AI-Glasses Companies Replace Aria?
It is difficult. Wearing the glasses for some time warms them slightly. Thermal expansion can subtly shift the relative positions of internal cameras, directly affecting calibration accuracy.
They model those thermal effects too. Device temperature and material properties inform continuing calibration and compensation of camera-relative positions. Calibration adjusts dynamically from thermal feedback, correcting errors due to expansion in real time.
That is the level of engineering detail involved, and one reason their visual perception and localization achieve such high precision.
6. What Would Widespread Wearable Glasses Make Possible for Embodied Data?
First, wearables are naturally suited to collecting egocentric data: first-person information about human interaction with the environment.
Second, capabilities such as SLAM, 3D reconstruction and hand-state estimation are essential to our research. They supply important supervision for training. Without those modules, obtaining reliable supervision and training the models effectively would be difficult.
It is therefore not just the hardware. The accompanying software—SLAM, reconstruction and hand estimation—is among the most important parts of the system.
7. Which Is More Useful: First-Person or Third-Person Data?
If I had to choose, I would put greater faith in first-person data.
First, many robots also observe the world from onboard cameras. A humanoid moving freely cannot depend on a camera in a room corner to observe everything it does. Sensors mounted on the robot make first-person observation a natural, sensible design.
Second, many problems concern interactions between hands and objects, which first-person views often show more clearly.
Third is scalability. Fixed cameras cover limited spaces, while body-worn devices collect wherever a person goes, making scale easier.
It is interesting that third-person video is more abundant online. In a CoRL talk around 2024, I suggested that widespread XR—AR and VR—might eventually generate enormous amounts of first-person data, since most YouTube videos are third-person.
Events then surprised me. I am not entirely sure why, but from 2025 onward egocentric data suddenly became very popular. At this interview checkpoint, I could count around fifty or sixty companies specifically collecting it.
These datasets are often clean, with action annotations and structured information already processed. Internet video from YouTube needs substantial cleaning and preprocessing. In the long run, first-person datasets could become extremely large, perhaps not smaller than internet-video collections.
8. Can First-Person Data Be RGB-Only, or Is Depth Essential?
Not necessarily. Some algorithms can now recover depth from monocular data.
There are other considerations too. Many companies do not fully know how to use the data. They see it as valuable, reason that others think so too and expect a market for collecting it.
Most people, however, do not know how to train machine-learning models from those data or which signals matter most during collection. At this stage, supply may even exceed demand overall.
9. Should Scalable Human Data Be Collected Deliberately or Incidentally?
There are roughly two considerations.
First, is the data usable? Format and whether postprocessing produces valid training data matter enormously. Different devices create very different processing burdens. With a first-person monocular camera, hand tracking and state estimation can be difficult: apparent hand size changes with distance, and monocular imagery makes the camera-to-hand distance uncertain, increasing estimation error. Stereo generally helps. Purpose-designed collection devices therefore have an advantage.
From another perspective, the most valuable human behaviors may not be deliberately designed. They often occur without conscious thought. You might close a refrigerator door with your foot rather than your hand, for example.
Robot collection is often standardized: push the upper part of the refrigerator door with your right hand, repeat one hundred times and keep each motion similar. That is not how people ordinarily act.
Large-scale capture of naturally occurring, unplanned behavior could therefore greatly enrich the data we use for robots.
10. Must Robot Data Collection Wait for Widespread Wearables?
In practice, companies collecting and selling data already form a large market. They may organize large-scale collection in regions with lower labor costs.
The main issue remains knowing what is useful. These companies explore alongside frontier labs and research companies such as ours. In a sense, collection itself is a research process.
They collect a batch and sell it to researchers or companies. If some data proves useful, customers provide feedback and the collection companies adjust their approach.
After exploration, a consensus might emerge that natural, incidental behavior is more valuable than one hundred identical repetitions under strict instructions. But that conclusion is not certain; the field may develop differently.
11. How Do Local and Whole-Body Data Work Together?
The main issue is that robot capability has not reached a point where whole-body control is handled well, particularly the complex whole-body manipulation tasks we study.
Opening a door while placing a tiny object precisely on a surface, for example, needs difficult whole-body coordination. Researchers therefore often split the problem.
One route studies whole-body control, largely 'blind': it does not depend on complex visual input but copies demonstrated human movements onto the robot.
Another trains vision-to-action models, taking visual information and outputting action policies, but usually focuses on hand manipulation rather than complete body control.
The routes remain largely separate, but I expect them to start converging soon. Once combined, human data from everyday, in-the-wild environments becomes more important.
Real activity is never just hands. Cooking, tidying and even tabletop manipulation involve natural body movement. Whole-body data is unavoidable.
We currently simplify collection with restrictions, asking people not to move their heads or bodies much. Such constraints are hard to sustain in large-scale, realistic learning. Many real behaviors are impossible under them.
As research progresses, more types of data will therefore become valuable.
Late last year, we also worked on EMMA, Egocentric Mobile MAnipulation. It learns policies from first-person human data to coordinate mobile-base movement and arm manipulation, including navigation.
It does not yet perform true whole-body control such as squatting or complex bodily movements, but explores base–arm coordination. I expect rapid development and more projects investigating this over the next few months.
12. Can One Device Collect Both Whole-Body and Local Motion?
In principle, some devices can recover whole-body motion from first-person views. Aria Gen 2’s forward cameras have very wide fields of view—roughly 170°–190° as discussed here—and can sometimes see much of the body. Recovering full-body motion from egocentric video is therefore theoretically possible. The quoted range is the interviewee’s approximate description, not a specification shared by every camera or every field-of-view convention.
In currently practical systems, additional devices may still be needed, such as wristbands or small foot trackers, for more accurate body state.
Quest systems already provide some body tracking without wrist or ankle devices, estimating pose from front-facing cameras. Accuracy may be limited, but basic body-pose estimation is possible. Editor’s note: Camera-observed joints and inferred, unobserved lower-body joints are different; inferred whole-body pose should not be mistaken for direct measurement of every joint.
Part Three: Modeling Human Behavior
13. How Can We Model Human Behavior? Can Manipulation Follow Behavior Cloning + RL?
The modeling I mean differs from whole-body-control modeling. People often mean dynamics or kinematics: geometrically describing body structure and joint movement. Those problems are closer to graphics or traditional robotics.
I am not distinguishing manipulation from whole-body control here. I am asking something more fundamental: how do we build a sensor-in, action-out system? We want robots not merely to walk somewhere, sit or squat, but to interact intelligently in everyday, natural environments.
That is quite different from dynamics modeling. I want a model to learn from extensive human input–output data and produce humanlike behavior.
That is difficult. Many current imitation-learning systems observe an instantaneous image, run a large neural network and produce an action, then repeat for the next image. They essentially map current observations immediately to actions. This describes the reactive models being discussed, rather than every imitation-learning architecture.
Humans do not work that way. We do not map the current instant of vision straight into an action. We model the world over time, maintain an ongoing environmental state and act from that model rather than only what we see this second.
Suppose I am cooking and chopping vegetables, then remember I have not switched off the oven. The present image of chopping does not prompt that action directly; I remember what I did half an hour earlier, stop and turn the oven off.
Human vision and action therefore are not a simple mapping. Between them lies a complex process involving memory, temporal modeling and continuing understanding of world state.
Building such a system from extensive human input–output data is a core question—perhaps a holy grail of robotics. Solving it might bring robotics close to its ChatGPT moment.
14. What Approaches Are Academia and Industry Trying?
Many people are quietly thinking about this. Machine-learning-based robotics became popular, especially after ChatGPT, largely because people began believing the path was feasible: enough data might solve very difficult problems.
Tesla FSD likewise learns a perception-to-decision mapping from large amounts of data. Many believe robotics could achieve a similar breakthrough.
There is no consensus on exactly how: how much data, which data and what learning method remain unanswered.
My judgment is that human data may be very important, but others disagree. Broadly, people believe sufficient data and research could make the problem solvable.
15. What About Collecting Data on Real Robots?
One further point: the robot data we collect is, in a sense, weakened human data. Often we teleoperate a robot to perform a task, meaning a person acts through the robot.
Operating through it constrains our own ability. Many complex or fine movements humans naturally perform become impossible. Those data therefore capture a diminished form of human behavior as an acting subject.
Starting directly from humans may provide richer, more natural data, although transferring it to robots is difficult.
One possible approach separates the problems: model human behavior and learn its perception–action patterns first, then study how to transfer that prior or model to robots. The two can be researched independently.
16. Recent Projects
EgoMimic, EMMA and EgoScale follow the same broad research path, with related projects such as EgoBridge and InMimic along the way. Their central logic is to treat human data as robot data and mix both sources when training robot models.
The distributions differ substantially, requiring processing to make human data resemble robot data or vice versa. Calibration, representation alignment and other learning techniques bring the two distributions into closer agreement.
EgoScale can be understood as a scaled-up extension of EgoMimic. What happens when human data reaches a previously unseen scale—more than twenty thousand hours? A longstanding difficulty was the human–robot gap: a robot had a gripper, not a hand. We mounted the same collection glasses on the robot to match viewpoints, but the hands remained different. Earlier dexterous hands lacked quality and degrees of freedom, making the gap difficult to bridge.
New NVIDIA projects use higher-degree-of-freedom dexterous hands and more humanlike upper bodies. Increasing both hardware capability and data lets us test the resulting system. In a sense, it is 'scale up everything and see what happens.'
Other approaches differ. Our collaboration with Physical Intelligence (π), Emergence of Human to Robot Transfer in Vision-Language-Action Models, assumes that a strong pretrained foundation model exposed to many robot datasets will learn more easily from added human data. Having seen diverse robot embodiments, it may find humans less exceptional. Structurally, the gap between a person and a seven-degree-of-freedom arm may not be so unusual.
The work tests whether adding human data after foundation-model training further improves capability.
At a higher level, the core idea is similar: bring human and robot data into alignment so one model can learn from both.
17. What Are the Main Lessons in Using Human Data?
It is difficult to explain quickly because there are many details. Most important, well-established information is already in the papers, particularly the findings we regard as reliable and valuable.
We are less certain whether smaller details have general value. Some help in particular situations but not all. We have not presented them as definite new contributions—not because we deliberately withhold them, but because we do not know how important they are.
For example, how much does hand-state-estimation accuracy help downstream learning? We have not fully established that. Experiments usually fix a model and dataset and vary algorithms or modules, but some factors have not received detailed ablations.
Time is limited and experiments have not reached that granularity. Our understanding of some factors is experiential, and it is hard to say confidently that a particular detail always helps or never helps.
18. How Do You View System 1 and System 2 for Manipulation?
That is a large topic. In current robot learning, what we call System 2 is often provisional.
Many systems use an LLM or VLM for higher-level planning. 'Make dinner' produces a sequence of steps, and that is called System 2. I see it as a makeshift System 2, not the full reasoning performed by humans. Much human planning and reasoning does not happen in language.
For example, moving a bottle may require roughly two newtons to overcome friction with a table. But we do not internally verbalize 'apply two newtons' while acting. Much of that reasoning is implicit and embodied rather than linguistic.
In that sense, I think genuine System 2 research in robot learning has barely begun. We mostly use language models’ language understanding to assemble a higher-level planner that is just about usable.
Difficult physical manipulation, such as interlocking or separating two rings in a physical puzzle, needs detailed physical reasoning. I do not think we have made meaningful progress at that level yet.
Breakthroughs may require learning from data, and I consider human data potentially the most valuable. Humans already solve complex physical-reasoning problems. Learning the behavioral patterns that cannot be expressed in language may provide a feasible route.
19. How Can System 2 Plan and Organize Behavior Around Long-Term Goals?
Discussing this makes me realize we are not solving only System 2 for robot learning. The broader category includes problems whose reasoning cannot be expressed through language.
We need to solve those nonverbal reasoning problems for robots to do many everyday activities.
They are very difficult. Language may be a local optimum for current research: strong natural-language models happen to be available, so much work centers on them. But nonverbal reasoning makes up a large part of daily behavior, and language alone is clearly insufficient.
We may need more data, particularly data reflecting genuine human behavior and reasoning.
20. Could World Models Provide the Foundation for System 2?
Many kinds of world model exist. Let me discuss two forms close to this question.
One takes a long-horizon task description such as 'make dinner' and generates a key state to reach next. It might be an image or another environmental representation: the state to reach in two seconds or two minutes. Generating that state suggests the model is already solving part of the problem. In output-space terms, it resembles the System 2 I described.
The other is a more traditional dynamics model of the physical world. If I push something forward five centimeters, what changes? It predicts a future world state from the current state and action.
Both directions have advanced in recent years and could form early versions of System 2, but both face major problems.
Video prediction struggles with abstraction. Human System 2 reasoning is not conducted in pixel space. When imagining kneading dough, you do not specify every future pixel. You think of a ball or a long strip, reasoning in an abstract state space.
How do we learn that abstraction? Which abstract spaces suit reasoning? Could a model automatically select different abstraction levels for different tasks?
Planning to fly to Los Angeles tomorrow, I do not think about each footstep; I think about booking a flight. That is a high-level abstract state.
Painting, however, may require precisely choosing where the first brushstroke goes, informed by the finished picture I intend. Different tasks need very different abstraction levels. Many current world or video foundation models essentially answer only what the next image looks like.
Dynamics models are also difficult because real-world dynamics are complex. Push a phone on a table at a slightly different angle, and its trajectory may depart entirely from the intended direction. Predicting state over a longer interval is then difficult.
Both video models and dynamics models therefore face very difficult problems.
Part Four: Human-to-Robot Transfer
21. How Can We Solve Human-to-Robot Transfer?
If we can model humans, the next question is how to put that model into a robot. Fundamentally, this is grounding: how does the prediction produce a real effect through the robot?
Broadly, if we want the same outcome a human produces—picking up a bottle, for example—what actions should the robot take? Since robots differ from humans, they may need different motions for the same result.
From first principles, some things humans can do are impossible for a particular robot. A two-degree-of-freedom robot cannot reproduce everything a seven-degree-of-freedom system does, at least in some cases.
A larger embodiment gap therefore makes retargeting or transfer harder. If we could build an almost identical humanlike robot, many transfer problems would disappear and a human-trained model could be used much more directly. This is an idealized argument, not a claim that present humanoids can use human models without adaptation.
My conclusion is rather unexciting: we need both more humanlike robots and stronger retargeting algorithms, advancing together. A very different body makes much transfer inherently difficult.
There are many transfer algorithms. Most of our previous work used learning-based methods such as representation learning. Recent approaches also resemble real-to-sim-to-real: place a real human demonstration in simulation, substitute a robot and use RL to generate motions that achieve the same object-level effect, such as lifting a bottle.
Many methods now address the problem, and I cannot say which will ultimately work best.
22. Are Truly General-Purpose Robots Still Far Away from Commercial Deployment?
This is not directly a human-data question, but I think some tasks are already deployable.
A typical category is tasks in machine learning’s sweet spot, such as folding clothes. Traditional robotics struggles because deformable garments are hard to model. Data-driven methods can address them through large-scale collection. This is a promising intersection: difficult for traditional methods but amenable to learning from data.
The key is finding more such sweet spots: tasks with real demand that suit data-driven methods. If they create enough value and offer convincing economics between investment and return, they may become practical applications.
Over the next year or two, an important question is which tasks resist conventional automation but support large-scale collection and training with new methods.
A major breakthrough in general robot models could remove the need to search for those narrow settings, because in principle many tasks could then be handled.
23. When Might an Early Usable General-Purpose Robot Appear?
One fairly general target would be a model that can attempt any reasonable task with about a 40% success rate. By that definition, I think it could be possible within two or three years. This is my forecast from the March 2026 interview, not a demonstrated result or guaranteed timetable.
At 40%, it might resemble GPT-2. When GPT-2 first appeared, I tried it for writing code and revising papers. It felt weak and usually needed extensive correction, but parts were becoming useful.
That feeling was: usable, but inadequate in many respects, requiring considerable editing. Yet you could see that it could do many things. A comparable level in robotics may emerge within two or three years.
Once an open scientific question becomes an engineering problem, understanding and solving it is much closer. The route is clearer, leaving scale, optimization and engineering to improve performance step by step.
The main point of my blog was that a GPT-like robot model is feasible.
It may be less simple than people assume. Continually scaling teleoperation data alone may not achieve it. The issue is not only quantity but data type.
Teleoperation data can solve commercially useful tasks such as folding clothes, plugging in network cables and checking cables. Those are deployable applications.
But for genuinely human-capable robots, I do not think teleoperation data alone is the right path. That was the central point of the blog.
Part Five: To Summon a Sensorimotor Ghost
In late 2024, I gave a talk at CoRL proposing a simple idea: human data is robot data in disguise. With sufficiently capable sensing and state estimation, human behavior can be recorded and treated as data from another kind of robot. The body, actuators and sensors differ, but it is still a perception-to-action mapping operating under the same laws of physics. Using Meta Reality Labs’ Project Aria glasses, EgoMimic provided early evidence that robot performance can improve as instrumented human data scales. With consumer wearables emerging, we can directly record much of that mapping’s input and output, from egocentric vision to head, hand and even full-body movement. I predicted that human-experience data at global scale was not far away. This section reproduces the essay as translated in the Chinese article, with Danfei speaking in the first person.
The industry has since changed. I expected consumer wearable adoption to be the main bottleneck, but dozens of companies are already collecting and selling egocentric human data, with new ones appearing almost weekly. Global-scale instrumented human datasets arrived faster than I expected. Crucial problems remain, however. We still struggle across the embodiment gap. Zero-shot robot execution from human data alone remains distant. Large video-prediction models are promising, but a plausible future does not guarantee effective action under real contact. Between prediction and physical execution lies the last inch—or newton: millimeters of error, milliseconds of delay or a newton of force can determine whether an object slips or stays stable. Bridging that gap still requires extensive robot data, careful human instrumentation and coordinated robot-hardware design.
Modeling humans is not only about better robots. It is an important scientific goal itself. Recovering human sensorimotor intelligence from large-scale data remains fundamental and unsolved. Robotics is one application; the larger question is whether learned models can recover that intelligence itself.
1. Modern AI: Summoning Ghosts
Modern AI is fundamentally a knowledge-transfer problem: moving human knowledge into machines at scale. Language models are the clearest example. They absorb global-scale text, compress it into a parameterized function and refine outputs through reinforcement learning and preference optimization. But their intelligence still originates in human-produced content. Andrej Karpathy has described training modern language models as summoning ghosts. These ghosts statistically distill human semantic knowledge, approximating our collective cognition. Imperfect though they are, they reproduce enough of its structure to reason, explain and create.
In robotics, we attempt something similar with much less success: summoning a sensorimotor ghost. We want not merely what humans have written but how they perceive, adapt and act physically. Teleoperation is the prevailing approach, but it distorts behavior. Much natural, subtle sensorimotor intelligence never enters the dataset, leaving a simplified version of human capability.
2. Where Is the Ghost?
There is an animal-like component of human behavior, much of it outside direct conscious access. It is fast, reactive and tightly integrates movement and perception—the behaviors associated with System 1. A tap on the shoulder makes you turn; a falling object makes you reach; something flying at your face makes you dodge; heat makes you withdraw your hand before words form. These reactions are not planned or derived through deliberate reasoning, but shaped by evolution and prolonged physical interaction.
Teleoperation struggles to capture this intelligence. It relies on continuous perception–action loops with millisecond feedback. Even low-latency interfaces push operators toward deliberate behavior. Reflexes become joystick or VR operations, and feedback is heavily filtered. Everyday human data also contains something teleoperation rarely captures: human–human interaction. Much real interaction is social—handing over objects, sharing tools, coordinating attention and conveying intent through movement. These are hard to reproduce through robots but occur naturally every day. To recover embodied priors for physical and social worlds, we need real-life behavior rather than only designed demonstrations. Ordinary actions may be more valuable than carefully staged ones.
3. How Do We Summon the Ghost?
Suppose we capture natural human sensorimotor behavior at scale. Can we learn the function producing it? With rich observations and sufficiently complete action capture, is there a learnable function mapping perception to action like a human? In other words, can we behavior-clone humans?
Language models learn by predicting the next token. A sensorimotor counterpart predicts intentions and actions from perception. Scaling that objective might produce capabilities analogous to those emerging in language models. There, scale brought in-context learning, reasoning and agency. In action prediction, it might bring physical common sense: objects fall, stacks collapse, heavy things resist movement and slippery surfaces require adjustment. Such understanding need not take the form of an explicit physics model. It could appear directly as stable grasping, subtle nudges and quick corrections. Just as a language model learns grammar without a rulebook, an embodied model might internalize physical laws through predicting behavior.
We have early evidence of feasibility. Recent NVIDIA research shows scaling trends in human data: more human pretraining improves behavioral modeling and real-world generalization. We also observe emerging capabilities such as learning a new dexterous task from one demonstration and strong language-instruction understanding. In EgoScale, increasing human pretraining data substantially improved both behavior prediction and human-to-robot transfer. One-shot transfer here still uses the study’s aligned human–robot training and downstream supervision; it does not mean robot execution from human data alone.
A further question is whether effortful System 2 can emerge from human cloning. Could a model trained only on multimodal inputs and outputs acquire long-horizon planning, long-term memory and complex physical reasoning? Language-model reasoning has appeared with sufficient scale and context. In embodied systems, an analogous physical agency might let a model do more than predict the next human action: organize behavior around goals, unfinished actions and future states. This is a research hypothesis, rather than a capability already established by EgoScale.
4. Giving the Ghost a Body
Even a successfully summoned sensorimotor ghost needs a body. Prediction is not control. Predicting human behavior does not automatically tell a model how to execute it through a particular robot. We must transform distributions of possible human futures into executable robot-control sequences. That requires effect-driven retargeting: reproduce the effect on the world rather than the human trajectory itself. The system must infer how a robot can achieve the same outcome under different kinematics, compliance, delays and force limits. This difficult inverse problem requires both inverse dynamics and real-time control.
Many approaches try visual alignment or video prediction, but the real challenge is physical interaction. A grasp stable for a human may fail for a robot; human millisecond reflexes may exceed a robot’s control bandwidth. A powerful inverse-dynamics model is sometimes proposed as the answer, but learning it may be as difficult as prediction, or harder. More humanlike robots reduce complexity, yet actuator, compliance and control differences persist even in humanoids. Another approach co-designs hardware and data, as UMI does by constraining human behavior to match robot capability. That works, but the more scalable long-term path is to adapt robots and models to real human behavior, rather than the reverse.
5. Conclusion
If the last decade of AI modeled language, images and video, the next will model behavior. The question is no longer whether we can capture human behavior at scale—that future has arrived. It is whether we can learn deeply enough from those data to recover the intelligence producing it. Success would mean more than better robots: it could yield the first learned theory of how humans act in the physical world. Robotics would be its first application.
References:
Danfei Xu’s Homepage
https://faculty.cc.gatech.edu/~danfei/
The Blog Post To Summon a Sensorimotor Ghost
https://x.com/danfei_xu/status/2027034645892624528
References to the projects mentioned in this article can be found in Danfei’s blog linked above.