Back to Human Data

Human Data

An Interview with Lei Han: Noitom Robotics and the Human-Data Challenge, Part One

Lei Han discusses reinforcement learning, the human-data pyramid, optical–inertial motion capture, multimodal collection and opportunities for robot manipulation.

Zha Dami2025-12-0118 min readWeChat original
Contents

In the Human Data series conclusion, Human Data—The Key to Robot Data, we summarized the human-data pyramid. For the first of our closing interviews, I am delighted to speak with Lei Han, chief scientist of Noitom Robotics, a company working on data challenges at different levels of that pyramid.

Noitom Robotics’ three-tier Human Data pyramid: high-precision motion capture, lower-precision motion capture, and human video/synthetic data
Figure labels, top to bottom: high-precision human motion-capture data; low-precision human motion-capture data; human video data and synthetic data. The surrounding frame reads “Noitom Robotics,” and the callout reads “Scale Human Data for Robots.” This three-tier illustration is distinct from the four-layer data-collection framework discussed in the interview.

Longtime Smart Note readers may remember my 2023 article, A Map of Chinese Researchers in AI and Robotics. For the final two interviews in the Human Data series, I had the privilege of an in-depth conversation with one of the researchers featured in that map: Lei Han.

At the time of this interview, Lei Han was cofounder and chief scientist of Noitom Robotics. Previously, he spent six years at Tencent's Robotics X laboratory as director of embodied intelligence and principal research scientist. Before that, he was a senior research scientist at Tencent AI Lab.

Before joining Tencent, Lei was an assistant research professor in the Department of Basic Science at Mississippi State University in the United States. He received his PhD from Peking University, advised by Professor Kunqing Xie, and subsequently conducted postdoctoral research at Hong Kong Baptist University with Professor Yu Zhang and at Rutgers University with Professor Tong Zhang. His research interests include large-scale statistical machine learning, reinforcement learning, optimization and multi-task learning, and their applications in robotics, games, natural language processing, computer vision and bioinformatics.

Before this formal interview, I had asked Lei on several occasions about combining reinforcement learning with robotics. I greatly appreciate the depth and breadth of his technical thinking. In this conversation, I asked him about robot data, particularly human data, as well as current algorithmic approaches in robotics. I strongly recommend listening to or reading the discussion.

Search for ReSpark on Xiaoyuzhou to listen to the podcast.

Podcast: Conversation with Lei Han: Reinforcement Learning and Motion-Capture Data—Noitom Robotics' Data Story, Part One. Listen in the original WeChat article.

Original article outline

Contents

↓

1. Introduction

2. Reinforcement Learning's Development Across Fields

3. Why Start a Company Focused on Robot Data?

4. Will Motion Capture Still Matter If Video Data Makes a Breakthrough?

5. The Importance of 3D Information for Video Data

6. Contributions to the Research Community

7. Comparing Motion-Capture Approaches

8. Opportunities for Motion Capture Plus RL in Manipulation

1. Introduction

I am chief scientist and cofounder of Noitom Robotics. Before founding the company, I spent eight years at Tencent, working first at Tencent AI Lab and then at Tencent Robotics X.

My early research focused mainly on reinforcement learning. We were developing game AI and trying to apply RL to games. This was in the years around AlphaGo's emergence. We studied game theory and the strategy and action spaces of complex competitive games, developing agents that could surpass human performance.

Around 2019–2020, I moved to Tencent's robotics laboratory and began researching robot motion control. Hardware across the industry was less mature than it is now, so the lab also developed its own hardware, including custom robot platforms and quadrupeds. On these platforms, we combined reinforcement learning, imitation learning and other algorithms for robot control and motion planning.

By the first half of 2025, the year of this interview, we could see the industry's development accelerating significantly. In embodied intelligence, the demand for high-quality, diverse multimodal data was becoming especially urgent. We realized that a core bottleneck in improving robot intelligence was not just algorithms, but how to acquire and manage high-quality data.

That is why Ruoli Dai and I founded Noitom Robotics. We hope to explore and build industry-level technical infrastructure for high-quality multimodal data collection and embodied-intelligence training.

2. Reinforcement Learning's Development Across Fields

I first encountered reinforcement learning around 2015–2016. AlphaGo had just emerged, and it had a profound impact on me. For the first time, I felt that an algorithm could learn strategies beyond the distribution of human experience, driven by a minimal zero-or-one signal such as winning or losing, without relying on a predefined dataset in the same way as supervised learning. I became fascinated with RL and have continued researching it ever since. I still believe it is a necessary technical path toward human-level or even superhuman intelligence. Editorial clarification: AlphaGo itself used human game data as well as RL; the purely self-play, from-scratch approach discussed below was demonstrated later by AlphaZero.

As the research problems became more complex, especially in the real world and robotics, I also became increasingly aware that pure RL is not enough. As with large language models, an agent learning efficiently in a complex environment must first accumulate substantial preparatory knowledge through large-scale data training. RL can then iterate more efficiently on that foundation. We had already learned this during our game-AI work.

Take Go. In 2015–2016, it was considered an extremely complex game, yet AlphaZero later showed that pure reinforcement learning through self-play could solve it. Looking back, that suggests Go's action space is still relatively small. Once the problem becomes something like a real-time strategy game, we cannot simply begin from scratch in the AlphaZero manner. We need human expert data or replays to obtain a good warm-start base model, then use RL to optimize it further. That greatly improves exploration efficiency and produces a better final strategy.

From 2020 onward, we tried combining robotics with AI, particularly reinforcement learning. We used simulators to train robots, almost as if they were playing a physics game. Our thinking was straightforward: could we approach robot control in the same way we approached game AI?

In simulation, this can indeed produce good results quickly. Xue Bin (Jason) Peng's 2018 work on reinforcement learning for human-motion control, for example, is a highly respected research direction. It allows virtual characters to imitate human movements and has had a major influence on subsequent virtual-character and robot control. See Interview with Xue Bin (Jason) Peng: Exploring a General Controller for Whole-Body Humanoid Control. But transferring these models from simulation to the physical world revealed problems much more complex than we had expected.

We were developing our own robot hardware because there were not yet many mature consumer-level platforms available. Exploring hardware and algorithms together showed us that integrating them requires long-term iteration. Simulation training produces results relatively easily, but making a robot execute reliably in reality exposes a large gap. The central RL challenge in robotics therefore shifted from designing complex strategy spaces to addressing the sim-to-real gap.

A game is a fully specified, idealized environment, with consistent training and testing conditions. Real physical environments are full of uncertainty. The industry has explored approaches such as domain randomization, making simulation cover as many real-world physical variations as possible. It is effective, but expensive: the model spends a great deal of time exploring situations it may never encounter in reality. Only after covering enough randomized scenarios may it work reliably on a real robot. This improves robustness but greatly reduces training efficiency.

Exploration efficiency is a core problem in reinforcement learning. We knew from game AI that learning entirely from scratch is inefficient. It is much more effective to begin with a few promising initial solutions and explore a smaller region around them.

Of course, supervised warm-start training introduces another risk: it may restrict the eventual strategy space and trap the model in a local optimum. Overall, however, I think the better route is to train a strong base model or initial policy first, then use RL for fine-tuning and policy optimization. This is more efficient and supports more robust, continuous improvement in complex robotic systems.

3. Why Start a Company Focused on Robot Data?

Let me explain why I think data is so important. Following the reasoning we discussed, even reaching human-level intelligence—without aiming for superhuman performance—becomes an increasingly complex large-model or agent problem. Robotics is such a problem. Its overall space keeps expanding, and high-quality data is the foundation supporting all of it.

I also spent some time working on large language models. The gap between the volume of language-model training data and robot data was enormous: they were not even on the same scale. Language models draw on the internet, Twitter, blogs and other sources. The new text generated each day far exceeds all the robot data currently collected together. That made me realize that robot or agent foundation models can only acquire even their most basic capabilities once the data reaches a comparable scale before we even consider subsequent supervised fine-tuning (SFT) or reinforcement learning.

Robotics is developing rapidly. Hardware continues to improve, and AI algorithms can already deliver good results in motion control and basic manipulation, producing impressive demos. But from the perspective of data volume and first principles, robotics is much more complex than language modeling. It involves more modalities—not just text, but video, images, touch and others—and a larger, more complex space. Yet the available robot data is far smaller than the text corpus. I realized that data would be central to solving robotics' key problems, while existing collection methods still had limitations.

The mainstream VLA approach has mainly used teleoperated real-robot data. People use remotes, handheld controllers, leader–follower arms or motion-capture systems to operate a robot and collect physical demonstrations. Early models such as π0 and much of the field followed this approach. We collected a great deal of teleoperation data ourselves and found it inefficient. Operating a robot does not match natural human habits. You need to turn your head to inspect the robot, sometimes adjust your posture because of occlusions, and work without tactile feedback and with only indirect visual feedback. A tabletop task that takes a person a few seconds may take the robot tens of seconds or more than a minute.

Second, data collection is tied to a specific hardware embodiment. Data may generalize somewhat between embodiments, but our mixed-embodiment training experiments found that much of the transfer occurred at the visual-understanding level. At execution, the data was often difficult to reuse. Data from one configuration could even impair another configuration's execution. With both efficiency and embodiment constraints, further scaling becomes very difficult.

We then saw two possible directions beyond the current VLA approach. One is to train a base model from teleoperation data and then apply reinforcement learning on the real robot. That is a reasonable technical route; Dyna, for example, is pursuing something along those lines. The other is to use a wider range of data modalities. Looking back at locomotion, we did not rely on teleoperation data in the same way as VLA. Researchers often use RL with extensive randomization in simulation to teach motion control. This route is closely integrated with RL, while motion-capture or video data provides imitation references and information for training.

Returning to the earlier question: data is the most important factor. High-quality data brings a model closer to the actual embodiment and makes real-robot demonstrations more efficient to achieve. To expand the data volume without continuing to tie collection to robot hardware, the next possibility is human data. Its volume could be at least one or two orders of magnitude larger.

Because it comes directly from people, this data is no longer tied to a particular robot embodiment and can use a unified format. People differ in body shape, but they share the same basic structure. We do not need to account for robot-specific differences in degrees of freedom, joints or individual component parameters. Human data can therefore have a more consistent format and greater generality.

I know motion-capture equipment is expensive, but the Noitom team has more than a decade of experience in the field, including efficient large-scale deployment and data collection. I previously bought its hybrid optical–inertial motion-capture equipment as a customer and had a good experience. That makes me confident this approach is feasible.

We now treat human-centered data as our second major technical route. In a motion-capture enclosure, we use a combination of inertial and optical tracking wherever possible. Participants wear lightweight trackers, and objects carry small optical markers. RGB-D third-person cameras, LiDAR and other sensors capture human movement, object interaction and 3D models of the environment. The resulting modalities are rich. We also equip participants with flexible tactile gloves that record touch without disrupting their normal manipulation habits. The result is a multimodal data system centered on human activity.

Compared with real-robot teleoperation, this approach is much less dependent on the robot embodiment. Data volume can potentially expand by tens or even hundreds of times. Another advantage is precision: in our motion-capture system, the spatial alignment error between a person's fingers and an object can be less than one millimeter, reaching submillimeter accuracy. A third advantage is the richness of modalities, recording visual, tactile and motion information together.

This route also creates challenges. The first is retargeting. Since the data no longer belongs to a specific robot embodiment, we must map human movements to different robot configurations. One approach combines imitation with further reinforcement learning and exploration. Another directly learns a neural model, treating human motion data as if it came from one particular robot configuration during VLA pretraining and enabling cross-embodiment training. In that sense, the human body becomes a unified robot embodiment.

This data also has applications beyond robot policy learning, including world models, 3D video generation and 3D scene generation. It records multimodal information about human interaction with the environment, giving it a very broad range of uses.

Although human-centered capture provides rich modalities, it remains constrained by a controlled environment. Such a collection system typically needs a laboratory or enclosed space and cannot reach internet scale. We are therefore exploring a third data layer: lightweight wearables. People could wear watches, glasses, hats or other devices to collect data without disrupting daily life. These devices record human–machine interactions in settings closer to everyday reality. Privacy and compliance are also important considerations for us.

A further direction is to use third-person videos from platforms such as YouTube or Twitter, building an internet-scale data system comparable in reach to internet text.

We divide data into roughly four layers, forming a pyramid from top to bottom. Moving downward, the volume grows, the range of modalities broadens and the distance from the actual robot embodiment increases. Each layer has its own strengths and limitations. Editorial clarification: the modality mix depends on the collection method; not every lower layer contains every higher layer's sensor measurements.

At Noitom Robotics, we want to integrate data across modalities and levels, providing a high-quality data foundation for the industry and supporting different technical approaches. Whatever direction AI models take, these modalities can support multiple routes. We hope to prepare the overall data system so that teams can explore different model approaches on top of it. That is what we are working on.

4. Will Motion Capture Still Matter If Video Data Makes a Breakthrough?

That is a crucial question. First, will Noitom Robotics work on video-related data? Absolutely. An important component of our human-centered collection is head-worn equipment that captures first-person, or egocentric, video.

This video comes from a person's viewpoint, unlike the third-person or casually filmed footage commonly found online. It focuses on the scene directly in front of the person: what their hands are doing, or what they see while walking. We will prioritize this first-person video closely connected with manipulation and embodied behavior.

We also capture synchronized depth information or fisheye imagery, obtaining a wider and more complete visual input. These are important components of our video-data collection.

On the second question, computer vision is indeed progressing very rapidly. If video processing becomes good enough, it may enable world models to reconstruct the physical world and make highly accurate predictions.

There is nevertheless a first-principles limitation: the dimensionality of the information. Estimating 3D information from 2D video involves missing information. Without multiple viewpoints, a single RGB view makes precise 3D recovery difficult. For example, close one eye and try to align your fingertips. You may notice errors, illustrating that one viewpoint does not capture all spatial-depth information.

Therefore, 2D video alone cannot provide precise 3D information. Vision should combine modalities, integrating 2D video with structured 3D data to give world models high-quality inputs for more accurate prediction and reconstruction. Information completeness constrains a model's capability. Only visual models that fuse multimodal data containing sufficiently rich information about the 3D physical world can truly reconstruct it precisely. Editorial clarification: this is Han’s argument for explicit physical information; monocular methods can estimate 3D structure using learned priors and motion cues, but a single RGB image does not uniquely determine absolute geometry without additional assumptions.

In the future, one of the most important uses of human-centered video may extend beyond embodied intelligence to world models or computer-vision models. Whatever the application, the premise is that the data supplies fundamental, essential information across modalities.

5. World Models Without 3D Information

There is already excellent research exploring related directions. NVIDIA recently released its Cosmos world models, followed by related work from its team and other researchers. One project that particularly inspired me is DreamGen.

DreamGen's central idea is to use a video-generating world model to roll out robot behavior in an imagined environment. It generates frames showing a robot acting in a manipulation scene and predicts what comes next. Researchers then extract action information from the generated videos and use this data to train robot behavior policies.

It illustrates a common idea: generate large quantities of video with a world model and use these imagined environments to learn behavioral regularities. Judging from the experimental results, however, this approach’s accuracy remains below that of training directly on teleoperation data, in my assessment. That is understandable, because the generated video lacks some critical details of real physical contact, such as precise 3D coordinates, joint configurations and velocities. Editorial clarification: the interview discusses a precision trade-off; this is not a claim that synthetic data never improves a real-data baseline.

Its strength is greater diversity and generalization, because the model can imagine different scenes and actions. Its weakness is the lack of precise real-world execution data, particularly at the end effector during physical action.

These projects are valuable explorations. As research advances, we will see different routes emphasize different things: some prioritize diversity and imagination; others prioritize precise control. A future world model that also contains 3D physical information could make this route fully viable. It still needs sufficiently large, high-quality datasets. That brings us back to the fundamental data problem.

6. Contributions to the Research Community

This is a very practical issue. Motion-capture data is expensive to obtain, and many laboratories face that difficulty. Yet the capture technology itself is mature and supports large-scale, efficient collection. Our near-term goal is to provide the community with hundreds to more than a thousand hours of high-quality motion-capture data. That would be an unusually large resource with broad coverage and high precision.

For comparison, established datasets such as AMASS have motion data on the scale of tens of hours, and LAFAN is smaller still in the comparison we are discussing. Because existing motion-capture datasets are limited, researchers often extract motions from videos to supplement them. That is convenient—a normal camera or phone is enough—but has clear precision limits. Large-scale, high-precision motion capture could substantially improve data quality and reliability.

The dataset Noitom Robotics plans to provide aims to be tens or even hundreds of times larger than existing resources, offering researchers and developers abundant high-quality motion-capture data. Work is actively underway. Some current datasets contain floating or foot-sliding artifacts, include a high proportion of walking, and have too little highly dynamic movement. Those are among the issues we are trying to improve.

We have invested heavily in data quality, repeatedly testing and refining the data through model training to improve accuracy and stability. The aim is not merely to provide enough data, but to meet standards for research and application. After thoroughly using and validating it, we will assess its quality and usability from a researcher's perspective, then scale it further to provide a reliable resource for more users.

7. Noitom Robotics’ Advantages in Motion-Capture Data Collection (Section 8 in the original numbering)

Noitom has more than a decade of technical experience in motion-capture equipment. It has substantial strengths in equipment details, cost control and operating large-scale collection. The system is also flexible. Noitom uses hybrid optical–inertial motion capture, one of the relatively few approaches worldwide that can combine high accuracy with continuous tracking.

Pure optical capture is accurate but can lose frames during occlusion or highly dynamic movements. Inertial capture maintains continuity but can have positional errors. Combining them allows the system to maintain both accuracy and continuity while avoiding tracking dropouts. The high-quality capture hardware and sensor-fusion algorithms are the core technologies.

During motion collection, we can also combine sensing modalities flexibly. Participants can wear additional equipment, such as head-mounted vision systems or tactile sensors on their hands, to record visual and touch information synchronously. This multimodal combination supports rich modalities, precise capture and high overall efficiency. It is one of our strongest areas of expertise.

8. Opportunities for Motion Capture Plus RL in Manipulation (Section 9 in the original numbering)

I believe there is definitely an opportunity here, and reinforcement learning will be essential when manipulation tasks become practically deployable. It remains a technical route worth pursuing.

Whatever data type they use, teams can generally begin by training a reasonably capable base model. What happens next is crucial. Some approaches apply RL directly on real robots, with people participating throughout training. This usually optimizes a specific task and can raise its success rate above 90%. Clothes folding, for example, has used RL with an evaluation function to improve the policy iteratively.

The advantages are clear, but substantial human involvement is needed throughout training. Both the labor cost and the challenge of scaling are considerable.

Another approach resembles locomotion research: train repeatedly in simulation. Simulation allows unrestricted exploration, with computing as the main cost. This is promising too, but has its own difficulties. Motion control is relatively simple because even on complex terrain, the environment is often static and only the robot moves through it. The robot's parameters and simulation model can also be made quite close, so the trained policy's mismatch can be small.

Manipulation is different. It requires interaction with movable objects that have their own physical properties. Collisions become a major issue. The robot is not merely facing stationary objects; it must handle moving objects and their dynamics. Contacts and dynamic changes occur frequently, making RL for manipulation in simulation harder and increasing the sim-to-real gap.

Other research explores alternative routes. Some teams use human-centered data, such as motion capture, without applying RL directly. They select keypoints from human hand motions and map them to corresponding robot-hand keypoints. They then combine robot teleoperation data with motion-capture data: establish correspondences between the human and robot hands, then merge the two sources for joint training.

This brings human motion into the manipulation policy. Through joint training, motion-capture data becomes part of the policy model. It introduces substantial human-motion data and yields a model with basic capabilities that can serve as a starting point for later reinforcement learning.

These are promising directions already visible today, particularly for manipulation, and they are technical routes worth exploring further.

References

https://www.leihan.org/

Smart Note · Human Data