Back to Whole-Body Control

Robot Control

A Conversation with Zhengyi Luo: SONIC’s Whole-Body Control Framework

Zhengyi Luo discusses SONIC’s model, motion data, retargeting, applications and the challenges of general-purpose humanoid whole-body control.

Zha Dami2026-04-1628 min readWeChat original
Contents

Our series on general-purpose humanoid controllers has explored this subject in A Conversation with Xue Bin (Jason) Peng: Exploring General-Purpose Controllers for Humanoid Whole-Body Control, A Conversation with Zhengyi Luo: Inside PHC, a General-Purpose Humanoid Controller, and the 2025 Human Data series wrap-up, Human Data—The 'Key' to Robot Data. This time, I invited Dr. Zhengyi Luo back to discuss SONIC, a widely known project in humanoid whole-body control. Editor’s note: This interview preserves the research and release status discussed in April 2026; subsequent releases may have expanded those capabilities.

Zhengyi Luo is a research scientist at NVIDIA’s GEAR Lab. He received his PhD from Carnegie Mellon University, advised by Kris Kitani. Before that, he earned his undergraduate degree at the University of Pennsylvania in 2019 and conducted research under Kostas Daniilidis. His interests lie at the intersection of vision, learning and robotics, including human pose estimation, human–object interaction modeling and human motion modeling. Readers can also follow his Xiaohongshu account for updates on SONIC’s open-source releases.

Zhengyi Luo’s Xiaohongshu profile, shown in the original interview.
Profile shown in the original article: Zheng He Shi Yi; Xiaohongshu ID 495366852; location listed as United States; 114 following, 15.5K followers and 20.7K likes/saves. Bio: NVIDIA humanoid-robotics researcher; Carnegie Mellon PhD and University of Pennsylvania computer-science undergraduate; zhengyiluo.com. These are historical profile details, not current counts.

This interview is also available on ReSpark through Xiaoyuzhou or Apple Podcasts.

Original article outline

Contents

👇

1. SONIC’s Core Idea

2. The Model and Method

3. What Distinguishes SONIC’s Performance?

4. What Distinguishes SONIC’s Training Data?

5. Data Retargeting

6. What Is Hardest About Retargeting?

7. Human–Object Interaction Data

8. SONIC’s Applications

9. Adapting SONIC to Different Robots

10. SONIC’s Open-Source Status

11. Further Questions

Retargeting versus SMPL

Inertial Motion Capture versus VR

VLA–Controller Integration

Wheeled Dual-Arm Robots versus Bipeds

A General-Purpose Controller Across Embodiments

Large-Motion and Strong-Interaction Scenarios

Generalists and Specialists

Further Scaling of Compute and Models

Multimodal Data

1. SONIC’s Core Idea

At its core, SONIC is a motion-tracking-based control method that scales motion tracking into a larger, more complete system, with the goal of building a general-purpose humanoid controller.

Its central idea and objective are therefore very close to PHC. Both follow the path of using motion tracking to drive humanoid control. PHC, however, was developed by one person under comparatively limited conditions and focused more on implementing and validating the method.

SONIC is being developed within a larger team, bringing in more practical and engineering considerations. It must work not only algorithmically but as a usable system. We put greater emphasis on making it practical so that it can enter concrete applications quickly and be deployed as a general-purpose controller.

It can support data collection on real robots and collaborate with higher-level models such as VLAs or a System One/System Two architecture. In that hierarchy, the higher-level model handles policy or decisions and SONIC executes them as the lower-level control module.

In that sense, SONIC extends PHC to physical humanoid robots, with further expansion in scale, system completeness and applications.

SONIC whole-body control and its motion-tracking interfaces.

2. The Model and Method

Overall, SONIC is a framework trained through reinforcement learning.

SONIC framework: human and robot motion pairs, three aligned encoders, a shared latent space and one robot-control policy.
SONIC framework: human and robot motion pairs, three aligned encoders, a shared latent space and one robot-control policy.

The first step is data. We have 700 hours of human motion-capture data. Retargeting it to the G1 produces paired data: human motion on one side and the corresponding robot motion on the other.

With those data, we train three encoders, each supporting a different interface.

The robot encoder receives the robot’s own motion representation.

The human encoder receives a complete human-motion representation.

The hybrid encoder takes three upper-body points, such as the head and both hands tracked in VR, together with robot or automatically generated lower-body motion.

The encoders learn to map these different input forms into a shared latent space. Although their forms differ, semantically they represent the same motion intent. For a humanoid, they can all ultimately be understood as expressions of the human motion space.

We align all three representations in that latent space and send the latent to a decoder. You can think of it as translating the latent representation into concrete robot joint commands: how the robot should move.

The entire process takes place in an RL framework. The three encoders train together rather than separately. We use a large number of parallel environments, allocating a subset to each encoder and training them simultaneously.

Besides the main RL objective, additional losses align the encoders’ latent spaces so that their representations agree as closely as possible. Different interfaces therefore share the same underlying motion representation, allowing seamless switching.

The result is the behavior you see in the demonstrations.

3. What Distinguishes SONIC’s Performance?

What are the baselines for the increases in network size, dataset volume and compute? What do these changes in scale actually mean?

There are two ways to understand the comparison. One is against animation or earlier motion-tracking work, such as PHC.

In my view, SONIC already clearly exceeds PHC in some aspects. PHC operates in virtual environments, while SONIC runs on physical robots in the real world. In that sense, it has crossed the boundary from simulation to reality.

In motion-tracking scale alone, SONIC is substantially larger than methods such as PHC and MaskedMimic. Its network is larger, its capacity stronger and its generality better. That is the comparison with motion tracking in animation.

The other comparison is within robotics. Platforms such as G1 already have excellent general-purpose motion-tracking work, including BeyondMimic, GMT—discussed in Human Data EP02—MoCap: GMT—and AnyTrack. These are all solid explorations.

Relative to them, SONIC emphasizes scaling three dimensions systematically and simultaneously: network size, dataset size and compute. Those dimensions are tightly coupled.

Start with data. The key question is whether there is enough human data. In whole-body teleoperation, we want the robot to reproduce what a person does, which is the same basic objective as PHC. We want this general-purpose controller, or motion tracker, to learn: whatever the person does, the robot can do.

Data volume is therefore crucial. PHC used an academic dataset of approximately 40 hours. SONIC increases that to 700 hours, an order-of-magnitude jump. At that scale, generalization clearly improves. Some results surprised even us: the robot may remain controlled after falling, or stay stable in more extreme cases. Those results suggest that the boundary of generalization has expanded.

Compute and model size are linked to data. More data inevitably requires more training compute. The paper describes our 700 hours as roughly 350,000 trajectories, a substantial increase relative to a previous scale of around 120,000 trajectories.

Because we train with RL in simulation, larger datasets require more parallel environments to reduce gradient noise. Our experiments showed that increasing the number of environments and compute produced more stable gradients, faster learning and better coverage of the dataset. In other words, the model could absorb the data more fully.

The same logic applies to model size. As behavior becomes more diverse—from basic walking to acrobatics, jumps and more dynamic motion—the behavioral space expands substantially. An undersized model cannot represent those complex behaviors. Data, compute and capacity therefore reinforce one another and must scale together.

There is also a comparison with humanoid controllers that do not center on motion tracking, such as HOMIE (transcribed as “HoME” in the original) and AMO. Many use decoupled structures: a locomotion policy for the lower body and inverse kinematics or neural IK for the upper body, rather than directly driving robot behavior from human-motion data.

SONIC instead treats motion tracking as a highly scalable task definition and uses human data directly to learn behavior. That breaks through limitations on behavior categories and supports richer, more unified humanoid behavior.

4. What Distinguishes SONIC’s Training Data?

What types of data make up the 700 hours—optical motion capture, inertial capture or others? What is the overall distribution, and how has motion diversity improved relative to PHC?

First, we are working toward releasing these 700 hours. A substantial portion will be made available to the community. For academia, this is a dataset larger by an order of magnitude, allowing more researchers to work from the same high-quality data.

Currently, all of it is optical motion capture, with very high accuracy and quality. We are discussing whether to add some inertial motion-capture data, but that is still being evaluated. I believe inertial data certainly has value; we need a better way to use it. If we add it, we will explore how to integrate it effectively into training.

Our collection process follows a comprehensive set of scripts, systematically covering different categories of motion to promote diversity. The dataset was originally intended for digital humans rather than robots. NVIDIA has long worked on animation and digital humans, and construction of these data began much earlier.

For robotics, we mainly developed high-quality retargeting so robots could use those data well. Because the data was designed for animation, games and digital humans, the range of motions is rich. It includes text labels and aims to cover behaviors that human characters might perform in games or virtual settings, such as opening doors and climbing stairs. We try to include any motion category you can imagine in those settings. Editor’s note: Coverage in the original human-motion library does not mean every clip was retained for robot training; stair-climbing capability is explicitly discussed as a limitation later in this interview.

Even a 700-hour dataset is far from complete. We found, for example, relatively few instances of bending down to pick something up from the ground, although this is common in real life. Coverage gaps remain even at this scale.

By releasing the dataset, we hope to encourage other institutions and teams to open their motion-capture data. The field benefits when the community forms a larger collective dataset. Expanding coverage together would support the long-term development of humanoid control and motion tracking.

Even with 700 hours and 350,000 trajectories, the space of human motion remains enormous. A movement such as hopping on one foot may have only dozens or hundreds of examples. Human behavior is open-ended and combines in effectively unlimited ways. Seven hundred hours is a major step in scale, but nowhere near complete coverage.

I think it represents a clear jump from earlier datasets, while leaving many behaviors to collect. More data should leave considerable room for controller improvements.

5. Data Retargeting

Retargeting remains unsolved in full. Our team invested an enormous amount of time in it, with help from NVIDIA’s digital-human teams to improve the pipeline. It is labor-intensive and time-consuming, yet still an open problem.

We also draw on open-source methods. As noted in the article, we used the newly released GMR and PyRoki as references, developing further on those frameworks and trying other retargeting approaches. Retargeting remains a crucial, incompletely solved step. 'Pirocky' in the Chinese transcript refers to PyRoki.

The core issue is the embodiment gap. When a captured human motion lifts the toes, for example, human feet are highly flexible, while a robot such as G1 lacks equivalent degrees of freedom. Structural differences inevitably introduce discrepancies in the mapping.

The issue becomes more obvious when switching platforms. In edge cases, how retargeting handles structural mismatches directly affects control quality. These details are difficult, and there is no general-purpose, fully robust solution yet.

Another important requirement is closing the loop after retargeting. You must train your own controller afterward rather than treating the retargeted motion as the final answer.

In practice, I gradually realized that robots, especially humanoids, demand even higher motion quality, smoothness and fluency than animation or games.

In animation or games, looking approximately right may suffice. A character’s overall pose can be correct even with minor mesh penetration or small stutters, and the audience may accept it. It is a visual medium viewed through a screen; if the overall impression works, slight defects do not feel too jarring. Robots differ in two fundamental ways.

First, they are physical and move in real space. A slight foot slip or hesitation at touchdown is amplified through sound, vibration and visual feedback. It is not merely unnatural-looking; it has a strong physical presence. Motion continuity and genuinely humanlike movement therefore face very high standards.

Second, robots are life-sized. In a game or animation, a character occupies only part of a screen, so details may be inconspicuous. With a robot 1.3 or 1.8 meters tall standing in front of you, even a slight backward lean or hesitation in a footstep is magnified. The unnaturalness is obvious.

Requirements for motion quality, precision and fluency are therefore even higher than in animation. I sometimes think motions generated in PHC or PULSE would not meet the quality threshold if placed directly onto a physical humanoid.

That is why retargeting must be followed by closed-loop controller training, adapting the policy to the robot’s dynamics and physical constraints rather than stopping at visual resemblance.

6. What Is Hardest About Retargeting?

Retargeting methods themselves have become fairly mature, with a reasonably clear overall framework. The bigger issue now is edge cases: a long-tail problem.

For ordinary walking, most available methods work if the cost function is designed well. Typical objectives match human orientation and position and solve them through optimization. That works for conventional motions.

Different motions, however, often need different costs. Normal walking uses one cost function, while sitting cross-legged may need an entirely different constraint structure. Solutions from kinematic optimization can fail if not validated in a physics environment.

Human motion-capture data may have no ground penetration, yet after mapping and optimizing it for the robot, the result may penetrate the ground. Human and robot structures differ; matching poses alone does not guarantee physical plausibility.

An important next direction is physics-based retargeting. After kinematic retargeting, solve the motion again in a physical environment to make it dynamically valid. This also removes undesirable artifacts, such as high-frequency hand jitter or unnatural instantaneous velocity spikes.

These defects may be inconspicuous in common motions but amplified in edge cases. Those long-tail cases may be the next problems requiring systematic solutions.

7. Human–Object Interaction Data

How much object-interaction data, such as locomotion and manipulation, have you included? Will you substantially increase data and investment in this area?

That is a very good question. I think we need to separate it into two parts.

The first concerns training a general-purpose controller such as SONIC. Once trained, using it for teleoperation or tasks does not necessarily require a large volume of manipulation-specific data to train that controller itself.

Many motions are dynamically equivalent from the controller’s perspective. Waving an arm freely and waving while holding a very light object differ little except for the object’s extra mass and inertia. If its weight is approximately negligible, a controller that reproduces the trajectory can already execute the motion through teleoperation or higher-level commands.

For locomotion and manipulation with light objects, a general-purpose motion controller therefore already supplies a large part of the required capability.

Other scenarios create problems. Pushing a large box or carrying something heavy introduces mass, contact forces and reaction forces that substantially affect dynamics. The policy must adapt to object properties instead of simply reproducing the original human motion. Additional interaction data and modeling may be needed.

Terrain and environmental interaction are another dimension. Slopes, steps, large obstacles and stairs substantially affect contact patterns and center-of-mass control. General motion-tracking data alone is insufficient; relevant environmental interaction data and training remain necessary.

From the controller’s perspective, light manipulation may not need large amounts of specialized data, while heavy loads and complex terrain still require additional data and training.

If a motion has little effect on the robot’s overall posture and dynamics, training a general-purpose controller such as SONIC does not necessarily require extensive task-specific interaction data.

If the motion mostly takes place in free space without significant external-force coupling or changes to system dynamics, general motion-tracking data can largely support controller learning.

Interaction data still has value; its purpose is different.

Once the general controller is trained, we may want task-level applications involving objects. SONIC demonstrates moving an apple from one place to another in a task integrated with a VLA.

Locomotion-and-manipulation data matters there because the task involves more than reproducing a trajectory: it includes object contact, grasp stability and trajectory planning.

Such data does not have to be optical motion capture. It can come from teleoperation, third-person video or videos of human demonstrations. All may help add task-level interaction capability above the general controller.

Training a general motion controller and using manipulation-specific data are therefore questions at different levels. The former concerns bodily movement itself; the latter concerns particular tasks and environmental interactions.

Can we understand it this way: the lower-level controller learns 'how to move' without needing extensive interaction data, while higher-level understanding of 'what to pick up and how' relies more on video and other task-level data?

There is another question here: whether to introduce perception, such as images or 3D perception, during training itself.

NVIDIA’s recently released EgoScale uses human-centric data to train a manipulation model. Those data are task-specific, organized around manipulation.

SONIC trains on optical motion capture to learn bodily movement and general control. The two data types are compatible but emphasize different things. Optical capture focuses on motion control without inherently requiring perception; human-centric manipulation data emphasizes the coupling of perception and action for specific tasks.

Again, these are two levels: general body control and specific manipulation tasks.

If controller training grows beyond 700 hours, would performance still improve substantially? At what data scale might the controller be considered mature or almost perfect?

We are exploring that very question. Seven hundred hours already works well for some basic, general motions. But ultimately it depends on what you want the robot to do.

Clothes folding creates new issues, for example. It needs many precise wrist movements, which were not a focus of our optical capture. Those data may need to be collected again or supplemented through augmentation. The dataset covers broad motion categories, but finer, more specialized tasks reveal coverage gaps.

Bending to pick something up from the ground is another example, and many people have tested SONIC on it. G1’s arm length limits its reach; sometimes even kneeling does not let it reach the object. In these specific settings, the controller still has shortcomings and may need targeted motion data. More data cannot remove a robot’s underlying geometric reach limits.

We iterate according to need: test SONIC in teleoperation demonstrations, assess particular motions and add data. The promotional video includes opening a trash-can lid by stepping on the pedal with one leg. That requires deliberate single-leg balance, a specialized capability our current data may not cover sufficiently.

The adjustment may involve not only data but controller formulation, such as fine-tuning rewards to support those special motions.

8. SONIC’s Applications

SONIC is an extension of PHC in some sense, but the extension concerns usability and practicality as much as scale.

Whole-body teleoperation is intuitive and powerful: the robot does what the person does, offering a complete expressive interface. It is less suitable for long-distance travel, though. Going a long way from A to B requires the operator to keep walking. Human and robot stride lengths and dynamics differ, making it inefficient and unnatural; the operator often has to follow along and continually adjust.

For navigation from A to B, we therefore need a planner. In a game, you push a joystick forward and the character walks automatically. We want the robot to have that capability.

We designed several more application-oriented interfaces. The same motion tracker underlies them all: give it human or robot motion, and it reproduces it.

The first interface is whole-body teleoperation, with human motion sent directly to SONIC for tracking.

The second combines a joystick with a kinematic planner. A high-level command such as 'walk forward,' 'move backward' or 'run' feeds a motion generator that produces a robot-motion sequence for SONIC. SONIC turns that sequence into joint-level control signals. You can walk or run the robot using a joystick without acting out the motion with your whole body.

We also provide a three-point interface for typical VR teleoperation. Many users track only the head and both hands rather than the entire body. We use those points for the upper body and have a planner complete the lower body—for example, generating a gait from forward/backward commands. The completed full-body motion goes to SONIC. Users control the upper body and direction while the lower-body motion is generated automatically, giving a lighter teleoperation setup.

Text-to-motion is another example. Our voice-controlled dancing demonstration uses NVIDIA’s earlier GENMO human-motion-generation model (now named GEM) to produce human motion from music or text. We send that motion into SONIC through the same interface used for whole-body teleoperation: a human trajectory to reproduce on the robot. Editor’s note: The transcript’s 'GEMO' refers to GENMO, now named GEM; the original framework diagram uses GENMO, while the updated project page uses GEM.

The design is therefore a unified motion-tracking controller underneath, with different upstream inputs: full-body capture, joystick plus planner, three-point VR or motions generated from text or music. That preserves a common foundation while greatly improving practicality and usability.

9. Adapting SONIC to Different Robots

A different robot platform will probably require retraining the lower-level controller. We plan to release the training code. G1 is currently the main platform, but we have preliminary support for other robots and hope the open-source community will help integrate more.

The interfaces themselves are very general and can support many uses.

For long-distance teleoperation, a planner could take the robot automatically from A to B. On arrival, the user seamlessly switches to whole-body teleoperation for finer actions such as opening a door, fetching an object or folding clothes. Afterward, switch back to the planner to travel onward. The transitions can be natural.

That is possible because we align the interfaces’ latent spaces during training. The input methods share one underlying motion-tracking controller, allowing smooth transitions rather than operating as completely separate systems.

A central idea is that full-body capture, planner-generated motion and three-point VR all aim to make the robot move. Since each ultimately produces target robot motion, we use one motion-tracking framework: given a target sequence, the robot tracks it. That is a very general task.

Interfaces differ only in where the motion comes from: captured from a person, produced by a generative model or generated by a planner from high-level commands. Sending the final target to the same controller unifies those input methods.

Without motion tracking as the shared mechanism, different tasks often demand different controllers: one for walking forward, another for bending and another for kicking. The system becomes fragmented and hard to extend.

Our key design choice is a sufficiently general motion tracker that unifies these different forms of input.

10. SONIC’s Open-Source Status

At the time of this interview, we had released the model and inference code, the full teleoperation system and essentially the deployment setup shown in the demos, allowing users to run the physical robot and reproduce the demonstrated results. Our own deployment stack and detailed usage documentation were included. Some users were already collecting teleoperation data with it. Training code and training data had not yet been released. Editor’s note: This describes the April 2026 interview checkpoint. Later releases in the linked official repository include a training stack; readers should consult the repository for current availability.

Using SONIC ourselves reveals issues and opportunities for improvement. Once we have verified internally that a change is stable and effective, we update the model. Through continued iteration, we hope to make it easier to use, more reliable and less prone to problems.

11. Further Questions

In VR whole-body teleoperation, you map human skeletal motion directly into robot control instead of using a separate conventional retargeting stage. What motivates that design, and what are its advantages?

Our earliest SONIC demos did have an intermediate retargeting stage. We used a neural retargeter and conducted experiments ourselves, with reasonable results.

We later chose not to make it the central path for several practical reasons.

Retargeting itself is hard. Producing smooth motion without foot sliding or ground penetration is difficult. Especially high-quality methods are often offline: they use a complete sequence and optimize it jointly for temporal smoothness and consistency.

Offline optimization works well because the entire temporal context is available. Whole-body teleoperation, however, receives streaming data and needs to be highly real-time. The robot should follow almost simultaneously as the person moves.

Real-time retargeting exists, but its results differ from offline optimization. The analogy is rendering: cinematic offline rendering can take a long time, while real-time rendering uses approximations. Retargeting with ten hours for optimization is fundamentally different from producing an answer in milliseconds.

Teleoperation is highly sensitive to latency. The robot must respond almost immediately. Adding either neural or analytic retargeting introduces delay. Online retargeting also remains open-ended: sliding-window size and the number of context frames affect the trade-off between latency and quality.

We chose a more direct route. With paired human and robot motions already available, we can establish the mapping in data and let the neural controller learn it directly.

Instead of running an online retargeting module, RL gives the controller the ability to perform the mapping itself, from human-motion representations directly to robot-control outputs.

In practice, that worked better. Developing our own neural retargeter also exposed difficult questions about online context, sliding windows and the balance between real-time performance and stability.

We therefore integrated the mapping into the RL framework rather than inserting retargeting as an independent real-time module.

How does VR whole-body teleoperation compare with wearing an inertial motion-capture suit in real-time performance, accuracy and deployment cost?

Our positioning is clear: SONIC is only the lower-level controller. It does not care how human motion is obtained; it receives motion and tracks it.

Motion-capture suits are compatible in principle. We chose VR mainly for ease of use, not because of a technical limitation.

We use PICO’s full-body setup. Adding two trackers at the feet, for example, enables fairly complete motion capture. Our tests showed very good real-time performance, high reliability and sufficiently good motion quality.

Setup effort is low. An operator can generally prepare the system within five minutes and start using it, without a complicated suit or repeated recalibration. That makes a large difference in practical applications.

VR equipment also allows fairly free movement without special clothing or elaborate extra equipment, giving a more natural experience. Our choice was about practicality and deployment, rather than an inability to use other inputs.

Essentially, SONIC is input-source-agnostic: supply a reasonable human-motion sequence in a supported representation, and it can track it.

With a VLA, is the relationship hierarchical: the VLA supplies higher-level action instructions and the controller executes the lower-level movement?

That is a direction we have been exploring. When designing SONIC’s latent space, we already considered how a future VLA would interact with a whole-body controller.

We deliberately had all three encoders map into the same space to create a unified action interface. The latent is structured, rather than an arbitrary embedding.

We intend to release the training code and the full pipeline connecting the model families referred to in this conversation as Flash/G: from data collection to VLA-integrated pick-and-place and trash-can-opening tasks. The published SONIC project identifies its demonstrated VLA connection as GR00T N1.5; 'Flash/G' is retained here as the transcript’s terminology rather than treated as a separate verified product name.

We have already tried some approaches. An early observation is that using the learned, structured latent space as the interface works somewhat better than making the VLA output full-body joint motion directly. The results remain preliminary, but that trend is visible.

I am particularly interested in how VLAs should interact with whole-body controllers. Full-body motion must be executed through a lower-level policy such as SONIC. The most direct theoretical approach is to have the VLA output the whole-body motion. Root velocity or local control signals alone cannot specify many fine-grained actions—for example, stepping on a trash-can pedal with one leg requires whole-body coordination.

Ideally, the VLA tells the controller, in some form, how the whole body should move. The key question is how to convey that information.

We tried direct full-body motion generation and latent generation by the VLA. The latent route currently seems better. It is a compressed motion representation, and our latent contains temporal information from more than ten frames rather than a single frame. Combined with a VLA, it produces more coherent, fluent motion instead of stitching isolated frames together.

We plan to release these exploratory results with the code and GR00T integration. The work is still early, but the direction is clear.

Compared with wheeled dual-arm systems, could legged humanoids benefit more from abundant human-video training data, improve their VLAs and iterate faster over the long term, perhaps eventually overtaking them?

Wheeled dual-arm and legged humanoid robots share manipulation data to a substantial degree. The crucial components are the upper body—hands, arms and grasping capability—and those data are broadly reusable.

The real difference is the legs. Legged humanoids must consider lower-body movement, while wheeled dual-arm platforms have stable support and do not need to solve gait and balance. The difference is primarily locomotion, not manipulation itself.

Our recently released EgoScale, for example, uses approximately 20,000 hours of human egocentric data to train manipulation models. Those human manipulation data do not inherently depend on the robot having legs. Dexterous-hand data and first-person human data can similarly support manipulation learning.

Adding legs makes the problem harder: manipulation must happen while maintaining posture and balance. The robot must stabilize itself and perform the task simultaneously.

Legged humanoids therefore have a higher potential ceiling: more reachable environments, movement and posture adjustments across spaces. But the floor can also be lower. An unstable system may perform poorly because it has so many problems to solve at once.

In the near term, wheeled dual-arm robots and legged humanoids will coexist. In many manipulation settings, a wheeled dual-arm robot resembles a person standing still. A legged humanoid can also stand at a table and use only its upper body. Their manipulation capabilities are closely related.

We essentially regard both as humanoids with different locomotion capabilities.

Could a future general-purpose controller support humanoids with different structures without retraining separately for each embodiment?

Over the long term, I believe that is possible. In the near term, however, a cross-embodiment humanoid controller may struggle to outperform one trained specifically for a single robot.

The reason is practical: network capacity is limited while the motion space is large. Some behaviors transfer between robots and some do not. Their joint structures, mass distributions and degrees of freedom differ.

For a single model to control both H1 and G1, the necessary capacity, representations and training methods remain uncertain. We do not yet have a mature solution for learning both embodiments equally well.

Online learning or some form of adaptation may be needed when deploying to a different robot. Recent work explores this, but it remains an open problem.

Cross-embodiment control is valuable but introduces further challenges. In the short term, it may not outperform a single-embodiment model whose full capacity is devoted to that structure.

It is difficult to make a definitive prediction, and we will explore it. Ideally, one controller would support G1, H2, Booster, EngineAI, Figure and Tesla—one model connecting all humanoid platforms. That is an appealing vision.

Achieving equally high performance on every platform remains an active, very open research problem.

How does SONIC perform on strong-interaction or changing-environment tasks, such as stairs, different terrain, balance recovery or ball sports? Is its capability in these settings mature enough?

Put simply, at this interview checkpoint it cannot climb stairs. Stairs and skateboarding are out of distribution for the current controller because it was not trained to execute those motions.

We tried it ourselves. Stair climbing is practical and useful but outside the current capability range. Skateboarding is even more extreme.

Training a controller specifically for skateboarding is feasible. The open problem is retaining generality at the same time: adding a highly specialized skill may affect existing generalization.

Humanoid motion control therefore still has much to explore. Recent stair-climbing and parkour projects are impressive, but often optimize for particular abilities.

The real challenge is doing those difficult motions while also supporting higher-level tasks from a VLA, performing manipulation and autonomously completing useful tasks. A genuinely general, deployable lower-level controller still needs substantial research.

What is the conflict between generality and capabilities such as parkour?

There is no fundamental contradiction. The question is how one controller can climb stairs and do parkour while supporting varied whole-body commands.

Stairs, obstacles and parkour may require perception in the underlying policy. The controller needs to see the environment, using depth, for example, to detect a bump or step and respond.

In work such as PHP, Perceptive Humanoid Parkour, the model observes depth and generates a specific gait from a higher-level command such as 'move forward.' It receives the command and generates lower-level movement internally.

SONIC follows a different approach: an external source supplies the complete target trajectory—where each joint should go—and SONIC tracks it accurately. It is a motion-tracking framework.

These approaches share elements but are difficult to merge. One is a perception-driven specialist generating its own motions; the other is a general tracker reproducing a supplied full-body sequence.

If perception lets a controller generate motion automatically from the environment, how do we preserve its ability to follow externally specified full-body commands accurately? The two control paradigms have different interfaces.

Fundamentally, this is about combining a generalist and a specialist. How does a generalist acquire specialist skills such as stairs and terrain perception without sacrificing its general motion expressiveness?

That still needs exploration; there is no particularly mature solution yet.

Can we say general controllers are largely sufficient for ordinary settings such as flat ground, while complex terrain or highly dynamic tasks such as dancing, parkour and stairs still require specialized policies?

Broadly, yes, in the near term.

General controllers now support walking, simple manipulation and basic dancing. More complex terrain, such as stairs, or extreme movements, such as difficult jumps and parkour, generally rely on specially trained policies.

We and others are trying to combine those abilities. SONIC dances, but in a general sense, rather than performing extreme jumps, flips or superhero landings as a specialist would.

There is a trade-off: a highly specialized model can excel at selected movements, while a general model that handles useful, practical everyday tasks may be less outstanding at extreme abilities.

A general controller covering productive everyday tasks and advanced specialist skills, while preserving wrist flexibility and motion quality, remains an open problem.

Given the current parameter count and compute investment, have you considered further scaling training to test whether performance still improves substantially?

We have used the current data fairly thoroughly. Covering more motion categories and task types will require further scaling. More varied behavior and more complex tasks need larger models and more compute.

Our approach is straightforward: push SONIC’s scale upward until infrastructure bottlenecks appear or increasing training scale stops producing clear gains. That suggests we are near the limit for that stage.

This is an intermediate result. Performance is good, but it is not the endpoint. We will continue exploring larger datasets, larger models and more systematic training, and share progress publicly.

SONIC currently trains mainly on 700 hours of high-precision optical motion capture. Will you add multimodal sources such as video and VR, or does good lower-level control require highly precise human-motion data?

We will certainly consider it. Work such as GAE also extracts human motion from multimodal data and ultimately constructs paired data. Its quality may be lower than optical capture, but its diversity and coverage are much greater.

We have considered adding more such human data but have not yet attempted it very systematically. Intuition and experience suggest that, with an appropriate integration method, more human-motion data should help.

For teleoperation, for example, more diverse data may improve robustness and stability in additional edge cases. The direction is clear: richer human-motion data is valuable. The key questions are how to add it and how to combine it.

References:

SONIC: https://nvlabs.github.io/GEAR-SONIC/

Zhengyi Luo’s Social Media and Research Profiles:

X/Twitter: @zhengyiluo

LinkedIn: Zhengyi (Zen) Luo

Website: https://www.zhengyiluo.com/

Google Scholar: https://scholar.google.com/citations?user=lHPTxGsAAAAJ&hl=en

Xiaohongshu: Zheng He Shi Yi

Smart Note’s Product Recommendation

As AI gradually develops new capabilities, being the mother of a young child lets me watch human intelligence grow alongside AI’s maturation. Last year, we encountered an interesting children’s conversational product, Kabao. Children can carry it with them, use different character cards and create characters themselves. The product has just launched, and I recommend it to Smart Note readers. It continues to evolve, and feedback is welcome.

Smart Note · Robot Control