This installment in my series on general-purpose humanoid controllers follows Interview with Xue Bin (Jason) Peng: Exploring General-Purpose Controllers for Humanoid Whole-Body Motion and A Conversation with Zhongyu Li: Building the “Cerebellum” of Legged Robots. This time, I invited Dr. Zhengyi Luo, the author of PHC, a widely known paper in this field.
Dr. Zhengyi Luo earned his PhD at Carnegie Mellon University, advised by Professor Kris Kitani. Before that, he received his bachelor's degree from the University of Pennsylvania in 2019, where he conducted research with Professor Kostas Daniilidis. His interests lie at the intersection of vision, learning, and robotics, particularly human pose estimation, human–object interaction modeling, and human motion modeling.
Original article outline
Contents
👇
Academic Journey
1. Character Animation
2. UHC
3. PHC
4. PHC2Real
About PHC
5. Introducing PHC
6. The Multiplicative Control Policy
7. Learning from AMASS Motion Capture
8. From PHC to Real Robots
9. Whole-Body Control after PHC
10. Which Reinforcement Learning Algorithm Does PHC Use?
About General-Purpose Controllers
11. What Does “General-Purpose Controller” Mean?
12. What Makes General-Purpose Control Difficult?
13. Representative Work on General-Purpose Control
Academic Journey
1. Character Animation
As an undergraduate at Penn, I worked on many AR/VR projects and had been using these technologies since 2016. In 2018–2019, I interned on Apple's team that went on to develop Vision Pro. At the time, I believed that further progress in AR/VR required more intelligent systems. I imagined wearing AR glasses, looking at an object, and immediately being told what it was used for. That struck me as a very meaningful capability.
During that Apple internship, I worked on character animation using a physics engine. Our group wanted to design character movements for AR environments, so that a virtual humanoid could move freely in an augmented reality scene. My project built on Jason Peng's DeepMimic and MCP, or Multiplicative Compositional Policies.
I then went to CMU for a master's degree in robotics. My starting point was still computer vision, so I worked on human pose estimation from video. As I did that, I gradually realized that I wanted to connect my work at Apple with my current research: to bring pose estimation and embodied AI together in simulation.
I noticed that without constraints, the movements produced by pose estimation systems could look unnatural. I began introducing physics simulation, using physical quantities and constraints to make those movements more realistic and natural. This direction was called physical grounding. My senior labmate Ye Yuan was working on related problems. With his guidance, I began combining computer vision with a virtual simulation engine to develop character motion systems grounded in physics.
2. UHC
I began working on UHC in 2019–2020. Inspired by Jason Peng's work, I wanted to build a controller that could execute any pose sequence. That demanded very strong generalization, and I wanted one controller to handle the entire process. I proposed and developed UHC with that goal. In a sense, UHC achieved it, but it also fell short of achieving it fully.

UHC
The reason was that UHC used residual force control, a mechanism developed by Ye Yuan. You can think of it as a marionette: extra forces help keep the humanoid standing, as though invisible strings were suspending it. In the interview, I describe this as adding a force at every joint. These auxiliary forces make the controller's learning task easier; you could even say they let the humanoid “fly.” They are additional assistance, rather than forces that a physical robot could necessarily produce through its own actuators.
I went on to do two related projects on this foundation. One was a kinematic controller, referred to as “UIC” in the original interview; the other was Embodied Scene-aware Human Pose Estimation, published at NeurIPS 2022. They addressed first-person and third-person human pose estimation, respectively. Both used the MuJoCo simulator and ran on CPUs. [Editorial note: the associated first-person project is KinPoly, published as Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimation at NeurIPS 2021.]
In 2022, I interned at NVIDIA, which was the second turning point of my PhD. Jason Peng was also there, so I began working directly with him and conducting research under his guidance. During that internship, I worked on Trace and Pace. Its humanoid controller, developed in Isaac Gym, could handle different kinds of terrain, including stairs and irregular ground.
Trace and Pace
That was an important milestone. Isaac Gym's parallel simulation reduced training jobs that used to take a week to less than a day, making experiments much more efficient. I could test different controllers quickly. I also learned important insights about controller training from Jason.
3. PHC
In fall 2022, I joined Meta as a visiting researcher. I worked with Reality Labs on character animation—in this case, human-shaped characters—and avatar systems. They wanted to develop virtual characters for AR/VR and generate their movements in a physics engine. I decided to revisit UHC and try removing residual forces: to take away the “crutches.” I used Isaac Gym for rapid training and AMASS as the training dataset.
By March 2023, I had completed a controller, PHC, that could learn and reproduce all the movements in AMASS. I realized that if motion tracking could make a humanoid execute those movements, I would have the motor skills needed for more complex tasks. I began thinking about running at a specified speed, crossing difficult terrain, or grasping objects. Those tasks could be built by combining the motion skills it had already learned—what we call motor primitives or motor control. [Editorial note: this is the interviewee’s description of the controller’s coverage. The published PHC demonstrations also show failure cases, so it should not be read as perfect success on every possible motion.]
PHC
Next, I thought about reusing these learned human movements. That led me to the follow-up work on a general motion representation—something we might now call a behavior foundation model. The key was to build a latent space in which learned motor skills could be reused. [Editorial note: the original names PHC again at this point; the latent-space reuse project discussed later in the interview is PULSE.]
The idea itself is not new in motion control. Humans do not relearn how to control their bodies every time they face a task. We already have basic motor skills stored in our brains, and some are even innate. I wanted robots to have a similar capability: acquire a set of basic movement abilities through pretraining, then call on them to accomplish new tasks. This is called motor skill reuse, and it is also common in character animation.
4. PHC2Real
The third turning point came around 2022–2023, as Elon Musk's humanoid robot initiative drew attention and suddenly made this field very popular. When I saw the news, I thought: I work on humanoids too, except mine are in virtual worlds. My original plan was to spend several more years in simulation and push their task capabilities as far as possible—for example, giving them vision and task understanding, so they could carry objects and tidy up virtual scenes.
I believe that building embodied AI systems in virtual worlds is a valuable research direction. In a physics engine, behavior is constrained by physical laws, which makes it meaningful to study. I expected this research to gradually feed back into AR/VR and computer vision before moving into robotics. But Optimus disrupted that timetable. In 2023, the Berkeley team published Learning Humanoid Locomotion with Transformers, and I began to realize that a route from simulation to reality was starting to become viable.
By the end of 2023, humanoid robots were commercially available. I began trying PHC-related methods on real hardware with Tairan He, another PhD student at CMU, in the H2O project. In some ways, our goal may have been too ambitious: the leap from PHC to reality was a big one. In ASAP, we took a step back and first worked on “DeepMimic to Real.” I think much of the field is now exploring the progression from DeepMimic toward PHC, and I am working on that too.
OmniH2O
ASAP
General-purpose control on real humanoid platforms remains very difficult. I have long felt that methods for real robots are roughly three to four years behind character animation. Even with the substantial technical foundations we now have, there is still a sim-to-real gap, although progress is accelerating.
About PHC
5. Introducing PHC
I have already touched on PHC, the work I developed. Its initial motivation came primarily from character animation—in other words, the problem of controlling a character. [Editorial note: the original expands PHC as “Perpetual Human Controller”; the published paper's title is Perpetual Humanoid Control for Real-time Simulated Avatars.]
When I began PHC, I already had an ideal demo in mind: point a camera at me, and whenever I move, the virtual humanoid on the screen makes the same movement in sync. Or, if I am controlling a robot, it should follow whatever I do. Today—in 2025, for example—we might call that teleoperation or mirrored control. But when I started this project, there was no ready-made controller that could do what I wanted, especially in terms of real-time performance and generalization across movements.
That goal requires a controller with very strong generalization. Its input comes from a camera, rather than a conventional motion capture studio or a high-precision multi-camera system. We are recognizing movements with just one monocular camera. Human pose estimates can therefore be very noisy, and the input is not tightly constrained. The controller must handle many kinds of motion while remaining reasonably robust to noisy input.
One design goal of PHC was to let the humanoid keep performing correctly even with these imperfect inputs. For example, if noisy input or a failed pose estimate makes it fall, it should be able to stand up again by itself. That is part of the reason for “Perpetual” in the name: the controller should not merely keep producing movements, but also have some ability to recover on its own.
This idea grew out of an earlier need. In the single-motion tracking setups I was using, systems such as DeepMimic could learn a reference movement, but each new clip required another trained model. I wanted a unified controller that could execute whichever sequence it was given without retraining each time. Since I also worked on human pose estimation, I especially wanted a system where any estimated pose could immediately drive the corresponding humanoid movement.
For that, I used AMASS, then the largest high-quality motion datasets. Developed by Professor Michael J. Black's team at the Max Planck Institute, it brings together multiple motion capture datasets. The collection used in this work contained approximately 10,000 diverse human motion sequences. I began trying to train a controller on it in 2020, initially in simulators such as MuJoCo. I worked on it intermittently for two or three years without getting the complete system to work.
The breakthrough came in summer 2022, when my NVIDIA internship introduced me to Isaac Gym, their GPU-based platform for massively parallel simulation. My reinforcement learning training became much more efficient and much faster than running on CPUs. That rekindled my determination to finish PHC.
In 2023, I returned to the project with two explicit goals. First, I wanted a real-time demo in which a camera captured human movements and the humanoid followed them, completing the mirrored-control pipeline. Second, I wanted one unified controller to learn all the motion sequences in AMASS, with coverage above 97%. [Editorial note: the percentage refers to success on the evaluated motion sequences; it does not measure the fraction of all possible human movements.]
These goals reinforce each other. If the controller learns all the movements in AMASS well, a pose captured by a monocular camera—even a noisy one—is more likely to resemble a movement pattern the controller has seen before. That can greatly improve the fidelity and stability of motion reproduction in actual use.
The ultimate vision for PHC was a humanoid control system with strong generalization and fault tolerance, covering most natural human movements.
6. The Multiplicative Control Policy
Progressive Multiplicative Control Policy (PMCP)
When I worked on PHC, I had 10,000 motion capture sequences. I decided to learn the entire collection rather than select a smaller subset. With Isaac Gym, I could simulate 2,000 environments—2,000 humanoids—at once. I randomly sampled 2,000 sequences from the collection for training and evaluation.
PHC's central objective was to learn every motion sequence successfully. That differs from the focus of traditional animation papers, where a model may count as successful if it reproduces some key movements and looks visually convincing. Because I was working on pose estimation and a universal controller, I wanted complete reproduction of each sequence.
I introduced several evaluation metrics. The first was success rate: could the model reproduce a sequence from beginning to end? A failure along the way, such as an excessive pose error or a fall, counted as a failed sequence. I also used mean per-joint position error, or MPJPE, to measure imitation accuracy—the average positional error between the joints and the reference motion. Acceleration error helped assess jitter and instability.
During training, I found that success rate was difficult to improve and stayed around 97%. Roughly 3% of the sequences—more than 300—still had not been learned successfully. I began examining the failed clips one by one.
The failures fell into two broad categories. Some were corrupted data: artifacts in the original motion capture made the movements discontinuous. Others were very difficult movements, such as cartwheels and backflips, which the controller struggled to learn.
But I noticed something else: if I trained on just one sequence, the model could overfit it. If I trained on ten sequences, it could learn those too. That observation suggested the next step.
I first trained on the full dataset until those roughly 300 unsuccessful sequences remained. Then I set them aside, initialized a new network, and continued training on them. After that, perhaps 30 even harder sequences would remain. I would isolate those and train another newly initialized network.
The approach resembles progressive networks, a method from Google's DeepMind: keep adding network capacity for more complex tasks, a little like growing new synapses in the brain to handle harder problems.
I ended up with three or four relatively independent small networks, each learning a particular group of sequences. To imitate a movement, I could use the corresponding network during inference. In that sense, the mechanism has something in common with a mixture of experts.
The distinction I am emphasizing is that PHC's component networks are frozen: once trained, their parameters no longer change. I train an additional control module, the composer, to determine how those networks are used during inference, avoiding catastrophic forgetting. [Editorial note: the published PMCP uses a multiplicative composition of learned primitives, rather than simply switching to one network; freezing is a feature of this design, not a defining difference from every mixture-of-experts system.]
The framework also supports an extra capability: learning to get up after a fall. That is quite different from imitating a reference motion. It is a less tightly constrained control problem, and reinforcement learning can also cause previously acquired skills to be forgotten.
Ultimately, I trained a controller that could both imitate high-quality movements and get up from the ground. I think this framework combines a high success rate with flexibility. I later developed more efficient training methods that do not necessarily require separate networks, but the structure remains useful when additional capabilities need to be incorporated.
7. Learning from AMASS Motion Capture
The AMASS data used here contains approximately 10,000 high-quality human motion capture, or mocap, sequences. They come from professional motion capture systems and provide detailed, high-precision pose information. My goal in PHC was to have the controller learn and reproduce every movement in this collection.
We use a simulated environment and learn control through reinforcement learning. AMASS supplies kinematic motion: the human pose at each frame, such as joint angles. It does not tell us how much force was used or what dynamics produced the movement. In other words, it does not include the dynamic information about joint forces or forces acting on the body.
Reinforcement learning therefore has to teach the controller how to produce those movements physically. In the interview's description, the input is the next frame's target motion—such as pose and velocity—and the output is the torque needed at each joint in the current frame to move the body toward that target. [Editorial note: in PHC's implementation, the learned policy outputs target joint angles for PD controllers, which then generate joint torques. The interview compresses these two stages into an end-to-end description.]
We call this task motion tracking. At each frame, the target motion is supplied as input; the control system produces joint torques, and the humanoid moves in physics simulation so as to follow that target as closely as possible.
PHC’s strength is that it does not train a separate controller for one fixed trajectory. A single controller learns to track any motion sequence in AMASS, or any part of one. This demonstrates strong generalization: it can accurately reproduce any motion type represented in AMASS. [Editorial note: this is the interviewee’s broad characterization; the unsuccessful clips discussed earlier and shown on the project page remain limits on that coverage.]
AMASS covers a broad range of common human poses and movement patterns. Once the controller fits most of those sequences well, it gains the ability to execute the great majority of natural human movements encountered in use. Of course, poor-quality input—for instance, excessive noise—also degrades control.
8. From PHC to Real Robots
We also wanted to transfer these movements into the real world using PHC and pose estimation. That led us to projects such as H2O, OmniH2O, ASAP, and HOVER. Working on real humanoid platforms is substantially harder. We have not fully matched the results in simulation, but we keep pushing forward.
My preferred research path is to refine the method and system in simulation first, exploring and validating them thoroughly in a virtual environment, then gradually transfer them to reality. This lets us work out animation-driven methods and early experiments before adapting them to real-world conditions.
In the real world, many numerical settings and modeling assumptions no longer hold, so deployment is much harder than in a simulator. Progress is therefore slower. Still, I think both simulation and real-world research matter, and both deserve sustained, in-depth work.
The ideal PHC demo I described is simple: give someone a camera, point it at a person, and the humanoid does whatever that person does. H2O is an attempt to realize that idea. It uses the same monocular human pose estimation approach as PHC. Time was tight, so rather than retrain that pose estimation model or develop a new perception system, we reused the existing pipeline. Connecting it to the robot let the robot follow the person's movements.
We did not then give H2O a name that explicitly positioned it as a PHC derivative, but what it accomplished was whole-body motion imitation. The final performance still had plenty of room for improvement, yet the pipeline and framework were in place. The route from human movement to robot movement—from monocular pose estimation through to physical execution—had been connected end to end. You could view it as a baby version of “PHC to Real.”
The main reason we did not simply call it PHC2Real was that its performance still fell short of simulation. There were several reasons, including hardware limitations. We used Unitree's H1, which struggled with some agile movements and may not have been capable of the high-precision imitation we wanted. The G1 platform available now might be able to execute some of those movements, but H1 could not deliver that performance at the time.
The main differences between OmniH2O and H2O concern their inputs and use cases. OmniH2O began as an effort to improve teleoperation. Operating through a third-person camera is difficult because the operator cannot easily perceive what the robot is doing. That view lacks immersion and synchronization. We wanted egocentric teleoperation: the operator sees what the robot sees and uses that shared view to control its movements.
For that, we wanted a VR interface, allowing first-person control through a headset. These tasks need an appropriate foundation model. In OmniH2O and the later HOVER project, we trained the base model in the same way as PHC: as a high-quality motion tracker.
OmniH2O also includes distillation. We transfer knowledge from a teacher model with complete inputs to a student with only partial inputs. PHC's controller receives full-body motion information. During distillation, however, we give the student just a small number of keypoints—for example, three points—and it learns a complete control policy by imitating the teacher's behavior.
This lets the student reproduce relatively complete human movements from limited input, while the teacher retains a structure and capability similar to PHC. The resulting training framework accommodates different input interfaces, including VR, rather than relying only on conventional full-body motion capture, which is difficult to obtain in everyday use.
9. Whole-Body Control after PHC
UHC and PHC were essentially attempting the same thing; PHC brought the idea to fruition and became the foundation for subsequent work. After PHC, I worked on PULSE and OmniGrasp in simulation, followed by PDC. This line moves beyond basic movement control toward reusable control priors—something we can understand as a behavior foundation model—and extends into manipulation. That includes visual manipulation, such as controlling actions using wearable visual equipment.
OmniGrasp
This is a connected series of projects. The aim is to control a wide variety of humanoid movements well in a virtual world, then use them for meaningful operations such as grasping objects. The broader direction is to make virtual humanoids more useful, explore higher-value tasks they can perform, and develop higher-level decision-making and control—what I call high-level brain functions.
One problem that I think robot control still has not fully solved is reusing skills that have already been acquired. Even when I train a new controller myself, I often still have to start from scratch. PULSE aims to reuse the movements PHC has learned. If those skills have already been acquired through training, I believe we should be able to use them again for new tasks. Humans do this too: rather than start from zero each time, we call on existing motor skills.
The core of PULSE is to build an efficient, usable skill library for humanoids. Once that library exists, reinforcement learning can solve new tasks by letting the humanoid select and use existing skills. But the library should not be a fixed, predefined set—like pressing a button in a game to perform one specified move. It should be highly general and diverse. That is why we embed the skills in a latent space.
This resembles an idea in image generation. Stable Diffusion, for example, encodes image information and models its statistical structure in a learned latent space instead of generating directly in pixel space. Motion control can use a similar approach: build a latent space that represents a large collection of human motor skills and gives those skills a structured representation.
Once that latent space has been learned, training a new controller or accomplishing a new movement or task can involve combining, searching, and fine-tuning within the existing skill space, rather than training from scratch.
10. Which Reinforcement Learning Algorithm Does PHC Use?
We use PPO throughout. Proximal Policy Optimization remains the mainstream algorithm in character animation. It is widely used because of its stability and general applicability. You can move from MuJoCo to Isaac Gym without substantially changing the algorithm's structure.
PPO also scales well. Whether you have 32 environments or 2,000, it can reliably train effective policies. That is one major reason it remains widely used.
I do think we may need to explore more advanced algorithms in the future. For now, however, PPO is so usable that even when alternatives exist, relatively few people actually adopt them.
About General-Purpose Controllers
11. What Does “General-Purpose Controller” Mean?
To me, “general-purpose” is not a particularly precise term. It is more an ideal we want to reach than a concept with a clearly defined boundary. Like artificial general intelligence, it is widely discussed, but people still interpret it in many different ways.
I have also encountered objections to using the term. My advisor, Kris Kitani, does not particularly like it because it is insufficiently specific. But what I mean by general-purpose is a controller that can produce a very diverse range of humanoid movements. Rather than optimizing only for one movement, a single controller should let the humanoid perform the different actions a person can perform in an open space.
When I wrote my first paper involving this idea, I wanted to express a simple vision: a person in an open area can walk, run, jump, and interact with objects in basic ways. I want a humanoid to do those things using the same controller, instead of depending on a separately designed module for every action.
In practice, many real humanoids we see today have control systems designed for a particular task. Some walk well but cannot jump. Others dance, perhaps through several different routines. Some can box. But ask the same robot to perform different kinds of tasks—to jump and box, or carry objects at the same time—and it cannot do them all.
The generality I am pursuing is a single robot that, like a person, can perform many tasks and cover most of the bodily movements we can think of in daily life. That capability should come from a unified controller.
My thinking was that once we are building humanoids, we should not artificially restrict them to a few movements. We should try to give them as much of the human movement repertoire as possible. The hardware is not yet fully mature, but demonstrations from many major manufacturers suggest it is approaching that level and should support attempts at general-purpose control. In my view, the problem is increasingly on the software side.
12. What Makes General-Purpose Control Difficult?
My own experience of the difficulty is that, once we actually make something work, it looks somewhat simpler in retrospect than I initially expected. The real challenge was deciding whether it was worth attempting, and whether it could work at all, without a precedent. That was especially true from 2019 to 2022.
At that time, the field was not sure whether one controller could truly generalize across different movements. I followed a lot of Jason Peng's work, starting around 2019. Humanoid controllers could already perform particular tasks very well: a controller specifically designed for walking, jumping, or boxing could excel at that individual task.
But how could one controller become general-purpose—generalizing across multiple kinds of movement? That remained an open question, and no one could confidently say whether it was possible. I was not sure myself when I began.
My research therefore centered on that question, working through it step by step. How could training be designed more effectively? How should inputs be handled to help the controller learn different kinds of movement? Which training strategies would enable real generalization across movement types?
I spent a great deal of time designing training mechanisms, structuring the controller's inputs, and deciding how broadly its tasks should be covered. Through repeated experiments and debugging, I gradually improved its generalization and moved closer to the goal of a truly general-purpose controller.
13. Representative Work on General-Purpose Control
My starting point in humanoid control was animation. Many of the humanoid demonstrations we see today also begin, methodologically, with animation-driven approaches. Their results often resemble animation-based methods from five years ago, except that movements previously achieved in animation are now being performed by real robots.
Other labs are exploring this direction too. Besides PHC, Meta published ScaDiver, which also aimed to build a generalizable controller. More recent work such as MaskedMimic—also discussed in Jason Peng's talks—tries to make one controller perform many tasks, moving beyond the pattern of one controller for one movement.
A new term is also gaining currency: behavior foundation model. It may be more appropriate than “general-purpose controller.” The idea is to train a unified behavior model that serves as a foundation at the behavioral level and supports many different complex tasks.
Here is an example of a real problem I faced in PHC. Systems such as DeepMimic could train a controller to reproduce a reference motion clip—single-motion tracking. In the setup I was considering, each new clip meant training another model. That could not meet my needs.
What I wanted was closer to pose-driven control. Whatever human pose I supplied—for example, an estimate from a pose estimation system—I wanted the humanoid to immediately reproduce that pose and movement. Training on a few specified motions was not enough. I needed coverage of almost every kind of human movement: an ambition of being able to execute 99% of common human motions.
Seen that way, I think this comes very close to the definition of general-purpose control. You could even say it is an early form of a behavior foundation model.
References
UHC: https://github.com/ZhengyiLuo/UHC
PHC: https://www.zhengyiluo.com/PHC-Site/
OmniGrasp: https://www.zhengyiluo.com/Omnigrasp-Site/
PULSE: https://www.zhengyiluo.com/PULSE-Site/
PDC: https://www.zhengyiluo.com/PDC-Site/
OmniH2O: https://omni.human2humanoid.com/
HOVER: https://hover-versatile-humanoid.github.io/
ASAP: https://agile.human2humanoid.com/
Zhengyi Luo's Social Profiles
X/Twitter: @zhengyiluo
LinkedIn: Zhengyi (Zen) Luo
Website: https://www.zhengyiluo.com/
Google Scholar: https://scholar.google.com/citations?user=lHPTxGsAAAAJ&hl=en
Xiaohongshu: Zheng He Shi Yi