Recently, I have enjoyed writing accessible introductions to humanoid whole-body motion control and interviewing researchers who have done representative work in the field. In the previous interview, Interview with Xue Bin (Jason) Peng: Exploring General-Purpose Controllers for Humanoid Whole-Body Motion, we discussed how Peng approached humanoid control through animation. This episode's guest, Zhongyu Li, approached it through model-based control theory, spending six years of his PhD exploring whole-body control for legged robots.
Dr. Zhongyu Li earned his PhD at the University of California, Berkeley, advised by Professor Koushil Sreenath. His goal is to build safe, intelligent, and agile legged robots that can be better companions to humans. His research combines model-based optimal control with model-free reinforcement learning to address control and planning for legged robots. He also works to expand their capabilities beyond walking.
At the time of this interview, Dr. Li was preparing to join the Department of Mechanical and Automation Engineering at The Chinese University of Hong Kong (CUHK) as an assistant professor in fall 2025. His research focuses on humanoids, highly dynamic robots, and embodied intelligence. He was recruiting PhD students, postdoctoral researchers, and interns, and welcomed interested students to his lab.
Original article outline
Contents
👇
Building the Cerebellum of a Legged Robot
1. Model-Based Motion Control
2. Motion Control with Reinforcement Learning
3. Whole-Body Loco-Manipulation
4. Multi-Agent Cooperative Control
5. Safety
Questions and Discussion
6. Defining the Robot Cerebellum
7. What Drove the Transition from Model-Based Control to RL?
8. General-Purpose Controllers for Legged Robots
9. What Does Poor RL Generalization Mean?
10. How RL Sim-to-Real Became Possible
11. Why Transparent Hardware Matters
The Robot Cerebellum
My PhD dissertation is titled The Robot Cerebellum: Toward Safe, Agile and Intelligent Legged Robotics. It advances three core capabilities in bipedal robotics: agility, intelligence, and safety. My research projects all revolve around those three aspects.
The central aim throughout my PhD was to build a “robot cerebellum.” The concept comes from the functions of the human cerebellum: coordinating and controlling whole-body movements, with some ability to learn and adapt. It also supports avoiding obstacles without conscious thought and certain automatic safety behaviors. These are the functions I wanted to build into a robot module. [Editorial note: this is the interviewee’s engineering analogy. Automatic avoidance and safety involve several neural systems, rather than being functions exclusively of the biological cerebellum.]
Over my six years as a PhD student at Berkeley, I explored this extensively. I feel we are very close to robots that can truly “think.” Of course, much follow-up work remains, but I think the next step may be to move from the cerebellum toward the cerebrum. By incorporating more perception and environmental information, those two systems could work together on harder tasks, giving robots real environmental understanding and more complex behavioral capabilities.
My research has three keywords: agility, intelligence, and safety. Within those directions, the technical approach breaks down into five modules.
1. Model-Based Motion Control
Our first module was what I call structured motion intelligence. The core idea was to embed the robot's dynamics model in a structured optimization framework. We wanted to run optimization either online or offline and have the robot directly execute the resulting solution.
We did a great deal of work along this path. Our earliest achievement was expressive motion on a bipedal robot, entirely through optimization. We first designed a trajectory by hand, then solved for a control strategy with an optimization algorithm, allowing the robot to execute the movements precisely without relying on learning.
We also explored automatic parameter tuning: adjusting certain parameters on top of an existing controller to improve overall performance. That work still sat within the model-based control framework.
But we soon ran into problems. Reinforcement learning had not yet become part of our approach, and we relied entirely on optimization for motion control. We gradually recognized some fundamental limitations of model-based methods. [Editorial note: the original says “there was not yet a concept of reinforcement learning.” In this research narrative, that describes its absence from the team’s approach at that stage, rather than the historical invention of RL.]
First, there is a substantial mismatch between the model and the real robot. Even a very detailed dynamics model has a gap from physical hardware. We later called this the model-to-real gap. However good the model is, it cannot fully describe nonideal effects such as friction, compliance, hysteresis, and other nonlinearities.
We tried several ways to close that gap. One was to increase the optimization rate. If the optimizer runs fast enough, many errors can be treated as disturbances without separately modeling compensation for them. But that introduces another problem: high-speed, real-time optimization with a complex model is difficult at the control layer. Most optimization models are too complex to meet the required control frequency.
A second approach was manual tuning. We began by adjusting many controller parameters by hand; sometimes we also put those parameters into an optimizer. In essence, though, it was like a person manually running reinforcement learning. You define something like a reward function, then use intuition, experience, and experimental results to tune parameters until the system's behavior matches what you want.
The biggest problem is how slow the development process becomes. Parameters that work today may fail tomorrow. A robot is a dynamic system that changes every day: motors wear, mechanical structures loosen, joints overheat, and friction changes. Those hardware factors directly affect controller performance.
This approach can really only serve as a demonstration-oriented route: keep refining it in one direction and you can certainly produce a beautiful demo. Boston Dynamics, for example, has very capable controllers and produces visually impressive, stable demos. But the problem is that this method is difficult to scale up or extend, and cannot generalize. That is my assessment of this development route.
2. Motion Control with Reinforcement Learning
We began asking whether we could dispense with an explicit control model and let reinforcement learning teach the robot highly dynamic, broadly useful motion control. Could it train itself to walk, run, and perform long jumps rather than have us tune each capability by hand? That was a radical idea at the time. People generally did not believe RL could run directly on physical hardware; the prevailing view was that it worked in simulation but would fail on a real robot.
Xue Bin Peng's work was a representative example. DeepMimic looked excellent in simulation and produced very expressive behavior. But as robotics researchers, we initially doubted whether it could transfer to real robots. One main reason was its idealized assumptions. In simulation, training can access the robot's complete state, including every joint's position and velocity, and the model matches the training environment. After training, the controller is deployed in that same simulator.
Those assumptions do not hold in the real world, so we felt the approach was still far from hardware deployment. Fortunately, we were all at Berkeley. Our group and the group led by Sergey Levine and Pieter Abbeel, where Xue Bin Peng worked, were tackling the same problem: how to deploy reinforcement learning on real hardware. We did a series of projects, mainly on Cassie.
Our first task on Cassie was walking. We first made it work around mid-2020; the paper was published in 2021, but the actual result was achieved in 2020. That put our work in the same period as early RL controller research at ETH and other teams. Looking back, that wave was among the earliest groups to apply RL in practical locomotion control. The systems were still relatively basic, but they created real momentum. [Editorial note: this describes an early wave, rather than claiming the first hardware deployment. ETH’s ANYmal work, Learning Agile and Dynamic Motor Skills for Legged Robots, was published in 2019.]
The major breakthrough in our paper was not only showing that an RL controller could run on hardware. More importantly, we demonstrated strong compensation and robustness, which surprised many people at the time. In our experiments, if the robot was almost falling and you helped it back a little, the controller did not break down. It kept running, recovered the robot from a highly unstable state, and continued walking.
That is extremely difficult for model-based methods. Once the system goes outside what is called the region of attraction, it breaks down and cannot recover. In other words, once a disturbance pushes it beyond that boundary, no amount of effort can bring it back. This is how I characterize the limitation we encountered. [Editorial note: a region of attraction specifies states from which a particular closed-loop system converges; being outside it does not universally prove that every model-based controller must fail.]
The RL controller behaved differently. Its policy implicitly captured extensive experience of responding to disturbances. Once you pulled the robot back, it could adapt to the current state and stabilize again. That recovery ability was unprecedented at the time and was very impressive.
Those results made many people reconsider the potential of reinforcement learning for real-world robot locomotion. More researchers entered the field because they saw that RL could actually run on physical systems and offer practical value, rather than remain confined to simulation.
We explored many methodological ideas that I think were valuable. Over these few years, our direction went through the whole progression from nothing to a first working result, then to many capabilities. It was not easy: systematic progress in RL for bipedal control has really concentrated into just these recent years.
We kept advancing the work. We started with basic walking control, then tried more complex jumps—long jumps, high jumps, and other dynamic jumping maneuvers.
Our goal was to accomplish many kinds of jump with one controller: a standing long jump, a jump onto a table, a backward jump, or a jump with a turn. It was an ambitious idea, aimed at pushing the generality of bipedal motion control to a very high level.
We spent a lot of time on it, recorded many videos, and ultimately achieved good results. For example, two controllers performed 19 different jumping movements. They did not require a separately programmed strategy for each jump. Given a landing target, the robot could jump to it. Training took place entirely in simulation, followed by direct deployment on the real robot without additional parameter tuning.
We developed several methods to support that generalization. First, we designed multi-stage training so that the controller could handle the different stages and objectives of the task. Second, we proposed a dual-history architecture: it encodes a longer history of the robot's inputs and outputs while also giving direct access to short-term feedback. Together, those histories help the controller infer the robot's current dynamics more quickly and accurately.
The design gives the controller implicit adaptation. Without external intervention, it can adapt to changes in the hardware, such as drifting dynamics parameters or changing contact conditions. That is a central technical reason a controller can perform so many different jumping movements.
We then worked on running control, including sustained runs of 100 and 400 meters. The point of that paper was not the number of demos, but the systematic reinforcement learning methodology we developed. We wanted to answer how a broadly applicable controller architecture, with suitable training and control structures, could support different kinds of task.
The resulting controllers were also stable in practice. We repeatedly tested one walking controller over a year and a half, redeploying and running it periodically, and its performance remained consistent. We have similar long-term validation for jumping and running controllers. They showed consistent behavior without obvious degradation. That consistency over time is crucial and valuable for deployment in real systems.
A crucial dividing line between a demo and a practically usable controller is that a robot always has a model-to-hardware gap. We can obtain a model, but it can never describe reality perfectly. Important parameters—joint friction, rotational inertia, and friction between the feet and ground—change over time on the real robot.
For example, in one running experiment, the robot wore through its sole after just one lap. The friction changed, and we had to replace the sole before running again. Its dynamic response was then different from training. Joints wear too. Long-jump experiments generate large impacts, and after several jumps the joints' condition changes. A controller that looked excellent in simulation or training can therefore fail a few days after deployment.
We realized early that without adaptation, a controller cannot truly be deployed. You might record a beautiful video, then have the controller fail completely a few days later as the robot changes. To demonstrate that our controllers could adapt to the body, we deliberately tested them repeatedly while the hardware's condition was changing. We wanted to know whether they could stay stable through those changes in dynamics.
Our robot control research followed a progression. We began with optimization and structured models, then gradually moved toward reinforcement learning, letting the robot learn policies from experience without an explicit model-based control law. We started from the agility of a single bipedal robot and pushed its capabilities as far as possible. Many projects now attempt similar things and draw on some of our early methods or ideas. I think that is natural; these were fairly frontier efforts at the time.
Of course, that work was still about a single agent: building the capabilities of one robot. But from the start, we knew that walking, running, and jumping were not enough. A robot ultimately needs to do useful work to deliver value as a system. So early in my PhD, we began thinking about more complex tasks, including manipulation.
3. Whole-Body Loco-Manipulation
We began asking early in my PhD whether legged robots could perform manipulation tasks. Around 2021–2022, humanoid platforms were much less mature than they are now, configurations with multiple kinds of embodiment were uncommon, and many robots did not even have “hands.”
So we asked: could legged robots still do useful work without hands?
We found a promising starting point: use the feet to kick, push, and pull, accomplishing tasks without grasping. We call this nonprehensile manipulation. Rather than rely on a dedicated end effector such as a robot hand, we interact with the environment through the feet or other body parts without grasping the object. [Editorial note: the original says “non-contact” here; these actions involve contact. “Nonprehensile” means without grasping.]
Opening, pushing, or pulling a door, or even switching off a light, does not always require a hand to grasp a handle. Sometimes a gentle touch is enough to open the door. We started implementing such actions on legged robots, initially using a quadrupedal platform. The first task was precise soccer shooting: kick a ball to a specified target. We then expanded to other scenarios, such as saving penalties. When a ball arrived from different directions, the robot used different control strategies to respond. It could take a step to intercept the ball, jump to block it, or switch gaits to chase or dive for it.
We tried many combinations of movements, effectively combining several low-level controllers into a highly dynamic manipulation task that looked very complex as a whole. We then explored harder tasks, such as dribbling while walking and going around obstacles or holes in the ground. Those tasks require dynamic coordination among locomotion, obstacle avoidance, and interaction with the ball.
We took a further step by using imitation learning to learn human manipulation demonstrations. The idea was to teach the robot to use its legs through human demonstrations. A person could guide a robot leg directly by hand through a movement, and the robot would learn the control pattern: the action sequence, contact strategy, and force level. This lets it both perform tasks and learn from human behavior how to deal with new manipulation settings.
The core idea is hierarchical control. Many people discuss hierarchy now, but we built our system around it. At the lower level, we use a general RL controller whose job is to make the legs track different end-effector trajectories. On top of that capability, we train a controller focused on task planning: planning the trajectories of those end effectors, in this case the legs.
The upper controller can take several forms. It might be an RL policy or an imitation learning model. It can learn in simulation, be trained on real-world data, or combine the two—for example, pretraining in simulation and fine-tuning on real data. We explored this framework extensively and validated the general applicability of its structure.
We conducted many experiments on quadrupeds in this loco-manipulation framework. Moving from quadrupeds to humanoids is not complicated: you can view the front limbs as hands and the hind limbs as legs. The underlying logic is similar. It is still a problem of interaction between a single robot and an object.
4. Multi-Agent Cooperative Control
We then asked: if one agent can accomplish a task, how can several agents cooperate?
My earliest work in this direction was human–robot cooperation rather than a multi-robot task. We used a robotic dog to guide a blindfolded person as they walked, functioning as a guide dog. That is a collaboration problem between a robot and a person.
We then had three quadrupedal robots cooperate to pull a heavy box. They used cables to tow it through a very narrow passage. That task used centralized optimization, with substantial work to accelerate computation.
Later, we published work in which several quadrupeds towed a load together without needing centralized information during execution. Each robot ran its own controller without depending on the states of the others, making the control decentralized. We could even remove a robot during the task—for example, reduce a team from four robots to three—and they would quickly adjust their formation and complete the task.
That work used multi-agent reinforcement learning. Multi-robot cooperation is difficult: controlling even one robot is challenging, and coordinating several in the real world is harder. We made progress through hierarchical architectures and reinforcement learning.
How multiple robots cooperate, especially on physical hardware in real environments, is a substantial challenge. All the projects I have described underwent extensive real-world experiments; they were not purely simulation results. That matters because many RL systems look excellent in simulation but encounter unforeseen problems when deployed.
The overall idea remains hierarchical control, which is especially effective for complex systems such as quadrupeds, bipeds, and humanoids. At the lower level is a stable, reliable controller that responds quickly and maintains dynamic stability. The upper level handles task planning: target trajectories, cooperation strategies, and other decisions. This division makes the system more scalable and adaptable.
We explored each direction extensively: optimizing single-robot tasks, learning cooperation among agents, and combining methods such as reinforcement learning and imitation learning. We enabled several robots to cooperate without centralized information during execution. Doing that in the real world is far from easy.
5. Safety
Beyond cooperation, I think the most important issue is safety.
That is especially true for humanoids, which are designed to operate in human environments. Safety has to be the highest priority there. If a large language model says something wrong, it may amount to incorrect information. If a robot controller makes a mistake, however, the robot could hit a person and cause physical injury. That is unacceptable.
We therefore focus on adding control-theoretic safety assurance to learning-based methods. Learning, especially RL, can perform very well on highly dynamic tasks, with flexible movements, fast responses, and strong generalization. But its behavior is not always predictable; we do not know exactly when it will fail. Control theory can provide a safety layer or boundary around the policy. For example, as the robot approaches an obstacle—a wall, a structure, or a person—the system can guarantee no collisions or violations of the specified formal safety constraints. [Editorial note: such formal guarantees depend on the model, assumptions and constraints used; they do not amount to an unconditional guarantee for every real-world situation.]
We have explored how to construct verifiable safe sets and how probabilistic methods can model a policy's reliability boundaries. This can ensure that even a learning-based controller does not make decisions that injure people when deployed in the real world.
6. Defining the Robot Cerebellum
I have always felt that “cerebellum” is somewhat vague in robotics, especially when we discuss humanoids; people do not share one understanding of it. The definition becomes easier if we narrow the discussion to legged robots or quadrupeds. Their first tasks are clear: maintain balance and perform basic movement control, including standing, walking, running, and jumping.
In the human nervous system, the cerebellum is primarily involved in motor control, coordination, and balance, while the cerebrum is more involved in perception, cognition, and decision-making. As an analogy for a robot architecture, the cerebellum corresponds to the low-level module that controls the body's movements. Locomotion control, whole-body coordination, adaptation, and rapid reflex-like safety responses can all be placed within what I call the robot cerebellum.
For instance, many controllers in our research enable highly dynamic movement and adapt automatically to changes in robot dynamics, such as joint damage or worn soles. That reflects the adaptive aspect of the biological analogy. Continuing to walk after “injury” does not require complex planning by the cerebrum; it happens within the motion controller.
Another example is a person's fast response to danger, such as pulling a hand away immediately after touching a flame. That happens through a reflex pathway rather than deliberate, complex thought. From a robotics perspective, this is one capability we want in the cerebellum-like system: automatic safety responses. [Editorial note: this is an architectural analogy. Biological withdrawal reflexes are mediated by spinal circuits and should not be attributed exclusively to the cerebellum.]
I would therefore include the following functions in the robot cerebellum: basic movement such as walking, running, and jumping; dynamic adaptation that implicitly compensates for friction changes, disturbances, and mechanical wear; immediate responses and safety mechanisms, including rapid obstacle avoidance, posture recovery, and collision avoidance without higher-level decisions; and low-level execution and feedback integration, such as trajectory tracking, stabilization, and processing proprioception.
The cerebrum covers higher-level cognition and task decisions. Examples include perceiving a target and deciding whether to pick up Coke or Sprite; processing multimodal information such as vision, language, and semantics; planning long-horizon tasks, such as getting to school through route planning and navigation; and coordinating multiple agents by deciding who does which task and how they divide the work.
My research aims to build a cerebellum-like module that gives a robot not just movement, but adaptation, safety, and reflex-like intelligence. Ultimately, it should provide basic, stable, reliable bodily control without intervention from the higher-level “brain.”
There are still gray areas in practice. Whether cooperation strategy belongs to the cerebellum, for instance, is open to interpretation. But from the perspective of engineering architecture and biological inspiration, I find the distinction fairly clear and useful: the cerebellum handles low-level movement and safety; the cerebrum handles cognition and task decisions.
7. What Drove the Transition from Model-Based Control to RL?
I still believe model-based control is very valuable in robotics. Companies such as Boston Dynamics have not abandoned it; they continue using it while also obtaining increasingly good results with reinforcement learning.
Over roughly the past five years, the robot control community has undergone a clear shift in thinking. Our group embraced RL quickly because we deployed policies on real hardware early and the results were excellent. That made the whole team excited.
Some researchers were initially skeptical. They felt RL control policies were not “elegant” in the way model-based methods are, with explicit mathematical structure, analytic forms, and strong interpretability. They preferred more constructive approaches such as optimal control and nonlinear model predictive control (MPC). As experiments increasingly demonstrated RL's performance advantages on real systems, however, acceptance grew.
Around 2020, groups at ETH, our lab, Oregon State University, Google, and elsewhere began systematically advancing real-world RL locomotion deployment. By 2024, the community had largely reached the view that model-based methods and reinforcement learning are complementary rather than mutually exclusive, with each suited to different tasks and system constraints.
I think two factors drove RL's rapid growth. First, the entry requirements for mathematical modeling are relatively lower. Compared with traditional optimization-based control, you do not need to formulate a complete dynamics model or solve a complex constrained optimization problem, so newcomers can get started more easily. Second, open-source work and toolchains have greatly reduced the barriers to learning and experimentation. ETH, NVIDIA, and others released simulation platforms, GPU-accelerated engines, and training frameworks, with plentiful tutorials and examples. Undergraduates and graduate students can quickly build RL control systems without implementing an optimization solver from scratch.
As a result, more newcomers now begin directly with reinforcement learning rather than traditional model-based control.
From a research perspective, though, I do not think this is necessarily a good thing. RL is practical to implement, but several important problems remain. First, adaptation is not reliably achieved. A controller may work well today, yet if its training or design is poor, changes such as friction shifts or motor degradation may prevent effective adaptation. Long-term consistency remains difficult. Second, formal safety guarantees are lacking. An RL controller is often a black-box neural network, and the robot itself is highly complex. Combined, their behavior in unknown scenarios can be hard to predict. The robot may take a dangerous action in the wrong state, collide with someone, or fall—outcomes that are unacceptable in real applications.
We have continued exploring control-theoretic safety for RL policies. One important finding is that, when a robot is trained well and its control performance is sufficiently good, the closed-loop behavior can be approximated by a globally linear system. That lets us turn a black-box problem into a linear control problem, then apply traditional model-based methods to formally verify and constrain safety. It is a pragmatic and implementable approach. [Editorial note: the published result identifies a low-dimensional linear approximation of the tested closed-loop Cassie behavior; it does not establish that every coordinate of the full robot is globally linear or that all trained robots have this property.]
Many questions remain. Which systems or training conditions produce linear-looking closed-loop behavior? Is the linear approximation universal across robot platforms? How can we extract and use that structure effectively? I think these are among the most promising questions in combining reinforcement learning and control theory over the next few years.
8. General-Purpose Controllers for Legged Robots
Adaptation can indeed be seen as a first step toward a general-purpose robot controller. The field is discussing generalist control, but generality has multiple levels and dimensions.
First: adaptation on one physical robot—generality over time
The most basic level is adaptation over time on the same embodiment. Take Cassie: the hardware platform stays the same, but its dynamics change with use as joints wear, friction decreases, and motor performance degrades. A genuinely general controller should automatically adapt to those changes without retraining and maintain stable performance. We repeatedly validated this in the experiments discussed earlier, making it a first foundation for generality.
Second: transfer across robot platforms—structural generality
The next level is cross-embodiment generalization, which attracts a great deal of attention. Can one controller operate Boston Dynamics' Atlas, Unitree's Go1, and even humanoids or quadrupeds with entirely different shapes and joint structures? That demands strong adaptation to underlying dynamics, the arrangement of degrees of freedom, and even mechanical configurations.
I believe it is possible; in theory, I am confident it can be achieved. But is it actually necessary?
One controller supporting many platforms sounds powerful, but training costs might grow exponentially. Whether performance can remain good enough on every embodiment is a practical question. It may also sacrifice the finely optimized performance of a controller dedicated to one platform.
RL already has limitations in generalization, particularly when the distribution has multiple modes. If the state–action distribution changes substantially, one policy struggles to adapt. That is a real bottleneck.
Third: task-level generality—adaptation across tasks and modes
Another level is generalization across tasks or skills. Whether the robot is walking, running, jumping, turning, or moving from locomotion into basic manipulation, and whatever the environment or goal, the controller should perform consistently and transfer some capability between tasks.
Here, I think we can borrow ideas from large models. If many tasks and robot configurations are jointly trained in one system, the controller should theoretically do more than remember them: some degree of emergent behavior might appear on new combinations and tasks.
For example, learning task A, long jumping, and task B, obstacle avoidance, might enable task C, jumping while avoiding obstacles. Control abilities learned on a humanoid might transfer to a quadruped. Or imitation learning might allow rapid generalization after seeing a human demonstrate a new task.
9. What Does Poor RL Generalization Mean?
I do think RL falls short in controller generalization, and our experiments provide very direct examples.
In our locomotion work, we trained separate controllers for walking, jumping, and running. The running controller is very capable, supporting changes in speed, turns, and other complex behavior, but it always operates within the basic running mode. It does not transfer to jumping or ordinary walking.
We tried unified RL controllers that learned several tasks, such as walking and running together. But their performance across those tasks was worse than that of separately trained and deployed controllers. The multi-task controller was usable, yet specialist controllers were clearly better. This exposes a limitation in RL task generalization, especially when task distributions differ greatly and involve very different dynamic regimes. It is hard to find one policy that performs well on all of them.
At first, we suspected insufficient data. But increasing the amount did not noticeably improve convergence speed or results. It is not something that simply training more will solve. We then observed newer humanoid work combining upper-body manipulation with lower-body locomotion—walking, running, and jumping—in one system. These controllers work in simulation and appear more general.
On closer inspection, however, most use a teacher–student architecture. RL first trains a task-specific policy for each task; imitation learning or policy distillation then combines those policies into one larger network. Each underlying policy can perform well, and the distilled controller gains some ability to switch and combine tasks. This avoids the training-efficiency and stability problems of directly training the general-purpose controller with RL.
Personally, I think that is a reasonable engineering path for now. We work around RL's limited native generalization by using imitation learning for policy fusion and general control, while RL remains responsible for training optimal task-level policies. That division is more pragmatic.
It also means that the goal of one controller learning every task with true emergent capabilities remains challenging, especially for highly dynamic, high-dimensional locomotion. We still have a long way to go in understanding better training structures, stronger policy representations, and more natural ways to unify multi-task learning.
10. How RL Sim-to-Real Became Possible
We initially thought RL could work only in simulation and could not succeed on real hardware. I find the shift from that view very interesting.
The first change was using history. Early work such as Xue Bin Peng's trained RL models using complete simulator state: every link's position and every joint's position and state were directly available. Such complete information is unavailable on a physical robot. Base linear velocity, for example, cannot be measured directly with the robot's ordinary onboard sensors. It previously had to be estimated with a state estimator, and those estimates still are not especially precise.
Rather than rely only on the current frame's state, we began providing a history of inputs and outputs—motor commands and sensor readings over a period of time. That history can be used to infer the current base state.
We further found that it can reveal internal dynamics properties, such as joint inertia and mass parameters. These estimation processes can be described mathematically. Once we can infer the relevant states and properties, we can adjust control to adapt to changes in the body, instead of having it fail as the robot's condition changes or wears.
Many subsequent projects studied whether to use history, how long it should be, and how to encode it as input. We also ran extensive comparative experiments. I feel our approach is somewhat more generally applicable, although platforms differ substantially, so it is hard to make a blanket claim.
The second issue is hardware. However good a controller is in simulation, it cannot be deployed if the hardware cannot keep up. Actuator performance is especially important. If the joints cannot respond properly and track the simulated reference trajectory, the system will not work. Some robots use inexpensive hobby servos with poor response and controllability. A requested velocity may be unachievable; the joint may oscillate or overheat and shut down.
An RL controller is fundamentally a module issuing commands at high frequency, often target joint positions or velocities. In simulation, we assume those targets can be executed as the simulation advances step by step. On hardware, if the actuator cannot achieve the requested target, control can fail completely.
Platforms such as Cassie and some high-performance quadrupeds have capable joints that respond quickly and track accurately. That hardware can support genuinely dynamic RL policies on real robots.
I therefore see real-world RL deployment as the result of two developments coming together. Algorithmically, we began using history to compensate for unobserved states, enabling estimation and adaptive control. At the same time, better hardware let actuators actually track the targets produced in simulation.
There is another gap: the simulator's model can itself be inaccurate. Mass parameters or friction coefficients, for example, may differ considerably from their real values. That is another source of mismatch between simulation and hardware.
Methods such as domain randomization can help close this gap: perturb simulation parameters to improve policy robustness. But it is even more important that hardware executes the controller’s commands well. In simulation, each step assumes that the ideal torque command has been executed before advancing to the next step. Real actuators have delays and nonlinear responses and may not keep up with that torque command. That introduces another sim-to-real error. [Editorial note: The original contains an apparent transcription error in the Chinese word for torque.]
If the hardware is not good enough, algorithmic improvements may be wasted. I see this transition as algorithms and hardware advancing toward each other until they meet.
11. Why Transparent Hardware Matters
That is an excellent question and deserves a detailed answer. We recently released Berkeley Humanoid Lite, an open-source humanoid with transparency down to the joint level, precisely because our control work exposed real problems with many commercially available joints. That experience motivated several recent projects in our group, including colleagues' efforts to redesign those underlying systems.
From a control engineer’s perspective, hardware must be transparent. From issuing a desired joint velocity or position—or a torque command—to its conversion into current, transmission to the motor and actuation of the joint, we need to know what happens along the entire chain. [Editorial note: the apparent transcription error in the original’s parenthetical is read as “torque command.” Position, velocity and torque are distinct command types.]
We may not care about every intermediate control detail, but we are highly sensitive to performance characteristics. For a robot of a given size, what is the motor's maximum rotational speed? At that speed, what acceleration or velocity can it actually deliver? Is the response linear? Are there significant dead zones or unstable regions? All of these directly determine the upper limits of controller performance.
When the actuator becomes a black box, controller developers lose their footing. We cannot tell whether a policy failed because of the algorithm or because of a nonideal actuator response. Externally integrated joint modules and commercial closed systems can both create this problem: we do not know what happens after a command is sent. If the hardware is poor, that response chain cannot be trusted. However good the algorithm is, poor execution below it means poor control. That is deeply frustrating for control engineers.
Our group recently developed two complete humanoid designs. One motivation was to gain more transparency and control over the hardware so that we could test more dynamic, higher-performance controllers. Only when we understand the complete chain, from issuing commands through the closed-loop current response, can we confidently push the limits of control.
That is also why we open-source the system. Everyone can see how a control command is ultimately executed. When something goes wrong, you do not have to guess. You can investigate the possible layer: torque calculation, communication delay, insufficient current-loop bandwidth. You can debug it step by step.
The idea resembles the distinction between end-to-end policies and hierarchical controllers. End-to-end is exciting, but a failure can be difficult to localize. Hierarchy is more complex, yet you can identify the problematic component precisely. The open-source joints in Berkeley Humanoid Lite follow a similar principle: make the hardware response system layered, with each layer traceable, debuggable, and verifiable, so that it can properly support higher-performance control policies.
The result is a solid foundation.
12. Recruitment
As noted at the beginning, Dr. Zhongyu Li was recruiting for his CUHK lab at the time of this interview, with places for PhD students starting in fall 2026, postdocs starting as early as fall 2025, and visiting undergraduate or master's students able to join at any time. He welcomed applicants interested in humanoids, embodied intelligence, and control theory. Interested applicants were asked to send an email in English to zhongyu_li AT berkeley.edu, including the following:
Subject format: [Application] PhD/Postdoc/Visiting Student - Name - Which Track (VLA/Humanoid/Control Theory/Design). In the email, briefly introduce your research interests and previous experience, and attach an English CV. Applicants who have received a National Scholarship should mention it.
References
Zhongyu Li's website: https://zyliatzju.github.io/