Two years into the surge of interest in embodied intelligence, we ended 2024 by discussing a question where the ambition far exceeds present reality: reliability. For this panel, we invited researchers representing two promising approaches—reinforcement learning and model-based control—to discuss bringing robot reliability closer to the 99.9x% threshold. The title expresses an aspiration, rather than a reliability level already achieved by general-purpose robots.
This article presents the livestream discussion in edited form. I also edited its audio into a podcast episode, available by searching for ReSpark in the Xiaoyuzhou app. ReSpark is a program I created with several fellow investors focused on robotics. As this industry develops, we want to document the achievements, stories, and personal journeys of the researchers, founders, and investors moving it forward at different stages.
When intelligence reshapes reality, the future sparks now! Welcome to ReSpark! We will continue publishing business-focused interviews on embodied intelligence. You are welcome to subscribe to the podcast.
Original article outline
Contents
👇
1. Introducing the guests
2. Creating the wave, and being carried by it
3. RL and control for locomotion
RL and control describe problems, not single methods
Viewing sim-to-real RL as a model-based approach
Sim-to-real RL advantage one: more computation before deployment
Sim-to-real RL advantage two: reducing explicit state-estimation requirements
Control advantage one: online computation
Control advantage two: useful structure
Combining RL and control
Recommended work combining RL and control
Comparing RL and control in practice
The advantages of RL
Connecting RL with robotics
4. RL and control for manipulation
Why is manipulation so difficult for robots?
Why use real-world data for manipulation?
The different challenges of manipulation and locomotion
Competing approaches to manipulation
Limitations of control methods for manipulation
Limitations of solving manipulation in simulation
Combining imitation learning and RL for manipulation
5. How can general-purpose robots approach 99.9x% reliability?
Combining imitation learning and RL for manipulation
Improving humanoid reliability with RL
Which is more common in industry: traditional control or RL?
6. How to Make RL Great Again?
1. Introducing the guests
Jianlan Luo: I am currently a postdoctoral researcher at Berkeley Artificial Intelligence Research (BAIR), where I also earned my PhD. After working at Google for two or three years, I returned to Berkeley. Since around 2015 or 2016, I have focused on robot learning, primarily manipulation. At that time, the field was not yet widely referred to as “embodied intelligence.” These introductions describe the guests’ positions at the time of the 2024 panel.
Guanya Shi: I am an assistant professor in Carnegie Mellon University’s Robotics Institute. I joined CMU in 2023, so I am relatively new to the faculty ranks. I earned my PhD at Caltech, where I trained in control and learned from leading researchers including Richard M. Murray and John Doyle. I began studying the intersection of control and learning in 2017, and have accumulated seven or eight years of experience across RL and control.
After my PhD, I worked with Professor Byron Boots as a postdoc at the University of Washington. I completed my undergraduate degree at Tsinghua University. I am very excited about today’s topic: I have been thinking about this question since my undergraduate years, nearly a decade ago. I look forward to the discussion, and may offer a few provocative views. [Laughs.]
Zongqing Lu: I am from Peking University’s School of Computer Science, and my main research area is reinforcement-learning algorithms. Since returning to China in 2017, I have focused on RL, gradually moving into robotics in recent years. I also follow research on large models, particularly visual tokenization and learning through interaction.
Qiuguo Zhu: I am an associate professor in Zhejiang University’s College of Control Science and Engineering, and the founder of DEEP Robotics. I have long worked on humanoids and quadrupeds, initially using traditional control theory. Since 2019, I have collaborated with researchers overseas on RL, primarily leading graduate-student teams in that direction.
2. Creating the wave, and being carried by it
Guanya Shi: What distinguishes robotics from many other fields is that it is a systems discipline. It comprises multiple modules, which we might compare to a brain, cerebellum, and muscles. The brain corresponds to high-level algorithms, the cerebellum to whole-body control, and there are also hardware and many other components.
During my PhD, my advisor Joel W. Burdick described a “wave theory.” Whenever robotics becomes a hot topic, a wave of interest pushes technology forward. It may not take us directly to AGI, but it leaves achievements that form a foundation for the next wave. I strongly agree with that view.
Why did I choose robotics? I think robots are excellent platforms for scientific exploration. You can push physical limits by building a running robot, approach the problem through materials science, or focus on algorithms. The field’s diversity keeps me motivated to pursue research.
I have consistently argued that learning in real environments matters, rather than relying only on end-to-end learning or large datasets. Last year, a discussion titled “Data Is All You Need?”, covered in Can Large Models Enable General-purpose Robots?, produced a 50:50 vote. That reflects how divided opinion remains. I believe real data and structured learning frameworks are still indispensable. That is both my view and the direction I pursue.
Qiuguo Zhu: I have worked in robotics for 15 years, mainly on humanoids, as well as quadrupeds. Early on, we collaborated with Tsinghua University to participate in RoboCup’s humanoid competition. Its long-term vision was a humanoid soccer team capable of defeating human champions by 2050. That set an ambitious horizon for robotics research.
Humanoids are excellent research platforms, but challenging, especially in maintaining stable control. We want them to leave the laboratory and handle complex real environments. Around 2010, our lab tried to get a robot walking outside on a sports field. That apparently simple goal took nearly ten years to achieve. It taught me that robotics requires long-term commitment.
Along the way, both theory and practice have to improve continually. Loving robots is the foundation, but persistence and patience are just as essential.
Jianlan Luo: I encountered robotics while studying control at Berkeley. My first project involved manipulation. Traditional approaches separate state estimation from control, but state estimation can be extremely difficult in complex manipulation settings. That limits what those approaches can do.
In 2015, I encountered Berkeley’s work on end-to-end learning, particularly Guided Policy Search. Optimizing from perception to control without depending on a precise intermediate state estimate was very inspiring. Manipulation remains difficult, but the broad applicability of end-to-end learning is making it an important direction for complex tasks. [The original calls the method “Guide Policy Search”; its established name is Guided Policy Search.]
Zongqing Lu: My starting point was a little different. I initially focused on RL algorithms, mostly tested in MuJoCo and games. With the development of large models, we tried combining them with RL to build general-purpose agents capable of multiple complex tasks. But we found that even with large models, many problems are better addressed through interaction with the real physical world.
We therefore turned toward robotics, especially locomotion and manipulation. Using real-environment data, we explore how large models can better understand the physical world. Their visual understanding, for example, still has substantial weaknesses in the settings we study. We are investigating whether multimodal interaction, including vision and touch, can improve that understanding.
As a means of interacting with the physical world, robots are important tools for pursuing more general intelligence. Our next research focus is how to integrate large models more efficiently with real-world data and situations.
3. RL and control for locomotion
RL and control describe problems, not single methods
Guanya Shi: My first point is somewhat provocative. I do not think RL or control should be treated as a single method; each describes a problem. In RL, an MDP specifies rewards and system dynamics, and we seek a policy that optimizes reward. That is fundamentally related to control: designing a policy for a dynamical system to meet objectives and constraints. Using the state–action–reward–next-state formulation associated with the Bellman framework does not, by itself, make the underlying problem entirely different.
The tuple (S, A, R, S′) describes a transition in a Markov decision process. S is the state of the system or environment at a particular time. A is an action available to the agent, such as a robot. R is the reward resulting from that action, used to evaluate its outcome. S′ is the next state reached after the action. This describes how an agent receives rewards and changes state through its actions, allowing it to improve behavior toward an objective. [The original attributes this formulation to Richard Bellman. This is a simplified historical description, and a transition tuple is not the entire definition of an MDP.]
Because RL and control describe problems, each has many algorithms. Sim-to-real RL is a much more specific workflow: train a policy using PPO in a locomotion simulator, such as MuJoCo, then deploy it in reality. That differs considerably from Jianlan’s work on RL directly on physical robots. I think those differences can exceed the difference between RL and control. [The transcript also names an “SR game” simulator, but that name is unclear; it is not silently replaced with a guessed product.]
My point is that we need more specific problem definitions. When discussing RL, which kind do we mean: sim-to-real model-free RL, learning a world model through model-based RL, or model-free RL directly in reality? Jianlan has produced several excellent recent examples of the latter.
I think these kinds of RL differ substantially. There is also offline RL: with a large demonstration dataset, you can do offline reinforcement learning as well as imitation learning. The transcript compares these differences with “the differences within RL,” an awkward repetition; the surrounding argument is that the broad RL-versus-control labels hide important distinctions. That is my first provocative point.
Viewing sim-to-real RL as a model-based approach
For locomotion, one of the most popular RL workflows is sim-to-real: train a policy in simulation using PPO or a similar algorithm, then deploy it in reality. My second provocative claim is that I regard this workflow as model-based. You first need a model you trust, such as a simulator; you learn a policy in it, then transfer the policy to the real world. [This is a system-level use of “model-based.” Under the usual algorithm-level convention, PPO can still be model-free when the simulator merely supplies sampled transitions rather than explicit planning or model gradients.]
Sim-to-real RL advantage one: more computation before deployment than conventional model-based control
I think it differs from model-based control (MBC) along two dimensions. The first advantage of sim-to-real RL is the extensive computation available before deployment. You can learn from hundreds, thousands, or tens of thousands of trajectories and spend substantial time optimizing a policy. In many MPC-style controllers, by contrast, optimization has to run online to decide the robot’s next action. [The original says offline compute is virtually infinite and MPC has none. These are contrasts in where the main optimization occurs: compute remains finite, and MPC systems can also use offline preparation or explicit solutions.]
That is the first major difference in how computation is allocated: before operation or during operation. Here, “offline” means calculating before the robot starts running, and “online” means calculating while it runs. This use of “offline computation” should not be confused with the separate definition of offline RL on a fixed dataset.
Sim-to-real RL advantage two: reducing explicit state-estimation requirements
The second major advantage, in my view, is that sim-to-real RL can partly bypass explicit state estimation. Traditional control often follows a separation-style architecture. To control a humanoid with MPC, for example, you will commonly first build an estimator for quantities such as linear and angular velocity. [The formal separation principle applies under particular assumptions; not every traditional control problem satisfies it.]
In traditional control, estimation can be just as substantial a problem as control itself. You need to understand the robot’s current state and position. RL can partly work around a separately designed estimator by learning policy behavior and state inference together.
Consider teacher–student learning, one of the most successful and popular patterns in locomotion. Since the 2020 Science Robotics paper Learning Quadrupedal Locomotion over Challenging Terrain, I estimate that thousands of papers have followed this logic. Over roughly five years, it has become highly developed. The paper count is the speaker’s estimate, and the time span is approximate at the 2024 panel date.
What does teacher–student learning mean? First, train a teacher policy in simulation with privileged information, such as terrain details and robot velocities. The teacher cannot necessarily be deployed directly, because those quantities may not be available on hardware. Then distill it into a student policy that uses the history of available observations—the past n steps. This approach is now relatively mature. “Knowing everything” is shorthand for having richer relevant state information, not necessarily every simulator variable.
That framework can remove the need to supply certain true-state variables explicitly at deployment. The controller instead uses a history of sensory observations. These are the two main advantages I see in RL. It still needs sensing and some form of implicit or explicit inference; it does not eliminate every state-estimation problem.
Control advantage one: online computation
Control also has two major advantages. The first is almost the opposite: online computation. In a conventional sim-to-real workflow, a trained policy’s parameters are fixed after deployment. If performance is poor, retraining or real-world adaptation may be necessary, and simulator errors can create a large reality gap. MPC, however, recomputes decisions as the robot runs. [A fixed-parameter feedback policy still changes its actions with observations. Online adaptation is also possible, so this contrast does not describe every RL controller.]
Control advantage two: useful structure
The second advantage is useful structure. PPO is widely used in the simulation-heavy RL settings we are discussing: it is on-policy, its sample inefficiency is often tolerable, and it can be relatively straightforward to use with good results. But an unconstrained learned policy may lack the structure we want and converge to strange behaviors. Control offers structural knowledge that can improve efficiency and behavior. That is an important advantage.
Combining RL and control
Much of my research tries to combine the two: bring control’s safety tools into RL, or use control structure to make learning more robust. I also share the other guests’ view that effective learning ultimately has to be carried out or validated in the real world.
I am not especially optimistic that progress in simulation alone will solve everything. Building a perfect simulator is extraordinarily difficult. We cannot expect an AGI-level simulator to appear and solve all our problems. Learning must connect to reality in some way. My approach is to combine RL and control for real-world learning.
Recommended work combining RL and control
First, I particularly like avoiding a direct n-input-to-n-output mapping in which RL produces every motor command. For a humanoid, why insist that RL output each motor’s torque? It can output higher-level quantities such as velocities or accelerations. I find that direction interesting and have several related papers. Search for CAJun or visit our lab’s homepage to see videos, including quadruped jumping, based on this hierarchical idea. CAJun specifically combines a learned high-level centroidal policy with a low-level control optimizer.
Second, I recommend real-to-sim-to-real work. Rather than learning in a fixed simulator and transferring straight to reality, first collect real-world data to improve the simulator, then learn in the improved simulator and transfer back. I am optimistic about this direction, and we have recently pursued it too.
Third, consider learned dynamics combined with MPC. We learn a world or dynamics model, then use MPC for control. We have several recent papers in this area. DIAL-MPC has attracted attention, and its code can be easier to use than some conventional MPC implementations. We also learn dynamics from real-world data and apply sampling-based MPC, which I regard as particularly flexible. [DIAL-MPC is itself a training-free sampling-based controller; it does not inherently require a learned world model. It is one example within the broader control discussion, not a claim that all these components belong to one algorithm.]
Comparing RL and control in practice
MPC + WBC
Qiuguo Zhu: We did extensive early work using MPC together with whole-body control (WBC). Looking at the overall results, however, RL can better express the robot’s full-body movement capabilities in the tasks we have studied.
MPC and WBC were mainstream approaches before the recent spread of RL in these systems. Humanoid and quadruped whole-body models are complex. Solving their high-dimensional optimization problems requires substantial computation while still meeting real-time deadlines.
MPC: longer-horizon optimization with a simplified model
We therefore often simplify the model for MPC’s longer-horizon optimization. We might use a single-rigid-body model or a more detailed robot model, predict the state trajectory, and use those predictions to support stable behavior. But the simplified model cannot fully capture whole-body dynamics, which creates limitations.
WBC: short-horizon optimization with a detailed model
The next stage uses a full-body dynamics model to solve for the current control action in real time. This is whole-body control, which compensates for some limitations of MPC’s simplified model. We have done substantial work in this area with good results, but flexibility remained constrained. In our experience, it was difficult to match Boston Dynamics’ Atlas, which I regard as demonstrating the strongest movement capabilities in that comparison.
In many settings, we still primarily use traditional MPC + WBC because it is stable, reliable, and easier to interpret and analyze. Nevertheless, it brings some problems.
Problems with MPC + WBC
First, a legged robot’s contact mode changes discretely. How do we maintain stability through repeated jumps, walking, or running? Traditional formulations have to address when the foot contacts the ground and how quickly the robot responds afterward. Including contact timing in the optimization can make it extremely complicated, and many implementations simplify away that issue.
Second, how can tactile or contact feedback affect the controller quickly enough? By the time relevant information is detected and acted upon, the robot might already have fallen. These are challenges we encounter with traditional model-based control. Some are very difficult, or impossible within the particular formulation we are using.
The advantages of RL
RL has advantages, and I will not repeat all those Guanya mentioned. I think of it as something like a stabilizer during contact with the environment: a learned policy can help maintain stable, reliable behavior. Our current humanoids, quadrupeds, and wheeled-legged robots use RL for such control. It can offer greater flexibility in exploiting whole-body dynamics and coordinating the body. This describes our systems and experience, not a general safety guarantee for RL.
Our main research emphasis is therefore RL, but that does not make model-based methods obsolete. At Zhejiang University, our graduate students are beginning to incorporate MPC ideas into RL to pursue more challenging tasks.
Moving from MPC alone, to RL, and now toward combinations of MPC and RL seems to reflect our continuing exploration of the field.
Connecting RL with robotics
Zongqing Lu: Let me first respond to Guanya’s point. Model-free RL and control are addressing related mathematical problems; they differ in how they solve them.
I have long worked on RL. For robots, especially humanoids, our emphasis may differ somewhat. Inspired by human behavior, we use a hierarchical framework. The upper level influences body pose or posture rather than controlling motors directly. A lower-level policy implements that target through motor commands or other control quantities.
We focus on interaction with the physical world, which requires visual information. Humans control their bodies using vision, touch, and other external sensory inputs. We want to understand how humanoids can similarly interact with their surroundings.
Humanoids also let us exploit large amounts of human-related data, including internet videos. How can those data help a robot control its body? At the lower level, either a PPO policy trained in simulation or an MPC controller is acceptable, as long as it realizes the upper level’s goals. This architecture may make it easier to build a useful system.
In locomotion, RL and MPC often solve the same underlying control task. I think the more important question is how to add modules that account for information about the physical world. Learning dynamics is difficult. Building a complete visual and tactile simulator or world model, as others propose, seems very challenging to me. From my perspective, learning in the real world may be more promising.
4. RL and control for manipulation
Why is manipulation so difficult for robots?
Jianlan Luo: Everyone has been discussing control. I also trained thoroughly in control, although I no longer primarily write control papers. It is extraordinarily effective and already part of everyday life. Cars use it; control technology helped send people to the Moon. Like Newton’s laws, we do not always explicitly cite it because it is already established and useful.
But robotics includes many different systems and problems. As Guanya said, it is a large systems field: a car, an aircraft, a legged robot, and a manipulating arm can all count as robots. Whether we use model-free or model-based methods, we have to examine the particular problem.
Broadly, people work on three kinds of task: navigation, locomotion, and manipulation. Applying the same methodological labels to those three settings can actually mean very different things.
Consider end-to-end learning. I am not sure how to formulate fully end-to-end navigation on a global scale. Mapping the whole Earth in that way would not be practical. I would instead expect visual foundation models to extract semantics, followed by mapping and techniques such as simultaneous localization and mapping (SLAM).
Locomotion is interesting too. Discussions of simulators are, fundamentally, about using models of physical laws. I agree with the view that sampling physical dynamics can be regarded as model-based at the overall-system level. A simulator is a better computational tool, but it does not produce a policy by itself; an optimizer is still needed. The algorithm-level meaning of model-free RL can coexist with this broader perspective.
Manipulation is another problem. Moravec’s paradox is a famous observation in robotics: over the history of AI, tasks that seem easy to humans can be difficult for machines, while some tasks people find difficult are easier to automate. We can teach machines board games—IBM’s Deep Blue played chess decades ago, and AlphaGo later defeated Lee Sedol—yet reliably picking up a cup can remain difficult. There are many explanations and debates about this contrast. [Deep Blue played chess, not Go, and defeated Garry Kasparov in 1997. The original’s “20 years ago” is approximate at the 2024 panel date.]
Moravec’s paradox, associated with Hans Moravec’s work in the 1980s, concerns the uneven difficulty of AI tasks. Perception and motor skills that humans perform easily can be very difficult for machines, while some formal reasoning and mathematical computations are comparatively easier to automate. One explanation is that sensorimotor skills reflect a long evolutionary history and are deeply embedded in biological systems, whereas formal reasoning and mathematics are relatively recent cultural developments. This is a broad observation, not a claim that every reasoning problem is easy for AI.
The paradox cautions us against underestimating supposedly “low-level” biological skills. A major challenge in pursuing general intelligence may be enabling machines to perceive and interact naturally with reality, not only to perform formal reasoning.
The evolutionary roots of our sensorimotor abilities extend far back through the history of life, while formal symbolic and numerical culture is much more recent. That helps explain why teaching machines movement can be harder than teaching some forms of reasoning. [The original speaks of humans evolving for billions of years and acquiring “true intelligence” only in the last few millennia. Humans themselves have not existed for billions of years, nor did human intelligence begin with recent formal culture.] In any case, manipulation asks how robots can act on the physical world as humans do, and do so reliably.
Manipulation differs from navigation and locomotion regardless of whether the method is end-to-end. It often involves particular objects in a bounded workspace. A mobile robot may first navigate to a location, then manipulate objects within that local area. In a factory, home, or another application, object dimensions and task scope may be relatively well defined within that workspace.
That gives end-to-end methods a plausible setting in which to succeed. The workspace is limited, and the objects and task can be constrained. Such methods can work well on particular tasks, provided they are trained and validated on suitable data to establish reliability and robustness within that scope.
I do not get deeply caught up in the end-to-end versus non-end-to-end debate. I care more about how to solve the problem. I recall Geoffrey Hinton describing neural networks in speech systems as gradually taking over adjacent components: replace one part with a network, and it expands through the system, almost like bacteria consuming the surrounding parts, until the whole system is dominated by neural learning. [This is the speaker’s paraphrase. Hinton received the 2018 Turing Award, but the precise talk and wording recalled here have not been independently established.]
My point is that manipulation can provide a setting where such a policy is learnable. End-to-end optimization can reduce biases introduced by intermediate interfaces and optimize directly for the objective. A suitably scoped manipulation environment makes that plausible. It does not establish that a single policy already covers every possible real-world situation.
Why use real-world data for manipulation?
Let us return to why real-world data matters. Put aside the technical details for a moment. Today’s topic is approaching 99.9%, or even 100%, reliability. Why do robots fail to reach 100%?
If they fall short, something in the model or system differs from the actual environment. We need to reduce that mismatch. In an environment with unknown information, my view is that the most general and scalable way to do that is to interact with it, obtain the missing information, and use that information to improve the system.
We can design a specialized system for particular requirements and construct a model for one or two tasks. Such dedicated systems can be very fast and may demonstrate 100% success over a particular test or operating range. A finite observation of perfect success does not prove an absence of failures under all conditions.
But we cannot hand-design a separate solution for a thousand or ten thousand tasks. For a general-purpose robot with an arm, the details differ but the underlying process is similar: start with an initial hypothesis, interact with the environment, and fill in what is missing.
RL offers one framework for that process. There are many ways to implement it, including model-based and model-free RL. I am not especially committed to one of those labels; each approach has strengths.
To summarize my argument for RL: I believe that achieving high-performance, highly reliable manipulation across real environments ultimately calls for reinforcement learning, rather than only passive learning from a fixed offline dataset. To close the remaining performance gap, the system needs interaction and improvement. That is my basic reasoning. [This is the speaker’s position, not a theorem that all interaction must be RL or that RL guarantees 100% reliability. Interactive data collection, system identification, and other control or learning mechanisms can also be used.]
The different challenges of manipulation and locomotion
There are deeper reasons for my preference for RL over traditional control in manipulation. A major theme today is comparing manipulation with locomotion; I see them as fundamentally different problems.
For locomotion, uncertainty is largely associated with the robot and its contact conditions. Stepping on a stone or slipping introduces uncertainty that we can often quantify. Control theory provides tools for such problems, including robust control: model uncertainty quantitatively, define a worst-case bound, and design a controller whose guarantees apply when the actual uncertainty remains within that bound and the other assumptions hold.
Manipulation faces uncertainty in the external world. The robot is a carefully engineered system, but the environment’s possible complexity is vast: visual variation, physical interactions, and tactile observations all matter. Fully and precisely modeling every such factor is generally impractical. We can model parts and achieve useful results, but not capture all external complexity. Calling the robot “deterministic” is a simplifying contrast; hardware also has uncertainty.
That is why I see a need for RL in manipulation. Interaction can help the robot adapt to external uncertainty. Existing control tools such as robust control can handle many locomotion problems effectively, while manipulation motivates data-driven approaches, including RL, for the complexity of the outside world. This distinction is the speaker’s argument, not a prohibition on control methods in manipulation or RL in locomotion.
Competing approaches to manipulation
Jianming Wang: I see several technical approaches. First is imitation learning on real robots: Diffusion Policy, ALOHA, and related methods can produce useful policies. Second is real-world RL, exemplified by Dr. Luo’s HIL-SERL. Third is RL in simulation for manipulation. Another direction improves simulation, for example by adding differentiable components to build a virtual world model.
Guanya Shi: Here is another provocative view: completely solving robotic manipulation may be no different from solving AGI. Consider “open”: opening a bottle, a cup, a can of cola, and a door share a word but require different physical actions. If a policy could open anything, I think it would be close to what we mean by AGI. This is a conceptual claim about broad generalization, rather than a formal definition of AGI.
When evaluating results, I often struggle to tell whether a method works because the researchers chose favorable tasks or because it represents a substantive breakthrough. I agree with the categories you listed. Without trying to rank their merits, let me give my perspective on them.
The first is a brute-force scaling approach: collect many demonstrations and use behavior cloning. In this part of robotics, comparatively few people are using offline RL; many simply clone demonstrated behavior. Diffusion Policy, ALOHA, and Physical Intelligence’s recent π0 follow that broad demonstration-learning logic.
The second is sim-to-real. Fewer people use it for these manipulation tasks, but I find it interesting. A famous example is OpenAI’s 2019 Rubik’s Cube project.
The third includes various fine-tuning approaches: pretrain a model, then adapt it further.
A fourth, which you did not mention, is conventional control. It does exist, but generalization can be limited. A pen-spinning controller, for instance, may work for one particular pen and fail when its mass or length doubles.
The fifth is model-based learning: learn a model, then use it for control. As Jianlan noted, predicting states or target outcomes and building an accurate model are both difficult. It is a relatively specialized direction, but researchers at places such as Caltech and MIT study particle dynamics and related models. (This is the learned-dynamics-model + MPC route. At the time of this article, we planned a dedicated interview about it.)
Jianlan Luo: Let me share my view too. Reverse-engineering human intelligence in pursuit of AGI could require an immense amount of computation, even just to capture the range of human movement and behavior.
Limitations of control methods for manipulation
Applying control to manipulation brings difficulties. Early robotics often had little visual input, almost like feeling an elephant in the dark. An important theoretical foundation in optimal control is the Hamilton–Jacobi–Bellman (HJB) equation, whose classical formulation relies on suitable smoothness or differentiability assumptions. [The original describes HJB as the foundation of all control and suggests inputs must become smoother than at the preceding step. That is not a general requirement: control encompasses many methods, and generalized HJB solutions need not be classically differentiable.]
The HJB equation is a central tool in dynamic programming for characterizing an optimal-control value function. It develops Bellman’s dynamic-programming principle in a continuous-time formulation related to Hamilton–Jacobi equations, hence the name Hamilton–Jacobi–Bellman.
My concern is that high-dimensional, nonsmooth sensory inputs can make the classical formulation difficult or impractical to use directly, undermining that route to manipulation control. [The original says HJB and therefore all control simply cease to work. That is too broad: nonsmooth optimal control, viscosity solutions, hybrid control, and other approaches remain available. The practical difficulty does not invalidate control theory as a whole.]
Manipulation necessarily involves perception. I think learning is currently the most effective way to address many of those perception challenges, so manipulation has a strong reason to incorporate learning.
That does not make control irrelevant. Consider very large robots at a dock, with four or five systems cooperating to move cargo. We should use motion planning, including efficient online planners, and verification mechanisms to check the resulting plans. The desired standard is effectively perfect accuracy within the specified conditions; otherwise robots could collide or injure people. Verification guarantees always depend on their models and assumptions, rather than establishing unconditional 100% safety.
Now return to an arm working on objects in a bounded workspace. Its policy-learning problem may not require the most complicated dynamics. Even solving 5% or 10% of useful tasks would be major progress toward robots entering ordinary homes and could make them highly useful.
Limitations of solving manipulation in simulation
My second point concerns simulation. Learning-based manipulation faces a problem: because robots are not yet deployed widely at the scale we want, we do not have enough real-world data to build better models.
People say manipulation can generate vast amounts of data. But if that data is simulated, the difficult part is making the simulation sufficiently accurate and complete.
I am not a simulation expert, so this is a basic conceptual argument. Building an all-encompassing simulator is like trying to simulate the universe. With real data, we instead need to build a robot’s “brain.” The universe contains brains; constructing the whole universe is harder than constructing a brain. So I think we should start with the brain. This is an analogy about modeling scope, not a literal requirement that a useful simulator reproduce the universe.
Take Tesla. Before deployment, it may have lacked data, but once its cars were on the road, the situation changed. I saw a figure yesterday suggesting the fleet generates 50 billion units of driving distance per year. At that scale, the issue is no longer whether data exists but how to handle or store all the camera data. [The original says “50 billion” without a clear distance unit or verifiable source and asserts insufficient disk capacity. These are the speaker’s recalled figures and claims, not confirmed current Tesla statistics.]
We often focus on simulators generating massive datasets, while real-world companies such as Tesla already produce more data than is easy to handle. The key question is what to do with that data: how to use it effectively, rather than only how much a simulator can generate.
I do think simulation is very useful. If someone started an autonomous-driving company today and said they would not build a simulator, I would ask why: software integration and the overall development workflow would be extremely difficult without one. Simulation also helps with comparatively simple, largely kinematic tasks such as pick-and-place.
But manipulation requires both handling visual input and dealing with complex physics: contact between objects, fluids, deformation, and more. Instead of investing everything in an enormous, universe-like simulator, I favor real data and thinking harder about how to learn better policies from it. Real data avoids the specific simulation-to-reality mismatch. [The original says real data has “no gap whatsoever”; it can still have coverage, sensing, distribution-shift, and deployment gaps.]
Guanya Shi: I strongly agree. People do not fold clothes or bedding by solving the Navier–Stokes equations. When we lift a glass of water, we do not explicitly know its exact dynamics. We rely on intuition, experience, and sensation.
Simulators can sometimes be overkill. We do not need to simulate everything; we need to capture the important aspects that support the behavior we want. This also connects to locomotion.
Why does locomotion work relatively well in today’s simulation frameworks? Much of it primarily concerns rigid-body dynamics, rather than more complex phenomena such as fluids and granular materials. This describes the mainstream tasks in the comparison, not every terrain or locomotion setting.
Jianlan Luo: Intuitive physics is especially interesting for manipulation. Useful behavior may lie on a lower-dimensional manifold, so we do not necessarily need to solve every underlying physical equation. Dynamic actions such as moving a whip, or the motion of deformable tissues mentioned in the transcript, can involve very difficult partial differential equations. Directly solving the full problem through a traditional formulation can be impractical. [The source’s term translated as “fascia” is unusual in context; it is retained as deformable tissue rather than replaced with an invented object.]
Optimizing in an outer loop around a PDE solver while making real-time decisions is not a plausible account of how people manipulate objects. I am confident that the brain is not explicitly solving those equations to do these tasks. Instead, intuitive physics develops through long interaction with the environment. Reaching toward an object or searching through a space looks simple, but reflects an intuitive understanding of the physical world.
Those skills predate modern science. People made fire through friction and used bows and arrows tens of thousands of years ago. The modern scientific and mathematical framework I am referring to developed much later, roughly over the last 300 years. We did not need PDEs to acquire these skills. Our brains developed an efficient way to connect with the physical world through intuition and interaction. Mathematics itself is much older than 300 years; the comparison refers to modern formulations, not the beginning of all mathematics.
5. How can general-purpose robots approach 99.9x% reliability?
Combining imitation learning and RL for manipulation
Let us return to imitation learning and RL. I do not regard them as mutually exclusive. Imitation learning provides a strong starting point. To approach the remaining performance gap, I believe we need RL—interaction with the environment to improve further. [The original says RL is necessary for 100% success. This is the speaker’s argument, not proof that any method can guarantee perfect reliability.]
RL traditionally needed large amounts of experience, but recent results show that some tasks can be learned through one or two hours of real-world interaction. I believe pretraining could reduce that further, perhaps to 20 minutes, because much of the time is spent acquiring basic manipulation skills. The shorter time is a forecast made in 2024, not an established result across general-purpose tasks.
Let me add one more point: why are we trying to push a metric toward 99.9%?
My rough picture is that about five million robots are deployed worldwide and roughly 95% are position-controlled, effectively “blind,” repeating programmed motions from A to B to C. Since the first industrial robot installation at General Motors in 1961, we have made improvements without fundamentally changing that pattern. [The original calls this the invention of the first production line; production lines long predate 1961. The robot totals and 95% estimate are the speaker’s broad, unverified estimates, not a documented count of every robot category.]
For a robot to be useful in such settings, I think success needs to be effectively perfect; 99% can still be too low. Every failure has a cost. On a production line, a failure may stop operations and reduce financial returns, and someone has to absorb that loss.
Someone might say industrial robots need perfect success, but a home robot only needs 70% or 80%. I have thought about that. Yesterday I went to play sports at Berkeley’s gym, where an automated vending system sells water. Tap your credit card and it delivers a bottle. I have used it frequently over the past month. But yesterday the bottle fell into a place where I could not retrieve it, rather than reaching the outlet. That failure had a very concrete cost: the bottle’s price, $2.75. Every setting has costs.
Imagine a Starbucks robot spilling coffee every five hours, or a home robot dropping cups or plates once a week. Those are losses because the robot interacts with the physical world. I am not saying we should abandon home robots and build only industrial ones. I think we need constrained or partly constrained environments at this stage, where we can develop solutions and scientific understanding, then extend them to homes. Home robots also raise data-collection and privacy concerns: people may not want a robot watching them sleep or sending household footage back for training.
I think we are beginning, for the first time in this particular wave, to systematically consider a more general solution. Data could eventually accumulate like a flywheel. Once it does, how should we address 99.9% success? I believe the answer will combine RL and imitation learning. They are not mutually exclusive. The key is interacting with a changing environment, rather than treating it as static.
Improving humanoid reliability with RL
Zongqing Lu: Let me first respond on manipulation. We did substantial work on it this year, initially training through RL in simulation. I found that once the problem was clearly formulated, RL could solve it relatively readily—in the particular setting where the hand also received tactile information.
But those simulated policies did not solve the physical task as we hoped. Much of the input was point clouds, and generalization was difficult: the point clouds captured by real hardware differed greatly from those in simulation. In our experience, that sim-to-real gap was hard to overcome.
The work nevertheless taught us something important: tactile information can make manipulation easier. That is intuitive—people with visual impairments can also carry out many manipulation tasks.
As Guanya said, manipulation is a major challenge. A particular task may be solved effectively with simulation, sim-to-real transfer, or real-world RL. The crucial issue is generalization: how do we obtain a policy that performs many different manipulation tasks? I find that extremely difficult, and currently do not see an adequate dataset to drive the broad model we would need.
For humanoids with two arms and two hands, we therefore use a two-stage approach. Since robot-collected data is limited, we first use human data: visual information goes in and human poses come out. We call this pretraining. Large quantities of comparatively low-quality internet data train a model to generate hand poses. We then perform post-training using real-robot data, for example teleoperation, to improve transfer and generalization.
I regard that as another promising route beyond the four approaches discussed earlier. Its emphasis is generalization, rather than only solving an individual manipulation task.
On reliability, I personally doubt that humanoids operating in open settings will readily exceed 99% with imitation learning and RL alone. Even a person may make one mistake in ten thousand grasps. Unlike an industrial robot, the embodied-AI applications I want are not restricted to a fixed scene. As Jianlan said, products in different settings have different requirements; in some cases, 70% or 80% success may be enough. [The human example corresponds to 99.99%, not 99%; it illustrates fallibility but does not mathematically establish the speaker’s proposed 99% limit. This disagreement among guests is preserved.]
In our architecture, RL mainly executes the upper-level module’s commands, whether those are poses or other instructions, by controlling motors or related quantities. I think the more important reliability question lies in the higher-level policy or pose-generation model, rather than only in refining the lower-level executor.
Even an ideally trained lower-level RL policy is bounded by the problem it is given. At best, it finds an optimal policy for that formulation. How we formulate and supply the problem is therefore something we need to examine more carefully.
Industry demands high reliability. Which is used more: traditional control or RL?
Qiuguo Zhu: Academic success rates such as 90% or 99% cannot simply be transferred to an industrial product. I think the other guests would agree: experimental results and deployment reliability are different things.
I would divide product reliability into several stages. One is the hardware’s own reliability, which we discuss internally using concepts such as failure-free operating time.
Failure-free operating time is an important measure of whether a robot is dependable. In power-infrastructure inspection, for example, a customer may require three months without a failure.
Hardware is the first consideration. More components, connections, and joints generally create more potential failure points. To keep a complex humanoid or quadruped stable over long periods, the hardware must be reliable. Mechanical structure, electrical control, and low-level software all need to operate dependably. That is a major systems challenge.
Second, we examine software and algorithms, including traditional model-based methods. Under their theoretical assumptions, correctly implemented controllers can be highly dependable. But adding autonomous navigation and precise localization introduces other challenges. [The original says correct code essentially cannot fail; correct implementation alone does not guarantee that models, assumptions, sensors, and operating conditions remain valid.]
Algorithms have to adapt to dynamic environments, including changes across seasons. We therefore need to evaluate reliability at the software and algorithm level too. Industrial reliability is different from a benchmark success rate; these are the major issues we encounter.
For deployed quadrupeds, traditional control remains common in our applications. We are gradually incorporating RL, especially to improve locomotion, alongside SLAM and localization/navigation components. By the end of 2024, we planned to try RL for adapting to different terrain within relatively fixed operating environments. This records the plan at the panel date, rather than asserting its later completion.
A key challenge is what happens when a robot tips over or fails. How do we keep it from damaging its surroundings or breaking nearby objects? We may need RL components or explicit rules for such contingencies, to prevent additional risks and preserve safety in deployment.
For many applications, quadruped locomotion is already adequate in my view. Demonstrations sometimes showcase extreme capabilities, such as landing stably after jumping from several meters up, but most applications do not need such spectacular actions. Current quadrupeds can meet the movement requirements of many practical settings. That assessment is specific to those settings, not a claim that every mobility problem is solved.
As Zongqing noted, humanoids have more complex joints. A typical quadruped in this comparison has 12 actuated joints, while a humanoid has several dozen. Keeping the system stable in use is a major challenge, so humanoids need more development time. People understandably want to see them working in everyday life, but progress must be gradual, beginning with suitable applications.
Humanoids can already serve as university research platforms. A robot standing at a coffee machine and serving a cup may also be feasible. But broad household or industrial use still has a long road ahead. It will require continued iteration in hardware, algorithms, and especially manipulation. We should take a longer-term view.
Guanya Shi: We have worked on humanoid sim-to-real transfer for about a year and a half, including MPC and RL, so I have a strong sense of the reliability problems.
I agree with Qiuguo’s assessment of what is achievable now and what may take years. Let me add several points. First, safety, like RL, is a broad concept with multiple dimensions. Hardware reliability and stability are fundamental. Professor Chris Atkeson’s work helped inspire Baymax in Big Hero 6. His view, as I recall it, is that truly safe robots should be intrinsically safe in their hardware—compliant enough not to hurt people even when something goes wrong. That is primarily a structural and design question, not only an algorithmic one. [Atkeson was an inspiration for Baymax, rather than the sole creator of the fictional robot.]
Second, recovery matters as well as preventing failures. For many robots, recovering may be harder than avoiding the initial problem. Consider Boston Dynamics and Hyundai, Agility Robotics’ Digit and Amazon warehouses, or Figure and BMW. In my view, being unable to get back up after a fall may be a more serious operational problem than the fall alone. Quadrupeds have relatively mature recovery behaviors; humanoid recovery has been more difficult. These are different dimensions of reliability, discussed in the context of the collaborations known in 2024.
Discussing safety without cost or generalization is like discussing toxicity without dosage. What safety level do we need? It depends on the task: SpaceX rocket recovery, for example, has different requirements from other robotic systems. Generalization matters too. I agree that reliable systems are more plausible for simpler or repetitive tasks with limited generalization needs, such as outdoor locomotion within a defined scope or repeatedly serving coffee.
Between fully constrained tasks and open consumer use lies business-to-business deployment. A factory robot making cars needs some generalization, but not unlimited generality. In the example discussed here, a Figure robot at BMW might repeatedly carry out a battery-installation sequence with five steps. There is uncertainty, but it can be constrained, and the range of vehicle models—say seven or eight—is finite. I think that is a plausible near-term direction for pursuing safety. These specific task and model counts are the speaker’s illustrative example, not a verified description of BMW’s deployed workflow.
But deploying humanoids in ordinary homes to sweep, care for children, cook, and perform many other tasks at 99.999% safety is beyond what I think current technology can deliver. Several major problems remain unresolved, and even humans may struggle to meet that standard.
I do not believe in “one-shot safety,” meaning a system that is perfectly safe immediately upon deployment. Open environments need mechanisms for correction. A system may initially have problems, then improve through human-in-the-loop assistance, perhaps including labeling and intervention. I think asymptotically improving safety is the best direction we can currently hope for. That is a research aspiration, rather than a proven guarantee of convergence.
There is also a narrower, formally specified kind of safety. Researchers may define a rule that can be verified, such as avoiding certain collisions. We have tools for that, including reachability analysis and other techniques from control theory, subject to their assumptions.
Humanoid manipulation often lacks such a simple safety definition. For opening a door, for instance, what exactly counts as unsafe? We could list ten thousand different failure modes. Defining the whole problem is difficult.
To summarize: hardware safety matters; failure recovery matters and can be operationally critical; safety must be considered alongside the application’s generalization requirements. Much of our lab’s controller work addresses specifically defined, verifiable safety properties, such as collision avoidance. Many manipulation hazards are harder to express that way. [The original writes “Feel safe”; the surrounding discussion clearly concerns fallback and recovery rather than a separate technical method with that name.]
6. How to Make RL Great Again?
Guanya Shi: May I ask the other guests a few questions? This is a rare opportunity. I have long wondered whether innovation in RL algorithms is slowing down. Is there still room for fundamental algorithmic progress? These methods were not designed specifically for robots; they are very general. Why can PPO be used both for RLHF in language models and for robot learning? Should students still study basic RL algorithms and look for methods particularly well suited to robotics?
Zongqing Lu: Let me respond first. I agree that there has been less conspicuous progress in general RL algorithms since PPO and SAC, from our perspective at this panel. At its core, RL is often formulated as an MDP, and we are choosing a method to solve that mathematical problem. The formulation is therefore quite general. This does not mean that no new RL algorithms or subfields have developed.
PPO is a useful practical method for that problem. It is commonly implemented with a policy-gradient actor and a value-function critic in an actor–critic framework.
Can we design better RL methods for robotics? I think we can start from robot learning’s particular characteristics. Its constraints differ from those in our usual algorithmic benchmarks. Incorporating those constraints should lead to different designs. Jianlan’s real-world RL, for instance, combines offline data and online learning in ways tailored to physical robots.
Finding an exceptionally strong universal RL algorithm is difficult. But there is still considerable room to tailor methods to robot learning.
Jianlan Luo: Around 2017–2018, I already felt that we should stop focusing only on MuJoCo benchmarks, because that did not necessarily solve practical problems. Why did progress seem to slow after PPO and SAC? We should ask what motivated them. They addressed problems such as getting a simulated character to run or a humanoid to stand. Once those benchmark problems were handled much better, the incentive for another major algorithmic change diminished. This is my interpretation, rather than a literal claim that the benchmarks were completely solved or that no subsequent algorithms exist.
For the next advance, we need to identify the fundamental problem we want to solve. A new benchmark or a major challenge should be motivated by a genuinely important problem, rather than repeating the same cycle.
Where is RL’s next frontier? For locomotion, I agree that control is part of the solution. Robots already have useful models—we could not engineer them without understanding something about their mechanics. We should use that knowledge through model-based control and ask how to combine it with RL for particular metrics. In an inexpensive simulator, sample efficiency may be less important; NVIDIA, which sells GPUs, may not worry as much about that either. [The last remark is the speaker’s aside, not a verified statement of NVIDIA’s position. A useful engineering model need not perfectly capture deployment dynamics.]
Performance and safety, and how to combine them, are interesting open questions. Even PPO with teacher–student learning still faces a sim-to-real gap. How can control make better use of modeled physical transitions and motion to obtain the behavior we want? The transcript’s final wording is imprecise, but its question is about exploiting physical structure rather than merely sampling more experience.
For manipulation, current methods may need one or two hours of task-specific training, and I think that can be shortened. Doing so may require general visual capabilities, which brings us back to dataset design, task design, and how to use RL. [The source calls this “zero-shot learning” while also describing task training. One or two hours of adaptation is not zero-shot deployment in the usual sense.]
Looking farther ahead, imagine a million robots deployed five years from now and sending back large quantities of heterogeneous data with partially hidden information and broad variation. How should we pretrain? RL pretraining differs from behavior-cloning pretraining. BC maximizes the likelihood of demonstrated actions in a dataset, while RL optimizes expected return, often through learned value functions. They may look similar at a glance, but their objectives differ; RL can seek behavior beyond what is directly copied from the dataset. The million-robot scenario is a forecast from the 2024 discussion, not a current deployment count.
Could we use physical modeling to make RL more scalable? With massive heterogeneous datasets, how do we pretrain and adapt quickly? I mean genuinely large-scale conditions. That may be an important next focus.
I expect this to be part of my own research plans for the next five years. But when developing open-source algorithms, we cannot test every candidate on ten thousand robots. The cost and operational impact would be far too high.
My overall point is that PPO and SAC addressed many of the problems that motivated them; now we need important new problems. RL provides a principled optimization framework. We should improve it rather than abandon it because of current limitations. It is not only about copying data, predicting outcomes, or extracting features. If we want very high performance—99.99%, or the aspiration of 100%—data alone is not enough; optimization matters too. Future solutions should include that element. Those percentages express the goal, not an established general reliability guarantee.
Papers and projects mentioned in the discussion:
Guided Policy Search
https://graphics.stanford.edu/projects/gpspaper/gps_full.pdf
DIAL-MPC
https://lecar-lab.github.io/dial-mpc/
CAJun
https://yxyang.github.io/cajun/
Jianlan Luo’s homepage: https://people.eecs.berkeley.edu/~jianlanluo/
Guanya Shi’s homepage: https://www.gshi.me/
Qiuguo Zhu’s homepage: https://person.zju.edu.cn/0011353/657053.html
Zongqing Lu’s homepage: https://www.ai.pku.edu.cn/info/1139/2783.htm
BC (behavior cloning)