When people hear robotics plus reinforcement learning, many probably think first of training robot controllers in simulation, especially for locomotion. For manipulation, the sim-to-real gap has made it difficult for simulation-based RL to achieve strong results on physical robots.
Before preparing this article, I had assumed that reinforcement learning for robotics happened only inside a simulator, with the resulting policy transferred to a physical robot through sim-to-real deployment. Most of the RL manipulation papers cited in my earlier review, Guiding Robot Hands Toward Human Hands: A Review of Chinese Researchers' Work on Dexterous Manipulation, followed that route. Some demonstrated only simulated results, while very few showed deployment on actual hardware.
I suspect many fellow non-specialists shared my assumption that a brute-force, try-until-it-works algorithm such as RL could operate only in GPU-powered simulation. As a longtime embodied-intelligence blogger, I can confidently say that today's article offers a clear introduction to training RL directly on real robots, without using simulation.
Our guest is Jianlan Luo, a researcher who has pursued real-robot reinforcement learning for seven or eight years and is a co-first author of SERL and the first author of HIL-SERL, two representative recent projects. At the time of this November 2024 interview, he was a postdoctoral researcher at the Berkeley Artificial Intelligence Research laboratory, or BAIR, working with Professor Sergey Levine. Before returning to academia in 2022, he spent two years as a researcher at Google X working with Professor Stefan Schaal. He received his master's and PhD degrees from UC Berkeley in 2020 and also worked at DeepMind and Everyday Robots. Editorial note: reported success rates and training times below refer to particular tasks and finite evaluations, not a guarantee of flawless performance under all conditions. The interview's forecasts and historical assessments are retained as the speaker's perspective at that time. Biographical clarification: the current primary biography lists Google X employment during 2020–2023 and a Berkeley PhD in 2020. The 2022 return date and master’s-degree timing above are retained from the original introduction, whose timeline differs from that biography.
I deeply admire Jianlan's choice of research direction and his persistence. Some people follow a road few others take, alone but determined, until more people recognize it and they become its leaders. I hope the conversation below helps readers understand this path in embodied intelligence and some of the story behind it.
Original article outline
Contents
↓
1. Introduction
2. What Is Reinforcement Learning on Real Robots?
3. The Development of RL in Robotics
4. How Does Real-Robot RL Avoid Brute-Force Search?
5. Is Simulation Used at All?
6. What Does the Training Process Look Like?
7. Generalization
8. Why Persist with Real-Robot RL?
9. Which Is More Popular in North America: IL or RL?
10. What Has Changed Since 2018?
11. The Limitations of HIL-SERL
12. RL and Simulation
13. Is HIL-SERL Open Source?
Some Personal Reflections
1. Introduction
I began my mechanical-engineering PhD at Berkeley in 2015, initially studying robot control. Robot learning was beginning to take off, particularly around 2015–2016. I gradually shifted toward it, working mainly with researchers such as Pieter Abbeel and Sergey Levine. My research focused on reinforcement learning, especially applying it to physical robots. Editorial clarification: this refers to the emerging deep-learning-driven wave of robot learning; the broader field predates 2015.
From 2019 onward, I worked in several robotics groups at Google, including Google X and DeepMind, collaborating with Stefan Schaal. He had been a professor at USC and a founding director at the Max Planck Institute for Intelligent Systems. Before 2015, he was a towering figure in robot learning. At that time, the work I am describing used no visual input and focused more on learning robotic dynamical systems. In my account, the entire European school, including Jan Peters and Aude Billard, developed within his academic tradition. Editorial clarification: the last statement is the interviewee’s broad characterization of an intellectual lineage, rather than a verified genealogy of every European researcher.
In 2022, I returned to Sergey's group as a postdoc, focusing on sample-efficient, high-performance reinforcement learning for real-robot manipulation and applications involving large models.
2. What Is Reinforcement Learning on Real Robots?
Simulation-based RL has been particularly successful in locomotion. Such work is often entirely state-based, or learns a latent representation and then aligns simulation with reality through sim-to-real or real-to-sim-to-real methods.
I focus on complex, important real-world situations, such as contact with objects and handling deformable objects. These require visual input, including RGB images and camera perception. Policies trained in simulation often perform poorly in reality, with a large gap between the two.
Using RL for manipulation is relatively uncommon now, but many people explored it around 2017–2018. Problems such as low sample efficiency made training difficult, and many abandoned the direction. I have continued because I believe the problem is important.
Broadly, I train RL algorithms directly on real robots with visual inputs. For particular tasks where imitation learning does not reach a 100% success rate—especially those involving contact and physical interaction—we can train a model in one or two hours that achieves 100% in the evaluation. This is task-specific, not a claim of general-purpose capability. But the policies are very robust to disturbances, something difficult to achieve through simulation alone.
3. The Development of RL in Robotics
RL has roots in the 1980s and 1990s, when the algorithms were relatively simple. Early researchers, including Stefan Schaal, tried applying it to motion control, combining ideas such as dynamic movement primitives, or DMPs, with policy-gradient methods. This work did not use visual input.
Around 2017–2018, Google began applying RL to real robots, mainly for grasping. But grasping also had other effective solutions, such as approaches using labeled data, and that line of work did not continue developing as much.
Around 2019–2020, two important papers from ETH showed how to train RL policies in simulation with domain randomization. They demonstrated that a simulator did not have to be perfectly realistic to produce a reasonable real-world policy. This work focused mainly on locomotion and helped establish today's common approach: train in simulation, then perform teacher–student distillation.
I want to emphasize that model-based control, including MPC, has been very successful in locomotion. The entry barrier is high, though. You may spend five years doing a PhD and studying dynamics and control before you can work with it effectively, and subsequent performance optimization requires strong engineering support. Simulation and RL lower that barrier and offer a more accessible tool.
RL locomotion performance can reach 90% or 95% fairly easily and then plateau. If it fails on a particular corner case, you may not know what to change beyond adding more data and optimizing the policy, whether with TRPO or PPO. MPC is difficult to tune and get working, but you know which parameters to adjust because each has a physical meaning. ETH's more recent work in 2023 and 2024 has combined MPC with reinforcement learning.
RL for manipulation has not seen the same degree of progress. The main difficulty is external uncertainty rather than uncertainty in the robot model itself. In locomotion, model errors might cause a robot dog to place a foot incorrectly, but those issues are relatively bounded. In manipulation, the arm's commanded movement is comparatively predictable: tell it to move left and it moves left. The concern is the outside world, much as in autonomous driving: is there a truck ahead, or another obstacle?
For example, picking up a USB plug and inserting it involves contact forces and complex rigid-body contact. Real-world visual information is continuous and environmental variation is effectively unbounded. That is the difficult part, and it is why real-world training matters. There are successful deep-learning simulation systems, but a policy learned from a simulator cannot escape the limitations of that simulator. Ultimately, the simulator bounds what the policy can learn.
Building a simulation identical to reality is extremely difficult. You need to interact with the real world to obtain its most authentic feedback. RLHF means reinforcement learning from human feedback. By analogy, we could call this RLPF—reinforcement learning from physical feedback—because the physical world supplies rich feedback. Solving a relatively simple problem in simulation does not prove the same approach will solve a much harder one. We should address the difficult problem itself rather than remain trapped at a saddle point, constructing a proxy for it and taking a detour. Over the long term, that may cost us the ability to find the global optimum.
Real-world training is certainly challenging. There are many factors to consider, sample efficiency must be very high and the whole system must work together. But I see it as the difficult and correct thing to do, because I believe it is the only effective path to a 100% success rate. Editorial clarification: this is the interviewee’s view; the reported 100% results are measured on specific tasks and finite trials, not a universal guarantee.
Before our recent work, including HIL-SERL, many people had largely given up on RL for real-world manipulation. They thought the policies performed poorly, training took too long and the process was too complicated, so they stopped pursuing it.
In January of the interview year, we released SERL. In the interview, I described tasks learning a policy with a 100% evaluation success rate in roughly a dozen to twenty minutes. The framework attracted attention from Boston Dynamics, Stanford, MIT and other universities and companies in China and abroad, and they began training RL on real robots again. Some European universities also tried it and shared training videos with us. Editorial clarification: the SERL project page reports training times of twenty-five to fifty minutes for its example tasks; times vary by task and setup.
4. How Does Real-Robot RL Avoid Brute-Force Search?
PPO is commonly used in simulation. It is on-policy: after using a batch for an update, you collect fresh data rather than reusing old experience indefinitely. That is less problematic in simulation, where sample efficiency is not constrained in the same way.
We need to reuse data much more efficiently. Human demonstrations are a major reason our method is sample-efficient. A person typically provides ten to twenty reasonable demonstrations as an initial solution. RL then searches around that region until it reaches a 100% evaluation success rate.
We use corrective demonstrations and online human feedback. The method does not require a complicated reward function. Think of AlphaGo: however many moves it makes, the final outcome is winning or losing. A zero-or-one criterion avoids imposing biases on the intermediate reasoning process and can work better.
Designing rewards is indeed difficult. We cannot simply create one function that handles every situation, and a hand-designed reward can differ substantially from the real physical objective. Our reward is sparse: whether the task is long or short, completing it gives one and failing to complete it gives zero.
The system is scalable and works on multiple tasks. Careful design choices integrate several systems and algorithms into an efficient whole. RL here is not merely brute-force search; it explores around an initial solution. Control theory has a related idea: use a reference trajectory to design a feedback controller around it, obtaining robustness.
With visual information, the improvement in sample efficiency is especially significant. On complex tasks such as precise PCB assembly or automotive assembly, our system can reach a 100% evaluation success rate in one or two hours in the real world, demonstrating efficiency and reliability in those settings.
5. Is Simulation Used at All?
We currently do not use simulation. We learn complex tasks directly in reality, often involving precise manipulation and multiple contacts—situations in which imitation learning can struggle. Our system reaches a 100% evaluation success rate in one or two hours. In this study, we completed twelve tasks, including dynamic manipulation and contact with flexible objects, such as installing a car-dashboard component.
Training RL directly on a real robot.
Dynamic manipulation requires real-time adjustment from feedback rather than executing a fixed sequence. We continually adapt to the information captured by the camera, rather than memorizing an action and running it open-loop. If execution deviates from expectation, we keep correcting it. Interaction with the actual environment is central to learning robot behavior.
Although we are not currently using simulation, I think pretraining could reduce the training time. Early in each task, the robot is often learning basic manipulation capabilities. Pretraining on readily available datasets could shorten that stage significantly. For example, through training we can bring the time needed for complex tasks such as installing a car’s flywheel and belt down to twenty minutes, although current robot capabilities in those areas remain limited. Editorial clarification: the twenty-minute example is the interviewee’s statement; training time depends on the task and setup, and should not be read as a benchmark for every automotive assembly task.
6. What Does the Training Process Look Like?
Real-robot RL training for automotive assembly.
This is an automotive-assembly task: two arms lift a component and install it on a dashboard. It is precise work involving several interfaces and carefully controlled operations. The robot cannot perform it initially; training begins from scratch for that task. Early behavior includes random exploration, almost like arbitrary interaction with the environment. We provide demonstrations, and people continually correct and guide the robot during the process.
Over time, it learns the basic operation. The video is uncut from beginning to end, showing the progression from early exploration to successful completion. After a period such as two hours, the success rate approaches 100% and mistakes become rare.
These behaviors are learned through interaction with reality. The robot learns how to grasp and put down a USB plug, for example, or pull a belt. Exploration produces these behaviors, while continued training and feedback improve them. RL's connection to dynamic programming helps it attain higher performance.
7. Generalization
This paper does not address generalization from one task to another. SERL and HIL-SERL primarily solve the problem of making RL for real-world manipulation work at all. For multi-task learning, earlier work has shown that single-task RL models can rapidly generate data, which can then be distilled into a larger model supporting multiple tasks. That offers a route toward broader adaptability.
We aim to provide a reliable, usable foundational tool that researchers can build on for more complex work. Many subsequent projects can develop from it, with a clearer understanding of how to implement the necessary components.
8. Why Persist with Real-Robot RL?
One reason people abandon this work is that it requires deep understanding of both robotic systems and RL algorithms. You need a high level in both, and that is not easy. Frankly, there is no obvious shortcut: you learn through mistakes and accumulate experience step by step. Fast results are difficult, so people may give up. I also think it is risky and may offer limited short-term rewards. Over the long term, however, RL will clearly be a major part of the solution. At the time of this interview, I predicted that startups relying heavily on imitation learning would, after experiencing its limitations, begin using RL the following year to improve success rates, cycle times and robustness.
Why have I persisted? Around 2019, while researching at Google, we used real-robot RL for an insertion-and-removal task. It achieved a 100% success rate across thirteen thousand tests: every test succeeded. It even outperformed the leading robot system integrator we had paid to implement the task. We genuinely saw RL's potential.
Much research on RL for manipulation is difficult to reproduce in practice. People struggle to make RL work, particularly when reproducing recent results. In June of the year before this interview, Sergey and I discussed whether we could give the community a mature, reliable tool to reproduce real-robot manipulation RL. That was the original motivation for SERL.
9. Which Is More Popular in North America: IL or RL?
Imitation learning can produce results quickly: collect some robot data and run experiments, so people are willing to pursue it. But improving it to much higher precision may require more than additional collection. Ultimately, imitation learning and RL should complement each other. If you want 99% accuracy, I believe RL is indispensable. Rich Sutton's Bitter Lesson argues that learning and search are methods that can scale without an inherent ceiling. Imitation learning can extract features from data, but without search or optimization through RL, it cannot readily go beyond the data's limits to solve new problems in new ways.
In North America, Sergey has consistently emphasized RL in real environments. SERL and HIL-SERL make a more convincing case for its necessity. They show its potential and why it matters in this process. Throughout my eight or nine years in North America, I have stayed with this direction. My wish is simple: for the technology to be widely used and receive real appreciation. I do not want exaggerated claims in a paper. I want the work to solve problems effectively and provide practical help.
10. What Has Changed Since 2018?
The core algorithms have not changed dramatically. Most progress comes from integration and optimization of the entire system. HIL-SERL contributes an end-to-end system and solution, rather than a single algorithm. Much earlier RL research evaluated performance in simulation and remained disconnected from reality. HIL-SERL covers everything from low-level controllers to high-level algorithms. It includes impedance and force-feedback control as well as algorithmic design. It is a complex integrated system whose components work closely together.
The design brings multiple factors together into a better whole, rather than optimizing one component alone. Reward-function design is one example: more complex reward mechanisms can help a model learn faster. All the factors work together to improve performance. That is the point I keep emphasizing. Editorial clarification: this general example in the transcript should be distinguished from HIL-SERL’s actual reward setup. The project describes a learned binary success classifier providing a sparse reward, rather than a dense hand-shaped reward.
Many newcomers approach robotics by collecting data and training or experimenting with a couple of robots or devices. Experienced roboticists would agree with me that a robot is a complex system: improving one component alone is not enough. Replacing a car's engine or computer does not automatically make the whole car faster. Hardware, software and control all need to work properly together for a robot to reach its potential. Working on one part while neglecting integration will strongly limit the result.
11. The Limitations of HIL-SERL
RL sample efficiency depends on several factors, particularly the state space, action space and task horizon. Greater complexity requires more samples. Short tasks lasting a dozen or so seconds are relatively easy to address with RL. Tasks lasting several minutes may need other tools or methods to help.
RL has advantages for complex contact, flexible objects and precise manipulation, but also limitations. A long task combining seven or eight subtasks—opening a door, closing curtains, pouring water and so on—may be too difficult to learn all at once. Splitting it into separate subtasks lets RL learn and execute more effectively.
12. RL and Simulation
People cannot learn many skills, such as tennis, merely by watching YouTube. They need to try, adjust their movements from environmental feedback and develop muscle memory. That is a bodily experience, rather than simply an intellectual understanding of physical laws. Reproducing it through simulation is difficult. Rather than spending ten billion to build a virtual world model, I think collecting real data can be faster and more effective, without having to solve a sim-to-real gap.
Even autonomous-driving companies with enormous, scalable simulators, such as Waymo or Tesla, still rely mainly on real data when training their final models. Tesla has accumulated so much driving data that the question is how to use it. I heard the previous year that it was running into limits on how much newly generated data its data centers could hold. There is almost a saying now that whoever has the most hard drives will win. These are observations and anecdotes from the interview, rather than independently verified current storage figures.
The interview also offers a historical analogy from computer vision in the 1990s: Adobe tried to solve image recognition with synthetic images, investing substantial resources in generation, but those images did not ultimately support practical visual models in the way hoped. Real-image datasets such as ImageNet later enabled major progress. Simulation has uses, but when thinking about the robot-data pyramid, I expect real-world data to make up the larger share. Editorial note: the transcript does not identify a specific Adobe study, so this anecdote is retained as the interviewee's recollection rather than a verified account of a named project.
A major argument for simulation today is that real robot data is difficult to obtain, whereas simulation can generate ten billion samples rapidly. But few would dispute the usefulness of real data if it were available. That is not the fundamental issue. Imagine a hundred million robots deployed in reality ten years from now, continually sharing physical experience. Looking back, many of today's difficulties may disappear and many arguments may become irrelevant. Current data volumes and deployment numbers are too small for firm scientific conclusions, which is why so many approaches compete. More immediately, whoever first deploys a thousand humanoids in factories may receive enough data around the clock to establish new paradigms and scientific findings. We can begin in these partially constrained spaces. Once we understand the problems more deeply, the resulting methods can extend to unconstrained environments.
13. Is HIL-SERL Open Source?
We are open-sourcing the entire project and deliberately chose the MIT license so that more people can use it. Everything, from robot control to RL, is available end to end. As for ease of use, even some undergraduates can set it up in a few days and extend it. We did not undertake this work simply to publish a paper. Publication by itself is not the meaningful outcome for me. License clarification: the interview names the MIT license, but the currently available official HIL-SERL repository identifies its license as Apache-2.0. The interview’s original licensing statement is retained as a historical statement, rather than presented as the current repository license.
We spent a great deal of time redesigning and validating the codebase so it runs smoothly in different environments. Repeated testing helps ensure people can take the tool, connect it to their system and use it successfully.
Some Personal Reflections
I have been in venture capital for seven or eight years. Some investments have done very well, but I have not backed a super-unicorn. Sometimes I wonder how to invest in the next Pinduoduo, and I still do not have an answer. I very much enjoy this profession, but achieving that ultimate sense of accomplishment is genuinely difficult. More often, there are regrets and missed opportunities. Some projects I deeply admired could not be pushed through; others, now enormously popular, were early-stage opportunities I missed by a single last-minute decision. Every investor experiences this. How do I make peace with myself? By turning each lesson into a basis for responding better next time. Writing this blog is another way. When effort does not directly turn into an outcome, I can reflect, build myself up and improve gradually. You have to endure the quiet years to be ready for the flourishing ones.
References
SERL
HIL-SERL