In earlier episodes of Robot Data, we introduced representative papers on robot learning from human videos, teleoperation and motion-capture data. These approaches use human expertise as their reference. Another major branch of robot learning combines simulation with reinforcement learning. Its effectiveness in training quadruped and biped locomotion is well established, but manipulation presents substantial bottlenecks: building simulation environments and reward functions takes considerable time and expertise, RL search can be slow, and the sim-to-real gap remains. Eureka and DrEureka use large language models to automate reward design and, in DrEureka, domain randomization, helping robots acquire skills in simulation and transfer them to reality. Editorial clarification: the original broadly describes this as automatically designing the simulation environment. Both methods start with supplied task-simulation code; they do not create the entire simulator or deployment system from nothing.
We were delighted to invite Yecheng Jason Ma, first author of Eureka and co-first author of DrEureka, for a detailed discussion of simulation, reinforcement learning and the potential of LLMs in this combination. I hope this extended interview helps you understand both projects.
The authors of Eureka and DrEureka include researchers from Penn's GRASP Laboratory, NVIDIA and other institutions. Eureka: Human-Level Reward Design via Coding Large Language Models is by Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi 'Jim' Fan and Anima Anandkumar; Jason Ma is its first author. DrEureka: Language Model Guided Sim-To-Real Transfer is by Yecheng Jason Ma*, William Liang*, Hungju Wang, Sam Wang, Yuke Zhu, Linxi 'Jim' Fan, Osbert Bastani and Dinesh Jayaraman. Jason Ma and William Liang contributed equally as first authors. The conversation retains its July 2024 timeframe.
Original article outline
Contents
↓
1. Eureka's core idea
2. DrEureka's core idea
3. How Eureka works
Initial reward-function generation
Environment as context
Rapid reward evaluation through massively parallel RL
Reward reflection
Reward functions that outperform human baselines
Novel reward functions
4. How DrEureka works
Reward-aware physics prior
5. Questions and discussion
What Eureka and DrEureka could improve
Comparison with other LLM-based robotics projects
Why are imitation learning and real-robot manipulation data more prominent?
Can Eureka address the challenges of simulation and RL?
Applications of Eureka
Rendering and the sim-to-real gap
How is domain randomization usually done?
Is Eureka limited to tasks such as locomotion?
How do simulation and RL handle long-horizon tasks?
How is data generated in simulation?
My research directions
A brief introduction to VIP
6. Some reflections
1. Eureka's core idea
As we know, data is one of the most important problems in robot learning. We can collect real-world data through teleoperation or collect data in simulation. In principle, we can scale up simulation data and then transfer the resulting capabilities to reality. But an important, underestimated issue is that a person has to design the simulation environment and somehow validate whether it is useful—particularly whether what we learn will transfer to the real world. Otherwise, even if we collect a great deal of simulation data, learning methods will not be very effective if that data is unreliable, whatever training strategy we choose.
How can we effectively design a simulation environment for a new task or robot? Traditionally, people have had to build these environments, a slow process requiring considerable expertise. In Eureka, we asked whether LLMs could automate the design work so that we could learn new skills quickly in simulation and transfer them to reality. That is the innovation we wanted to explore. More precisely, Eureka automates the reward-design component within an existing simulation environment; DrEureka addresses the additional sim-to-real design problem.
Simulation offers abundant data: we can run it for a long time and explore different states. Rather than having a person provide demonstrations using a keyboard or game controller, a popular approach is to run reinforcement learning, allowing the policy to collect its own data, observe feedback and improve. That is why RL is a common robot-learning method in simulation. But RL needs an environment, and here is the problem: you must design a reward function. This mathematical function tells the policy which states and actions help accomplish the task and which do not. People have traditionally designed it, but doing so is slow and unintuitive.
When you read RL papers, you may find that reward functions receive little discussion in the main text—perhaps just a brief explanation or a mention in an appendix. In practice, reward design is crucial. With a well-designed reward, RL can often find a solution. Otherwise, the results may be poor.
Eureka uses an LLM to write reward functions automatically for simulated robot tasks. A reward function is executable code, just like the simulator's other code, and LLMs such as GPT-4 are very good at writing code. We therefore use an LLM to produce the reward code. But that is not all: we also ask it to improve that code.
Concretely, the LLM writes several candidate reward functions. We train policies with these rewards in the simulator and construct feedback showing which work better and which work worse. Using this feedback and the policies' actual performance, we guide the LLM toward better reward functions. Because LLMs follow instructions well, they can use feedback to improve their outputs. This is an iterative process of self-improvement through simulation feedback. In the paper, we call it an evolutionary algorithm: we let the LLM evolve the rewards. You can also view it as a form of learning from AI feedback, or RLAIF (Reinforcement Learning from AI Feedback). The ultimate goal is a policy that solves the task. Along the way, both the policy and the environment's reward function are updated to find a better policy. That is Eureka's high-level mechanism. Here, the RLAIF comparison is an analogy for the feedback loop; the underlying LLM's weights are not updated.
Using the same algorithm without task-specific prompt engineering, we can address 29 tasks across ten different robot morphologies. We also demonstrate novel skills, such as a simulated robot hand spinning a pen for the first time. Combining Eureka's reward generation with established RL algorithms, we can teach the hand several ways of spinning the pen, with very good results. These demonstrations use curriculum learning; '29 tasks' describes the evaluation suite, not a claim of perfect success on every task.
Eureka: pen spinning
Eureka: cube rotation
2. DrEureka's core idea
Eureka gave us an automated method for designing rewards in simulation. In principle, a language model that generates rewards for desired behavior could support a very broad range of behaviors. Eureka received encouraging feedback. I presented it at several NeurIPS 2023 workshops, and it was accepted at ICLR 2024. But when we first showed it, the most frequent question was: does this work on real robots? Everything you show is simulated—can it happen in reality? To answer that key question, we went further and developed DrEureka, which I also call 'Doctor Eureka.' Receiving a doctorate means graduating, so Eureka's transition from simulation to reality is like its graduation ceremony. For sim-to-real RL, designing rewards is not enough; you also have to make the transfer work.
Simulation and the real world usually disagree in many ways, especially in their physical models. A simulator cannot perfectly reproduce gravity, friction and other real-world properties. A popular solution is domain randomization: randomize physical parameters in simulation so that the policy learns across a broader distribution of physical conditions. A policy trained over that broader distribution may be more robust and more likely to deploy successfully in reality. Some form of domain randomization is therefore used very widely in sim-to-real reinforcement learning.
But domain randomization still has a problem: people have to design its parameters manually. Consider teaching a quadruped to walk. Intuition and common sense tell us that friction, gravity and the robot's center of mass are important physical properties to randomize. Yet designing that randomization requires substantial domain knowledge. More importantly, a new task may require us to redesign the whole process.
In DrEureka, we automate domain-randomization design as well as reward writing. First, we provide the LLM with a list of physical parameters that can be randomized. For the particular task, the model chooses which parameters to randomize and how far to vary them from their default simulation values. We found that language models are good at selecting hyperparameters. Other research has shown that they can generate useful hyperparameter samples—for example, suggesting numerical values in linear-regression tasks. More excitingly, our research shows that they can generate useful domain-randomization parameters zero-shot, without additional model training. This greatly simplifies the design process, reduces dependence on human expertise and opens possibilities for automating RL tasks.
Once reward functions and domain-randomization settings can be chosen automatically, we can deploy the policy that performs best among the candidate reward and randomization configurations. Surprisingly, we found that direct real-world deployment could work. This gives us an automated design pipeline from simulation toward reality, with the LLM handling the iterative reward and randomization choices. Editorial clarification: 'fully automated' in the original refers to those design and training stages. Task specification, simulator setup, the evaluation measure, robot hardware and safe physical deployment are still supplied by people.
This progress shows the potential of LLMs to automate reinforcement-learning design, from reward functions to domain-randomization choices that support deployment. Removing repeated expert tuning from these stages can improve efficiency and open a path toward more complex real-world problems. It does not mean that all robot engineering or physical deployment is performed without people.
DrEureka demonstrates both in-hand cube rotation and forward walking. But I think the really interesting question is this: given that language models cannot automate everything, what if we use them to teach a robot a new task? That is what we did by having a quadruped walk on a yoga ball, a task that had not previously been demonstrated in this setting. We placed a simulated quadruped on a ball and ran DrEureka, asking it to walk on the ball and remain there for as long as possible. Without human supervision of the design iterations, it produced reward functions and domain-randomization parameters. Real-world deployment also worked fairly well, although a person held a tether. This was mainly a safety precaution to avoid damaging the robot. Even without the tether, it could stay on the ball for roughly five to ten seconds or longer across different terrain conditions.
DrEureka demonstration

Cube rotation
Quadruped walking on different terrain
An important point is that our Isaac Gym setup uses rigid-body simulation and cannot model the deformable surface in this task. The simulated ball is rigid and does not deform, whereas a real yoga ball is clearly deformable and elastic. Because we could not make the simulator as close to reality as we would have liked, the algorithm had to be robust to simulation inaccuracies. A policy that does not accommodate those inaccuracies may fail in reality. Yet the LLM could generate an effective randomization distribution that made transfer possible. This illustrates the potential for LLMs to automate new simulation tasks and support real-world transfer. DrEureka is a first step toward that goal. The rigid-body limitation describes the setup used here, rather than every physics capability available across NVIDIA's simulation products.
3. How Eureka works
However you set up a simulation environment or reward function, the result is code. Most of these programs are written in Python, and LLMs such as GPT-4 are good at writing Python. With appropriate prompting, they can produce that code. But we need the code to be useful and have to control hallucinations: an LLM might generate poor, unhelpful environment or reward code. Feedback is therefore important. We need a way to determine whether a reward is good or bad. We can train RL policies, observe how they behave in simulation, summarize their behavior and feed that information back to the LLM so that it generates better rewards.
Eureka's framework
There is an important qualification: the dense reward function we generate guides reinforcement learning, but we also need a true evaluation measure independent of that dense reward. Otherwise, we will have policies doing different things and no way to tell which is better. Eureka assumes access to a function that measures task success, letting us determine, for example, whether a policy succeeds 80% or 50% of the time. Typically, this is an indicator function evaluated over attempts.
For a door-opening task, the success function simply tells us whether the door is open: yes or no. A dense reward must capture intermediate steps, such as approaching the handle, turning it and opening the door. These rewards are difficult for people to write, while the success checker is usually much easier. But the checker alone offers a signal too sparse to be useful for learning in this setting. Eureka generates a dense reward, receives feedback and keeps improving it.
Alongside the success function, Eureka summarizes policy statistics for the reward function it generates. Each reward comprises several components: A, B and C, for example. A might encourage approaching the handle, B might encourage turning it, and so on. Summarizing the individual components gives detailed feedback about which terms are not being optimized properly. A policy might have a 0% success rate because it never approaches the door handle.
If we summarize these components in detail and feed the results back to the LLM, the next iteration can identify terms to change or new terms to add, producing a better dense reward and improving the policy. Eureka's LLM feedback thus combines the true success measure with statistics for individual reward components. Together, these signals let the model generate improved rewards autonomously and revise existing ones.
Initial reward-function generation
Zero-shot reward generation is important because, when data is scarce, we still need a reward to get started. Designing a reward for a new task is difficult, and finding a suitable starting point is often especially hard. An LLM can generate one directly and then initiate the self-improvement process. This is one of the key capabilities we need from the LLM in Eureka: it can start the reward-design process without manual reward writing.
DrEureka's framework
Eureka can also improve an initial reward written by a person, and the eventual result can be better than using Eureka from scratch. People may find reward design difficult, but they have experience recognizing which environment variables are useful for writing a reward. That is a valuable signal for the LLM, helping it write progressively better rewards.
If the LLM writes everything from scratch, it must decide not only how to design the reward but also which inputs that reward needs. This is especially difficult when the environment is poorly described and the state variables' meanings are unclear. A human-written initial reward simplifies the task: instead of generating an initial candidate, the LLM can improve it step by step. If an initial candidate is very poor, even an improved version may be unsatisfactory. A search process benefits from a good starting candidate so that self-improvement has something useful to work with.
Environment as context
A reward function is ultimately code that must run inside the original environment. To generate executable code, the LLM needs to know which variables are available and how the environment works. The environment itself is already code. Assuming that code is available, we can put it into the LLM's context. Modern LLMs are good at identifying useful variables and relevant lines in the source. The source supplies the information needed to recognize state and action variables, which the model can then use to write the reward.
This automates the explanation: here is the environment code for which you need to write a reward. Without that source as context, we would have to write a prompt describing, for example, a humanoid's head, torso, two hands, two legs and the corresponding variables before asking for a reward. We show that alternative in the appendix, but it requires a person to compose the description. The simpler method is to copy the already available environment code into context. It is a lazy person's prompting method, but we found that it works well.
Today's large models are strong at understanding language and code. Given the original environment source, the model can identify the variables by itself, without a person manually explaining every detail. We need not separately describe the humanoid's limbs, arms, hands and head. The model can identify the relevant information by reading the code.
This makes the process much simpler and reduces manual intervention. It is why we can use the same prompt across ten robots and 29 tasks. Each environment has different source code, but we need not create an individual prompt for each one. This offers a substantial scaling advantage: the same prompting approach applies directly across multiple robots and tasks.
Rapid reward evaluation through massively parallel RL
Running reinforcement learning in simulation generally takes considerable time. Before NVIDIA introduced Isaac Gym, it was relatively slow. Efficient tools such as Isaac Gym can now make evaluating a reward function take only one or two hours, allowing five or six iterations in a day. These are example timings for the setup discussed here.
Without GPU-parallel simulation, iteration and evaluation become much slower. Training a policy might take ten hours or a whole day, so five iterations could take three or four days. This greatly slows algorithm development and can make the wait unacceptable to an end user. If a good reward or policy takes three or four days, they may lose interest. As you mentioned at the start, they could instead spend those days collecting more real-world data. Editorial clarification: the Chinese transcript says 'if using parallel simulation,' contradicting the preceding paragraph and Eureka's documented use of GPU acceleration. 'Without' restores the intended contrast.
Reward reflection
This is the feedback mechanism I described earlier. The success function helps the LLM assess the quality of each reward. We turn that feedback into text and provide it to the model, which can then revise the reward and generate a better one.
Eureka's results: reward functions that outperform human baselines
We evaluated 29 open-source tasks written by RL researchers, rather than tasks we created ourselves. These include the original NVIDIA Isaac Gym environments, whose rewards were written by RL researchers and NVIDIA staff. Those rewards are reasonably good. Researchers' own benchmark rewards should be good: otherwise, their benchmark would not work and they could not demonstrate the result in a paper. Editorial clarification: the interview says 'ten original environments.' The project describes ten robot morphologies and 29 tasks, consisting of nine original Isaac Gym environments and twenty tasks from the bidexterous-manipulation benchmark.
Again, the 29 tasks in our paper are open source and were written by RL researchers, not by us. We use NVIDIA Isaac Gym environments with rewards created by researchers and the NVIDIA team. We regard these benchmark rewards as good: if they were poor, nobody would use the benchmark. They represent a human-expert baseline. Comparing our method with those expert-written rewards helps demonstrate Eureka's effectiveness. The original repeats the figure of ten Isaac Gym environments here; the corrected nine-environment, ten-morphology distinction above applies to this passage too.
Choosing the comparison baseline was difficult. At the time, we did not have another closely comparable algorithm that automatically generated rewards in the way Eureka does. Comparing against existing human-written rewards was more meaningful, especially on tasks people find difficult. Such a comparison better demonstrates LLMs' potential for automation, which we considered valuable.
Eureka's results: novel reward functions
There are two ways to understand how an LLM can generate rewards people might not think of. First, we compared human-written and Eureka-generated reward functions through correlation analysis. Treating them as mathematical functions lets us assess their similarity. Sometimes the correlation is low or even negative, yet the generated reward produces a better-performing policy than the already carefully designed human reward. Low correlation by itself does not establish better performance; the policy evaluation provides that evidence.
We also found mathematical expressions in Eureka rewards that people rarely use. One reward, for example, used a nested exponential expression, described in the interview as 'e raised to e raised to negative distance' and as a double negative exponential. This is unusual compared with the simple L2-distance-based terms people often write to minimize the distance between the robot and its target. The spoken description does not specify the complete formula, including all signs and scaling, so I have not reconstructed an unverified equation.
That nested exponential reward was surprising and uncommon. On studying it, we found the expression used in the reward to be smooth over the negative-value range being discussed. Smooth rewards can help stabilize RL training. The lesson is that an LLM-generated reward may look strange yet contain useful mathematics and perform better than one we wrote ourselves. This is an account of the particular observed reward, not a guarantee that every nested exponential improves learning.
Someone pointed this out after a talk I gave at MIT. An audience member had found this strange-looking reward on our website, tried it and found that it worked better than their own. I had not initially paid much attention to that generated reward, but others had noticed it and discovered its practical value. I found that both surprising and interesting.
4. How DrEureka works
Reward-aware physics prior
DrEureka first uses Eureka to generate a reward, then configures domain-randomization parameters. These parameters have ranges, and a very broad search could in principle consider enormous intervals. The interview gives an illustrative interval from −100 to 100 for physical parameters such as gravity. Asking the LLM to search such a broad space can produce an excessively wide, unhelpful distribution: too much variation during training may cause the policy to fail. This is an illustration of an unconstrained search, not a claim that every physical parameter has a valid range from negative to positive infinity; quantities such as friction have physical constraints.
We address this with a physics prior. Starting with an initial reward and the default physical parameter values, we test how the policy trained with that reward performs as those values are increased or decreased. This identifies a smaller, usable range before the policy fails. The resulting reward-aware prior tells us which perturbations the initial policy can tolerate.
We can then constrain the LLM's search so that it produces a useful physical-randomization distribution. In other words, we first need upper and lower ranges within which the reward-trained policy remains effective, then guide the search with that information. Otherwise, an overly large search space may yield unusable results. This makes domain-randomization design more effective; it does not guarantee success on every task.
5. Questions and discussion
What Eureka and DrEureka could improve
Neither Eureka nor DrEureka uses visual input in these experiments. The LLM receives text and code, while the learned robot policies rely on state or proprioceptive inputs rather than seeing the scene. In real-world manipulation, a robot often needs vision to locate objects and understand how to interact with them. These projects do not solve that part of the problem. But I believe visual-language models such as GPT-4V, GPT-4o and Gemini could help us design rewards using visual feedback and supply feedback from images. Editorial clarification: the original says the robot solves problems from text or code. Those are inputs to the reward-design LLM, not direct control inputs to the robot policy.
DrEureka also does not use real-world feedback to update its domain-randomization parameters. At the time, we thought that would make the iteration cycle too slow. We were interested in zero-shot performance: how much could an LLM do out of the box? For a more complete future system, however, I think real-world feedback would be very useful for adjusting both rewards and domain-randomization settings.
Comparison with other LLM-based robotics projects
Broadly, we developed a way to use LLMs to accelerate and automate RL design. We demonstrated skills such as pen spinning in simulation and a previously unseen skill in reality: a quadruped walking on a yoga ball. This suggests substantial potential for LLMs to help robots learn complex skills.
I want to place this in the broader context of LLMs in robotics. Many projects already use these models, including SayCan and RT-2. Much of this discussion concerns high-level planning, whereas it is harder to demonstrate how a language model can teach high-frequency, low-level skills. I see Eureka and DrEureka as early demonstrations that an LLM can do more than tell a robot what to do: it can help teach the robot how to perform a complex skill. That opens a new direction. Editorial clarification: SayCan is a high-level skill-selection approach, but RT-2 also predicts low-level actions from vision and language. Grouping both solely as high-level planners, as the original does, is too broad. Eureka's distinction here is reward-based skill learning, not that all earlier projects lacked action outputs.
How do you view imitation learning and real-robot manipulation data's greater prominence in academia?
Simulation and RL do have problems. As you say, setting up the environment and reward can account for 80% to 90% of the work. If you are trying to write a paper, environment design may not look novel enough to provide a method to describe. From the perspective of writing papers and producing demonstrations, that is painful. The percentage is our experience-based estimate, not a measured field-wide statistic.
But I think both simulation-based and real-data learning paradigms matter. In my view, relying only on imitation learning from real data eventually leads to a performance plateau. At a certain stage, imitation learning may be the easiest way to reach 70% to 80% performance. That is enough for a demonstration, where you only need to show the system working once or twice. For a robot that must actually work in reality, however, 80% success and the corresponding execution speed are insufficient; it needs greater robustness. In sim-to-real RL research, controllers are often robust. Quadrupeds and drones displaying robust behaviors can also be agile and adaptable, largely because they can try many interactions in simulation and RL can use suboptimal experience. These percentages and the comparison express my assessment of the approaches discussed, not a universal ceiling for imitation learning.
By contrast, imitation learning generally needs high-quality demonstrations, and those demonstrations teach the robot to act within the expert's distribution. Outside it, the robot can get stuck and not know what to do. That may be acceptable for a demo, but RL can take us further when robustness matters. For DrEureka, we made some revealing videos: we kicked the robot, kicked the ball, and moved the ball from grass to flat terrain. The robot could respond to these disturbances, which is harder to obtain through imitation alone. But there is no denying how difficult it is to build the simulation environment and design the reward. Responding robustly here does not mean the robot predicts a kick before it happens.
Can Eureka address the challenges of simulation and RL?
Algorithms such as PPO are mature and widely used in RL, but environment construction and reward design remain major challenges in practical robot simulation. We still need to work out how to build suitable environments, create effective rewards and perform domain randomization. Different robots and tasks need different configurations, which often require a great deal of manual work.
There is an academic joke about 'graduate student descent': graduate students and PhD students manually tune parameters and optimize models, spending considerable time and effort. Automating that work with LLMs could save both. I think this is an important contribution that can help scale simulation and sim-to-real learning.
Previously, each task took substantial time. With LLMs, we could potentially automate design across hundreds or thousands of tasks. That could address a key bottleneck in large-scale simulation and sim-to-real transfer. Traditional approaches work well for one task: you can spend hours on it or write a whole paper. But the result may not carry over to another task or robot. An LLM-based general platform could support many different simulation tasks, which I think will become important. Hundreds or thousands is a scaling prospect, not the number of tasks demonstrated in these papers.
Applications of Eureka
There are academic examples of using Eureka with different kinds of robots. Someone used it to design a reward for a snake-like robot that could move forward successfully, with good results. We also use Eureka to improve rewards in our own research, and it has worked well in several projects.
Eureka's influence extends beyond robotics. The approach is general and need not be limited to reward design. Recently, others have used the idea of letting an LLM evolve candidate algorithms. Some Google papers at ICML explore related ideas: the LLM proposes solutions, those solutions are tested, and further solutions are generated. This could also have considerable potential in scientific discovery, including protein synthesis and coding. The interview does not identify the particular Google papers, so this remains the speaker's general description.
Eureka is a cutting-edge technology, but commercialization may require another iteration or two. There are many considerations beyond the LLM itself, including whether the robot hardware is good enough to support the algorithms effectively.
Overall, I think Eureka could have a major influence in robotics and science. I would not give Eureka all the credit: many other projects are exploring LLM-driven evolutionary discovery in parallel. But I consider it one of the most persuasive examples of this approach producing genuinely creative and interesting work.
Rendering and the sim-to-real gap
For visual realism, we could run a policy in the simulator, make a video and pass it to a multimodal language model to judge how realistic the visual distribution is. This could extend simulation's usefulness by improving rendering. It follows Eureka's overall idea: use a language model to generate content, evaluate it and iterate. Thus, beyond improving domain randomization through DrEureka, we can consider iterative LLM feedback to improve rendering realism. Such components were already technically possible at the time of this interview, and I knew teams working in this direction.
This approach could improve simulation realism and multimodal processing, helping us obtain better results across more applications.
How is domain randomization usually done?
There is a degree of guesswork when choosing domain-randomization parameters. You guess a value or range, train a policy and test it in simulation or reality. If it works, you have found useful parameters; if not, you adjust them and try again. This process has problems, especially for dangerous tasks that could damage the robot. Simulation testing is therefore important, and ideally we should do as much tuning as possible there.
This is one reason combining simulation and reality is difficult and time-consuming, and why robots may be damaged early on. It would be better to resolve everything in simulation. In practice, people have to guess and adjust initial parameters before risking the hardware, which clearly does not scale. The contribution of the Eureka line of work—specifically DrEureka for randomization—is to automate these parameter choices, reducing manual guessing and repeated experiments and making larger-scale application more practical.
Is Eureka limited to tasks such as locomotion?
Fundamentally, nothing prevents us from doing manipulation tasks in simulation. Many difficulties concern visual input: a manipulation policy needs to know where objects are, while the locomotion and in-hand tasks discussed here operate with ongoing contact and state information. Bridging the gap to visually guided manipulation is a substantial challenge. If we solve it, there is much more manipulation we could transfer from simulation to reality. Projects are already using 3D information such as point clouds for this purpose. More research is needed, but there is considerable promise. Progress here could bring simulation and reality closer together and solve many practical problems. The original's phrase about locating objects in 'local movement' is unclear; this passage follows the question's contrast between locomotion and visually guided manipulation.
How do simulation and RL handle long-horizon tasks?
For long-horizon tasks, the fundamental issue is not simulation itself but how to decompose the extended sequence of manipulation effectively. This must be solved in both simulation and reality. There is extensive work on hierarchical reinforcement learning, although much of the work I have in mind is theoretical and has not been fully applied to sim-to-real transfer. It is a research frontier that deserves more time. This assessment reflects the interview's 2024 context.
How is data generated in simulation?
A major advantage of RL is that it can generate its own data. A policy explores the environment and collects experience autonomously, which differs substantially from imitation learning.
In imitation learning, another agent—usually a person—collects the data, and the algorithm learns by imitating that agent's behavior. RL instead lets the policy explore and produce its own experience. This allows learning without a pre-collected demonstration dataset in the online setting discussed here.
Using the reward function, the RL algorithm can distinguish useful from unhelpful experience and iteratively improve. Through exploration, it can adapt its policy and improve performance under different conditions. Autonomous exploration and data collection are major advantages in complex, dynamic environments. Editorial clarification: the original says RL 'designs its own reward function.' Ordinary RL optimizes a supplied reward; in this work, Eureka's separate LLM-based design loop generates and revises that reward.
My research directions
My work is not limited to simulation; much of it is not conducted in a simulator. I began in reinforcement learning, and some of my early papers concerned RL algorithms and theory. Later, I also worked on imitation learning, including learning from human videos. Before the Eureka series, for example, I worked on VIP, which pre-trains useful value and visual representations from human videos. Eureka then took me into simulation. Overall, I care a great deal about developing robot-learning algorithms, whether or not they use simulation. Editorial clarification: VIP stands for Value-Implicit Pre-Training, not the original transcript's description of 'selecting manipulation options' or 'explicit pre-training.'
Could you briefly introduce VIP?
VIP learns a value representation that can provide visual rewards. Unlike a reward written as task-specific code, its learned value estimate is a numerical signal derived from videos. During robot execution, that signal can indicate how well the task is progressing. It comes from visual input, making it particularly useful for real-world applications. More precisely, VIP implicitly defines a goal-conditioned value function through distances in a learned visual embedding; those values can then supply a dense reward for downstream tasks.
6. Some reflections
Starting with Eureka, we will devote two episodes to simulation data and reinforcement learning. In conversations with robot-learning researchers, many have raised questions about this route. But if we consider the three engines behind scaling—large compute, large datasets and scalable model architectures—data is currently the biggest bottleneck in robotics. A robot interacts with the physical world, and the corresponding data modalities were largely absent from the internet and mobile-internet era. Teleoperation and motion capture alone are unlikely to produce enormous, diverse datasets quickly. Generating data that approximates physical-world interactions in simulation may therefore be an indispensable step toward embodied foundation models. Many problems still need to be solved along the way.
One point from my conversation with Jason was especially thought-provoking: in imitation learning, high-quality data tells you how to complete a task within the expert distribution, but outside it a robot may get stuck and not know what to do. We have also seen AI capabilities that astonish us emerge from reinforcement learning. Earlier, AlphaGo defeated Lee Sedol and dominated Go. More recently, AlphaProof and AlphaGeometry 2 together achieved silver-medal-level performance on the 2024 IMO problems. The large language models popularized by ChatGPT also make substantial use of RL during alignment. Editorial clarification: the original abbreviates AlphaGeometry 2 as AlphaGeometry. The IMO result was a silver-medal-equivalent score on competition problems, rather than a medal awarded to an AI contestant. AlphaProof uses RL for proof search; AlphaGeometry 2 combines a language model with symbolic deduction. These examples do not mean that RL alone accounts for every system's abilities.
Perhaps emerging intelligence is not confined to the range of human experience. I believe that training with human experience and ability as the reference inevitably produces a discounted version of human capabilities. To match those capabilities, perhaps we need methods that reach beyond human understanding and experience. Simulation and RL may be going through their darkest hour. But what person or endeavor that eventually achieved something great has never faced a darkest hour? This is my reflection on the research direction, not a proven limit on learning from human demonstrations.
References
VIP
https://sites.google.com/view/vip-rl
Eureka
https://eureka-research.github.io/
DrEureka
https://eureka-research.github.io/dr-eureka/