Back to Robot Learning & Foundation Models

Robot Learning & Foundation Models · EP01

RFM, Episode 1: Physical Intelligence, π0 and the Robot Foundation Model Recipe

Complete notes from Sergey Levine’s CoRL 2024 talks on cross-embodiment data, π0, pre-training, post-training and reinforcement learning on real robots.

Zha Dami2024-11-1828 min readWeChat original
Contents

Welcome to the first episode of the Robot Foundation Model, or RFM, series. Its abbreviation is also used by Covariant's robotics model, which I covered a few months ago in Inside RFM-1: An Interview with Covariant Co-founder Peter Chen. With Physical Intelligence releasing π0 and Tsinghua releasing RDT, I have been considering a series on robot foundation models. Here, PI refers to the company Physical Intelligence, while π0, pronounced pi-zero, is its model.

Over the past six months, results from North American embodied-model companies such as PI and Skild have begun to emerge, and the fundraising valuations of similarly capable Chinese companies have risen rapidly. PI and Skild give investors a reference for the paradigm and potential of robot foundation models, making it easier to identify comparable capabilities in China. I spent several weekends and evenings on this article, listening to both of Sergey's CoRL talks at least twice. I hope these notes clarify what embodied-model companies do, what representative model architectures look like, and what PI and π0 actually are.

These are notes from two talks Sergey Levine gave at CoRL 2024 workshops: 'Generalist Robot Policies' at Mastering Robot Manipulation in a World of Abundant Data, and 'How Real-World Cross-Embodiment Data Will Lead to Robotic Foundation Models' at the First Workshop on X-Embodiment Robot Learning.

The talks introduce robot foundation models and how to build them. Sergey emphasizes the importance of data types and the role of real-world experience in developing these models. He also discusses π0's design, including pre-training and post-training.

I think the reasoning behind these talks matters, so I have retained the extended arguments rather than heavily condensing them. The Chinese original runs to roughly ten thousand characters. It is a long read, but I strongly recommend reading closely. It should help you understand a mainstream robot foundation-model framework and what PI and π0 are doing. I hope you find it useful. The talk notes and forward-looking assessments retain their November 2024 context.

Original article outline

Contents

↓

1. Opportunities and challenges in robot learning

2. The Open X-Embodiment dataset

3. What is a robot foundation model?

4. Why do we need robot foundation models?

5. Where can we obtain the large datasets they need?

The central question of data choice

Why not simulation data?

Why not video data?

Why real-world data?

6. π0

π0's architecture

π0's training recipe

π0 pre-training

π0 post-training

π0 in action

7. Pre-training robot foundation models

8. Supervised and reinforcement-learning fine-tuning

9. Reinforcement learning on real robots

10. Closing points from Sergey's talks

11. Questions and discussion

1. Opportunities and challenges in robot learning

How did AI systems work ten or fifteen years ago? To solve a particular problem—image segmentation, classification or visual question answering—we would create a large dataset specifically for it. We would then train a specialized model and hope it performed well. That was labor-intensive because it required extensive data, precisely annotated by experts for the individual task.

Today, that has changed. Instead of building a separate dataset and model for every task, we collect large quantities of internet data, often without strong labels: raw text or loosely associated images and text. Large-scale self-supervised objectives, such as next-token prediction, train a foundation model that can support many real-world tasks. We can then fine-tune it for downstream problems or sometimes use zero-shot prompting. This has transformed AI by reducing the need for large manually annotated datasets in every domain and allowing many applications to share a general model.

I think we are beginning to see a similar shift in robotics. A few years ago, the prevailing view was that every task—making pancakes, picking up a box or anything else—needed a separate model for its particular robot and application. Training usually happened in a controlled laboratory and required substantial data, creating a major obstacle to deploying robot-learning systems.

Robot learning could look very different in the future. We can pool data from many robot sources to obtain much larger datasets, then train a large foundation model using self-supervised or action-supervised learning. That model can be fine-tuned or prompted for downstream tasks. At a high level, this is attractive because it addresses a major deployment barrier: collecting data and training separately for every task.

But important challenges remain. How should we design the models? Where does the data come from? How do we combine different sources and scale the process effectively? How do we adapt and specialize the model for downstream tasks? We need answers to all of these questions.

2. The Open X-Embodiment dataset

One piece of the puzzle is collecting data from many different robots to train a foundation model. There is no 'internet of robot data' yet, and an individual robot may not have a large dataset. Pooling sources could create something with a similar role.

We explored that idea in the RT-X project, which began roughly a year before these talks. Instead of simply scraping the internet, we contacted thirty-three academic labs and worked with Google DeepMind to collect manipulation datasets. This produced data with substantial diversity in tasks, camera viewpoints and robots. The manipulation-policy experiments emphasized single-arm systems with parallel-jaw grippers. Editorial clarification: the original says all robots in the dataset have that form. The broader Open X-Embodiment collection also includes other embodiments; the statement describes the aligned manipulation setting, not the entire collection.

That diversity also creates a challenge: the data is highly heterogeneous. We need learning methods that work across embodiments and adapt to different robot systems. But the benefits could be substantial. We trained a model on the pooled data and sent it back to contributing labs. Each tested the cross-embodiment model on its own tasks and compared it with its specialist policy.

The results were exciting. In most cases, the cross-embodiment policy performed better, with an average relative success-rate improvement of about 50% in the reported RT-1-X comparisons. This resembles what we see in language modeling: a general model trained on diverse data can outperform a specialist on the specialist's own task. A 50% relative improvement is not an increase of fifty percentage points.

Another way to use abundant data is to inherit internet-scale image–language knowledge. We can train a language-model-based system to process images, producing a vision-language model, and train it on captioning or visual question answering before adapting it to robotics. Asked how to pick up a soda can, for example, the model can output an encoded robot action based on the visual observation instead of an ordinary verbal answer.

This idea was explored using vision-language models such as Google's PaLI-X and PaLM-E as backbones for action prediction in RT-2 and RT-2-X. Fine-tuning on diverse cross-embodiment data produced impressive behavior. The policy could follow a spatial instruction such as 'put the apple between the can and the orange' even without explicitly seeing that kind of instruction in its robot-training data, transferring knowledge learned elsewhere. Editorial clarification: 'PolyX' in the original is a transcription error for PaLI-X; PaLM-E is also corrected to its standard spelling.

In the reported comparison, the cross-embodiment model performed roughly three times better than a baseline trained on the most represented robot's data alone. This shows the potential of pooling diverse sources. But those tasks were relatively simple academic tests emphasizing semantic understanding, rather than complex dexterous manipulation. The multiplier applies to that evaluation, not every task or robot.

There is much more to do. We need to understand how vision-language-action models, VLAs, handle more complex tasks. That requires architectures designed for robot control rather than simply reusing a VLM designed to produce language. We also need to handle different robot forms and action spaces; many initial experiments focused on single-arm robots with aligned action representations.

This is only one part of a larger puzzle. Many questions remain about designing, scaling and adapting robot foundation models to realize their potential.

3. What is a robot foundation model?

A robot foundation model aims to perform many tasks, usually across different robot systems. Octo is one early generalist-policy project. The particular networked experiment described here, however, used RT-2-X: the model ran at Google DeepMind, while a robot at UC Berkeley called it over the internet. It was a VLA trained using Open X-Embodiment data. Editorial clarification: the original first names Octo and then describes an RT-2-X demonstration. These are distinct models, not two names for the same system.

That experiment was exciting. To my knowledge at the time, it was the first demonstration of accessing a robot foundation model through a network API. A VLA processes images as well as text. Its foundation comes from language and visual understanding trained on internet-scale tasks such as question answering and captioning; it is then adapted to generate robot actions from language and vision.

The model was trained to answer a question like 'how should the robot perform this task?' with action tokens rather than a description. Such models inherit powerful semantics from internet-scale pre-training, while diverse robot data grounds those capabilities in control. Their breadth and generality are central to the approach.

4. Why do we need robot foundation models?

There are many reasons, but the most convincing to me is that they address robot learning's biggest limitation. Learning lets a robot perform tasks that are difficult to program manually, yet it traditionally requires a great deal of task-specific data. There are several possible responses to that problem.

One is to use a small, high-quality dataset. Some research demonstrates methods on small domain-specific datasets. For deployment, however, their scope is limited. The real world is broad; a narrow dataset may support excellent performance in one domain without preparing the model for the wider world.

Another option is a large dataset that may be lower quality. Simulation data is easier to generate at scale, for example, but often differs in distribution from reality. The training set may be large while the deployment environment is substantially different. Both approaches have limitations.

Foundation models bridge that gap by learning broad knowledge from large, diverse datasets, then specializing with smaller, narrower data. Pre-training builds a basis for manipulation across varied real-world situations, and fine-tuning makes the model effective for a particular application.

5. Where can we obtain the large datasets robot foundation models need?

This raises another question: where do we get large, high-quality pre-training datasets? That remains a central challenge. Pre-training need not use highly specialized data, but it still needs a great deal of it. The internet is not full of robot demonstrations, and researchers disagree about the best source.

One route is synthetic data from simulation. Another uses internet videos as a substitute or precursor for robot-specific data, covering many situations. The route I advocate and consider most promising is real-world robot experience.

First, real data has worked well in other fields. Simulation has contributed to perception training, but in my view the most effective practical models have consistently been trained on real data, matching the conditions encountered at test time. More importantly, real-world robot data may become the easiest and cheapest option in the long run. Once robots do useful work, there will be a strong reason to deploy many of them. Those robots could produce large amounts of experience while doing their jobs. After the initial bootstrapping problem, that could be the most economical and useful data source. This is an argument about a future deployment regime, not a claim that all successful models already use only real data.

The central question of data choice

Returning to the main question, I believe real-world data is essential to developing robot foundation models. That does not make simulation or video useless. I see real robot data as more promising over the long term, not because simulators or human-to-robot transfer are impossible, but because physical robot experience contains so much practically relevant information.

Why not simulation data?

Simulation is attractive because data generation can be almost free apart from compute. It offers a direct way to turn computation into capability. To control a rocket, for example, we can write equations of motion, implement a simulator, and use RL to 'invert' the simulator into a control trajectory that follows a desired flight path. But this resembles classical model-based control more than the usual motivation for machine learning: describe a system mathematically, then derive a controller. 'Invert' is an analogy for optimizing control through the model, not necessarily computing a literal algebraic inverse.

Classical methods work well when we can characterize the system, often through explicit equations. Machine learning relaxes that requirement: we can use executable code to simulate dynamics without a closed-form solution. Still, sim-to-real control in this view optimizes against a known modeled system, even when that model involves complex simulation or rendering. This perspective helps explain its strengths and weaknesses.

To some extent, simulation can repackage traditional model-based robot intelligence. But the challenge in robotics is not only controlling known systems; it is learning behavior people cannot readily formalize or encode. We might derive dynamics for folding a T-shirt, but writing equations for how a robot should interact with a hotel guest is much harder. Human behavior is complex and difficult to simulate. That is a fundamental problem when modeling real-world situations: however advanced the simulation, it differs from the phenomena it represents.

Why not video data?

Videos offer a more direct, potentially useful source: take real-world footage from the internet without needing a robot or simulator. Other foundation-model fields have made progress with internet-scale data. But this has drawbacks. It can resemble a 'couch potato' solution—sit and watch videos, hoping passive observation will teach motor skills. Depending on how it is used, it could combine the strengths of simulation and reality or inherit limitations from both.

My concern is that relying too heavily on video could be the least effective choice. Like simulation, it has a distribution gap, while lacking the simulator's ability to produce high-quality data for a specific task. A video dataset may also be insufficiently specific to the robot problem we need to solve.

Another possibility is that conventional visual pre-training already captures much of the useful information in internet images and videos. Computer vision is good at training on these sources and continues to improve, so the boundary between using video data for robotics and ordinary visual pre-training may become blurred. Can watching videos alone make you an expert tennis player? Watching Roger Federer may teach you something, but it will not make you a professional. Videos can be valuable, yet may not alone provide the performance and generalization needed for robot tasks. I think their value grows when combined with substantial physical interaction, much as in human learning. This is the speaker's hypothesis about the limits of passive data, not an established claim that video learning has exhausted its potential.

Why real-world data?

I see real-world data as a practical way to pre-train and fine-tune. It looks prohibitively expensive mainly if we depend on a single robot. That brings us back to the workshop's cross-embodiment theme: once many robots work in reality, collecting experience becomes easier. Each generates data, and aggregation means we are no longer confined to the deployment scope of one robot.

Cross-embodiment training is key because it lets us use data from many robot types. This resembles the pooling of diverse sources and web scraping that enabled powerful computer-vision models. Robotics could see a similar transformation. Fundamental research should address problems with long-term potential, rather than only today's collection cost. If we are optimistic about robotics, we can imagine millions or even billions of deployed robots worldwide. Real-data collection could then become cheap and easy. These deployment numbers are a future scenario, not a forecast with a fixed date.

Pre-training data need not be perfect; it should be diverse and cover many situations. Fine-tuning needs more specialized, high-quality examples matching the desired task. Fortunately, post-training may need far less data: perhaps tens of hours, some of which can be collected in a few days. In the future, all of that experience might even be collected automatically through RL.

In my view, real-world experience from many robots is central to improving both pre-training and fine-tuning. Better cross-embodiment learning and broader deployment could make it an efficient, practical resource for advancing robotics.

6. π0

π0 is another robot foundation model we developed, aimed at flexible, complex manipulation and tasks extending over many stages.

It is trained across multiple embodiments and designed to perform tasks such as laundry handling: unloading a dryer and folding individual garments.

Its diverse training mixture combines data collected on different robot platforms with RT-X and other public datasets. It includes more varied configurations than the earlier aligned datasets: dual-arm systems, six- or seven-degree-of-freedom arms, and mobile robots with different action spaces. The model is designed for dexterous tasks that can require control at approximately 50 Hz. That is action execution frequency, not necessarily fifty fresh full-model inferences per second.

The recipe has several stages. We start with a vision-language model pre-trained on internet-scale data, then pre-train a flow-matching VLA using robot data across embodiments, including RT-X. The resulting base model can follow text commands in a zero-shot evaluation setting, without dedicated downstream fine-tuning. We then post-train on selected data to improve performance and specialize it for demanding or new tasks, such as laundry folding. This resembles language-model specialization and alignment, though the initial π0 post-training here uses supervised robot demonstrations rather than RLHF. Zero-shot does not mean the model has never received any robot training or never encountered a related skill.

I want to emphasize that architecture is only one part of the system. How we collect and structure data, pre-train, and fine-tune can matter more than the architectural details.

π0's architecture

The architecture is interesting. It uses flow matching, related to diffusion, to generate continuous actions without discretizing them into categorical action tokens. It can model complex multimodal action distributions and output action chunks. For high-frequency control, generating a sequence—roughly one second of actions at 50 Hz in this example—is important. Here, 'multimodal' means multiple possible modes of behavior, not merely multiple sensor modalities.

The vision-language backbone is initialized from PaliGemma, an open-source VLM. It receives camera images—the talk notes discuss one to three, depending on embodiment—and a language command tokenized in the usual way. Some setups have a base camera; others also have two wrist cameras. Editorial clarification: the original incorrectly calls π0's backbone 'PolyX.' It is PaliGemma, not PaLI-X; the published π0 setup uses two or three RGB inputs with missing slots masked as needed.

Actions come from a component we call the action expert. Think of it as a robot-control extension of the VLM that produces continuous actions rather than language tokens. The action sequence slots use a flow-matching loss rather than the usual categorical next-token loss. This creates a model that combines a vision-language backbone with diffusion-style continuous action generation.

The action expert uses separate weights for the action-related part of the network while matching the transformer's layer structure. The experts exchange information through attention at the corresponding layers. It is not a conventional encoder–decoder architecture; it is closer to one decoder-only transformer with different expert weights. The attention mask lets action tokens attend to visual and language inputs, while preventing those earlier inputs from attending to future action blocks. The components work together despite their different weights. The original's statement that every action layer attends to every VLM layer is too broad; the interaction occurs through the transformer attention layers.

You can roughly view this as a two-expert model: one handles images and language, while the other handles robot state and actions. This design aims to preserve efficiency while supporting complex tasks and adaptation. It is an analogy to mixture-of-experts, not a claim that it uses a conventional learned routing gate for arbitrary experts.

Joint angles are supplied to the action expert, which specializes in continuous robot quantities, while the other expert processes image and text representations. The model generates chunks spanning about fifty time steps, with the action dimension depending on the robot. The talk notes give a maximum of twenty-four dimensions; the published October 2024 paper instead describes eighteen-dimensional state and action vectors for its largest robot, with smaller embodiments zero-padded. I retain that distinction rather than treating twenty-four as the paper's specification. Action generation uses flow matching, which can be understood here as related to diffusion.

π0's training recipe

We begin with a broad pre-training dataset totaling roughly 10,000 hours. At the time of these talks, we believed this might be the largest robot-model training dataset outside autonomous driving. It gives us a model that can perform tasks through prompting. We can then specialize it for a downstream task, typically using one to twenty hours of post-training data, depending on complexity. The largest-dataset assessment is historical and qualified by what the speaker knew then.

Complex tasks such as folding clothing or assembling a box may require about twenty hours of post-training data, while simpler ones can work well with one to five hours. Those quantities are manageable to collect in a day or two. For laundry folding, the pre-training mixture already includes many examples, so post-training adds high-quality, task-specific demonstrations. For box assembly, the relevant skill was absent from pre-training, and its task-specific examples came entirely from post-training.

The distinction is what each stage emphasizes:

Pre-training uses broader, mixed-quality data, helping the model learn diverse strategies and recover from mistakes.

Post-training uses narrower, higher-quality data to teach consistent, effective task execution.

With only polished post-training demonstrations, the robot may not learn to handle unexpected states or recover from errors. Combining them with broad pre-training can provide both general experience and task-specific expertise. The analogy is a language model that acquires broad knowledge during pre-training and specializes for an application afterward.

Comparing the complete pre-trained-and-post-trained π0 with a model trained only on the post-training dataset highlights the value of pre-training. The latter can work on simple tasks, such as packing eggs, but does poorly on harder ones. Broad prior experience supplies recovery strategies and substantially improves complex-task performance.

π0 pre-training

An exciting result of this recipe is its ability to support complex, dexterous tasks. At Physical Intelligence, π0 is pre-trained across platforms including dual-arm robots and mobile manipulators. It can unload laundry from a dryer, put it in a basket, move the basket to a table and fold the garments. It is not flawless and sometimes makes mistakes, but diverse pre-training helps it handle these tasks impressively.

The experiments also teach us about the required data. Box assembly is challenging because the robot must stabilize parts while folding them. Post-training teaches the specific strategy, while pre-training provides a broader basis of physical manipulation experience.

Video demonstration from the original article

In the experiment, the robot first braces the box against the table, then folds its sides methodically. The behavior illustrates how broad pre-training can support physical manipulation while post-training refines a particular complex strategy. It is behavioral evidence, rather than proof that the network explicitly represents physical laws.

π0 post-training

Complex tasks such as box folding and car-dashboard assembly illustrate the potential of pre-training and post-training with real experience. In π0, general pre-training provides a starting point, and post-training improves more demanding skills. During box folding, for example, if one side slips out, the robot retries and corrects the situation. Better task-specific data and training improve reliability and precision. The dashboard assembly discussed later belongs to the separate real-robot RL work, not a demonstrated π0 task. Retrying during deployment also does not itself imply online updates to π0's weights.

The efficiency is notable: tens of hours, far less than the pre-training volume, can specialize a policy, making collection for niche applications practical. Autonomous improvement is another possibility. Work by Zhiyuan 'Paul' Zhou and colleagues, SOAR, uses foundation models to propose and evaluate tasks and uses generated goal images to support control. A robot can collect useful experience autonomously and specialize for a new environment. The talk describes roughly a twofold performance gain in the environments where the robot collected data. The SOAR project page separately reports an average success-rate improvement of 31% across its evaluated scenes, so the twofold description should not be read as that published overall average. This is a separate project illustrating the prospect of autonomous post-training, not an existing π0 feature.

π0 in action

Consider laundry folding, a complex, delicate task. Garments come out of the basket crumpled in many ways, and the robot has to handle each configuration. It spreads out a garment, folds it and places it in a stack at the upper-right of the table. The demonstrations cover different garments, starting configurations and even a mobile platform. The robot sometimes fails but can persist and retry. These examples illustrate recovery; they do not guarantee eventual success in every attempt.

Video demonstration from the original article

Another laundry task is unloading a dryer. The robot approaches, opens the door and removes the garments. This highlights pre-training's value. The high-quality post-training set is under twenty hours, but pre-training includes many less-polished clothing-manipulation attempts. That broader experience helps with recovery when garments are piled unpredictably.

We deliberately disturbed the robot to test recovery. Michael threw a shirt onto the table midway through the task. The robot was initially confused, then moved it aside and resumed folding. Even with repeated interventions, it adapted and returned to the task. The examples show how broad pre-training supports a general policy, while post-training develops task-specific proficiency.

7. Pre-training robot foundation models

Many people know the multi-robot pre-training idea from RT-X. Diversity matters because a model can learn useful behavior despite substantial variation in the data. Most excitingly, a general model trained on diverse sources may outperform specialists trained separately in each domain. Early experiments, however, addressed relatively simple tasks—putting a carrot in a pot or opening a drawer—and often used similar single-arm robots with parallel grippers.

We are exploring more varied mixtures: mobile robots, mobile manipulators and stationary arms. The key question is whether combining them produces positive transfer. Jonathan Yang at Stanford, working with Catherine Glossop and Arjun Bhorkar at UC Berkeley and other collaborators, studied whether navigation data could help manipulation. Editorial clarification: 'K. Glossop' and 'Archon Boar' in the original are corrected to Catherine Glossop and Arjun Bhorkar, as listed in the project and paper.

The study combines different action representations and egocentric observations, including wrist-camera views connected to robot motion. We compared training on manipulation data alone, adding navigation data collected by multiple robots, and also adding driving data. The navigation mixture is GNM, General Navigation Model, rendered 'G&M' in the original notes. It includes sidewalk and other indoor and outdoor navigation environments. Pooling these domains improved the evaluated manipulation tasks, suggesting that mobile-robot and driving data can help train stronger, more robust robot policies. A wrist camera is an observation source, not by itself a 3D motion-tracking device.

The benefit of adding navigation and driving is particularly interesting because we might not expect it to help manipulation. One explanation is shared geometry: as we approach a strawberry, it appears larger and closer, information that also matters when reaching to grasp it. Cross-embodiment training can transfer spatial regularities that apply across physically different systems. This is an interpretation of the observed transfer, rather than proof of an explicit internal physics model.

We recently broadened this idea further. CrossFormer, developed by Ria Doshi and colleagues, trains across manipulation, navigation, quadruped locomotion and other embodiments. It uses a large transformer with embodiment-specific action heads, allowing learning from systems with different action spaces. Its results exceed single-embodiment baselines and earlier approaches in the evaluated settings. Editorial clarification: the original calls it 'CrossForer'; the correct name is CrossFormer. The project’s overall claim is that it matches embodiment-specific specialist policies while significantly outperforming prior cross-embodiment methods; this does not mean it beats every specialist on every task.

8. Supervised and reinforcement-learning fine-tuning

Supervised fine-tuning may need only tens of hours of demonstrations, and RL may require even less interaction time for certain tasks. This is an area where I hope future research makes further progress, analogous to the rapid development of language-model post-training. As the community improves the recipe, we may see significant advances in robot foundation models over the following years.

Minimizing adaptation data matters particularly for tasks requiring precision, robustness and efficiency. SERL illustrates real-world RL for tasks such as electronics assembly. The talk discusses high success after roughly thirty minutes on tasks such as cable routing, with more demanding training runs taking up to two hours. Results can substantially outperform imitation learning with comparable collection effort. These timings refer to particular SERL-family tasks and demonstrations, not a universal training requirement.

This suggests a broader recipe combining real-world pre-training with efficient task adaptation. Autonomous collection and continued improvement could improve performance and reduce the manual effort needed to deploy increasingly complex behavior. The RL examples discussed here, however, are not all fine-tuned VLAs; integrating them with large pre-trained policies is a research direction.

9. Reinforcement learning on real robots

I want to focus on improving robot policies through real-world RL. This work uses smaller models rather than VLAs at this stage. I believe combining RL with VLAs in the future could produce faster and more effective policies.

HIL-SERL was developed by Jianlan Luo, Charles Xu and colleagues at UC Berkeley for tasks including electronics assembly. We began the research roughly a year before these talks. The sample-efficient method starts from a small number of demonstrations, often fifteen to thirty, then improves through real interaction. Tasks can be learned in about thirty minutes to two hours. Some simple tasks develop successful policies in thirty minutes, and in a demonstrated insertion task the robot begins inserting components reliably after only fifteen minutes without damaging delicate pins. For related discussion, see Jianlan Luo on Real-World Reinforcement Learning for Robot Manipulation. Editorial clarification: the original calls the method 'HI-SERL'; its name is HIL-SERL, for human-in-the-loop SERL. These are task-specific examples, not guarantees of training speed or damage-free execution.

Video demonstration from the original article

The video shows the complete training process.

We evaluated tasks including microchip insertion, cable routing and object relocation. For comparison, the imitation-learning baselines received comparable data-collection effort, typically thirty minutes to two hours depending on the task; in some cases we provided additional demonstrations. RL performed substantially better in the evaluated tasks. These require precision, and RL's execution cycles were also two to three times faster than the imitation baselines. Comparable collection time should not be confused with identical transition datasets.

Now imagine combining these methods with robot foundation models. The integration could be more effective and powerful. We also extended real-world RL to demanding dual-arm tasks such as assembling a car dashboard, requiring precise alignment and insertion of plastic pins. Training took about two hours, and the policy achieved 100% success in the reported evaluation. That means all tested attempts succeeded, not that all future executions are guaranteed to succeed.

We explored other tasks: using a whip to remove a block, which is not especially practical but requires dexterity; assembling a transmission belt involving deformable material; flipping an egg; and assembling IKEA furniture. The robot could not yet use a screwdriver, but it aligned the pins and components successfully.

The RL policies' robustness is also notable. During belt assembly, Charles deliberately disturbed the belt, and the robot recovered. That ability came from varied RL experience during training, rather than from a foundation-model pre-training stage. Combining such RL with pre-training could create stronger systems in the future.

One particularly interesting demonstration involved assembling several computer components. Separate policies were trained for each part, such as RAM, SATA connections and a USB plug, then sequenced for smooth execution. I asked the students to show that the RL process had not damaged the computer. After assembly, they powered it on successfully. That demonstrated the functionality of the particular assembled system, rather than establishing a universal reliability guarantee.

10. Closing points from Sergey's talks

The emerging recipe for robot foundation models increasingly resembles that of LLMs: pre-train on large, diverse data using a flexible architecture, then post-train with high-quality supervision or reinforcement learning to align behavior with human preferences. For robotics, we are developing the analogous components—diverse experience, architectures designed to handle it, and adaptation procedures for individual tasks.

We aim to consolidate broad pre-training knowledge in the model, then post-train it. This is often supervised fine-tuning; instruction tuning, including reasoning-oriented examples in language-model settings, offers another analogy. RL may play a larger role in robot post-training in the future. The reference to chain-of-thought in the original describes that broader analogy, not a specific chain-of-thought training feature of the initial π0 model.

A robot foundation-model recipe is taking shape that resembles the LLM recipe, while tailoring its data, architecture and fine-tuning methods to robots' particular needs.

11. Questions and discussion

Q: Could vision-language training and robot post-training happen together, rather than completing vision-language training first? A: We did not do that for π0, for practical reasons: we started from an already pre-trained, available model. But I suspect joint training could be more effective. It might reduce forgetting of vision-language capabilities and let us explore post-training objectives better suited to robotics. I think it is worth trying, though the setup would be more complicated.

Q: You mentioned the importance of data filtering and cleaning. How does that apply to robot models? A: It is an open problem. Methods and algorithms are important, but the practical challenges of making a model work often concern data management. I can describe our approach: selection is much more permissive in pre-training than in post-training.

For post-training, we want demonstrations in which the operator performs well: consistent strategy, efficient execution, and no major delays or pauses. We select data that clearly shows the desired way to complete the task.

Pre-training prioritizes broad coverage over perfection. Examples that include mistakes can help—not to teach the robot to make them, but to expose it to recovery. Wider state coverage helps it handle varied situations and recover from suboptimal states.

Q: Could post-training be accelerated with reinforcement learning and real-time human feedback during training? A: That is a good idea. Real-time feedback can guide RL. Because these training runs can be short, supervising them is practical and could significantly speed up learning. We have already explored human interventions, but there is room to improve how efficiently the algorithm uses them.

Q: You described real-world data as the cheapest option. Why would it be cheaper than a deliberately curated dataset? And why not use simulation for pre-training?

A: At the level of a large deployed robot fleet, I see real-world experience as potentially cheap, although practical issues such as privacy need to be addressed. Once useful robots generate experience while operating, collecting it could become more like gathering readily available internet data. Calling it 'free' in the original expresses this future marginal-collection-cost argument; it does not eliminate hardware, operation, processing or governance costs today.

That does not mean abandoning simulation. I expect a situation somewhat like computer vision: synthetic data remains useful, but the richness and availability of real-world data draw much of the attention.

In some domains, simulation may be preferable. Rocket flight is an example: collecting real experience is difficult and expensive. But where simulation is not the most natural or efficient option, real-world data may be more practical.

12. Some reflections

People have recently asked why, as an investor, I spend time explaining frontier research. More bluntly: you have interviewed all these people, yet you have not invested in any of the leading companies in this wave. It is direct, but a good question. At first, I was simply learning and wanted to organize what I learned; writing became a habit. Looking back, there are two reasons I continue and consciously try to expand the effort. First, I am curious. I cannot always understand a paper without guidance, and the content creates its own loop: I learn, write it down so others can understand, give the research some visibility, and perhaps make a tiny contribution to the field. Second, creating independent media is, in a way, entrepreneurship. One of entrepreneurship's attractions is that when you push something forward, unexpected connections and opportunities find you. Some commitments are not about short-term ROI. Watching 20VC recently has encouraged me to keep going.

References

First Workshop on X-Embodiment Robot Learning, CoRL 2024

https://www.youtube.com/live/ELUMFpJCUS0?t=16866s

CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data

https://www.youtube.com/live/GhwCIjEEBBs

Smart Note · Robot Learning & Foundation Models · EP01