In the first installment of the Robot Foundation Model series, RFM Episode 1: Physical Intelligence and the π0 Embodied Foundation Model, we explored π0, the robot foundation model released by the US embodied AI unicorn Physical Intelligence, through two talks by Sergey Levine at CoRL 2024. This second installment looks at an earlier generalist robot model, Octo, and the development path from Octo to π0.
This article is based on From Octo to π0: How to Train Your Generalist Robot Policy, a talk by Kevin Black, a lead author of both projects, at the First Workshop on X-Embodiment Robot Learning at CoRL 2024.
The talk provides an accessible account of Octo’s successes and failures and how π0 built on those lessons. Particularly valuable is Kevin’s willingness to share what did not work in both projects. Alongside a detailed discussion of Octo, he offers distinctive insights into training π0.
This article mainly translates and organizes Kevin Black’s presentation. There are details I do not fully understand, so I welcome corrections if readers notice factual mistakes. Editorial note for this English edition: the original notes contain transcription errors in model names. Gemma, SigLIP and PaliGemma are used below, consistent with the π0 paper. The smaller precursor was developed after Octo and before the full π0 model. π0 uses flow matching, which the paper describes as a variant of diffusion.
Original article outline
Contents
👇
1. Why We Built Octo
2. Octo’s Model Design
3. Evaluating Octo
4. Did Octo Work?
5. Lessons Worth Keeping
6. What Went Wrong
Data
Model Design
7. Scaling at PI: From Octo to π0
Scaling the Model and Improving Pretraining
Improving the Decoder
Addressing Distribution Shift with Action Experts
Training and Results
Did π0 Work?
1. Why We Built Octo
Many people across different fields have heard of the Open X-Embodiment dataset. Released a year earlier, it was a major milestone for robotics researchers. It was far larger than any dataset we had previously used, bringing together contributions from 21 institutions and data from 22 robots, totaling roughly 4,000 hours of recorded activity. In data-driven robotics, data is the crucial ingredient, and this dataset offered the scale and diversity we had been waiting for.
Google did not stop there. It also released impressive results from training RT-2 on Open X-Embodiment, demonstrating remarkable generalization across laboratories and tasks. That was an extraordinary achievement. When I first evaluated the model, I said it was unstoppable—a line Sergey likes to quote on his slides, much to my continued embarrassment. Naming note: The cross-embodiment version trained on this collection is RT-2-X; the talk uses RT-2 as shorthand.
My main criticism of RT-2 was the number 55 billion: the parameter count of its language model. That raised two major problems. First, it effectively made the model proprietary and unlikely to be open-sourced. Second, even if it were released, most robotics researchers would not have the resources to deploy it, let alone train it. Most of us are not accustomed to launching enough GPUs to handle that scale of computation.
Obtaining enough GPUs just to run inference with a 55-billion-parameter model, never mind training it, is no small undertaking. Google did release a smaller model, RT-1, but it had limitations too. Its training infrastructure was not open-sourced, for example, and it was not particularly well suited to fine-tuning.
At that point, some friends at Berkeley and I saw an opportunity. Why not train our own model on this incredible data? It was not a particularly novel or profound idea, but it was worth trying. That became the Octo project.
Octo’s main goal was flexibility. Cross-embodiment learning requires adaptability across systems and tasks, and we wanted our model to support that. Another cornerstone was open-sourcing the entire infrastructure. Our idea was to create something people could actually use. For many researchers, this would be their first hands-on experience with a model trained on such a large, diverse dataset.

We wanted people to use the model—that was the central idea. One analogy was that Octo could be robotics’ GPT-2. GPT-2 was a relatively small model and not especially useful for zero-shot tasks: it could generate coherent text, but that was about it. It was very useful for fine-tuning, however. Its smaller versions had on the order of 100 million parameters, making it easy to download and adapt to tasks such as writing poetry.
That was roughly the ambition for Octo. Perhaps it was a little optimistic, but we hoped it could deliver a similar kind of practical usefulness in robotics if people could use it and obtain good results.
2. Octo’s Model Design

Octo’s design is fairly straightforward: essentially, a large transformer. A language encoder processes instructions, observation encoders process sensor inputs and other observations, and everything goes into the transformer. The main design choice is the action heads: small modules, such as MLPs, that output actions. This increases flexibility. If a robot has an action space absent from pretraining, you can simply replace the action head. Our thinking was that the large shared encoder would still transfer useful knowledge while the heads adapted easily to new action representations.
3. Evaluating Octo
We ran extensive evaluations. For fine-tuning, we tested tasks with new action spaces, such as Berkeley Pick-Up and Berkeley Bimanual. Those spaces were not represented in pretraining. Pretraining used conventional end-effector position control, whereas these tasks involved joint positions and joint torques.
We also evaluated zero-shot performance: no fine-tuning, simply testing in settings represented in the training data. That was not intended as a demonstration of generalization, but as a check that the model could handle the kinds of data it had seen during training.
We put considerable effort into accessibility, releasing an open-source codebase and complete infrastructure, including data loaders.
4. Did Octo Work?
Did it work? Were people actually using Octo? Well, sort of—but mostly not for fine-tuning. Most used it as a comparison baseline. Honestly, that was fine. It was still a form of impact and showed that we had made inference simple enough for people to use.
But people were not really fine-tuning Octo because its performance was not good enough. The improvement did not justify the effort of formatting data, setting up training infrastructure and learning how everything worked. The benefit of fine-tuning was insufficient to make those investments worthwhile.
In many respects, Octo failed. We did not achieve the breakthrough we had hoped for. That does not mean it failed in every way. Before discussing what went wrong, I want to reflect on what we got right and what we learned. This judgment concerns our goal of widespread practical fine-tuning, rather than an absence of positive results in Octo's published benchmarks.
5. Lessons Worth Keeping
One lesson was that diffusion works extremely well. Diffusion policies were already fairly well known when we began, but this project made their value clear to me personally. Initially, nothing worked. About two weeks before the deadline, we added a diffusion head, and suddenly things started working. That experience reinforced the idea that diffusion is especially useful as datasets scale, perhaps because it handles multimodal action data effectively.
Another important lesson was action chunking. This was not new: Diffusion Policy and ACT already used forms of it. But the project made me appreciate its importance. At Berkeley, we had traditionally avoided action chunking for some tasks. We found that chunks spanning roughly half a second to one second substantially improved performance.
There were also mundane but crucial lessons—details that may not look impressive enough for a paper but matter enormously when training these models. Data shuffling, for example, proved essential. Robotics data has an unfortunate property: it is highly correlated. Each trajectory is strongly correlated with itself, and trajectories from the same scene are often very similar. Without proper shuffling, the loss curve does not improve as expected. We learned that through trial and error. More effective shuffling substantially improved performance, sometimes even doubling it.
Another seemingly minor detail with major consequences is action representation. How should gripper state be represented: absolute or relative position? For joints, should we use position, position increments or torque? Which reference frame should end-effector control use? These questions may appear trivial, but they matter greatly, especially in cross-embodiment training. At PI, we spent countless hours discussing them. Perhaps they will matter less as data scales, but today they make a substantial difference.
6. What Went Wrong
Why did Octo fail to become robotics’ GPT-2? I will divide the problems into two categories: data and design.
Data
Data is always the most important factor in data-driven robotics, as it is in machine learning more generally. Much has already been said about this, so I tried a different way to illustrate it. I did something slightly silly: converting text datasets into hours to compare them with robot datasets.
Using the number of tokens in a text dataset and an average human reading rate, you can estimate how long an ordinary person would take to read it. GPT-2’s training dataset, for example, was considered relatively small at the time, yet it far exceeded any robotics dataset.
This comparison has many limitations. Diversity and quality matter much more than raw size. Nevertheless, it highlights a key point: we have nowhere near enough robotics data. Even GPT-2, which now seems comparatively weak, encountered more data during training than any current robot model can access.
When we started Octo, we overestimated the size of Open X-Embodiment, thinking it enormous. In reality, it remained tiny beside language datasets. Even at PI, collecting data around the clock, we had accumulated only about 10,000 hours at this point—a small fraction of the data available to language models. For comparison, a modern language training dataset corresponds to roughly 790 million hours under this conversion. I made a chart to illustrate the difference, and the disparity was startling.

This does not mean robotics can never obtain that much data, but it does mean we need to adjust our expectations. We cannot expect the same emergent properties seen in language models when we lack comparable data volume, diversity and state-space coverage.
Model Design
Although data is the most important factor, design matters too, especially as we scale. If robotics datasets grow over time, it remains valuable to explore suitable architectures and how they scale with additional data.
A major mistake in Octo was not using visual pretraining. This is closely connected to data: large internet datasets are an effective way to compensate for limited robotics data. We thought Open X-Embodiment was large enough, so we pretrained only the language encoder, not the observation encoder. That proved wrong. Even with a comparatively large robot dataset, visual pretraining can bring substantial improvements.
Early internal experiments at PI showed that even simple ImageNet-style pretraining with a ResNet or ViT produced significant gains. This was not merely better generalization or resistance to distractors: it improved performance on tasks within the training distribution. Pretraining helped much more than we expected.
Another problem concerned the action head. We initially made it very small—less than 5% of the total parameters—so replacing it for a new action space would discard little information. That became a limitation, especially with diffusion-based objectives. As the data becomes more complex and multimodal, a larger decoder is needed to capture the distribution effectively. In Octo, the large transformer processes the inputs in one forward pass, while the action head, or decoder, runs repeatedly during denoising. A more powerful decoder is better suited to those complex distributions.
High-frequency control posed another problem, especially in increasingly popular ALOHA-style setups. These often called for action chunks of 50 or more steps, which Octo could not handle. The loss would not decrease; it could not even fit the data. After Octo’s release, I joined PI. In my first week, I proposed a new architecture similar to Octo but with a much larger decoder: roughly 30% of the parameters, and probably an even larger share of computation, although I did not calculate the exact proportion.
It remained transformer-based because we knew transformers worked well. The design drew inspiration from the Diffusion Transformer paper, which had been successful in computer vision. We also addressed visual pretraining by initializing the vision backbone from ImageNet training.
This architecture worked very well. In the π0 paper, it is called π0-small. It came after Octo and served as our smaller precursor for a long time before we scaled to the full π0 model. Its performance was exceptionally good. Interestingly, several other projects independently developed almost the same architecture, including ALOHA Unleashed and a DiT-based policy. Multiple teams arriving at a similar solution suggested that it was a strong design.
7. Scaling at PI: From Octo to π0
Scaling the Model and Improving Pretraining
Our next goals at PI were to scale the model and improve pretraining. ImageNet pretraining helped, but we quickly realized that a vision-language model backbone—the approach now called a VLA—would be more effective. The basic idea is to take a VLM and fine-tune it to output actions. This showed considerable promise.
I began with simple changes to turn our architecture into a VLA while retaining its diffusion-style objective and overall design. First, I replaced the large transformer encoder with a language model, Gemma. Second, I upgraded the visual backbone to a stronger model, SigLIP.
VLMs commonly use decoder-only transformers, combining visual and language inputs through attention mechanisms. Our model now resembled PaliGemma, which combines the SigLIP vision encoder with Gemma. PaliGemma is a Google model designed to be adapted to new tasks through fine-tuning.
Our initial attempt retained the same action decoder but replaced the encoder with PaliGemma. It worked reasonably well and could fit the data, but did not improve much over the previous small model. It also showed problems such as occasional loss instability. I tried fixes including freezing the encoder and warming it up more slowly, without much success.
I then realized the problem: Gemma is a decoder-only transformer, but we were treating it as an encoder. Its final layers are trained with an autoregressive objective to predict the next token, rather than specifically to provide rich representations. In the limiting case of perfectly solving that objective, the final layer would tell you the next word, rather than necessarily retaining all the contextual information needed for action generation. This is an architectural intuition, not a theorem that next-token representations cannot encode context.
Improving the Decoder
The next logical step was to let the action decoder access representations from every layer of the language model, rather than only the last. This made the overall system look more like a large decoder. Observations such as images entered the model, and action inputs were added as well.
A key feature was retaining a separation between the original encoder-like prefix and the action decoder. At inference time, we run the prefix once and the action decoder repeatedly during denoising. For efficiency, we use a KV cache to store the prefix’s keys and values. This greatly accelerates the decoder because it need not recompute attention for the entire prefix.
This was similar to ideas in Transfusion, a paper released around the same time. When I saw it, I thought: they had the same idea! Independent teams converging on a concept is often a good sign.
The new architecture performed much better. Generalization improved, especially when making use of VLM pretraining. It could recognize previously unseen objects and showed stronger robustness. Its diffusion-style objective also let it handle high-frequency control.
One major problem remained: convergence was very slow, with training taking much longer than expected. My hypothesis was distribution shift. The VLM had seen only certain kinds of inputs during pretraining, so introducing randomly initialized linear projections for new inputs could disrupt it during fine-tuning.
Addressing Distribution Shift with Action Experts
I proposed a solution inspired by mixture-of-experts architectures: use different weights for the new inputs. Conceptually, imagine a transformer sequence whose different token types use different sets of weights. Original inputs, such as PaliGemma’s vision and language inputs, use pretrained weights, preserving compatibility with what the model saw during pretraining. New inputs, such as robot states and noisy actions, use a new set of weights initialized from scratch, reducing the disruption caused by distribution shift.
This separation also improves efficiency. Because the components interact through shared attention, we can make the new action expert’s MLP blocks much smaller. That saves parameters, reduces training time and greatly accelerates inference, since the action expert runs repeatedly during denoising.
The results were promising. Training became faster, convergence improved substantially and performance increased. This architecture became π0, as described in our paper. The only difference was that the published diagrams looked much nicer.
Did π0 Work?
We trained π0 using multiple data sources: PI’s own robot data, Open X-Embodiment, and the internet-scale pretraining inherited from PaliGemma. We fine-tuned it across multiple robot platforms, demonstrating flexibility across embodiments and tasks.
The larger question was whether π0 could become robotics’ GPT-2: imperfect, but highly adaptable and useful. The results showed progress, but not yet a transformation on the scale of GPT-2.
One example of the continuing challenges was the microwave task. Interestingly, a model trained from scratch consistently outperformed the pretrained model there. We suspected that undesirable actions in the pretraining data had biased the pretrained model and caused it to get stuck during execution, whereas the model trained from scratch did not have that bias.
Robotics is complicated, and evaluations can be noisy. These results show that much remains to be done, particularly in understanding how to make full use of pretraining and post-training. Although π0 is imperfect, it can perform impressive tasks, many absent from pretraining, after only a few hours of fine-tuning.
References
The First Workshop on X-Embodiment Robot Learning, CoRL 2024.
https://www.youtube.com/live/ELUMFpJCUS0?t=16866s