Back to Robot Learning & Foundation Models

Robot Learning & Foundation Models

Inside RFM-1: An Interview with Covariant Co-founder Peter Chen

A March 2024 interview with Peter Chen on RFM-1, robot foundation models, real-world data, world models and commercial deployment.

Zha Dami2024-03-2516 min readWeChat original
Contents

March 2024 will surely be remembered as a milestone in the development of general-purpose robots. On March 11, the prominent AI robotics startup Covariant unveiled RFM-1, its first Robotics Foundation Model. I described it as the world’s first robot foundation model trained on real task data, and the model closest to solving real-world tasks. Editorial clarification: that was the author’s assessment at the launch, rather than an established global-first claim; earlier systems such as RT-2 and RT-X had already used real-robot data. Covariant’s own positioning was the first commercial Robotics Foundation Model. The next day, Sergey Levine and Chelsea Finn—respectively a student and an academic descendant of Covariant co-founder Pieter Abbeel—announced Physical Intelligence, or Pi, an AI robotics startup aiming to provide a common brain for different kinds of robotic hardware. For a sense of the importance of Abbeel, Levine and Finn in AI robotics, see my Map of Chinese Researchers in AI and Robotics. Soon afterward, North American humanoid company Figure released a Figure 01 demonstration, showing the robot smoothly performing tasks under human instructions with the help of GPT-4V. It caused a sensation across Chinese technology media.

It was hard not to wonder: had the moment for robot foundation models finally arrived? Progress in this field was changing our expectations by the day.

In June 2023, I wrote an in-depth profile, Covariant: The AI-for-Robotics Unicorn Co-founded by Three Chinese Researchers. At the time, it was the most comprehensive account of the company available on the Chinese-language internet. My own interest in AI robotics also began with Covariant. After the company released RFM-1 last week, I was fortunate to interview its co-founder Peter Chen, exploring RFM-1, which I described as the first model of its kind and the one closest to the real world. That wording expresses the launch-time assessment qualified above.

Original article outline

Contents

👇

1. An Overview of Covariant RFM-1

2. About Peter Chen

3. Inside RFM-1

Where RFM-1 Came From

Is RFM-1 Robotics’ ChatGPT Moment?

Competition in Robot Foundation Models

Understanding the RFM-1 Architecture

What Does Any-to-Any Mean?

Does RFM-1 Use Third-Party Language Models?

Why Generate Multiple Modalities?

What Data Does RFM-1 Use?

RFM-1 and Figure 01: Comparing Approaches

Accelerating Data Collection

The RFM-1 API

RFM-1 as a World Model

Where Should Embodied AI Entrepreneurs Focus?

Where Should Embodied AI Investors Focus?

4. A Vision Taking Shape

1. An Overview of Covariant RFM-1

Founded in 2017 and headquartered in California’s Silicon Valley, Covariant aims to build general-purpose AI that enables robots to see, learn and interact with the world around them. Its founding team came from OpenAI and Berkeley Artificial Intelligence Research, or BAIR. The four co-founders are UC Berkeley robotics and AI researcher Pieter Abbeel and three of his doctoral students: Peter Chen, Tianhao Zhang and Rocky Duan. For a more detailed history, see Covariant: The AI-for-Robotics Unicorn Co-founded by Three Chinese Researchers.

Inside RFM-1: An Interview with Covariant Co-founder Peter Chen · Original article illustration

In March 2024, Covariant announced RFM-1, a robot foundation model trained on general internet data alongside rich data from interactions with the physical world. TechCrunch described it as ChatGPT for robots. RFM-1 combines a world model for understanding our physical environment with the ability to interact with that environment through modalities such as human language and images. Its release marked an important step in incorporating large amounts of real robot task data into foundation models and extending AI into the physical world. Date clarification: the Chinese overview says March 13, while its opening says March 11. This paragraph retains the month rather than silently resolving those inconsistent dates. Its description of a first incorporation of robot task data is also the author’s launch-time assessment, subject to the earlier-model clarification above.

Video demonstration from the original article

RFM-1 introduction video. Source: Covariant’s website.

RFM-1 is an eight-billion-parameter transformer, trained as an any-to-any sequence model on text, images, video, robot actions, sensor information and other modalities. It tokenizes these modalities into a common space and uses autoregressive next-token prediction. Its broad range of inputs and outputs supports many applications. For example, it can perform image-to-image learning for scene analysis tasks such as segmentation and recognition. It can combine text instructions with image observations to produce the desired grasp or movement sequence. It can pair an image of a scene with an image of a target grasp to predict the resulting video, or simulate numerical sensor readings that might occur along the way. We explore what any-to-any means in the interview below.

Video demonstration from the original article

An introduction to any-to-any modeling. Source: Covariant’s website.

2. About Peter Chen

At the time of this March 2024 interview, Peter Chen was CEO and co-founder of Covariant.ai, a leading AI robotics startup that had raised more than $200 million. Covariant was building foundation models for robotics, enabling robots to perceive, reason about and manipulate their physical environment. Before founding Covariant, Chen was a research scientist at OpenAI and a researcher at UC Berkeley’s BAIR Lab, focusing on reinforcement learning, meta-learning and unsupervised learning. He had published more than 30 papers in leading academic venues, with over 20,000 citations.

3. Inside RFM-1

Where RFM-1 Came From

A robot foundation model was one of the central ideas behind founding Covariant. Three of our four co-founders came from OpenAI: myself, our CTO Rocky Duan and our chief scientist Pieter Abbeel. One of the main reasons we started the company was our realization that general-purpose intelligence in robotics requires collecting data in the real world. From the beginning, Covariant therefore worked to deploy real robots at customer sites and collect data at scale in actual operating environments. From a data-collection perspective, the idea behind RFM-1 goes all the way back to our original thinking when we founded the company.

The specific architecture—a transformer-based, any-to-any multimodal model—is something we have explored over the past year or so. As large language models and multimodal foundation models advanced, our original vision became increasingly achievable. RFM-1 is what we have been dreaming about for six years; it was only during the past year that we settled on the architecture and methods to make it concrete.

Is RFM-1 Robotics’ ChatGPT Moment?

That depends on how you define a ChatGPT moment. If you mean whether robotics will develop rapidly in the coming period, the answer is definitely yes. If you mean whether progress in robotics will draw on and adopt technologies and methods like those behind ChatGPT, I also think the answer is yes.

But if you are asking whether robot intelligence today is already as general-purpose as ChatGPT, I do not think we have reached that point. That situation will change quickly, though.

Competition in Robot Foundation Models

I think the founding of Physical Intelligence, or Pi, which you mentioned earlier, is exciting. The potential robotics market is enormous. Robot foundation models could operate at a scale of billions: billions of robots powered by these models. In a market that large, it is good for everyone to have more capable companies succeed in the field.

At this stage, however, Covariant has a clear lead. The ability to adapt across settings and hardware is essential for any robot foundation model, and we have accumulated extensive experience doing that. For example, before leaving Google, Chelsea Finn and others created Open X-Embodiment, the dataset associated with the RT-X project. They brought together more than 20 robotics laboratories around the world and collected roughly one million trajectories from 22 different robot embodiments. In actual production environments, Covariant has already deployed more than 30 different types of robot. In other words, the range of robots deployed by our company alone exceeds the range represented across that entire dataset.

RFM-1’s training data is therefore not restricted to one kind of hardware or one setting. A truly excellent robot foundation model needs to work across different hardware, tasks and environments, so its training dataset must also cover different hardware and settings.

Understanding the RFM-1 Architecture

In simple terms, you can think of RFM-1 as a large language model, except it does not predict only language tokens. Its tokens also represent robot actions, sensor data, images, video and other modalities. Each modality has a corresponding tokenizer that converts it into tokens. Ultimately, the model works much like an LLM, but its inputs and predicted outputs are not limited to human language. That is the simplest way to understand its input and output structure.

What Does Any-to-Any Mean?

Any-to-any is one of the reasons RFM-1 is so powerful. Traditional AI robotics models usually take images as input and produce actions, or take 3D data as input and produce actions. Those models have substantial limitations. A major advance with RFM-1 is that it can act as a world model and predict what will happen next. Ultimately, everything it does is next-token prediction: the next action, image or video simply corresponds to a different kind of token. Given the right prompt telling it what you want, it can make the corresponding prediction. We can train it to learn a world model, understand the physical world and use that understanding to produce better robot actions. Its understanding of the physical world can also serve as a simulator, allowing us to avoid undesirable actions and plan within that simulator, in a way analogous to AlphaGo’s planning.

You can think of RFM-1 as a very general-purpose model. Unlike conventional AI robotics models, it is not designed for a single use. We introduced the term any-to-any because there are so many possibilities. Text-to-text is a conventional language model. Image-to-robot-action is easier to grasp: that is a robot policy. Text-to-image-to-policy means a person communicates with the robot through language, and the robot decomposes the language task and uses relevant task images to carry out the actions. You could even provide the current image and specify a target image, then have the model output actions. The essence of any-to-any is that it can accommodate different input and output modalities.

It is admittedly an abstract concept, so we provided concrete examples in our blog to explain it. As we deploy RFM-1 with customers, we will keep releasing capabilities that make any-to-any easier to understand intuitively.

Does RFM-1 Use Third-Party Language Models?

It does not call any third-party APIs. RFM-1 is itself a multimodal foundation model that understands text and can respond in language. You could even use it purely as a large language model. Of course, it would not be as effective as ChatGPT for that purpose, because much of its parameter capacity is devoted to understanding robotics.

Why Generate Multiple Modalities?

As a robot foundation model, RFM-1 is not satisfied with producing only actions. Its ability to output multiple modalities—the any in any-to-any—is intended to make it more general-purpose.

If we look at RFM-1 only as a policy, it maps different inputs to actions. You might give it a natural-language instruction, a goal, or even a video demonstrating a task, and ask the robot to learn and execute it. A general-purpose model should understand what happens in the video and be able to reproduce those situations. Yes, its output is still actions. From that perspective, there is nothing especially unusual about it.

But if I want the model to understand video better, or even learn a world model, a very effective approach is to give it the first half of a video and ask it to predict what happens in the second half. That is video input and video output. After extensive training of this kind, it develops a deeper understanding of video and the physical world, which it can also use when executing actions. Another application is to use this world model as a simulator.

What Data Does RFM-1 Use?

RFM-1 uses internet data and real robot data from Covariant deployments at customer sites around the world. Internet data provides common-sense understanding of text and images, but it is generally insufficient for a world model because it tends to lack action information. When you watch publicly available videos, for example, the underlying action information is usually missing. Whether the video shows human behavior or a robot operating, you are observing passively without knowing the specific actions involved. Real robot data is crucial because it contains extensive information about robot actions and their outcomes. You cannot obtain that detailed action information from internet data, which shows states before and after an event but does not capture the action details in between. Here, “action information” means recorded robot control commands and physical sensor quantities, not the absence of visibly observable movement in video.

For example, watching someone pick up an object lets you infer that they are doing so, but it does not tell you how much force they applied, where their fingers were placed, or exactly how they grasped it. Those details are not available in ordinary internet data.

Internet data can therefore train video, image and text generation models, but learning a complete world model from it is difficult because action information is missing. Covariant’s deployed robot hardware provides large amounts of sensor and joint data for training. We can also change how we collect data when necessary, because we control the hardware. If we discover that a particular modality is more useful, for example, we can update the robots already deployed to collect it.

RFM-1 and Figure 01: Comparing Approaches

Background: the Figure 01 approach connects a vision-language model such as GPT-4V to a robot action model. RFM-1 takes a more end-to-end approach, directly mapping multimodal inputs to action outputs.

Ultimately, I think the key is that any technical approach is a good one if it can be deployed and deliver value. Success is possible in different directions. A robot’s success ultimately depends on whether it benefits customers in production environments and whether the technology can scale across multiple settings.

With approaches that connect multimodal foundation models such as GPT-4V or Gemini to action models, demonstrations can work perfectly well. But the largest gap between a lab demonstration and a robot usable in production lies in reliability and performance. At this point, even GPT-4V, which I consider the best multimodal model available at this point, has high inference costs and reliability problems. Consider a crowded scene: put ten apples in a basket, or bunch three or four pieces of clothing together, and ask GPT-4V to count the objects in the image. Our tests show that its success rate is not very high. If the higher-level decision made by GPT-4V is wrong, subsequent low-level control cannot deliver the required reliability.

Accelerating Data Collection

First, we are scaling rapidly ourselves—not only by acquiring new customers, but also through rising order volumes from existing customers. If we can handle all those orders, the rate at which we collect data will increase by more than an order of magnitude.

Second, as RFM-1 matures, we plan to make it available to other robotics companies, following a model similar to OpenAI’s GPT APIs. Opening it to more people will let us accelerate data collection.

The RFM-1 API

Covariant plans to release a public RFM-1 API in the near future, suitable for established robotics companies and startups alike. In the LLM world, if I want to build something involving natural-language processing, I would not train a separate model myself. I would use OpenAI’s API, perhaps with prompting, fine-tuning, or retrieval and generation, then build my new application on top of it.

We want to provide brains for the millions, tens of millions, hundreds of millions and eventually billions of robots that will exist. This is about more than a single robot application or just hardware. Many robot developers and companies will connect to our API, and we hope to become their GPT platform. They can rely on us to solve most of the difficult, data-intensive intelligence problems, but that does not mean they have nothing else to do. They still need to develop hardware, design human–robot interaction and understand their application settings precisely. We hope to support them in doing that.

RFM-1 as a World Model

RFM-1 is a world model in the full sense of the term. Sora has the potential to become one, but is not one yet, because it lacks action data. If Sora were trained only inside game engines, without real-world video, the physics it learned would be biased: the physical behavior in game-engine-generated video generally differs from the real world. These are the interviewee’s March 2024 judgments. “World model” here emphasizes action-conditioned physical prediction; the term has broader definitions elsewhere, and this is not an assessment of later Sora versions.

If Sora were trained only on YouTube, with real videos comprising its entire training set, it should learn some of the physical world’s operating principles. But what is missing? A crucial concept in the middle: understanding actions. It has videos, perhaps with long descriptions generated by GPT, but from those alone I cannot identify the specific agent and the actual action information behind the video.

For example, we can see a person punch a wall, but we cannot determine how much force was applied. Whether the person’s fist is injured or the wall breaks, the model can only describe the sequence of events: the fist touched the wall, then the wall broke. It does not understand the underlying action process. Sora therefore has the capacity to become a world model, but its lack of action understanding makes it difficult to learn the causal relationship between actions and outcomes. It should still learn plenty of related information, but that information may not be sufficiently accurate.

Where Should Embodied AI Entrepreneurs Focus?

Foundation models are similar across fields: ultimately, only a few companies are likely to emerge as winners. If you do not think you can become one of those winners, whether in robotics or another field, you should consider the application layer or tools. This is an excellent time to enter robotics through those areas.

As machine-learning foundation models mature rapidly, progress in robotics will accelerate too. Regardless of what others are doing, we can see robot foundation models advancing extremely quickly within our own company. Hardware will also mature rapidly. Companies such as Figure are attracting investment, and investment in humanoids, mobile robots and robot arms in China is increasing as well. That will help hardware develop. As more products launch and companies communicate their progress, public acceptance of robots will rise quickly. It is a very good era for developing robotics applications and building infrastructure. Focusing on those areas is a sensible choice.

Where Should Embodied AI Investors Focus?

At this moment, robot foundation models and the tools and infrastructure around them may be the most worthwhile areas to watch. But I think that could change quickly within a year. A year from now, robotics could offer many more opportunities.

From another perspective, investing in an application company now could make sense if you believe it can get through an initial one or two years of learning. Look at application companies in the language-model space, such as Character.ai and Perplexity: they began building applications before ChatGPT launched, when the underlying foundation models had not yet fully matured. They iterated quickly and gained experience building on evolving foundation models, allowing them to break through rapidly. If we take a longer view and start experimenting and learning this year, I do not think it is too early. Teams that cannot achieve commercial traction quickly may face fundraising difficulties. But with a sufficiently strong team and market, starting application-layer experiments this year is feasible.

4. A Vision Taking Shape

After the interview, I reread my earlier article, Covariant: The AI-for-Robotics Unicorn Co-founded by Three Chinese Researchers. I was struck by how closely the founding team’s original vision, described in their account of starting out in 2018, aligned with the launch of RFM-1 in 2024. I will close by quoting Pieter Abbeel’s words again: Chronology clarification: Covariant was founded in 2017; the quoted 2018 passage describes the beginning of its real-world data-collection journey.

“Two important factors drive rapid progress in artificial intelligence: the experience available for AI to learn from, and research breakthroughs in AI architectures.

The key to experience is that robots must learn in the real world. They must interact with the limitless variety of objects that humans encounter and perform the limitless range of tasks that humans carry out.

The key to architecture is that real-world data is more diverse than laboratory data. We need fundamentally new architectures capable of learning from that diversity.

So, in early 2018, we began a journey to expose robots to the real world and research new AI architectures that could absorb those experiences—experiences far richer than those of a typical laboratory.”

References

https://covariant.ai/insights/introducing-rfm-1-giving-robots-human-like-reasoning-capabilities/

https://covariant.ai/insights/rfm-1-a-world-model-that-understands-physics/

https://techcrunch.com/2024/03/11/covariant-is-building-chatgpt-for-robots/

Smart Note · Robot Learning & Foundation Models