For our online forum, Embodied AI: How Artificial Intelligence Is Leading the Robotics Revolution, we were delighted to invite four outstanding entrepreneurs and researchers to discuss the field's most pressing questions: Peter Chen, cofounder and CEO of Covariant; Xiaolong Wang, an assistant professor at UC San Diego; Huazhe Xu, an assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences; and Qiuguo Zhu, founder of DEEP Robotics. I have edited the conversation into the transcript below. To hear the guests directly, search for SOTA! on Xiaoyuzhou, or use the audio link below. Their introductions and forecasts reflect the time of the original conversation, published in January 2024.
Podcast: Embodied AI Forum (listen in the original WeChat article)
Original article outline
Contents
↓
Introducing the guests
What drove the embodied AI boom in 2023?
Model-based versus learning-based methods
How can we make robots reliable?
Safety
Prospects for deploying general-purpose robots
1. Introducing the guests
Peter Chen: My name is Peter; my Chinese name is Xi Chen. I am the cofounder and CEO of Covariant. During my PhD at UC Berkeley, I focused primarily on reinforcement learning for robotics, while also working on generative models. After my PhD, I joined OpenAI in its early days, around 2016, when it may have had only about ten people. I spent some time growing with the organization. At the end of 2017, I left to start Covariant with my PhD adviser, Pieter Abbeel, and two other cofounders.
Our reason for starting Covariant was straightforward. At OpenAI, we could already see the trend toward large foundation models, and believed they would also be the future of robotics. We thought that building a good robot foundation model required owning our data and developing robot applications that actually worked, rather than technology confined to a lab. We founded Covariant in 2017, more than six years ago now. What has happened largely matches our expectations then: the foundation-model approach has been strongly validated in AI development, especially in language and images, where progress has been much faster than expected. That also creates many opportunities in robotics.
Huazhe Xu: I am Huazhe Xu, an assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences. Before that, I was a postdoctoral researcher at Stanford, and did my PhD at UC Berkeley. My research concerns today's topic, embodied intelligence—more specifically, dexterous manipulation, tactile perception, and using reinforcement learning and imitation learning to improve generalization and learning efficiency.
I would like us to explore the future of intelligence together, bringing academic and industry perspectives into the conversation and asking whether we can turn intelligence into practical applications. At this moment, we all hope to see these technologies reach real deployment.
Xiaolong Wang: I am Xiaolong Wang, a professor at UC San Diego. I did my PhD at Carnegie Mellon and then a postdoc at UC Berkeley. During my PhD, I mainly worked on computer vision, gradually moving toward reinforcement learning and robotics. At UCSD, I established a robotics research group. We work across robot vision and control, trying to connect these areas.
Our lab's research broadly divides into vision and robotics. In robotics, we focus on learning and low-level control for dexterous hand manipulation and legged locomotion. I think progress in these areas depends heavily on hardware development. Chinese companies such as DEEP Robotics have made impressive advances in hardware, and our research benefits from the platforms they provide. In the future, I hope to see progress not just in hardware, but also in developing hardware and software together. I believe China has considerable advantages there.
Qiuguo Zhu: I am Qiuguo Zhu from Zhejiang University, where I completed my undergraduate, master's and doctoral studies. I primarily worked on humanoid robotics, especially model-based control for legged locomotion, including MPC, or model predictive control. After my PhD, I noticed substantial changes in the field. While supervising students, we also began experimenting with reinforcement-learning-based control and made new progress. Editorial clarification: MPC specifically means model predictive control, rather than being the acronym for all model-based control.
I founded DEEP Robotics in 2017. At the time, Boston Dynamics was advancing rapidly, whereas getting robots to operate outdoors was still relatively difficult in China. We decided to establish a company developing products for industrial applications. We also work on educational products and research platforms. One challenge we face now is applying learning-based methods stably and reliably in industrial settings.
2. What drove the embodied AI boom in 2023? Have the underlying technologies made clear breakthroughs?
Peter Chen: AI and robotics are a huge subject, which we can divide by application—for example, manipulation centered on robot arms, or robots primarily concerned with locomotion. These applications have similarities as well as differences. AI addresses lower-level problems such as motion control, alongside higher-level questions such as how to give a robot an abstract task description and have it complete the task autonomously. That covers a wide range of problems. Our focus is arm-based manipulation, so I will answer from that perspective.
From our perspective, there have not been many fundamental hardware breakthroughs in recent years, especially in manipulation. More of the breakthroughs have come from models and intelligence. There are two aspects. One is solving relatively low-level control problems: issues that previously required model-based methods can now be addressed with learning-based methods.
The other is higher-level reasoning, connected to large language models such as GPT. These models can understand abstract, even incomplete task descriptions and turn them into tasks a robot can execute. That is a capability we did not previously have, and high-level reasoning has made tremendous progress over the past year. Technically and academically, this is a major advance. But returning to why robotics has suddenly become so popular, I think much of it is simply that AI itself has become more popular, and robotics is a very tangible application of AI. It therefore attracts more attention, rather than the excitement being explained solely by numerous breakthroughs in underlying technology.
Huazhe Xu: I think the growing popularity of embodied intelligence is inevitable. Building intelligent robots has always been an ultimate goal of AI; when that goal still looked far away, fewer people paid attention. The main change now is the maturity of technologies across AI. During our PhDs, the field was divided into modules, with different groups handling object segmentation, video classification, natural language processing and so on. Now, a PhD student or even an undergraduate can quickly learn to use knowledge from these modules and undertake related research—for example, using SAM. Once AI reaches this level of maturity, the pursuit of intelligent robots naturally attracts attention.
Large models also give people conviction and confidence in the future. Although they are mainly used for language, they demonstrate the power of the Transformer architecture: it can absorb vast quantities of data and produce qualitative changes in capability. That invites us to imagine whether enough robot data might bring similar changes, as large models did for natural language processing. The current answer may not be a complete solution—whether we look at RT-1, RT-2 or RT-X—but these results at least give us something to believe in.
I differ slightly from Peter on hardware. There may not have been a fundamental technological breakthrough, but prices have fallen. The Franka arms our lab initially used could cost more than RMB100,000. Now, some Chinese manufacturers sell arms with a two-kilogram payload for RMB8,000–9,000, although they do not match Franka in precision or force feedback. With an RMB8,000 arm, I would even dare to run reinforcement learning in the real world. That is an important change for online RL. Lower hardware costs greatly reduce the barrier to experiments on real robots.
Xiaolong Wang: Much of the excitement comes from successes in vision and language, which make people eager to apply those successes to robots. Robots have to go from perception to action; this is a very intuitive and reasonable direction. A few months ago, I asked Pieter Abbeel why he was now working on NLP. He said, 'everything is the same': language and robotics are both AI, and AI's maturity brings these fields closer together. But a robot is a complex hardware-and-software system. Many challenges remain before it actually works. Transferring AI directly to robots is still difficult.
I also think the successes in vision and language make people want to bring them into robotics. But there are many hidden pitfalls in this complex hardware-and-software system. Over the past few decades, robotics has progressed less dramatically than computer vision. Perhaps one reason is its focus on impressive demonstrations: many papers or projects aim to show a beautiful demo. That matters even more in today's AI boom, as increasingly sophisticated demonstration videos attract people more easily. But whether the system really works behind that polished demo is a question worth asking. We need more research into what is actually happening, rather than pursuing appearances alone.
In low-level control, we really are seeing learning-based methods perform increasingly well. Problems that traditional control struggles with can now be handled better through learning. There are positive signs in simulation development and in the wider availability of hardware and software. Companies such as DEEP Robotics have lowered the price of quadruped robots, allowing more people to buy similar platforms and run comparable experiments. That makes apples-to-apples comparisons more feasible in robotics research. Hardware progress and lower costs let us accumulate more knowledge and make further incremental advances. It is an exciting direction.
Qiuguo Zhu: I want to emphasize Xiaolong's point about demos. When demonstrations look so impressive, they attract considerable attention, especially from investment firms worried about missing the wave. Those investors then rush in. That may be the main reason robotics has become so popular.
From an industry perspective, Tesla's humanoid robot, for example, has drawn global attention. People are asking what humanoids might do in the future and where their intelligence will come from. Technologically, the arrival of ChatGPT leads people to imagine whether combining AI and robots could answer these questions. That scope for imagination drives the industry. There are many technical demos, and industrializing them will take time, but I believe it is a viable path worth attention. Reinforcement learning is another key direction: we see rapid iteration in using it to improve robot skills. We are increasingly experiencing how learning methods let complex robotic systems accomplish tasks that traditional control could not. For activities such as walking, where every movement need not be prescribed with exact precision, this expands where robots can operate. Combining large and small models creates more possibilities, both industrially and technically. That is my personal view of why robotics is so popular.
Peter Chen: In the past, relatively simple AI problems—segmentation or basic semantic analysis, for example—could take a great deal of time to solve. Now you can use a model such as SAM much more easily. I think large language models and models like SAM have already been trained on many tasks, making new problems relatively easy to address. That is part of what foundation models offer. It is especially meaningful for robotics, because a robot must solve many small, tedious AI problems. Existing foundation models reduce that workload. This does not necessarily mean doing something previously impossible; it means doing things we could already do much more easily. For robotics, that is a very substantial improvement. Editorial clarification: SAM is trained for promptable segmentation; it can support varied downstream applications, rather than being trained separately for every task mentioned here.
Xiaolong Wang: As someone with a computer vision background, I have to discuss SAM. There are misconceptions in treating it as a visual foundation model. Its dataset contains a huge number of annotated masks, yet training on around 10% of the data can produce nearly comparable performance. To me, this suggests that localized segmentation does not require that much data and is actually a relatively simple problem. That may point in the opposite direction, rather than proving the foundation-model premise. Many people also may not realize that SAM is not especially strong at tasks such as semantic and instance segmentation. I think it offers a new approach, but has not fundamentally solved many problems. One advantage is its accessibility: people can use it directly. But it certainly is not the best method for many problems, and that deserves attention. Editorial clarification: these are Wang's interpretations. SAM's paper reports a 1-million-image versus 11-million-image ablation with comparable performance on its evaluated metrics; this does not establish that additional data is universally unnecessary. SAM provides promptable masks and does not by itself assign semantic class labels.
Peter Chen: SAM is not a complete foundation model that solves every problem in vision. As Xiaolong says, it has limitations in segmentation and in combining with other modalities. At least in the original paper, text-based segmentation, such as combining it with CLIP, was not particularly strong. What I mean is that SAM offers a very extensible representation. Later work, such as LISA, combines a large language model with SAM and shows that some of these limitations can be overcome to solve problems effectively.
I agree that it is not the ultimate visual foundation model. But it suggests what might happen if I train beyond a single task. Localization plus segmentation is an incomplete representation, yet still very flexible. If you look at work combining large language models with SAM, a small amount of training can give it a variety of additional visual capabilities found in other models. That is powerful in itself. Editorial note: the Chinese transcript calls these 'convolution capabilities'; the discussion concerns added visual capabilities, not a claim that SAM changes into a convolutional architecture.
Xiaolong Wang: In manipulation, grasping does not always require extreme precision, especially when a multifingered hand grasps an object. Effective grasping can still be possible without highly precise control. Multiple contact points can make the problem easier, allowing a relatively rough grasp to succeed. We see interesting examples with soft objects such as towels: a multifingered hand can grasp a towel more easily than a gripper. So I see similarities between grasping and locomotion. Both can be improved through learning and better models.
3. Learning-based versus model-based methods
Xiaolong Wang: I believe large-scale real robot data is certainly useful, but I also want to promote simulation data and sim-to-real methods. Running manipulation at scale in simulation is itself a form of large-scale data generation. This is something the robotics community has underexplored, perhaps because people generally find the sim-to-real gap too difficult. They feel that starting in simulation and then moving to reality is too hard to solve real problems, and much less effective than collecting data and making changes directly in the real world. But I think we can gradually invest more effort in improving that situation.
Huazhe Xu: If I had to bet ten yuan, I would definitely back learning-based methods. AI's development has shown us a trend: data always beats human ingenuity—or ingenuity in using data always exceeds ingenuity applied to solving a problem directly. We went from being unable to play board games well, to search methods for chess, and then to Go; all of these were upended by reinforcement learning trained at scale in simulated environments. As Xiaolong just said, large-scale interaction in simulation is also large-scale data. Through RL, accumulated data has solved problems that once seemed unassailable. Editorial clarification: this is Xu’s argument for learning and scale. Historical chess programs did not all use reinforcement learning; systems such as AlphaZero combine learned policies and values with search, rather than eliminating search.
Our challenge now is embodied intelligence in robots, and I believe the eventual solution will be learning-based. Of course, 'model-based control' here may mean models in the narrower sense: modeling a robot using human knowledge of physics or mechanics. In the broader sense, a control model can itself be learned. While model-based methods emphasize data efficiency, we can also learn the model and make it more intelligent.
When there is no data, model-based methods can get a system working well enough to solve some problems. Then the collected data can improve it through learning. It is like a person becoming more intelligent while growing up. Insisting on traditional model-based methods may require substantial human effort, whereas learning-based methods lend themselves more readily to scaling.
Peter Chen: First, I do not agree that model-based and learning-based methods are fundamentally opposed. To me, the more relevant distinction across AI is between data-driven methods and explicitly programmed methods. Traditional control or optimization algorithms can be considered programmed approaches: people use their understanding of the world to implement some form of intelligence.
As for whether data-driven or programmed intelligence will matter more in future robotics, I would back data-driven intelligence, as Huazhe does. I would raise the stakes, potentially without limit, especially over a longer time horizon. But these approaches are not mutually exclusive. A commercial robot system can combine them. There is no requirement for the entire system to be data-driven or entirely programmed; we can integrate them flexibly.
For me, this is more a question of products and engineering. As Huazhe said, data-driven learning requires a sufficiently large dataset, and also faces constraints such as response time and power consumption. We think the best approach is to decide which parts use data-driven intelligence and which use programmed intelligence according to the product's needs and engineering constraints. A sensible architecture assigns these roles well, so that increasing compute and data can gradually move more of the system's intelligence toward learning. That can make it more flexible, capable and reliable. I do not see this as a philosophical opposition.
Qiuguo Zhu: I strongly agree that model-based and learning-based control are not opposed. They are more like a progression in capabilities and algorithms. We should look at this from the product perspective. Choosing a method depends on the system and application. For example, before the current AI wave, industrial systems and even rockets operated using traditional models. Rockets launched by Musk's company may still use conventional models for control.
Peter Chen: Indeed, their landing control uses more traditional model-based control methods.
Qiuguo Zhu: In process-industry control, many applications make it difficult to collect large quantities of training data. In those cases, traditional control algorithms and traditional fault-diagnosis methods may still be appropriate.
In robotics, learning may be the more promising trend, so we need to distinguish the systems and settings to which each method applies. I do not regard learning-based and traditional methods as opposed. To be clear, I have actively embraced and joined this trend, because we have experienced the problems and limitations of traditional models. Better learning-based methods can produce useful results. When choosing an approach, we should distinguish the application, environment and product. Personally, I believe the future direction of robotics will certainly involve AI and learning-based methods.
4. How can we make robots reliable?
Qiuguo Zhu: From a product perspective, ensuring stability and reliability in what we deliver to customers is essential. Even after doing that work, however, reaching 99.99% reliability is not certain—even with model-based methods. Product development has a series of standards and procedures, including hardware and software testing, factory acceptance tests and reliability testing over long-term use. Where did these practices come from? They reflect experience accumulated since the Industrial Revolution. Following them helps us manufacture high-quality products.
With learning-based methods, as Xiaolong just mentioned, there may be opaque behavior and hidden pitfalls. From an application perspective, I do not think this is mature yet, though there may be deployments in limited domains. At least for now, it remains challenging.
Peter Chen: Stability and reliability are indeed important, especially when turning an impressive demo into a product that creates value for customers. One of the biggest challenges is achieving 99.99% reliability. But learning-based methods did not create this problem. Often, you have to use them because model-based control or other traditional programmed approaches simply cannot solve the problem.
The real world has enormous diversity. Without sufficiently adaptable intelligence, model-based control or simpler vision models may not even exceed 90% reliability. So learning is not the obstacle to high reliability and success rates; it is actually one of our most effective tools. There is no simple recipe for achieving this, but I can share approaches we have found useful. One is transfer learning. An individual customer's dataset may be limited, but combining data across customers gives you a larger dataset. In machine learning, more data often improves generalization and can therefore improve success rates.
Another important principle is not to treat the product as a one-off delivery. AI-driven robotics has to be a system that keeps updating and learning. Data collected from problems encountered in production can support continued model training, allowing improvement and learning over time. These two directions still require a series of technical advances, of course. But in our experience, implementing them successfully makes reliability achievable through learning-based methods.
Huazhe Xu: Reliability is indeed one of the biggest challenges for learning-based methods. From a product perspective, it is crucial that robots be stable and reliable when delivered to customers. But when no other method can solve the problem, we may have to tolerate some risk of unreliability. For a home robot, for example, traditional model-based algorithms or programmed methods may not solve every task, while learning could provide a viable solution.
We need to consider the scope of the task and environment. Think of smaller and larger circles of problems: traditional control can solve the smaller circle, whereas the larger one may require more flexible learning methods. In my experience, problems that model-based methods can solve can also be solved with learning if enough data is available, though that comes with costs.
Learned models face the black-box problem, particularly neural networks. Our current approach is to give them large quantities of data, whether collected in reality or simulation. With that data, iteration can make them increasingly capable, somewhat like the patience we have with people as they learn.
Xiaolong Wang: I agree with Peter about reliability. It is not a problem unique to learning; it is shared across fields. We have tools for evaluating it, such as studying uncertainty in classification, including confidence and the trustworthiness of an estimate. This is a widely explored direction in machine learning. Introducing safety constraints in AI safety and reinforcement learning is another approach to reliability.
In areas such as interactive object manipulation, safety is especially important. Robustness, risk and estimation are already major topics in learning. Reinforcement learning, natural language processing and computer vision have substantial research in these areas. Robotics is gradually adopting similar directions, though perhaps on a smaller scale. As technologies are applied and real problems addressed, uncertainty and risk estimation will become more mature. Editorial note: the original transcript uses 'IOM (Interactive Object Manipulation)'; it does not identify a particular paper or a standardized field acronym.
Overall, learning-based methods share common reliability challenges, but applications and research in each field still require deeper work, especially on robot safety and reliability.
5. Safety
Huazhe Xu: Although Xiaolong mentioned substantial safety research across fields, it has not become standard practice. For example, when using a vision model, we rarely see a default neural network with uncertainty estimation that leaves performance unaffected. An uncertainty estimate must first avoid degrading performance, and second, be trustworthy. The uncertainty predicted by a neural network can itself be uncertain. That is an unresolved problem, although some research and projects address it.
On safety, my view is similar to Xiaolong's: first build the system to a sufficiently capable state, then introduce uncertainty handling and safety considerations. Iterating in this way—developing quickly, then adding safety constraints where needed—may allow faster progress than placing many restrictions on a still-immature system. That is how I see the development process.
Peter Chen: There are several dimensions here: reliability, physical safety, and the safety of large language models, including alignment—ensuring that AI does not pursue intentions that cause harm to people or anything else. Different methods can address these dimensions.
First, for robot reliability and physical safety, we can use classical methods and programmed safeguards during interaction with the world. Collision detection, for example, can constrain decisions that would otherwise lead to unsafe situations. This is a typical form of programmed intelligence that can provide safety guarantees. Editorial clarification: such guarantees depend on the modeled conditions, sensing and the enforcement mechanism; collision detection alone is not an unconditional guarantee of physical safety.
For the longer-term questions of general intelligence, I do not think robots urgently need to focus on that kind of safety yet. Getting reliability and flexibility right is already very difficult. First, we need high reliability and stability on new tasks; then we can consider higher-level safety questions related to AGI. Editorial note: the original ends with the unclear phrase 'the singleness of AGI'; the transcript does not specify a technical safety criterion.
6. Prospects for deploying general-purpose robots
Huazhe Xu: In the short term, embodied intelligence will mainly be applied in complex settings, where the system must cope with varied objects and environments—for example, supermarkets, logistics within campuses or industrial parks, and production lines. Manipulation in these complex settings is especially well suited to its strengths.
We should distinguish embodied intelligence from robotics generally. Robots may be useful in a broader range of settings, while embodied intelligence stands out when a setting is complex. In the short term, applications may concentrate on narrow AI tasks, logistics and factories. Over the medium to long term, we hope it will enter homes, especially kitchens. Why not use squirrel-shaped mandarin fish as a ten-year benchmark? If a robot could make that dish for me—cutting the flesh into little squares without severing the skin, removing the scales, and frying it—I would feel that the future we want had arrived.
Peter Chen: I think embodied intelligence will first be applied in more industrial settings, then expand into more varied, less frequently used applications. Intelligence still needs a suitable body. Frying a fish, for example, may require a particular hardware platform, and the cost of that hardware matters especially in a home.
I differ somewhat on the timescale: I think it may be achieved sooner than ten years. Over the next two or three years, I hope that most problems solvable with existing hardware, but still lacking sufficient intelligence, can be addressed. I am optimistic about near-term progress in frequently used applications of embodied intelligence. These are my expectations at the time of this conversation.
Qiuguo Zhu: I am more concerned with traversing complex terrain, where legged robots can offer better mobility. I hope future robots will adapt better and move autonomously across different terrain without manual operation. Better traversal could benefit emergency firefighting, indoor and outdoor patrols, and exploration in mountainous areas. I expect significant breakthroughs over the next one or two years, bringing tangible improvements to robots. This is a forecast made during the original conversation.
Xiaolong Wang: Walking is very important for robots. We do not currently see robots walking around the streets; perhaps they fall too easily, or cannot open doors. The other guests have already made good points about these issues. I want to approach the question from another angle: society. Social acceptance of robots matters too. An interesting observation is that food-delivery robots are already common in Chinese hotels, but not in the United States, in my observation. Installing them requires modifying elevators, which may raise safety concerns. In the US, such concerns may be harder to overlook. This is my observation about deployment settings, rather than a universal comparison of regulations.
I think progress may be easier in the short term where people are absent; once people enter the picture, things become harder. Interaction with people is a key issue, and some companies focus on solutions with high levels of compliance. Good human–robot interaction is crucial. Even after we solve all the technical problems, social interaction can raise others. Here is an interesting example: a performance in which humanoid robots dance with actors could gradually change society's view of AI, making people more comfortable with robots' presence.
That concludes my edited transcript. It really is an article of more than ten thousand Chinese characters: the conversation was so compelling that I could not bring myself to cut much of what the guests said. I also host a podcast sharing frontier industry developments and academic research in AI and robotics. You can find it by searching for SOTA! on Xiaoyuzhou. This is the podcast name given in the original article.