In the Robot Foundation Model, or RFM, series, I summarized the VLA architecture represented by π0 in RFM, Episode 4: Comparing π0, Helix and GO-1, Three VLA Architectures. In RFM, Episode 1: Physical Intelligence and the π0 Embodied Foundation Model, I also introduced π0's overall approach based on Sergey's talk.
Today we introduce Hi Robot, a project Physical Intelligence announced in late February 2025. Its core idea is to let robots reason about complex instructions, feedback and interruptions. A vision–language model can be used to guide a robot, but what happens when an instruction is too complicated to understand and execute directly? Hi Robot addresses that problem. I am honored to interview Lucy Xiaoyang Shi, first author of Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models.
At the time of this interview, Lucy was a computer-science PhD student at Stanford, advised by Chelsea Finn, and a researcher at Physical Intelligence, or PI. She had previously worked with Yuke Zhu and Jim Fan at NVIDIA Research. She received her bachelor's degree in computer science from USC, where she worked with Joseph J. Lim and Youngwoon Lee. Her interests broadly cover robot learning, with the goal of building general-purpose robots that naturally perform complex, long-horizon tasks.
Readers often encounter publicity describing a startup as building a brain to rival π0. After reading some of those announcements closely, I am puzzled: the business being promoted seems to have little connection with π0. Before reading today's article, consider this question: do you understand a VLA such as π0 primarily as a high-level policy or a low-level policy? In the brain-and-cerebellum analogy commonly used in China, which role does it play?
Original article outline
Contents
↓
1. Introducing Hi Robot
2. How Do Hi Robot and π0 Differ?
3. How Do Their Instructions Differ?
4. Why Not Simply Connect a General-Purpose VLM?
5. Why Does Hi Robot Understand Instructions Better?
6. Addressing Grounding
7. Combining VLA with RL
8. The Hi Robot Method
9. Data Challenges for π0
10. Collecting and Generating Data for the High-Level Policy
11. Understanding the VLM Data-Generation Step
12. How Is Hi Robot's Dataset Built?
13. Is a Hi Robot-Style Module Necessary for VLA?
14. Is Hi Robot Necessary for Real-Time Interaction?
1. Introducing Hi Robot
Before the interview, here is an overview of Hi Robot based on PI's website. The motivation is to let a robot reason through complex tasks, as if a small voice were telling it what to do next. Hi Robot integrates a VLA such as π0 into a two-level reasoning process. π0 is the fast, action-executing System 1, or S1, while the high-level VLM acts as the slower System 2, or S2, handling reasoning and language understanding. Through a form of self-directed dialogue, it breaks complex tasks into simple intermediate steps for π0 to perform. These system labels are functional analogies.
An introduction to Hi Robot.
Hi Robot's high-level policy is a VLM using the same underlying vision–language backbone as π0. It is trained to process complex prompts, observe the scene and divide a task into subtasks π0 understands. It effectively whispers concrete instructions to the VLA, such as “Pick up a slice of whole-wheat bread.” The high-level policy can also incorporate feedback from the ongoing interaction.
A VLM pretrained on large-scale internet data is already good at producing text from images and linguistic context—for example, answering which object a robot should pick up next. Hi Robot uses that ability, giving the robot commonsense knowledge for interpreting what a person says in relation to what it sees. It need not depend entirely on its previous action experience; it can bring broader knowledge to the task.
2. How Do Hi Robot and π0 Differ?
π0 is a low-level policy developed by PI. Its central function is to generate robot-control actions directly: action trajectories that let a robot interact with different objects in reality. For example, it can manipulate a T-shirt and fold it, executing a complex physical operation. In designing π0, we especially emphasized
dexterity.
π0 also has some language-following ability: it can perform tasks from natural-language instructions. A user gives an instruction, and the system understands and executes it. But this understanding is limited by what the model has learned from its training tasks. If the user says “Fold the clothes,” the robot follows the procedures it learned, for familiar clothing types in similar settings. The language it handles is also comparatively closed and limited to the vocabulary or expressions represented in its training data. Here, “closed” describes the demonstrated task and instruction distribution, not a literal restriction of the tokenizer to those words or an inability to process any new phrasing.
Deployment exposes this limitation. In real environments, people speak naturally and openly rather than restricting themselves to the training vocabulary. An older family member might say, “Bring me that glass of water.” If that expression was absent from the model's training, the system may not understand it.
Hi Robot therefore aims to support open-vocabulary instructions. Users can express requests naturally, including unfamiliar wording and long, complex instructions. The key is a high-level policy that understands human language and converts it into an intermediate representation the robot can execute. A low-level policy such as π0 then generates the physical actions. The language interface is broader, while execution still depends on the robot's available skills.
The architecture offers three main advantages. First, it makes interaction more general, letting non-experts instruct robots naturally. Second, it provides language reasoning, enabling more complex understanding and transformation within language space. Third, it strengthens task generalization during training through knowledge sharing and transfer across tasks.
3. How Do Their Instructions Differ?
For example, to make a sandwich with π0, the instruction is typically a concrete operation such as “Pick up a slice of bread.” It needs to be an explicit, specific and structured action description.
In the demonstrated tasks, with Hi Robot, you can instead say, “I am vegetarian. Please make me a sandwich.” The system reasons that being vegetarian means it should not use the meat on the table. You can also say you are allergic to a particular ingredient, and it can interpret that constraint and exclude it from the preparation.
Making only a vegetarian sandwich.
In other words, Hi Robot understands more natural, complex and personalized requests and uses them to reason about suitable actions, rather than relying only on predefined action scripts. At the level of task and instruction understanding, it performs better than π0 alone.
Hi Robot and π0 are being developed in parallel. We also work on semantic understanding within π0—for example, improving annotation quality, especially label definitions and precision. At the same time, we refine the quality of synthetic data for Hi Robot.
The team is exploring how π0 and Hi Robot should interact and how each module should be designed to strengthen the overall system's semantic understanding and generalization.
4. Why Not Simply Connect a General-Purpose VLM?
For the high-level policy, we tried directly using large models such as GPT-4, particularly image-capable versions such as GPT-4V, which were among the strongest general-purpose multimodal models available at the time.
The problem is that they do not understand the robot's capability boundaries: what it can or cannot do. They lack grounding. Their spatial understanding is insufficient, and they struggle to maintain an understanding of the task over time. These observations concern the models and robot-control settings discussed in this interview.
Hi Robot is different. We have our own VLA and robot dataset, and can use VLMs to generate synthetic data more closely aligned with actual capabilities. Combining these resources lets us train a high-level policy designed specifically for embodied intelligence and robot control.
In our task-understanding and instruction-execution evaluations, it clearly outperformed directly using external general-purpose models such as the GPT-4 family.
5. Why Does Hi Robot Understand Instructions Better?
The central issue is grounding, a longstanding research topic. Large models can be very strong at mathematics or coding, yet still be unable to control a robot, interact physically with the world or carry out a seemingly simple command such as “Pick up the phone.”
They lack feedback from actually interacting with the world. Training on static data does not let them affect the environment through perception and action or build the same understanding of the consequences of their actions.
At the time of this discussion, even many state-of-the-art VLMs and VLAs remained weak in spatial reasoning.
We can think of grounding at two levels. External grounding concerns understanding the physical world in space and time: spatial relationships between objects and how interactions work. Internal grounding concerns the entity the model controls, such as a robot arm or mobile platform. The model needs to know what it can and cannot do, and what is likely to succeed or fail.
Without both forms of grounding, strong language and logic abilities do not readily translate into practical embodied behavior.
6. Addressing Grounding
This is still a very open research question. No one has a definitive answer. In my experience, it is fundamentally a question of both data and training methods. With enough high-quality data—especially real human–environment interactions containing clear feedback—a model may learn effective grounding.
Reinforcement learning is often suggested as a training method because it lets an agent actively interact with its environment. It can try one action, then try another, repeatedly receiving rewards or penalties and gradually learning the relationships between actions and consequences.
That mechanism is a prominent direction in embodied intelligence and robot control, and an important route being explored to address grounding.
7. Combining VLA with RL
One possible analogy is the progression seen in approaches such as DeepSeek: begin with a large-scale pretraining stage, then refine the model with reinforcement learning. For a robot policy, the initial stage could use imitation learning on a large dataset.
We are considering a similar idea for π0. Having completed an imitation-learning pipeline, could we place the model in a real or simulated environment and let the robot try things autonomously? It could obtain rewards or estimate a value function, then use that feedback to fine-tune the existing model. That is one possible route.
The reason RL has been used comparatively little with these models in embodied, interactive robot-control settings is not that no one studies it. Data and platforms supporting environmental interaction at the required scale are limited.
Compute is another practical constraint. A typical university laboratory has difficulty training a model of this size. There are not many places able to train these VLAs in the first place.
Even among institutions that can, many have not yet fully worked out the VLA training recipe at the imitation-learning stage. Most are still exploring that stage, where many relatively straightforward improvements can yield substantial gains.
RL remains a highly research-intensive direction. AlphaGo is an example of how impressive it can be when it succeeds, but making it succeed is difficult. It is a little like trading a very volatile stock: enormous potential, but hard to control.
When training RL models is expensive, organizations are more cautious about allocating resources. That is one reason many had not yet tried applying it at scale to these models at the time of the interview.
8. The Hi Robot Method
The paper describes a hierarchy of high- and low-level policies. The high-level policy receives open-ended natural-language instructions and images from base and wrist cameras, then produces low-level language commands and, optionally, a language response. The low-level policy combines the commands with images and robot state to generate physical actions. Editorial correction: the original introductory description attributes the optional language response to the low-level policy; the architecture and the explanation below place it at the high level.

We separate a high-level policy from a low-level policy. The high level resembles System 2: it does the thinking. The low level resembles System 1: it executes.
In the diagram, the images linked to the high-level policy are the robot's observations of the world: its cameras show what is happening around it. The user prompt and instruction can be whatever the person says—for example, “I want a sandwich with beef, tomatoes and greens.” Those images and that request are the inputs.
One output is a low-level language command: a concrete instruction for execution. The model might translate the request into “Pick up a slice of whole-wheat bread” or “Pick up a slice of tomato.” Another output is the robot's reply to the person. If you say you are allergic to something, it can acknowledge that and say it will leave the ingredient out of the sandwich. There is a verbal response as well.
The low-level command, such as “Pick up a slice of tomato,” goes to the low-level policy, π0, together with the images. It also receives robot-state information, including joints, observations and internal values such as its current velocity. Model inference predicts the subsequent action trajectory, which the robot then executes. The state examples here are illustrative; the input channels depend on the robot configuration.
The high-level policy can indeed be understood as the brain in this analogy, while π0 is closer to the cerebellum. We train it end to end, treating the perception-to-action mapping as one integrated model performing the S1 function rather than separate functional planning and execution systems. Editorial clarification: π0's implementation still contains a VLM backbone and an action expert. The interview's point is their integrated sensorimotor role, not the absence of distinct architectural components.
9. Data Challenges for π0
Training the low-level policy is difficult because its data must be collected through interaction with real hardware. First, enough data requires many robot platforms. Second, and more importantly, the data must be good. Quality has a major effect on training, and there is still work to do.
For projects such as π0, we have accumulated experience about what constitutes good data. We have worked on this for years and developed intuition about which data is valuable and which is inadequate.
We use that experience to actively control collection, improving collection strategies, environments and operating procedures to obtain data that is high quality and useful for training.
Quality depends on details throughout the system. Camera placement is crucial. We often see serious placement problems when companies collect robot data. A camera may be too far away to see what the robot is doing. Human operators can judge with their own eyes, but the camera's coverage and resolution may not support that. Or it may be so close that it misses the surrounding context. When the robot makes a mistake, it then lacks the information to recognize what went wrong or how to correct it.
We therefore usually use at least three cameras rather than just one. Their placement and coordination directly determine what information the data contains and what important information is missing.
A wrist camera is one effective example. It observes the manipulated object at close range, which is extremely important.
That is only one placement example, but it illustrates broader questions about coordination across robot configurations and the design of perceptual information, all of which affect data quality.
There are also differences between data collected by experienced teleoperators and beginners. We cannot employ only experts at large scale; there will inevitably be substantial novice data. How should we combine these sources and model their variability and instability? That is another important question in π0.
This is somewhat analogous to ChatGPT. Why did other models released around GPT-4 perform worse overall? I do not think the explanation is simply that OpenAI had ten times more data. Many teams draw on broadly similar volumes of internet data. This is an analogy about training practice, rather than a verified comparison of private training-corpus sizes.
The differences lie in countless details, particularly data processing. Perhaps a thousand details must all be right, with each one important. Cleaning, filtering, organization, weighting and formatting can profoundly affect training results.
Papers from organizations such as Google often devote considerable space to model architecture. But the final performance can also depend on deep, careful data-processing work whose details are not fully described in the paper.
Architectures attract attention, and sometimes changing the architecture really does improve performance by 20%. Researchers are therefore more interested in architectural innovations and more willing to discuss them in papers.
But data cleaning is often not described. Cleaning, selecting and organizing data may be what determines whether performance improves by 60% or 70%. Those details can be the real secret weapon.
10. Collecting and Generating Data for the High-Level Policy
The paper describes collecting teleoperated robot demonstrations and segmenting them into short skills, such as “Pick up a KitKat.” A VLM uses the labeled data to generate synthetic user instructions—“Could you get me something sweet?”—and robot responses. The resulting dataset trains the high-level policy to map visual observations and user instructions to language responses and skill labels.

Step zero is real-world data collection: control the robot to perform a task, such as shopping in a supermarket. It might navigate autonomously to a shelf, then be remotely operated for the manipulation. The original data is a long sequence, perhaps ten minutes for one example.
In step one, we divide that ten-minute sequence into clips a few seconds long, each corresponding to an action or task. We label them in natural language, describing what the robot is doing. A clip showing it picking up a piece of chocolate receives that action label.
An action description alone is not enough. A vague request such as “I am in a supermarket and want something sweet” does not directly tell the robot how sweetness relates to chocolate. It cannot reliably infer the task from such an ambiguous expression without learning that connection.
Step two supplements this with synthetic data. We give a strong VLM an image and a clear action label, such as “Pick up a piece of chocolate,” and ask it to imagine what a user interacting with the robot in that scene might say.
The user might ask, “Please buy me some sweet snacks.” We also ask the model to imagine the robot's response, such as “Sure, I will get you some chocolate.”
In step three, we combine the collected, labeled and synthetic data to train the high-level policy—the thinking part of Hi Robot. Its inputs are images and possible user requests. It learns to predict two outputs: how the robot should reply and which action it should execute.
11. Understanding the VLM Data-Generation Step
The generated content includes user instructions. Returning to the sandwich example, imagine the sandwich contains lettuce, tomato and cheese, but no cabbage or meat. The model can infer what kind of user intent that combination might express. It might imagine a request such as “I would like a sandwich without meat.”
Its ability is to infer plausible user requests from the current action and visual information, associating an ingredient combination with a semantic situation and establishing a correspondence between language and action.
We could collect actual human–robot interaction data, but that is expensive. One person must control the robot while another continually speaks—“I want this,” “I want that,” “Wait a moment,” and so on. Labor costs are high.
The strongest VLMs available at the time already have relevant capabilities from internet-scale training. Even if they have not seen human–robot conversations, they have seen abundant human–human interaction: how one person asks another for help, how the other replies and how they perform the request.
Even when a request is not explicitly shown, the model can infer what may have been said from the subsequent behavior. This lets synthetic data simulate human–robot dialogue without depending entirely on collecting real conversations.
12. How Is Hi Robot's Dataset Built?
Much of the foundation comes from data collected on actual robots. Learning to understand actions remains a supervised-learning process.
The model's ability to understand actions and instructions depends substantially on large amounts of manually collected and annotated data from real environments. That addresses the question raised later: much of its capability rests on earlier high-quality real-interaction data. At the time of the interview, we said Hi Robot might be open-sourced if time allowed.
13. Is a Hi Robot-Style Module Necessary for VLA?
I do not think so. It depends on the application. If a model does not need to interact with users or generalize to new settings, it may not need a high-level module such as Hi Robot at all. It may not even need a full VLA; a simple model might be sufficient.
The question is where it will be deployed. Does the setting require interaction? Does it require logical reasoning?
14. Is Hi Robot Necessary for Real-Time Interaction?
I can say that Hi Robot provides this capability, but it is not the only way. A future single large model might integrate all of these abilities. This remains an open-ended research question, and no one has a final standard answer.
References
Hi Robot
https://www.pi.website/research/hirobot
π0
https://www.physicalintelligence.company/blog/pi0
Lucy Xiaoyang Shi's Website