A while ago, a reader left a message through Smart Note's WeChat account: 'Could you do an episode specifically about CoPa, ReKep and VoxPoser—methods that perform robot tasks zero-shot, mainly through inference? I personally feel this may be the main route toward generality and generalization, although it still needs improvement for learning specific complex skills.'
We previously interviewed Yang Gao about the CoPa project mentioned in that message, in Conversation with Yang Gao: ViLa and CoPa for Embodied Foundation Models. The other two projects, ReKep and VoxPoser, have also attracted considerable attention from Chinese technology media. We were delighted to invite their first author, Wenlong Huang, to discuss this route of zero-shot robot manipulation through reasoning and inference.
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation is by Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang and Li Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models is by Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu and Li Fei-Fei. Wenlong Huang is first author of both papers.
At the time of this February 2025 interview, Wenlong Huang was a third-year computer-science PhD student at Stanford, advised by Fei-Fei Li in the Stanford Vision and Learning Lab, SVL. He earned his undergraduate degree in computer science at UC Berkeley, studying there from 2018 to 2021, working with Deepak Pathak, Igor Mordatch and Pieter Abbeel. He worked at NVIDIA's Seattle Robotics Lab in 2024 and interned with Google Brain's robotics team in 2022. This introduction preserves the interview's historical timeframe.
Original article outline
Contents
↓
Part I: An overview of VoxPoser and ReKep
1. What kind of approach do ReKep and VoxPoser represent?
2. How should we understand rewards and task representations at different levels?
3. Is this the same reward concept as in reinforcement learning?
4. Does this route have more practical potential than end-to-end learning?
Part II: How VoxPoser and ReKep work
5. VoxPoser's core idea
6. ReKep's core idea
7. How are the keypoint relationships established?
8. How should we understand temporal relationships?
9. Why is the clothing not folded neatly in ReKep's demonstrations?
10. How do we connect spatial information with an LLM?
11. How far is ReKep from practical applications?
Part III: Hierarchical models, end-to-end models and world models
12. End-to-end and hierarchical models
13. Learned dynamics and MPC
14. What is a world model?
15. Could large models become the world models robots need?
16. Should 3D data come from 2D observations or 3D scanning hardware?
17. Combining these methods with RL and imitation learning
Part I: An overview of ReKep and VoxPoser
1. What kind of approach do ReKep and VoxPoser represent within today's robot-manipulation research?
It may be clearer to understand robot manipulation from the perspective of data. A major challenge is deploying autonomous robots in the open world: homes, places where people are present, and other unstructured environments. We can divide the challenges and data sources for manipulation in these settings into several categories.
The first is real-world robot data, collected by physical robots in actual environments. It is the most direct and closest to the intended application, but usually difficult to obtain at large scale with rich diversity.
The second is simulation data, collected in virtual physical environments that reproduce different manipulation situations. It can be adjusted and expanded more flexibly than real data and used to train robots, particularly when real-world data is insufficient. But it faces discrepancies in both directions: real-to-sim reconstruction and sim-to-real transfer.
The third is internet-scale data, mainly involving humans as the acting embodiment and encompassing vision, language, video and other modalities. Examples include YouTube videos, data from VR environments and the data used to train existing large models. Its sources are broad, and its visual and language content offers rich semantics and common-sense knowledge. For physical interaction, however, it lacks precision and presents an embodiment gap between humans and robots.
These are the principal data sources in robotics. End-to-end imitation learning that receives an image and directly outputs a robot action typically relies on real-world data. End-to-end RL often relies on simulation because it needs an environment with which the policy can interact repeatedly. These describe common choices, rather than restrictions that prevent imitation learning in simulation or RL on real robots.
Many projects also use modular robot stacks to incorporate internet data or models trained on that data. My research—from earlier work on using large models for task planning to VoxPoser and ReKep—belongs to this category, as do related projects such as CoPa, MOKA and PIVOT.
The richness and diversity of internet data create opportunities for open-world generalization. But in a modular system, a key question is: in what form should that data, or the models trained on it, enter the robot stack? My research proposes that they supply task representations for manipulation. These can be high-level semantic representations, such as decomposing a complex task into subtasks, or lower-level representations in three-dimensional space and time, such as where a teapot and cup must be relative to each other and in what sequence during pouring.
To understand and define this mathematically, we can borrow a classical decision-making framework: the Markov decision process, MDP. It generally defines several elements.
First is the state, or the observation. They are different, but for simplicity I will call them 'state' here. A robot's sensors, such as cameras, observe its surroundings and measure its own condition. Together, these provide the information we are discussing as state. In a formal MDP, observations need not reveal the full underlying state; that distinction is being set aside for this explanation.
Second is the action. In robot manipulation, this is often defined as a command to the robot's joints.
Third is the reward. We want the robot to accomplish a task. What that task is, which subtasks it requires, and how each subtask is defined can all be encoded in the reward function.
Each step has a state, action and reward. Another essential component is the transition function, describing how the current state and action determine the next state. Depending on the context, this can refer to the physical world itself or to a world model, dynamics model or simulator representing it.
The MDP connects these elements. The robot takes an action and, through the transition dynamics, moves from one state to the next, receiving a reward associated with the task. Both end-to-end and modular approaches can be understood through this framework. Even in imitation learning without an explicitly defined reward, we can think of matching the expert's action as the objective, analogous to a reward.
At deployment, we usually need to infer an action from the current state. An end-to-end approach directly learns a feedback policy mapping states to actions. The task objective and the effects of the dynamics are absorbed implicitly into the learned behavior, integrating the control mapping in one neural network. This greatly simplifies the system and can be advantageous in contact-rich tasks. But the semantics of a particular task are embedded in a black box, making generalization to a task absent from training difficult. 'Implicitly absorbed' does not mean that the policy necessarily learns an explicit reward function or reconstructs a transition model internally.
Motivated by that issue and by the semantic information in internet data, I explore how to separate task-specific semantics from task-agnostic physical information as much as possible. Pushing that separation leads to an interesting scientific question: how can a robot generalize zero-shot to a new task, even if forgoing task-specific examples means accepting a success-rate disadvantage compared with learning from them? This separation corresponds naturally to the task-specific reward and the task-agnostic transition dynamics in an MDP. The success-rate tradeoff is the research premise here, not a theorem that zero-shot performance must always be lower.
With this separation, we do not map state directly to action. Instead, we use an established modular robotics framework and optimize using a task objective and transition model to obtain actions. It is a more structured approach. My current research primarily asks how to extract the needed reward—or, intuitively, task representation—from large models and apply it to robot manipulation.
2. How should we understand task representations at different levels, from high-level planning to subtask execution?
Task representations can be defined at different levels of abstraction. Fundamentally, we translate an abstract, language-described goal into a function over a robot's state and action spaces: a reward. In task planning, the action space can consist of combinations and sequences of subtasks described in language. For 'make breakfast,' the actions might be 'open the refrigerator,' followed by 'take out the eggs.'
For finer joint commands or end-effector motions, the representation must address actions in three-dimensional space and time. VoxPoser and ReKep investigate how to turn a language instruction into a spatial and temporal objective. Large models' semantic and common-sense knowledge from internet training is crucial: an instruction is often underspecified, and the model must turn it into a function the robot can use in 3D space and time. This can support zero-shot execution of a previously unseen task.
3. Is this reward the same concept as the reward in reinforcement learning?
Yes, the concepts are essentially the same. In RL, a reward is usually defined for a particular environment, often a simulated one, and people typically design it. For navigation, a simple objective might encourage progress from point A to point B while avoiding obstacles.
To improve learning efficiency, we often use privileged simulation information when defining the reward. In clothing folding, for example, the simulator can directly tell us where the cuffs and collar are. That information helps guide reinforcement learning.

In VoxPoser and ReKep, the task objectives are grounded more directly in real-world perception. We do not have an exact 3D object model or direct access to all physical quantities such as contact forces. Instead, we use visual perception to define the objective, which we can call a visual reward. From that visually grounded objective, an optimizer can derive the robot's actions. ReKep represents its objectives as costs and constraints, while VoxPoser uses value maps; 'reward' is the broader explanatory term in this discussion.
4. Does this route have more practical potential than end-to-end methods?
It depends on the goal. For a clearly defined setting—tightening screws in a factory, installing batteries or folding clothing—repetitive work may be addressed by collecting a large dataset, such as a thousand or ten thousand trajectories. In those settings, I expect end-to-end methods to be more robust and effective.
But for open-world tasks, such as entering an unfamiliar home to make coffee or breakfast, we cannot collect data for every possible environment in advance. More structured methods introduce stronger inductive biases and may have an advantage early on, when data is scarce.
Part II: How VoxPoser and ReKep work
5. VoxPoser's core idea
VoxPoser uses powerful LLMs and VLMs to address task representation and reward specification. The key question is how to represent a task effectively, decompose an abstract instruction and connect it to the robot's concrete manipulation behavior.
Tasks contain subtle distinctions. 'Open a rice cooker' or 'open an electric kettle' may sound simple but require considerable common sense. Sometimes the robot must press a button rather than lift the lid directly. LLMs and VLMs can capture such details, but they generally express this knowledge in language rather than directly telling the robot what to do in three-dimensional space.
VoxPoser's core idea is to use model-generated code to map that common-sense knowledge into 3D space. We represent the environment as a three-dimensional grid, and generated code indexes particular regions within this discrete space. This grounds language knowledge in the spatial information needed for manipulation.
For example, to open an electric kettle, the model may determine that the robot should first locate and press a button instead of lifting the lid. This lets us compose knowledge spatially. The model can indicate where to interact, where to avoid contact and what environmental constraints apply. Near a stove, for example, it might specify that the stove is hazardous and the robot should keep its distance.
The model ultimately generates a 3D value map indicating where to interact and where to avoid, forming a spatial reward. But an objective alone does not produce an action. There are two common ways to obtain actions from it.
One is model-based planning. A world model, or dynamics model, lets us imagine how different actions would affect the environment, then plan a route toward the goal.
The other is reinforcement learning. In the model-based setting we are discussing, the world model can serve as an environment for RL. Rather than planning directly through it at each step, we learn a feedback policy: a neural network mapping states to actions. Editorial clarification: the original says RL also requires a world model. RL generally requires interaction or experience, but model-free RL can interact with the real world or a simulator without learning a separate dynamics model.
For our application, model-based planning is simpler and is what VoxPoser uses. We combine the 3D objective with planning to generate robot actions. The project focuses on grounding language models' common-sense knowledge in concrete manipulation through spatial maps, helping robots execute tasks in real environments.

Our website includes an interactive visualization resembling a thermal image. Highlighted regions in an affordance map indicate where to interact, while a constraint or avoidance map identifies regions the robot should avoid. These 3D objectives guide manipulation. For example, the robot might open the upper drawer while avoiding the vase beside it. The drawer's interaction region is highlighted, and the nearby vase is represented as something to avoid. Map colors depend on the displayed layer; a constraint is not universally defined by the absence of highlighting.
We have been calling this a reward, but the model-based planning community usually calls its objective a cost. One is maximized and the other minimized; under a sign change, they can express the same preference. Lower cost is better. Thus, the target region at the drawer has low cost, whereas the region near the vase has high avoidance cost.
6. ReKep's core idea
ReKep extends the representation to capture temporal relationships and addresses limitations of VoxPoser. Many tasks have dependencies across stages. Pouring water, for example, involves grasping the kettle's handle, moving the kettle above the cup, and then pouring. The relevant value map changes from stage to stage: first highlighting the handle, then the region above the cup.

But expressing detailed relational constraints with VoxPoser's spatial maps is difficult. The kettle should stay upright while being lifted. During pouring, the cup must remain correctly positioned; if someone moves it or the robot loses control of the kettle, the system must adjust. A position-based voxel grid does not naturally express these coupled orientation and temporal relationships. We therefore propose ReKep, which represents tasks as relationships among keypoints. Editorial clarification: VoxPoser also includes rotation, velocity and gripper maps. The limitation discussed here is the expressiveness of its relational and temporal representation, not a complete absence of rotation support.
For pouring, we can specify several keypoints and relationships. First, a point on the robot's hand should approach a point on the kettle's handle. Second, points on the handle and the spout should be at the same height in this illustrative example to keep the kettle steady during grasping. Third, the spout should be vertically aligned with the cup opening. Fourth, the directions associated with the cup opening and the kettle’s spout should form the required pouring angle. The transcript repeatedly uses a term literally meaning 'the web between thumb and index finger' for the kettle part; the task context indicates the spout. An angle is defined by directions between keypoints, not by two isolated points alone. The specific height relation depends on the kettle’s geometry; the example illustrates how a relation can be expressed, rather than a universal kettle constraint.
These keypoint relationships and their order define the task constraints. The task succeeds only when its relevant constraints are satisfied. Standard constrained-optimization tools can turn them into robot actions. Successful physical execution still depends on the constraints, perception and motion assumptions being suitable; optimization alone is not a universal success guarantee.
7. How are ReKep's keypoint relationships established? Are they generated automatically?
Different tasks, and even different environments, need different keypoints and relationships. Our core idea is to use large models—VLMs or LLMs—to infer and specify those relationships, rather than having a person define each one. This relies on the models' reasoning and prior knowledge. Their broad training lets them infer relationships appropriate to different tasks and scenes.
Clothing folding is an intuitive example. Different garments may require different folding strategies, but those strategies can be expressed as keypoint relationships: first align certain points, then bring others together. The representation can capture the patterns and relationships across clothing types, even though accurately executing them remains challenging.
8. How should we understand temporal relationships?
Consider the tea-pouring video. The system must check the stage dependencies. If the cup is moved during pouring, the robot should stop, return to the preceding alignment stage, line up with the cup again, and resume. ReKep queries the large model once at the start to generate the task constraints. It then uses optimization and updated perception to respond to disturbances and adjust execution.
VoxPoser's representation does not by itself supply the same dependency structure. For a disturbance requiring changed task logic, one option is to query the large model again, generate an updated task specification and re-optimize. This adds computation and waiting time. Efficiently handling such dependencies is therefore important. Editorial clarification: VoxPoser already replans in closed loop with cached model-generated code. It does not require a fresh LLM query for every ordinary positional disturbance; the issue here concerns revising stage logic or relationships that the original representation does not capture.
If we could query large models in real time—say, dozens of times within a second—this dependency problem might become easier to handle. Nevertheless, VoxPoser's representation still has significant limitations for coupled rotation-related constraints. That is why I think keypoint relationships offer a better task representation for this purpose.
9. Why is the clothing not folded neatly in ReKep's demonstrations?
There are two important reasons. First, the system lacks a good dynamics model, or world model, for clothing. This project focuses primarily on the task cost or reward. But a cost alone is insufficient for folding well: we need a model that predicts how the garment moves so that we can plan actions. Where should that model come from?
An important assumption in our system is that the grasped keypoints move with the gripper. If it translates five centimeters and rotates thirty degrees, we apply that same rigid transformation to the relevant keypoints. This is only an approximation for flexible objects such as clothing or scarves. If the robot holds one end of a scarf, a point in the middle may move too, but it will not follow exactly the gripper's rigid motion. That prediction error is a main reason the fold is untidy. A better dynamics or world model capturing deformation could help address it. More precisely, the paper uses a local rigidity assumption for the group of keypoints on a grasped object or part, while assuming other points static; observed keypoints are updated continuously, rather than enforcing that approximation for the entire task.
The second reason concerns the zero-shot setting we study: how far can we push generalization without task-specific data? Generalization often means manipulating an object in a new pose or a similar object in a new environment. Here, even the task can be new, which adds further challenges.
That is one of my main research themes. Many of my questions concern generalization, particularly across tasks. Suppose I encounter a new task described simply in language, such as 'make breakfast' or 'fold the clothes.' How can I use existing tools and data to generalize as far as possible without collecting a dedicated dataset? That is what interests me.
This does not mean I oppose using data. I think data is very important. But the zero-shot setting encourages us to ask where precious robot data is indispensable. If we use robot data to reproduce abilities existing large models already have—locating objects or specifying how different garments should be folded—that may not be its best use. We can instead use those models and their priors as a basis for generalization, then ask which essential problems robot data should solve that existing models cannot cover.
Insufficient data has long been a pressing problem in robotics. I suspect it will remain one until we deploy robots at scale. That is why zero-shot settings interest me: they push us to find more efficient and precise uses for data.
10. How do we connect spatial information with an LLM?
VoxPoser and ReKep use different environment representations and interfaces. VoxPoser represents the environment as a 3D array or grid—for example, 100 × 100 × 100. Once we fix the coordinate frame, we can index a location; the handle might be at grid index (15, 20, 30). The language model then generates code to map its knowledge into the 3D environment.
ReKep uses a more direct route through a VLM's image input. We overlay numbered keypoints, such as 0, 1, 2, 3 and 4, on the image. We explain the point labels to the model so it can reason about their task meaning: a point might identify where the robot should grasp, while another indicates something to avoid. Rather than relying on a fixed 3D grid, the task is represented by relationships between points.
The points cannot be arbitrary; they need to occupy meaningful locations. Self-supervised visual models such as DINOv2 help capture image semantics and propose suitable keypoints. This improves the keypoint-selection process.
11. How far is ReKep from practical applications?
I regard this series of projects as exploratory. Robustness has considerable room to improve, so the system is not ready to apply directly in industrial deployments. But its modularity makes the directions for improvement clear.
First, it relies heavily on large models. We need stronger models with richer priors about three-dimensional space and time. We began this line of research two months before GPT-4 was released. When GPT-4 and then GPT-4o became available, we observed substantial improvements in reasoning. Future VLMs could provide better task objectives and guidance. This refers to the research line's development, not a claim that the 2024 ReKep paper itself preceded GPT-4.
We also need a general world model that works in open environments, not just a closed laboratory. Popular video-generation models such as OpenAI's Sora operate primarily with 2D image representations. ReKep needs a world model with a usable 3D state representation that works in real, open scenes. Like the visual task objective, it should not rely on task-specific data. Calling a video model '2D' here concerns its output representation; it does not rule out internal modeling of three-dimensional structure.
For clothing folding, a pre-trained world model should predict how a previously unseen garment deforms under different actions. That is difficult, but the learning objective is clear and separated from the semantics of a particular task. It may therefore become an important research direction.
Part III: Hierarchical models, end-to-end models and world models
12. End-to-end and hierarchical models
Both approaches may use large models. The difference is whether we explicitly assign a role to an existing model—the hierarchical route—or fine-tune it for a robot-control setting—the end-to-end route. Physical Intelligence's π0 and Google's RT series are examples of end-to-end policy learning. Training on expert actions does not necessarily provide an explicit decomposition of task representation and dynamics. The processing happens largely in latent representations, making it hard to isolate which weights handle vision, which encode the task objective, and so on. 'Unstructured' here describes the absence of that explicit functional decomposition, not the absence of an internal neural architecture.
A structured alternative separates elements such as the reward and transition model, following the MDP view. That makes it clearer which component needs improvement.
13. Learned dynamics and MPC
In learned dynamics plus model predictive control, MPC is closely related to model-based planning, though their details and emphasis differ. Both use a model and an objective, such as a cost or reward, to predict how actions change the environment and choose a useful current action. More specifically, MPC repeatedly replans over a finite horizon using updated observations and executes the next portion of the plan.
A conspicuous missing component in this framework is a high-quality, general world model. Learning such a model is itself extremely challenging.
Researchers disagree about the best route. Some think it is simpler and more effective to learn a general-purpose policy mapping observations directly to actions. Others think a world model is more general and has richer available training data. Either route could potentially lead to a general-purpose robot.
14. What is a world model?
In the sense we are using here, a world model has a clear definition: it takes the current state and action as inputs and predicts the next state. It is the transition or dynamics model in the MDP framework.
The distinction lies in how we define states and actions and represent them. For a video-prediction model, the state might be one or several image frames, while the conditioning instruction might be 'open this drawer.' The model would then predict an image showing the drawer after opening. A text instruction in this example is high-level conditioning, rather than necessarily a precise physical motor command.
For robotics, actions are usually represented as motions in 3D space or joint commands, and the model predicts how they change the environment. State can take several forms: 2D images, as in video models, or 3D representations such as point clouds, meshes and voxel grids.
The world model's central job is to predict environmental dynamics using these state and action representations. The challenge is to choose suitable representations and predict changes accurately in complex environments.
15. Could sufficiently powerful large models supply the world models robots need?
This is an open question, and researchers have different views and approaches. Personally, I do not think existing language-based models, including LLMs and VLMs, can serve as the robot world model we need.
Their strengths primarily concern understanding and reasoning over discrete symbols, such as natural language and code, and understanding how images relate to those symbols. The robot's world model needs to predict the next state given an action. That state is often continuous, high-frequency and three-dimensional.
Whether those abilities can be integrated into existing large models remains unknown. I think we may need a new approach, or a separate kind of model independent of an LLM, specifically for predicting 3D spatial states.
Such a model could complement an LLM. An intuitive analogy is the difference between common sense specific to humans and common sense shared by humans and cats. LLMs and VLMs are better at capturing human symbolic knowledge and mapping it into visual space—for example, knowing that opening a kettle may require pressing a button and finding that button. A cat may not understand that semantic concept deeply enough to generalize it zero-shot to a new scene.
At the same time, animals, people and even young children can make rapid predictions about their three-dimensional surroundings. I think these two model types may therefore serve different fundamental purposes, though they could be combined. Recent projects have integrated 2D generation with VLMs and obtained encouraging results. So I do not rule out unifying these capabilities in a single model eventually.
16. Should 3D data come from 2D observations or 3D scanning hardware?
For a purely 3D world model, data may not be the biggest bottleneck right now. Robot-action data is certainly difficult to obtain, but visual data is abundant. I see it as less scarce than language data: text and code depend on what people produce, while cameras can record the natural world around the clock. The number of cameras and the density of the data they generate are enormous compared with text. This is my assessment of availability, not a quantified global dataset comparison.
I therefore do not see visual data itself as the bottleneck. Although much of it is 2D, research shows that it is possible to learn 3D representations from 2D observations. We may not need direct 3D measurements in every setting: a 2D image is a projection of a 3D world. Better algorithms could infer useful 3D structure, though considerable algorithmic progress is still needed. Inferring structure does not mean that every single image uniquely determines the scene's geometry.
The 2D-to-3D route depends heavily on algorithms. Mature 3D-capture hardware can therefore be an effective shortcut in the near term. It is somewhat like the contrast between Tesla and Waymo: Tesla emphasizes using camera observations to solve 3D perception and scaling the data, whereas Waymo uses lidar to obtain direct 3D measurements and deploys and improves the system incrementally. This is a simplified analogy, not a complete description of either company's sensor stack or a claim that Waymo uses no cameras.
Both have advantages, and I have no strong preference. Either could work depending on the scene and technical conditions. My sense is that the goal matters. If we target a few specific tasks or environments and can afford to deploy robots for demonstrations or collect 3D data to build a world model, a higher-cost solution focused on those limited tasks may make sense.
For a general-purpose robot, however, I think we need to rely more on existing internet-scale data. Better algorithms could learn from 2D and other data without direct 3D measurements, reducing dependence on robot-collected data.
Ideally, we would learn a good 3D world model, stronger VLMs and better reward models from those sources, building on approaches such as ReKep and VoxPoser.
With those models, we could use planning and RL to deploy a first generation of relatively general robots. They might not fold clothing perfectly, but they would provide a starting point. Continued optimization and accumulated experience could improve performance and address initial shortcomings. As more data becomes available, the system could become progressively more end-to-end, gaining generality and robustness.
17. Combining these methods with RL and imitation learning
RL fits this structured approach well in certain respects. Given a reward and an environment with which we can interact repeatedly—a simulator or world model, for example—we can use trial and error to search for useful action sequences or policies. Editorial clarification: the original calls RL a model-based optimization method. That describes RL within the world-model framework discussed here; RL also includes model-free methods. Nor does trial and error guarantee a global optimum or literally unlimited interactions.
For example, an LLM or VLM could provide a useful reward and task representation. Combined with a world model that we can query repeatedly, we could train through RL to obtain a policy meeting the task requirements, such as pouring water or folding clothing. The language model provides the task representation; this does not assume that an ordinary LLM is already the accurate 3D dynamics model described above.
Imitation learning is harder to combine directly with this particular decomposition. Behavior cloning commonly learns an end-to-end observation-to-action mapping, from visual inputs to behavior, without an explicit intermediate structure. That does not directly match a task-modeling framework with separate functional modules and models. It describes the form of behavior cloning being contrasted here, not every possible hierarchical imitation-learning architecture.
But that does not make combination impossible. Each approach has strengths and weaknesses. One robot system can use different techniques for different tasks, and the experience it collects can train all its data-driven components, improving execution.
References
ReKep
https://rekep-robot.github.io/
VoxPoser
MOKA
https://moka-manipulation.github.io
PIVOT
https://pivot-prompt.github.io
CoPa
Wenlong Huang's website