How fortunate the founders and investors who have spent the past three years in embodied AI have been. Amid a growing bubble, we are witnessing prosperity and spectacle whose reality is not always easy to judge. This may be the beginning of an enormous technological transformation—or it may not. Either way, viewed against the next twenty or thirty years of a career, these years will be precious experience.
As investors following this sector, many of us spent the first half of 2026 anxious that we were “too late again” or wondering whether a round still had room. However hard we work, deals keep closing their first two rounds before we have covered them, and founders we have not yet met may soon become difficult to reach. The tremendous pressure of FOMO has pushed me into frequent travel too. Smart Note feels more like a quiet personal garden amid the noise. After a stretch of travel, I can retreat to the island and reflect on these conversations.
Two months ago, I spoke with Wenlong again. Our previous interview was VoxPoser & ReKep: Exploring Zero-Shot Task Generalization Through Task Representations. As I edited this conversation, his calmly articulated technical insights kept drawing me into individual passages, leaving me thinking about them for a long time.
Among the many categories of world models is the 3D world model. Using PointWorld as our starting point, Wenlong and I discussed a wide range of topics. This interview also led me to speak with Bowen Wen—A Conversation with Bowen Wen: How FoundationStereo Is Transforming Stereo Vision—and to interview the researcher behind another influential 3D reconstruction project, to be revealed in the next installment. Through these three conversations about 3D foundation models, I hope to draw attention to recent breakthroughs in 3D.
This is a long interview. Set aside an hour and read it slowly.
Original article outline
Contents
👇
1. Introduction
Part I · PointWorld
2. PointWorld's central idea
3. Unifying the state and action modalities
4. A shared modality for representing actions and states
5. Is this similar to humans acting and perceiving in visual 3D space?
6. How could joint forces be represented beyond pose?
7. Handling different numbers of cameras in 2D and 3D
8. 2D-to-3D reconstruction
9. Recent 3D advances that made PointWorld possible
10. Processing training data
11. Can a monocular camera plus ToF replace stereo?
12. Why is FoundationStereo's depth estimation so good?
13. PointWorld's architecture
14. Applications of PointWorld
Part II · 3D World Models
15. Why use a 3D world model? Is 2D insufficient?
16. What do you think of today's egocentric data?
17. Controllability as a defining strength of 3D
18. Could we obtain a 3D world model comparable to the world humans inhabit?
19. Do we really need world models?
20. Must we first have enough robot data to build such a world model? [Editor's note: this original outline includes a separate controllability topic; the retained interview body groups the discussion differently and ends with Question 19. The page's navigation follows the body headings.]
1. Introduction
At the time of this interview, I am a fourth-year PhD student at Stanford, advised by Professor Fei-Fei Li. My research is mainly in AI and robotics. Over the past few years, I have worked in two main directions.
The first is foundation models for robotics. I began exploring this around 2021, while I was still an undergraduate at Berkeley, mainly asking whether large language models could support task understanding and decomposition. I subsequently worked on a series of related projects, including collaborations with Google and further research in Fei-Fei's lab at Stanford: Code as Policies, SayCan, PaLM-E, and VoxPoser. See our earlier conversation: VoxPoser & ReKep: Exploring Zero-Shot Task Generalization Through Task Representations.
The second direction is world models, which I began focusing on in summer 2024. I found the area interesting in itself, and it complemented my previous work. Foundation models had mainly addressed task understanding and abstract priors, but were limited at a fine-grained level: executing specific actions and making counterfactual predictions, such as “How will the world change if this action is taken?” World models can fill that gap. Combining task understanding with models of future state changes allows robots to generate and choose actions more effectively. Those are my two main research directions today.
Part I · PointWorld
2. PointWorld's central idea
The central question of this work is whether we can train a general-purpose 3D world model.
“World model” is a broad concept, defined differently by different people. From a robotics perspective, I tend to think of it as a dynamics model: we have a state representation and an action representation, input the current state and action, and output the next state.

The essential differences between world models lie in how they define states and actions. In a video-based world model such as Genie, for example, the state is a sequence of 2D pixels—video frames—and the action may be navigation, such as moving forward, backward, left, or right. The output is the next image.
PointWorld explores 3D state and action representations. There is a somewhat philosophical issue here: action representations vary enormously across robots and tasks. Some use task space, such as the end effector's 3D pose; others use joint space. The differences among quadrupeds, humanoids, single-arm robots, and multiarm robots are substantial.
One of our central ideas was therefore to find a unified modality that could represent both states and actions, be easily obtained from robot cameras, and accommodate different robots.
PointWorld investigates 3D point flow as that shared representation. Specifically, its input includes a point cloud from a static frame. This can be obtained from the robot's onboard cameras—for example, from RGB and depth. The point cloud need not be complete: it can be partial, even from a single camera.
We consider an action-conditioned world model, so the action is known. The robot's corresponding 3D point flow forms part of the input. The output describes how every point in the input environment point cloud changes when the robot executes that action.
The formulation is simple and intuitive. Both the environment and robot are represented through 3D points and flows, reducing world modeling to the unified problem of predicting how points interact. We aim to simplify the method while investigating whether this 3D representation can scale to support a general-purpose 3D world model.
The project took a year and a half overall, and we spent a great deal of time processing data. Data processing and quality remain major bottlenecks in 3D. But recent 3D models are becoming mature enough to support training these world models, especially by supplying annotations such as depth, camera poses, and point tracks.
The main contributions are twofold: exploring how to scale 3D world models, and providing a relatively stable pipeline for constructing 3D interaction data and annotations.
We also made some preliminary investigations into what value such a model could bring to robotic manipulation. We can discuss that later.
3. Unifying the state and action modalities
There is relatively little disagreement about state representations. A robot operating in the real world mainly relies on onboard cameras, so RGB video is the most common input. Many robots also have depth cameras and receive RGB-D information, which can be represented as a partial point cloud. It is partial because we cannot observe points behind objects. That is the state representation used in our work.

Actions are much less standardized. The two common choices are the end effector's task space—for example, a change in 3D pose—and joint space.

Both have the same cross-embodiment problem: they are not unified. Robot structures differ greatly. A humanoid might have dozens of joints and a correspondingly high-dimensional action. A single arm might have seven joints, or eight action dimensions including its gripper; two arms might have sixteen. Three or more arms add still more dimensions. End effectors can also be grippers, dexterous hands with different numbers of fingers, or specialized tools such as suction cups or rods.
As a direct consequence, training a world model or policy that transfers between robots becomes very difficult.
4. A shared modality for representing actions and states
As I mentioned, most work represents actions through end-effector poses or joint space. Our goal is to unify these different action representations in a single modality.

Specifically, we assume that the robot's URDF is known. You can think of a URDF as a robot blueprint or structural description: it specifies the joints, their ranges of motion, and the corresponding mesh geometry.
Given joint-space actions, we can use the URDF to calculate the corresponding changes in the robot's 3D point cloud, mapping joint actions to 3D point flow. The URDF bridges joint motion and geometric change, allowing different robots' actions to be converted into one shared 3D representation.

In this sense, the URDF converts the robot's action space into a point-cloud representation. A human analogy is useful: when carrying a large box, we do not think about how many degrees each joint should rotate. We think directly in 3D space about how to move our hands, where to place them, and how to lift the box.
Likewise, when teleoperating an unfamiliar robot, we do not first think about the dimensionality of its joint space or the value of each dimension. We naturally think about how the physical robot—even the surface of each finger—should move through 3D space to have the desired effect on the environment. Representing actions as point flows aligns with that intuitive expression of spatial motion.
5. Is this similar to humans acting and perceiving in visual 3D space?
Yes—that is our core philosophy. A sequence of numbers does more than raise the problem of differing joint counts: it also ignores geometry. A gripper's opening is often represented by a single number, yet grippers vary substantially in shape, size, and structure. Dexterous hands may have the same degrees of freedom but differ in size, thickness, and other geometric properties. These details matter to an action. Low-dimensional control signals compress them away and lose some important information.
We want the action representation to be closer to the observation space. Rather than describing an action in low-dimensional control coordinates, we express it in perceptual space. You could say that we represent actions in a manner resembling observations, instead of as abstract control signals.
6. How could joint forces be represented beyond pose?
That is a good question. We do not explicitly handle forces in the current work. Methodologically, however, there are fairly direct ways to extend it. The present input specifies the robot's future poses. If we wanted force-related inputs, we could instead use joint torques and the robot's dynamics information in its URDF to calculate an acceleration or force representation at robot points, incorporating that information into the model. [Editor's clarification: this is a proposed extension; accurately calculating forces also requires appropriate dynamics and contact modeling, beyond geometry alone.]
Our current tasks are relatively simple. We focus more on intuitive understanding of environmental physics and generalization than on precise force control. In other words, we predict how objects move through space, and have not yet extended the model to different forces applied along an identical spatial trajectory.
7. Handling different numbers of cameras in 2D and 3D
Once trained, our model can accommodate arbitrary numbers of cameras. That is an advantage of 3D representations. In 2D, different numbers of views—one, two, or several cameras—increase modeling complexity. In 3D, they all become a point cloud: a collection of points. Processing is therefore simpler and more unified.
We assume depth information is available in this work. 3D modeling generally requires two key elements: depth, and the extrinsic transform between camera and robot. The latter is the familiar hand–eye calibration problem: determining where the camera's observations lie in the robot coordinate frame.
From a modality perspective, a multiview 2D model can also be understood as an implicit 3D model. PointWorld explicitly unifies this information in the robot's point-cloud space, bringing multiview observations and hand–eye coordination into one space.
In terms of information, multiview RGB-D observations with their camera parameters and the corresponding 3D representation are equivalent. A point cloud constructed this way adds no information beyond the multiview RGB-D itself. Its more important role here is allowing the model to handle actions within the same observation space.
PointWorld also works with just one camera, since a single viewpoint can supply a point cloud.
8. Does PointWorld require 3D reconstruction?
“Reconstruction” has different definitions. In our setting, it means using available depth and observations to obtain a point-cloud representation, rather than seeking a high-precision mesh reconstruction.
Goals also differ within 3D computer vision: some projects focus on meshes, others on 3D Gaussian Splatting or point clouds. Our work leans toward point-cloud representations.
Another distinction is that we introduce the robot's own embodiment. In many vision tasks, reconstruction mainly aligns multiple views. Here, the more important objective is to express every observation in the robot's coordinate frame, not merely align views. That lets the robot identify which part of the observed space is its own hand and how that hand interacts with the environment.
9. Recent 3D advances that made PointWorld possible
This project depended considerably on recent advances in 3D models.

We used VGGT, a 3D reconstruction model that is relatively mature and familiar within computer vision.
We also used FoundationStereo, which estimates depth from stereo images and supplies a depth map. See A Conversation with Bowen Wen: How FoundationStereo Is Transforming Stereo Vision.
Another component is CoTracker3, a point-tracking model that can track each pixel's trajectory through a video. We mainly use it for supervision: generating training signals during data labeling.
Those are the three main models used for annotation.
During model training, we also used Point Transformer V3, a Transformer-based point-cloud model.
The project brings together several recent advances in 3D vision. In a sense, these models had to reach their current maturity before we could combine them to train a 3D world model.
10. Training data
We mainly used two datasets: DROID and BEHAVIOR.
DROID is a real-world manipulation dataset covering open-world settings. It contains a few hundred hours, mainly of relatively simple robot manipulation tasks in real environments. BEHAVIOR is simulated and somewhat larger, with over 1,000 hours. [Editor's clarification: these are source-dataset scales; the paper reports approximately 500 hours of curated training interactions after filtering, not the sum of all source footage.]
Simulation is relatively straightforward because all the 3D information has ground truth that we can use directly. The real difficulty is DROID's real-world data, for which we must construct our own 3D labels.
Many DROID recordings use stereo cameras, so we first use FoundationStereo to predict depth from stereo images. Its depth predictions are excellent—better even than the estimates from some stereo cameras themselves. That was crucial for us.
Next, we need each camera's pose relative to the robot base. We initially use VGGT to obtain an approximate pose, but errors remain, on the order of ten to twenty centimeters.
That was already a relatively good result at the time, but still insufficiently precise for our purposes. We use a prior: the scene contains a robot whose mesh we know. Based on that, we optimize and refine the camera poses, reducing the error further.
With depth and camera poses in hand, we track points. CoTracker3 tracks each pixel's trajectory through the video, revealing how each point changes over time.
Those are the three steps: FoundationStereo provides depth; VGGT plus optimization provides more accurate camera poses; and CoTracker provides point tracks. Together, they let us construct point-level 3D labels for real-world data.
The process in simulation is similar in principle, but those models are unnecessary because the 3D information is already supplied directly.
11. Can a monocular camera plus ToF replace stereo?
Our method treats different depth-acquisition techniques the same way, because ultimately we need a unified point-cloud representation. A time-of-flight (ToF) depth camera can provide such a point cloud directly, so it is usable too.
My personal view of acquiring depth changed substantially after models such as FoundationStereo appeared. See A Conversation with Bowen Wen: How FoundationStereo Is Transforming Stereo Vision. We found that learned depth estimation clearly outperformed traditional hardware. The input is stereo RGB and the output is depth. Although it is a predictive model, its practical results are better than those of many commercially available depth cameras.
Traditional depth cameras—many based on emitting light and measuring its return time—can struggle with reflective objects, stainless steel, or complicated materials that produce reflections.
We found FoundationStereo very stable in these situations, without obvious problems on reflective objects or complex materials. That matters because it substantially improved the quality of 3D information in our real-world data.
12. Why is FoundationStereo's depth estimation so good?
It is not that no one had worked on this before; the field has always had plenty of research. The bigger issue was that it had not really been done well enough. In some sense, models like these could have been built many years ago, but people had not refined them to this degree.
One important reason it works so well is that it solves a more well-defined problem: stereo depth estimation, rather than monocular depth estimation. There is extensive research on depth, but monocular estimation typically relies heavily on priors. Knowing roughly how large an apple is, for example, we infer depth from its appearance, position, and lighting. There is inherently an element of guessing.
Stereo is different. There are two input cameras with a known distance between them, analogous to human vision: we have two eyes with a fixed separation and use them continuously to observe the world. For fine-grained manipulation, such as threading a needle, stereo is particularly important. At close distances, priors alone are not sufficiently precise.
Once the stereo geometry is known, depth estimation becomes a feature-matching problem. Good features and correspondences between the views can produce accurate depth estimates.
A model can therefore estimate depth correctly even for an object it has never encountered, because the approach rests on geometric relationships rather than semantic priors.
These models are typically trained at scale in simulation, with extensive randomization, to learn general matching capabilities. This can produce something like a depth-estimation foundation model that works well across many settings.
13. PointWorld's architecture
We keep the architecture as simple as possible. The central idea is to choose a good modality that unifies states and actions. Beyond that, we do not particularly want to design a new architecture ourselves.
We directly adopted Point Transformer V3, a state-of-the-art point-cloud model. It is essentially a Transformer and does not differ dramatically in form from other models.

Our emphasis was to take this model and scale it. The philosophy was to avoid extensive architectural modifications and instead ask how well a model could perform in a simple setup once the data scale increased.
Previously, 3D research was constrained by data; large-scale, high-quality training sets were hard to obtain. With the development of these 3D models, we have our first opportunity to obtain comparatively large amounts of 3D data.
We care more about what simple models and large datasets can achieve under these conditions. Our experiments did reveal challenges particular to training 3D world models, but most required adjustments to training details rather than changes to the overall Transformer architecture.
14. Applications of PointWorld
World models are a mature concept in robotics, with many established applications. Our main concern here is the 3D world model itself: how to make it general-purpose for the first time.
Most earlier robotics world models were trained separately for a particular environment, rather than generalizing across settings. We therefore concentrated on training and scaling the 3D world model itself.
For robotics applications, we chose only the most basic scenario: given a new task, without task-specific priors or demonstrations, can the model work directly?
The key is its counterfactual prediction capability. Given an action, it predicts how the environment changes. Conversely, we can ask what action would produce a desired change in the environment.
That is fundamentally a planning problem. Traditional robotics typically solves it using hand-designed models or physics simulators.
We replace those traditional approaches with a learned world model.
I think we should evaluate it differently because a world model is not itself a policy, and solves a different problem from a VLA or policy. Our primary concern is counterfactual prediction: can it correctly predict how the environment would change if a particular action were taken?
People often compare VLAs and world models, but I do not find that entirely appropriate. They are fundamentally complementary. A world model focuses on modeling the environment and does not directly make decisions; a VLA or policy is used for decision-making.
A world model can assist decision-making at several levels. Without expert demonstrations for the current task, we can use reinforcement learning or planning within it to derive a policy. If a policy already exists, we can improve it using the world model. Or we can evaluate policies inside the model instead of running every evaluation on a real robot.
All these applications depend on counterfactual capability. The most direct test is whether, starting from randomly initialized actions, the model can guide gradual adjustments until the task is accomplished. That is commonly called planning.
If we begin with an existing policy instead of random actions, gradually refining it is policy improvement. Another test is whether a policy's evaluated performance inside the world model agrees with its performance in the real environment. These examine the model's counterfactual capabilities from different angles.
Part II · 3D World Models
15. Why use a 3D world model?
I keep thinking about this question. We can consider it in two parts: state representations and action representations.
For states, we first need to distinguish the observation space from the state space. Observations usually come from the robot's cameras and other sensors: there may be multiple cameras, but the images are generally 2D, sometimes with depth or stereo. The full state is not directly observable, so we need to construct a representation from observations and past interactions. Suppose I just closed a drawer after seeing what was inside. The model should still remember those contents, even though neither its current 2D nor its current 3D observation can see inside. The true state is therefore more than the current observation; it must include hidden information.
I think the best representation would be a latent state, rather than one prescribed as 2D or 3D. We should let the model learn the most appropriate representation instead of hard-coding a particular form.
Shima: I have long been confused about observations and states in world models, and Wenlong's explanation is very clear. I do not know whether readers share my confusion, but I want to offer another perspective that helped me understand. Recently I read some books about quantum physics, and thought of Schrödinger's cat as an analogy. A cat is sealed in a box containing radioactive material; before observation, it may be alive or dead, and on observation its state collapses to one outcome. The cat's state and our observation of it are different things. Before observing it, can we really predict which definite state it is in? [Editor's clarification: this is the author's analogy, not an equivalence between classical partial observability and quantum measurement. A world-model state can be hidden even when it is classically definite.]
Actions are what interest us most about 3D. Looking toward robotics' ultimate goal, I have not found a better representation. We may want to control any unfamiliar robot to perform any task, much as humans can teleoperate many different robots. Its joints and shape might be unknown, so we need to model the effects of different robots' actions on the world from the observation space.
I do not think 2D is enough for that. We must always judge which part of the image is our own hand—or the hand of the robot being teleoperated—and roughly what the robot's geometry is where it is occluded. That requires an explicit or implicit 3D hand–eye calibration process and a corresponding 3D representation.
Suppose I teleoperate a robot to carry a box. First I identify its hands in front of me and model the relationship between my actions and the controlled robot in my stereo observation space. When a hand reaches behind the box, I may not see the contact itself, but I know the approximate hand length and box size. Even without seeing it, I can infer roughly where the hand is and how it makes contact. This reasoning takes place in 3D space, implying a need for explicit or implicit 3D modeling.
From a data perspective, though, I agree that 2D has an enormous scale advantage. Scaling requires more data sources, and 2D clearly has the upper hand there.
Relevant 3D data is currently largely concentrated in robotics, where collection includes depth or stereo cameras and corresponding hand–eye coordination information. Most internet videos are 2D and lack action data. There is therefore a potential tension between 3D's advantages for representing actions and 2D's advantages in data scale.
One possible direction is to train a backbone on 2D data and then fine-tune or align it using 3D action data.
For me, this project's starting point was to find an action representation I was satisfied with. Subsequent work considers how to combine it with larger-scale data.
16. What do you think of today's egocentric data?
Much human egocentric data is effectively “2D plus 3D,” rather than purely 2D. Collection commonly uses a camera, perhaps worn on the head, to obtain a 2D RGB stream, but additionally records hand poses. Those poses are generally 3D, not 2D.
The action in that setup is therefore still 3D. Our work uses a more expressive representation: point flows on the hand's surface, closer to where contact happens, and adaptable to arbitrary embodiments. Fundamentally, the two approaches are fairly close; both represent 3D actions.
This is not limited to egocentric data. Handheld-device data such as UMI is similar: the visual inputs are 2D images, but methods such as SLAM recover a 3D action representation.
I personally find this direction very promising. Combining such data to train a 3D world model would be a natural extension.
17. Could we obtain a 3D world model comparable to the world humans inhabit?
I think it is very possible, because this is the most scalable objective currently available. All kinds of real-world data could help train it: internet video, human data, robot data, and even cameras distributed throughout the world.
For that goal, 3D is unavoidable, but the distinction between explicit and implicit 3D representations may not matter. What matters more is strong counterfactual prediction across different interactions, and the ability to use those predictions for decision-making.
Much discussion of world models still focuses on video generation or prediction: how good the generated pixels look. With enough robot data, though, I think a model should focus more on interaction itself than on merely producing attractive video.
Over the longer term, even toward the ultimate goal, the ideal would be to obtain enough robot data to train a more robot-native world model.
The problem today is that we lack that amount of robot data, so we rely heavily on internet video. It is 2D, and it also has strong biases. When do people take out a phone to film? Usually when something interesting happens. The viewpoints and distributions differ greatly from what a robot actually sees. Many robot interactions are boring and repetitive, and relatively scarce online. Robot data and egocentric manipulation footage capture more of them.
A practical short- to medium-term path is therefore to use scalable sources such as internet video to train a 2D backbone, then fine-tune on smaller amounts of data closer to robotics—including 3D or action-labeled data—to introduce controllability.
Over the longer term, if enough robot data becomes available, I think it is worth starting over and training a world model centered entirely on robots and interaction.
18. Do we really need world models?
Whether we really need world models is an important and interesting question that I care about quite a lot.
Today's interest in robotics world models often amounts to using them as better backbones, or extracting actions from generated videos. These approaches usually still require fine-tuning with task-specific demonstrations, so the paradigm has not changed very much.
What I find truly important is counterfactual prediction. I think it is essential to planning and discovering new behaviors and strategies.
For example, I recently saw a Physical Intelligence demo in which a gripper-equipped robot turns a sock right-side out. When folding socks, we may find one inside out and turn it back. Humans have multifingered hands: we can hold the edge with the thumb and index finger and use the other fingers to turn it out.
A gripper robot cannot do that in the same way because it lacks those finger degrees of freedom. A suitable strategy is to put the sock over one gripper, gradually push it up, and then have that gripper grasp it from inside and turn it out.
This is difficult: it involves a deformable object and very fine-grained, contact-rich manipulation. They ultimately trained an autonomous policy that works well, which is very impressive at that level.
What I find more interesting, however, is that a human devised the strategy while teleoperating. The data collector first thought of the method and then taught it to the robot through data.
Since robots previously could not accomplish these tasks, the paradigm may already have enormous economic value. But looking toward the ultimate goal, I am more interested in whether, as data increases, robots could devise such strategies themselves at test time.
Given a new task and situation, without a human demonstration, could a robot construct a solution? What if it has grippers of different thicknesses, or a dexterous hand with a different number of fingers? How should it act under the different constraints of each task?
At a deeper level, this may be a longer-term scientific pursuit. The goal goes beyond distilling human behavior to achieving human-level physical intelligence, even though distilling humans will probably be an important means toward it.
My conjecture is that world models become very important in this setting. They supply an ability to imagine, and to use that imagination in decision-making, somewhat like reasoning in LLMs.
As humans, we may never have performed a particular manipulation, but we can simulate possibilities mentally and improve our ideas through interaction. If I were a robot with a gripper rather than a dexterous hand, what could I do to reach a goal? That is counterfactual prediction.
The behavior is not one we would automatically execute in everyday life; it emerges from reasoning through different possibilities.
For me, the importance of world models in robotics may therefore go beyond better backbones, priors, or more data. They could provide a longer-term capability: counterfactual reasoning, generating novel strategies, and feeding that reasoning back into, or internalizing it within, predictive policies.
19. Must we first have enough robot data to build such a world model?
Robot data will certainly help us build better world models. But recently I have come to doubt that such a model necessarily needs vast quantities of robot data from the very beginning. That is partly a reflection from working on PointWorld.
Once we regard actions as part of the observation space, we can learn how the world works from many nonrobot datasets. That includes physical laws, as well as the behavior of people, objects, animals, and other agents or entities.
In internet videos or footage from cameras around the world, we see different agents interacting with the physical world. An action need not be my action. If I watch a cat play with a toy, for example, I neither control the cat nor know its embodiment, yet I can learn about the toy's physical properties and the cat's behavioral characteristics.
Of course, this is more like pretraining, providing behavioral and physical priors. LLM pretraining similarly learns from many kinds of data, but post-training is still needed for an agentic system. Robot post-training may well involve counterfactual reasoning, but it also requires actual interaction with the real environment. Just as LLMs cannot improve purely through reasoning, they need feedback and rewards from a real environment too.
I particularly like a philosophical point made by Stanford Professor Dan Yamins:
“No one observer is the only true agent in the environment.”
In other words, no observer is the sole agent; many different agents exist in the environment.
A cat is an agent, and other people are agents too. Observing their behavior lets us learn the regularities of the world and of behavior itself.
References
Wenlong Huang's homepage: https://wenlonghuang.com/
PointWorld: https://point-world.github.io/
Paper presentation: https://www.youtube.com/watch?v=0vfgm8LshmY
FoundationStereo: https://nvlabs.github.io/FoundationStereo/
VGGT: https://vgg-t.github.io/
CoTracker3: https://cotracker3.github.io/