All writing

3D Vision

A Conversation with Bowen Wen: How FoundationStereo Is Transforming Stereo Vision

Bowen Wen discusses FoundationStereo, Fast FoundationStereo, depth estimation and the role of 3D perception in robot manipulation.

Zha Dami2026-05-1838 min readWeChat original
Contents

Does a robot actually need depth information to perceive the world? Should its perception be based on 2D or 3D? Why does the industry currently make relatively little use of 3D? Can the shortage of training data for 3D models be addressed? Despite AI’s rapid development, why have visual sensors changed so little over decades? Why do we not have visual sensors designed for today’s physical AI? With these questions in mind, Smart Note undertook its first exploration of 3D vision for robotics.

We interviewed Bowen Wen, first author of FoundationStereo and Fast FoundationStereo—work with the potential to reshape today’s stereo-camera landscape. Bowen researches 3D and embodied intelligence at NVIDIA. In recent years, he has contributed to several representative projects in 3D perception and robotics: FoundationPose, now widely used in industry; FoundationStereo for high-quality depth estimation; and this year’s Fast FoundationStereo, which brings depth estimation into real time. Earlier, he developed BundleSDF for 3D object reconstruction and pose tracking. His work starts from visual perception but focuses more specifically on 3D perception for manipulation: how can perception actually help solve the problems a robot faces when manipulating objects?

This is a long article—about 18,000 Chinese characters in the original—but well worth reading.

Original article outline

Contents

👇

Part One: 3D Vision for Manipulation

1. Some Concepts in 3D Vision

2. Which 3D Representations Are Relevant to Robotics?

3. What Are the Main Approaches to Depth Estimation?

4. How Does Manipulation-Oriented 3D Vision Differ from 3D Vision More Broadly?

Part Two: FoundationStereo

5. Why Develop FoundationStereo?

6. FoundationStereo’s Training Data

7. How Are Different Types of Data Mixed?

8. Problems with Synthetic Data

9. Processing and Annotating Synthetic Data

10. Comparison with Other Simulation Environments

11. Sources of Synthetic Data

12. FoundationStereo’s Method

13. Model Performance

14. Comparison with Other Depth-Estimation Methods

15. FoundationStereo’s Applications in Embodied Intelligence

16. Can FoundationStereo and Two Ordinary Cameras Replace a Depth Camera?

Part Three: Fast FoundationStereo

17. How Does Fast FoundationStereo Differ from FoundationStereo?

18. Fast FoundationStereo’s Architecture

19. Can Fast FoundationStereo Support Robot Policy Learning?

20. How Can FoundationStereo and Its Fast Version Expand the Use of 3D Data in Robotics?

Part Four: The Need for 3D Vision in Embodied Intelligence

21. Should a Robot Model’s Visual Input Be RGB or RGB-D?

22. How Do We Explain Tesla FSD’s Use of Monocular Cameras?

23. Future Directions in 3D Vision and Manipulation

Part One: 3D Vision for Manipulation

1. Some Concepts in 3D Vision

3D vision is a very broad concept, and different people understand it differently. Let me briefly explain my own definition.

Generally, 3D vision uses prior knowledge such as multiview geometry and has a relatively explicit three-dimensional representation. Examples include point clouds, feature matching and 3D structure. More recent representations such as NeRF and Gaussian splatting also developed from these geometric priors.

3D vision usually emphasizes knowing explicitly the state of an object in three-dimensional space: its position, rotation and depth. Depth tells us how far an object is from the camera. Looking at a photograph and deciding which object is nearer or farther is a form of depth perception. More concretely, it concerns the distance of points on an object’s surface from the camera.

Many living creatures have this ability. Most creatures, as I am describing them here, have two or more eyes and can perceive depth through disparity. For example, a deer that sees a car approaching runs away because it senses the distance getting smaller. Humans likewise judge near and far quite naturally. Editor’s note: This biological analogy is illustrative. Eye configurations vary widely across species, and depth perception also uses monocular cues; not all animals have two eyes or rely on stereo disparity.

A 6D pose builds on this idea: 3D position plus 3D rotation. It usually describes a rigid object, such as an industrial part, whose state can be represented by a single 6D pose.

An articulated object needs to be divided into parts, each with its own pose, or described using a more complex representation. Generally, when we discuss 6D pose, we mainly mean rigid objects.

NeRF and 3D Gaussian splatting are essentially two different technical routes to a similar goal: given a set of images, reconstruct a 3D scene that can be viewed from arbitrary new viewpoints. They lean more toward computer graphics and visualization applications such as entertainment, sports broadcasting and AR/VR. In embodied intelligence, they are currently used more often for real-to-sim, making simulated environments resemble reality; their direct connection to manipulation itself is less strong.

2. Which 3D Representations Are Relevant to Robotics?

There have always been different approaches and debates about 3D information. '3D' is itself a large category containing many representations. Some researchers use 6D poses for grasping and rigid-object manipulation, while others use depth maps directly.

There is no universally agreed solution. In many practical applications today, however, 6D pose is often a more direct and effective representation for tasks such as rigid-object manipulation.

On the one hand, methods such as FoundationPose can now provide sufficiently accurate pose estimation. On the other, 6D pose is a compact representation that generalizes readily. In many cases, there is no need to collect new demonstrations and retrain: give the system a model of a new object, and it can be deployed directly on a new task.

That is why companies in many practical settings, including retail, tend to use representations such as 6D pose directly. Overall, though, the field has not converged on a single unified choice of 3D representation.

3. What Are the Main Approaches to Depth Estimation?

Monocular depth estimation has been a prominent approach in recent years. These methods primarily learn depth relationships in natural scenes from internet data, and relatively few are used directly for robot manipulation. The central difficulty is obtaining depth with a true physical scale. A model may tell you which object is nearer or farther without accurately establishing whether it is one meter or ten meters from the robot. Manipulation is very sensitive to this: in grasping, for instance, different distances require entirely different motion plans. Editor’s note: Some monocular models do estimate metric depth; the point here is their remaining scale and accuracy limitations in demanding manipulation settings, not that metric monocular estimation is impossible.

Then there is stereo depth and multiview depth. Stereo recovers depth from two cameras; multiview methods use more cameras and known camera poses to recover 3D structure.

Recently, some foundation-model approaches to 3D have also emerged, including feedforward networks that recover 3D information directly from images without strict calibration. Like monocular methods, they often lack a reliable physical scale, making them hard to use directly for manipulation; they are more often used as features.

RGB-D cameras, such as RealSense and ZED, are more common in industry. They use approaches including infrared stereo, active projection and structured light, although many implementations are closed-source. Editor’s note: These are different hardware families: ZED cameras use passive stereo, while RealSense depth technologies vary by model. The listed mechanisms do not apply identically to every device.

LiDAR is another option, common in autonomous driving but less common for manipulation. Its point clouds can be relatively sparse, its field of view limited and its cost high, making it difficult in many cases to supply the dense, fine-grained 3D information manipulation needs.

4. How Does Manipulation-Oriented 3D Vision Differ from Traditional 3D Vision?

We can look back to roughly 2018–2022, before the GPT era. Perception still had many unresolved problems, and much work focused on familiar vision tasks such as object detection, 2D detection and segmentation. These could help manipulation, but they were essentially only the first step. There was still a long way to go before visual perception could truly support robot manipulation.

Even basic pick-and-place remained far from satisfactorily solved as a research problem. Many systems could grasp objects, often using point clouds, but struggled with placement. The more mature imitation-learning approaches that came later were not yet available; methods were generally more basic, straightforward behavior cloning.

The prevailing approach was a modular pipeline: perception first, planning next, with multiple modules assembled to carry out manipulation.

A clear gap remained. Traditional vision could recognize and segment objects, but usually supplied 2D information. Manipulation, motion planning and control require representations and understanding in three-dimensional space. Simply 'seeing an object' is far from enough: the crucial questions are where it is in 3D, what its pose is and how the robot should interact with it. Making that transition with 2D information alone is extremely difficult, so there was a substantial gap.

Computer vision subsequently produced developments such as NeRF and Gaussian splatting, major advances in 3D vision in recent years. But there remains some distance between those methods and manipulation.

Their more common application now is constructing digital twins. For example, 3D Gaussians can represent a highly realistic scene for real-to-sim mapping. Such environments can support evaluation without repeatedly experimenting in reality, and training: reconstruct a real scene in simulation, train a policy there and reduce the visual difference between simulation and reality.

However, these methods’ direct contribution to manipulation capability is still relatively limited.

In contrast, FoundationPose, which I worked on earlier, has a more direct effect on manipulation. We now see demonstrations of complex actions such as tying shoelaces and folding clothes, but earlier even rigid-object manipulation had not been well solved.

Work on 6D pose estimation, including FoundationPose, was therefore an important step forward for rigid-object manipulation. Tasks such as pick-and-place and industrial bin picking fundamentally depend on obtaining object poses reliably and accurately.

For example, in our collaboration with Vention on industrial sorting, a typical scenario is a bin containing many industrial parts in disordered piles. The robot must grasp accurately and then place the parts precisely at designated locations, such as a downstream conveyor. Placement has specific requirements: for a tubular part, a small hole might have to face upward, with all parts arranged in a consistent orientation to interface with the next production stage. That requires knowing the object’s 6D pose.

Vention industrial robot sorting and placing parts with object pose information.
Vention industrial robot sorting and placing parts with object pose information.

A 6D pose consists of 3D translation and 3D rotation. Translation is an object’s position—for example, its x, y and z coordinates in an XYZ coordinate system. But an object is more than a point: it has an orientation. Is a pipe pointing up or down? Along the Z axis or X axis? That is rotation, which itself has three dimensions. Three translational plus three rotational degrees of freedom give us a 6D pose.

In our Vention collaboration, FoundationPose solves this sorting-and-placement problem so that robots can arrange the objects neatly in the required way.

One way to understand the distinction is that traditional 3D vision emphasizes 'understanding the world' through environment reconstruction, object geometry and pose estimation. For manipulation, the core is not only understanding the object but understanding the interaction between object, environment and action: how can this object be manipulated in its present environment, and what will happen afterward?

That is the more direct value of 3D vision in manipulation. My central goal in this perception research is not merely to help during training. I want it to influence policy decisions directly at inference and execution time, solving problems robots encounter during actual execution.

Another class of tasks is high-precision industrial insertion and assembly—for example, fitting a gear onto a shaft—with tolerances often below a millimeter. Traditional planning alone struggles with these tasks because the system can tolerate extremely little perception error.

Our approach was to estimate the object pose with FoundationPose and use that pose as input when training a reinforcement-learning policy. We use 6D pose information rather than raw RGB images. RL generally requires extensive trial-and-error training, mostly in simulation. RGB inputs can introduce a large sim-to-real gap that makes direct transfer to the real world difficult.

Using 6D pose as an intermediate representation makes things much simpler. In simulation, we can obtain the true pose directly and add perturbations. In the real world, FoundationPose estimates it. The policy receives the same representation on both sides.

From this perspective, 3D perception, such as 6D pose estimation, is not an isolated vision problem. It can directly provide a robot policy’s input and serve real manipulation tasks.

Part Two: FoundationStereo

5. Why Develop FoundationStereo?

First, I have long used depth cameras for manipulation, so I know how strongly depth quality affects point-cloud quality and task success rates.

I had also worked extensively on sim-to-real. I found that RGB-only policies often had a large sim-to-real gap and relatively weak generalization. Policies based on depth information generally trained more efficiently and converged faster. In some tasks, such as locomotion, their sim-to-real transfer had already proved better than that of RGB-only methods.

For manipulation, however, which needs finer depth detail, performance still declined substantially on transfer to reality. That shows another side of the problem: depth quality is an important bottleneck. Geometry in simulation is perfect and depth data very clean. If real-world depth could approach that ideal, many sim-to-real problems would become easier.

From an engineering perspective, sufficiently clean point clouds also make many geometry-based methods work well.

Second, there was the state of the field. Monocular depth estimation had been advancing rapidly and beginning to generalize well, but stereo depth did not yet have a genuine foundation model. Most methods trained on fixed datasets and tested in the same scenes, with little work on large-scale training and zero-shot generalization.

That made stereo models hard to deploy. Using one in your own setting often required collecting data, annotating it and retraining. I thought this was worth exploring: if monocular depth had shown that large-scale training worked, could we bring the same idea to stereo depth?

Third, there were the engineering conditions. At NVIDIA, we could collaborate with engineering teams and use good data resources and infrastructure. That gave us an opportunity to scale up data and train a stronger stereo-depth foundation model. Those were the main motivations.

Stereo depth had been studied for many years before FoundationStereo, but most methods remained within a train-on-one-domain, test-on-that-domain paradigm and struggled to generalize across scenes. For a robot, every new environment could require another round of data collection, annotation and training, making deployment expensive.

Our central motivation was to make stereo depth a genuinely generalizable foundation-model capability: something usable directly across domains, like today’s detection and segmentation models, instead of requiring retraining every time.

A key context is that both academia and industry increasingly emphasize generalization in embodied intelligence. If all you want is one fixed task, you can lock down the entire environment using industrial automation. What matters for the future is enabling robots to adapt cheaply to new objects, scenes and lighting conditions.

There were several main reasons stereo depth had not achieved good zero-shot generalization.

First, data. Real stereo data is difficult to collect because it requires not only stereo images but pixel-level ground-truth depth. That kept past datasets from growing large.

Second, architecture. Many methods used conventional convolutional networks without designs specifically intended for large-scale data and cross-domain generalization.

Third, resources. Both high-quality synthetic rendering and large-scale training require enormous compute and engineering resources. We used extensive photorealistic rendering and ray tracing, which consume substantial GPU resources.

FoundationStereo therefore tries to do one thing: bring stereo depth into the foundation-model era through large-scale data, scalable models and adequate engineering resources.

6. FoundationStereo’s Training Data

3D data is extremely scarce. What specifically did FoundationStereo use for training? The paper mentions one million stereo pairs and synthetic data, for example. How was the data pipeline built, and how does it alleviate the shortage of 3D data?

We generated our data on Omniverse. One of our central goals was to build highly realistic scenes that resembled the real world as closely as possible.

A key issue in stereo depth is that conventional stereo matching essentially looks for pixel correspondences between left and right images, which is difficult in many real scenes. Strong light, reflections and exposure differences can make matching on pixel brightness alone unreliable. It is therefore not only a geometric problem; there is also an element of semantic understanding.

We designed more realistic, logically structured environments such as factories, hospitals, offices and warehouses, with objects arranged in ways resembling the real world. The network can then infer depth relationships using contextual structure, rather than relying only on pixels.

Photorealistic rendering was another crucial element. We tried to reproduce real-world textures, lighting and shadows, using extensive GPU-based path tracing. That requires a great deal of compute.

The dataset was also large: approximately one million stereo image pairs, substantially more than many conventional stereo datasets. It covered diverse scenarios, including indoor environments, flight and driving.

Another important design choice was randomizing camera parameters, including baselines and focal lengths. This helps generalization to different camera configurations, even stereo systems users assemble themselves.

Together, these factors form the main advantages of FoundationStereo’s data pipeline.

7. How Are Different Types of Data Mixed?

Where do the real and synthetic training data come from? Roughly what proportions of each do you use?

We did not collect real data ourselves. We mainly used publicly available real datasets, such as InStereo2K. We later found that these real samples helped, but not enormously. The main improvement came from the FoundationStereo Dataset we built, which is a central contribution of the work. The interview’s shortened 'Stereo2K' name refers to InStereo2K.

For training, we mixed the FoundationStereo Dataset with existing data, including classic synthetic datasets and a small amount of real data. The pipeline also relies on NVIDIA’s infrastructure for large-scale rendering, simulation and data generation, which supports dataset construction at this scale.

One important advantage is photorealistic rendering, which clearly helps reduce the visual gap between simulation and reality. Overall, the training data is predominantly synthetic.

8. Problems with Synthetic Data

Does synthetic training data introduce problems? People worry that it is not real, so might it create a sim-to-real gap or even train the model in the wrong direction?

That depends on the task. For end-to-end policy learning, many methods feed images directly into the policy. They may perform very well in simulation and then fail in reality, with the large visual difference between the two being an important reason.

High-quality rendering consumes substantial compute. For perception, we can generate photorealistic data offline, render everything in advance and repeatedly reuse it for training, without real-time interaction. Policy learning is different: the robot must continually interact with the environment in real time. Each action requires rendering the environment again, making expensive photorealistic rendering difficult to use.

Most policy-learning simulations therefore look relatively simple: basic ground planes, a few obstacles and limited variation in lighting and materials. That produces a clear visual gap relative to the real world.

One advantage of 3D perception is that high-quality rendering can happen offline. Once the perception model is trained, its stable 3D representation can feed an RL policy. In some earlier manipulation projects, for example, we used estimated poses directly as policy inputs. Policy training then does not have to confront complex visual differences directly; it uses a more stable, unified 3D representation.

From that perspective, synthetic data is extremely valuable for 3D perception. Many projects in 3D now train extensively with simulated data instead of relying solely on real data.

9. Processing and Annotating Synthetic Data

What does the subsequent processing and annotation pipeline look like? For stereo depth in particular, how do you obtain ground truth?

Such annotations are relatively easy to obtain in simulation. The simulator already has the true 3D information for every object, so ground truth such as depth can be generated directly.

10. Comparison with Other Simulation Environments

Systems such as Isaac Lab also use fairly photorealistic rendering, with underlying graphics capabilities similar to some of ours. But their main purpose is reinforcement-learning training, emphasizing large numbers of parallel environments and real-time interaction.

In principle, they can certainly generate high-quality data. The issue is that RL requires robots to interact with environments in real time, usually across many environments simultaneously. Using extremely photorealistic rendering at every step would be prohibitively expensive.

They therefore tend to prioritize speed at the expense of some rendering quality. That reflects a practical limitation in graphics: highly photorealistic rendering is difficult to deliver in real time at these training workloads.

The objectives also differ. RL environments generally target specific, relatively narrow tasks, such as insertion or machine tending, so scene variation is limited.

FoundationStereo, by contrast, aimed to train a model that generalizes and therefore needed many different types of scenes. We built hospitals, warehouses, offices and other environments, placed different objects in them and captured data from different viewpoints, on a much larger overall scale.

The issue is therefore not an absolute lack of capability. It is that combining photorealism, large-scale scene diversity and real-time interaction becomes very expensive.

11. Sources of Synthetic Data

Do you select similar scenes from existing datasets and then use generation tools such as Omniverse to expand, synthesize and diversify them?

The scenes were not simply taken ready-made. We bought and built many 3D scene resources, including commercial art assets. The generation pipeline runs on Omniverse and currently relies mainly on conventional computer-graphics methods.

One possible future direction is to render images in simulation and then use generative models to improve their realism, bringing simulated data closer to the real world.

12. FoundationStereo’s Method

This is a high-level overview of the complete method.

FoundationStereo architecture: stereo feature extraction, cost-volume filtering and iterative disparity refinement.
FoundationStereo architecture: stereo feature extraction, cost-volume filtering and iterative disparity refinement.

Disparity Estimation

The model takes a stereo pair—a left and a right image—and outputs disparity. Camera parameters, such as the baseline between the two cameras and the focal length, convert disparity into depth. I sometimes use 'disparity' and 'depth' interchangeably here because they are convertible in a calibrated stereo system.

The encoder has two branches: Depth Anything V2 and a side-tuning CNN.

At the time, monocular depth estimation already showed strong generalization. Models such as Depth Anything V2 were trained on internet-scale data using ordinary color images and had learned strong geometric priors. Given a picture of a chair, for example, they can roughly judge which parts are closer to the camera.

We use that model as a frozen backbone rather than retraining it. Since our own data is primarily synthetic and contains relatively little real-world data, this branch supplies additional real-world prior knowledge.

Frozen features alone are insufficient, however. The actual downstream problem is finding correspondences between left- and right-image features: stereo matching. If the features are poorly adapted to stereo, learning very fine-grained matches becomes difficult.

We therefore add a side-tuning CNN, both to adapt the features to stereo and to capture local details. Architectures such as vision transformers are stronger at global context but less sensitive to high-frequency local detail, where CNNs are stronger.

There are two complementary relationships here. At the data level, Depth Anything V2 contributes real-world geometric priors, while our CNN adapts to synthetic stereo data. At the architecture level, the ViT handles global context and the CNN local detail. We concatenate the features into a two-branch representation. Both left and right images pass through this encoder to obtain their respective features.

Building the Cost Volume

Next comes a classic stereo step: constructing a cost volume. For a point in the left image, we only need to look along the corresponding horizontal line in the right image. In a calibrated and rectified stereo system, corresponding points lie on the same epipolar line, which is horizontal here. A cup in the left image appears on that same line in the right image, shifted horizontally. That shift is disparity. Editor’s note: Calibration alone does not make epipolar lines horizontal; rectification is also required for this horizontal-search formulation.

That is one of stereo’s essential properties. The network exploits it by ignoring irrelevant regions and searching only along the matching horizontal line. For each left-image pixel, possible positions along the same right-image row form the candidate set. Organizing those candidates produces the cost volume, which can be understood as containing every possible matching location.

The network then learns over this volume to decide which candidate is the correct correspondence. Intermediate layers process the volume into better features. We also borrow the iterative-refinement idea from RAFT-Stereo.

The model first makes an initial disparity guess, then looks up features at the corresponding cost-volume location. This resembles a self-check: if I guess that a table corner matches a particular position in the right image, I retrieve that position’s features and compare them with the left-image features. If they do not match, I adjust the position, retrieve features again and compare again.

The process iteratively improves the correspondence until the left- and right-image features agree as well as possible. The ConvGRU module on the right performs these repeated updates. The final output is disparity—the offset between the images for every point—which can then be converted to physical depth.

Attentive Hybrid Cost Filtering, combining long-range disparity attention with axial-planar convolutions.
Attentive Hybrid Cost Filtering, combining long-range disparity attention with axial-planar convolutions.

Having constructed the cost volume, we come to AHCF: Attentive Hybrid Cost Filtering.

Stereo can be understood as a correspondence-search problem. Given a red point in the left image, we need to find it in the right. Consider a difficult example containing many repeated fence bars. The left-image point lies on one bar, but the right image contains many similar-looking bars, creating many candidate locations.

The difficulty is deciding which bar in the right image corresponds to that left-image point. This is no longer a purely local problem: because of disparity, the correct match is not necessarily at the nearest position and may be farther away.

We therefore need long-range context. We designed a disparity transformer that reorganizes all candidates along the epipolar line and applies self-attention. The model can then use global information to compare candidates and identify the correct correspondence more accurately.

AHCF also contains an axial-planar convolution branch. Conventional 3D convolution often uses a 3 × 3 × 3 kernel, with a limited contextual range. Increasing it to, say, 5 × 5 × 5 would consume much more memory.

We split this into planar and axial convolutions, enlarging the receptive field without substantially increasing memory consumption so that the model can access longer-range context.

The core design goal of AHCF is to make better use of large-scale data and improve scalability.

We found that training existing models directly on FSD improved performance but did not fully exploit the dataset’s value. This design therefore aims to use the data more effectively and improve zero-shot generalization. Editor’s note: Here FSD means the FoundationStereo Dataset. The original text’s parenthetical expansion as 'Fast FoundationStereo' is a naming error.

Iterative refinement follows earlier methods. Our main architectural changes are twofold. First, we introduce a monocular depth prior, a design that was largely absent from previous stereo methods. Second, we introduce a new approach to cost-volume modeling and filtering to learn and process the stereo-matching candidate space more effectively. Those are the main methodological contributions.

From the paper’s perspective, the two key innovations are combining monocular foundation-model priors with stereo matching and developing a new way to learn over cost volumes, improving matching in complex scenes. The latter requires more background in stereo matching and is more technically detailed.

13. Model Performance

The method is open-source on GitHub and can be used directly. Essentially, all you need as input is stereo images.

Your earlier question was right on target: can you run it with almost any two cameras? The basic answer is yes.

People here have tried buying two very cheap cameras on Taobao, fixing them into a stereo rig with a 3D-printed mount and calibrating it themselves. Calibration is mature enough to handle through OpenCV’s pipeline. Feed the left and right images into FoundationStereo, and it runs.

That is an extreme low-cost option. More commonly, labs use ZED or RealSense, both of which work and offer a reasonable balance of cost and performance.

One point worth emphasizing is that although FoundationStereo was trained on RGB images, we found some generalization to grayscale and infrared images as well.

For example, the RealSense devices we tested obtain stereo depth from infrared images rather than an RGB stereo pair. We ran FoundationStereo on their infrared stereo images and generally obtained good results. So the method can also run on infrared stereo input from RealSense, not just RGB. This description concerns the stereo models used in these tests, rather than every RealSense product.

14. Comparison with Other Depth-Estimation Methods

The paper compares your method with RGB-D cameras, conventional stereo matching and methods for transparent-object scenes. Why does FoundationStereo perform so much better on historically difficult cases such as transparent objects? Where do its main improvements over conventional methods come from?

You can see some transparent objects here, including two bottles. Our project website, listed in the references, lets you drag the viewpoint to inspect point clouds. It is intuitive and closer to how people understand 3D structure. Returning to why our method works better on transparent objects, I think there are several reasons.

First, we introduce monocular depth priors. Models such as Depth Anything V2 already generalize reasonably well to transparent objects—not perfectly, but quite well. Bringing a strong monocular geometric prior into stereo matching helps with these objects.

Second, our training data includes some transparent objects, although not a particularly large proportion. Time constraints prevented us from building a dedicated large-scale dataset for them. FoundationStereo already has some generalization to transparent objects, but there is room to improve further.

Conventional methods struggle partly because of data and model generalization. Another issue, when comparing with commercial RGB-D cameras such as RealSense stereo models, is that many use active infrared stereo, projecting an infrared pattern to recover depth.

Transparent objects are especially difficult because they refract and transmit light rather than reflecting it in the usual way. Light emitted by active systems may be refracted or pass through, instead of providing the expected return to the sensor. That creates fundamental imaging difficulties and explains why many depth cameras clearly fail on transparent objects.

The paper mainly evaluates standard stereo metrics. BP1 and BP2 measure the proportion of pixels whose prediction error exceeds one or two pixels. EPE, or end-point error, measures the average deviation between predicted and ground-truth disparities.

Overall, our method substantially improves these metrics over earlier methods, particularly in zero-shot settings. Many conventional methods train and test in a fixed domain and show much larger errors when the scene changes. FoundationStereo remains comparatively stable across domains.

A particularly intuitive evaluation is simply looking at the point cloud. Conventional outputs can be sparse and fragmented, with discontinuous boundaries and considerable noise. FoundationStereo’s clouds tend to be denser, smoother and more geometrically complete.

FoundationStereo depth and point-cloud comparisons in challenging scenes.

Watch the video above to see the difference.

For industrial parts, our point clouds show clear object edges, threads and planar structures. A tabletop looks like a clean, flat plane. With many conventional stereo or RGB-D systems, it may instead contain bumps, noise and holes; structures that should be straight and flat can become tilted or excessively smoothed.

The contrast is especially obvious with reflective objects. For a box of metal screws, our method still distinguishes individual structures and boundaries fairly clearly, while many conventional methods blur them together or produce large holes.

That is a major advantage of 3D: its interpretability is very strong. Often no additional explanation is needed; the point clouds make the differences immediately visible.

In a sense, users have been struggling with RealSense for a long time. People working with point clouds often spend enormous amounts of time on postprocessing—filtering, hole filling, interpolation and parameter tuning. That is one reason many had poor experiences with depth cameras.

So FoundationStereo substantially improves depth maps and point clouds through models and algorithms, without changing the existing optical hardware. Is that right?

That is an accurate summary. It was not that good cameras did not exist, but that good devices were very expensive. Equipment such as Photoneo could cost tens of thousands, or even hundreds of thousands, of US dollars in the examples I have in mind. Editor’s note: This is the interviewee’s contextual cost comparison, not a verified price list for every Photoneo device or a current purchasing quote.

FoundationStereo improves performance in software. In an extreme case, two ordinary cameras forming a stereo system can achieve depth quality approaching state of the art. Instead of depending on expensive hardware, algorithms make the capability more broadly accessible.

15. FoundationStereo’s Applications in Embodied Intelligence

FoundationStereo is already being used in many fields, and we collaborate with several companies. Beyond embodied intelligence, customers use it for surveying, construction and spatial management, such as room and object management. These are essentially 3D reconstruction problems, and they have begun using FoundationStereo.

In embodied intelligence, use is currently more often offline. Some recent projects process previously collected datasets with FoundationStereo, automatically generating depth or 3D labels, then use those data to train new models. That is a typical application.

Another application is collecting teleoperation data. In hand tracking, alignment between the hand and objects is often inaccurate because many methods rely on monocular vision. FoundationStereo can improve motion-capture precision and spatial alignment.

We are seeing increasing offline use, both in feedback from companies and in the research directions of papers citing the work.

I think there are several reasons it has not yet been widely integrated directly into policy learning.

First, some researchers prefer a 2D approach and do not want to introduce 3D information.

Second, many may not yet understand how FoundationStereo differs from conventional depth methods. With new papers and demonstrations appearing every day, people become somewhat overwhelmed and may not have taken the time to try it.

A very practical issue is speed. FoundationStereo prioritizes quality and generalization and runs relatively slowly, so it does not meet real-time needs well. Many robot policy-learning and deployment settings demand real-time operation.

That is why we later developed Fast FoundationStereo. The original is better suited to offline data generation and constructing training datasets; the fast version is better suited to a robot operating in real time.

16. Can FoundationStereo and Two Ordinary Cameras Replace a Depth Camera?

If methods such as FoundationStereo already outperform many RGB-D systems in depth quality, could ordinary stereo cameras plus a model eventually replace dedicated depth cameras such as RealSense?

That is a very good question. I agree with your point that many RGB-D cameras still use methods developed decades ago.

I started using RGB-D cameras early, around 2018 when I began my PhD. My first project used RealSense. Seeing a point cloud for the first time was striking: unlike a 2D image, you could put it in a 3D viewer, rotate it freely with the mouse and inspect it from different angles, rather like today’s virtual property tours. I found it fascinating.

I subsequently tried many RGB-D cameras, from different RealSense models to other manufacturers. Initially they looked promising, but using them for research and manipulation exposed many problems.

A typical example is imaging a tabletop. From a distance, the table looks flat and has four legs, so everything seems fine. Zoom in on objects such as a bowl or block, however, and the point-cloud quality can be very poor.

For instance, a cubic block may look like a ramp in the point cloud rather than a clean cube, and merge into the tabletop’s cloud. That depth quality is nowhere near sufficient.

With a sufficiently clean point cloud, many manipulation problems become much easier. Remove the tabletop, and geometry alone can reliably estimate object position and pose. In reality, many ordinary depth cameras produce noisy clouds with merged boundaries, making classical geometry methods unusable.

During my internship at Google X, I used industrial cameras such as Photoneo and saw very clearly how directly and substantially high-quality 3D perception improved manipulation.

We tackled difficult tasks such as inserting a gear onto a shaft or a battery into a spring-loaded slot, which require high precision and tiny error tolerances. We also used a single-demonstration imitation-learning setup: a person demonstrates once, and the system represents the action using 3D information and generalizes to gears or batteries with different shapes.

A major lesson was that people who consider 3D unhelpful have often encountered poor depth quality. Ordinary RGB-D point clouds can be so noisy and unstable that they seem to add little useful information beyond RGB. With accurate, clean depth, many tasks become markedly easier. That was another important motivation for FoundationStereo: obtaining genuinely high-quality depth through algorithms and cheaper hardware.

Part Three: Fast FoundationStereo

17. How Does Fast FoundationStereo Differ from FoundationStereo?

Fast FoundationStereo primarily targets real-time performance. The original solved zero-shot generalization and now has strong capability there, but it lacks real-time speed.

Previously, the field largely offered two choices: real-time models with poor zero-shot performance, or good zero-shot models too slow for real-time use. We did not have a model that handled both well.

The motivation for Fast FoundationStereo was therefore a model that runs fast enough while retaining useful zero-shot capability and accuracy. Speed without accuracy is not meaningful.

Compared with FoundationStereo, the fast version runs much faster while sacrificing a little accuracy to obtain that speed.

18. Fast FoundationStereo’s Architecture

We divided the original model into components and optimized each separately.

FoundationStereo has roughly three components: feature backbone, cost filtering and disparity refinement. We use a suitable acceleration method for each.

Fast FoundationStereo accelerates the feature backbone, cost filtering and disparity refinement separately.
Fast FoundationStereo accelerates the feature backbone, cost filtering and disparity refinement separately.

Start with the feature backbone. The original has two branches: Depth Anything V2 for monocular priors and a side-tuning CNN adapted to stereo. Running both is slower and uses more memory.

We use hybrid distillation to compress both branches into a single student backbone, retaining their capability while reducing computational cost.

Cost filtering is the second component and a major bottleneck because it processes a 4D volume with large tensors.

We perform a simplified form of neural architecture search. The original uses a 3D hourglass and a disparity transformer; we search for a more efficient architecture to replace that design.

Third comes disparity refinement, a process repeated over many iterations. A small speedup at each step can save substantial time overall because the step runs repeatedly. We prune this component.

Because it is recurrent and iterative, pruning must account for dependencies across the computation graph. We construct a recurrent dependency graph, use it to perform pruning and then retrain.

We also work on data. With a strong FoundationStereo model available, we use it to produce a large real-data training set from Stereo4D. Those stereo images lack depth annotations, so FoundationStereo automatically generates pseudo-labels. We generate labels for existing imagery; we do not synthesize the underlying real-world images.

Those labels are imperfect, so we add mechanisms such as local adaptation, cross-validation and sky segmentation to filter and check them.

Finally, we train Fast FoundationStereo using those pseudo-labeled real samples alongside the original data. That is the overall approach.

19. Can Fast FoundationStereo Support Robot Policy Learning?

Could this work directly support embodied-intelligence policy training, such as perception inputs for offline training or online reinforcement learning?

That is a very direct application. Fast FoundationStereo is now essentially a plug-in alternative for depth cameras such as RealSense.

The original FoundationStereo could not do this mainly because it was too slow. RealSense, for example, might run at around 30 Hz, while the original model did not reach real-time speed. Fast FoundationStereo can now approach that rate, making replacement practical in many scenarios. Actual throughput depends on image resolution, hardware and the model configuration.

Many policies previously relying on RealSense can therefore incorporate Fast FoundationStereo into the control loop, unlocking a new way to use it.

Previously, in the comparisons motivating our work, there was no method that combined better depth quality than RealSense with the required real-time performance.

20. How Can FoundationStereo and Its Fast Version Expand the Use of 3D Data in Robotics?

How important is 3D data for physical or embodied intelligence today? How will FoundationStereo and Fast FoundationStereo advance its practical use in robots?

There are already mature examples of why spatial information matters in embodied intelligence. Robot vacuum cleaners are a successful deployment: they commonly use LiDAR information for obstacle avoidance in systems that have proved reliable. Editor’s note: Many vacuum cleaners use 2D planar LiDAR rather than a full 3D sensor. This example establishes the value of geometric sensing, not that every such robot uses 3D LiDAR.

Autonomous driving offers another example. There are different approaches, such as Tesla’s vision-oriented route and Waymo’s use of LiDAR. Waymo’s progress shows that using 3D information is a feasible and effective path.

There are many specific uses of 3D in embodied intelligence. One is data collection for real-to-sim: reconstructing an environment in 3D, such as a building map, for inspection or delivery robots.

It also supports simulation training. Gaussian splatting, for example, can construct high-fidelity environments for training policies in simulation.

Another use is motion capture and retargeting: cameras recover human 3D joints and object trajectories, which generate trajectories for policy training. The 4D reconstruction you mentioned similarly recovers human–object interactions from video and imports them into simulation for training.

At inference time, 3D information can directly serve as a policy input. Using 6D poses, for example, can reduce the sim-to-real gap and accelerate deployment.

Some projects also expose perception modules as tools. Depth estimation, pose estimation and 2D detection can provide intermediate capabilities and representations that an agent calls.

Sample efficiency and inductive bias are another important aspect. With RGB alone, it is often hard to avoid collecting new demonstrations in a new environment. RGB lacks explicit physical scale, making generalization difficult. Adding 3D information makes it easier to share knowledge across environments and achieve zero-shot generalization.

The same applies to sim-to-real. Depth-based policies generally transfer more easily than RGB-based ones because geometry is relatively consistent between simulation and reality, while RGB appearance varies enormously with lighting, materials and other factors.

There are also safety and obstacle-avoidance issues.

With RGB alone, reliably constructing a 3D occupancy map is difficult, making it harder to ensure safety in complex environments—for example, avoiding walls and obstacles.

Finally, there is interpretability. 3D information is easier to visualize and helps people understand what a system has actually 'seen.' With a completely end-to-end system and no intermediate representation, users find it harder to establish trust.

Those are some of the main roles of 3D information in embodied intelligence.

Part Four: The Need for 3D Vision in Robotics

21. Should a Robot Model’s Visual Input Be RGB or RGB-D?

Starting from first principles, my intuition is that if you randomly select a species in nature, it usually has at least two eyes. I even looked into which species have only one eye. Examples I considered, such as deep-sea fish and earthworms, may have simple light-sensitive structures or no useful vision in environments where sight is unnecessary, such as permanent darkness. Some tiny organisms have a single 'eye' that does not form images but senses light intensity; a light detector is sufficient for them. Editor’s note: These statements express a biological analogy, not a verified classification of species. Many deep-sea fish have highly developed eyes, earthworms lack image-forming eyes, and animal visual systems have diverse configurations.

Computation is another consideration: such organisms cannot support more complex visual processing. Human binocular vision requires substantial processing resources, including a dedicated visual cortex.

In the analogy I am drawing, many species have two eyes, and insects’ compound eyes can be thought of as providing multiple views. Spatial understanding, or 3D perception, is therefore a fundamental and important ability for living creatures. Compound eyes are not literally equivalent to a calibrated multiview camera rig.

Of course, people point out that engineering need not copy nature completely: airplanes do not fly by flapping wings. But in intelligence, including our current work on humanoids, choosing two legs still reflects biological structure to some degree.

One could even argue that the importance of multiple eyes is more widespread than that of two legs: not all species have two legs, while vision and multiple viewpoints appear across many kinds of animals. From this first-principles intuition, embodied intelligence may ultimately need stereo-like or richer 3D information. The argument is an intuition about useful sensing, rather than a literal claim that almost every species has multiple eyes.

Will world models truly suited to embodied intelligence lean toward 2D video models or toward 3D models? What is the fundamental difference between those paths for physical intelligence?

Most earlier visual models were 2D. PointWorld is among the projects that began exploring 3D representations, such as point clouds, relatively early. The direction as a whole has not yet converged.

Whether a world model needs 3D depends on how you use it. For high-level decisions, it mainly provides a prior and may not accurately characterize the physics of the particular scene. In folding clothes, for instance, video generation can 'imagine' what happens next based on prior experience of folding. But each garment has different materials and properties, and many details of handling a new garment cannot be specified precisely by that model. It provides high-level guidance rather than all the information needed to solve the task.

So the need for 3D depends on how far you want the world model to take you. For high-level planning, 2D may suffice. For fine-grained tasks where you need to nail down the details, I think 3D information remains necessary.

22. How Do We Explain Tesla FSD’s Use of Monocular Cameras?

Tesla is not a single-camera system; it uses multiple views rather than a standard stereo pair. Adjacent cameras have some overlap, even if not much, and that overlap can help infer depth and complete information in nonoverlapping regions.

Tesla’s autonomous-driving approach uses standardized hardware configurations, allowing data to be shared across vehicles. Training still uses multiview images. Editor’s note: Camera layouts differ across Tesla models and hardware generations; the interview’s statement that all mounts are the same is a simplification about consistency within a configuration.

It is therefore too simple to say 'a monocular camera is enough.' Each camera is monocular, but the overall system is multiview. Overlap enables 3D inference: monocular cues infer part of the scene, assisted by multiview information.

Its 3D occupancy maps and bird’s-eye-view representations also show that it constructs a spatial environment representation rather than simply staying in 2D. At the time of this interview, Tesla had not solved autonomous driving with 100% reliability and the approach was still being tested. Whether it ultimately becomes the most robust route remained uncertain. This is the interviewee’s assessment at the article’s publication date.

In a sense, FoundationStereo also addresses the source of 3D data. By lowering the depth-hardware barrier, does it enable simpler, cheaper stereo equipment to collect and use 3D data at scale?

Previously, obtaining high-quality 3D data was difficult because every sensor had limitations. Ordinary RealSense point clouds could be poor, while LiDAR was not well suited to fine-grained manipulation. There were few easy alternatives for obtaining such data from the real world.

That often left simulation as the option, but simulation struggles to reproduce complex scenes or object physics. FoundationStereo offers a new possibility.

More people are now recording stereo video. One reason is recognizing stereo as a potentially valuable modality. Even if the data is not immediately useful, collecting it now may pay off as stereo-depth technology improves.

In other words, accumulate data first, and it becomes useful 3D data when depth estimation matures. FoundationStereo provides relatively dense, pixel-level, accurate and generalizable depth, making real-world acquisition of 3D data more feasible.

What do you think about current 'monocular camera + time-of-flight' depth systems? Compared with stereo, what are their strengths and limitations for embodied intelligence and manipulation?

A possible issue is sensing range. Some time-of-flight (ToF) sensors we used gave good depth nearby but substantially poorer results at longer distances. That was a significant problem.

Scalability is another issue.

I saw particular potential in FoundationStereo because it is fundamentally algorithmic. It can inherit developments across RGB vision: stronger future monocular-depth models or feature representations can feed directly into stereo matching.

It can therefore iterate rapidly at the algorithm level. ToF systems remain constrained by physical hardware, which generally evolves much more slowly.

That is why I think stereo depth has the potential to achieve very strong performance.

If sensor hardware makes further breakthroughs—for example, a new depth-sensing approach—could that fundamentally change or even solve the acquisition of 3D information?

Hardware can help, but I doubt it can solve everything. Hardware is rarely perfect: strength in one respect generally comes with weakness in another.

A highly precise device may have a narrow field of view or limited range. Others struggle with black, reflective, metallic or transparent objects. It is hard for one physical device to perform well in every scenario.

The second issue is scalability. Better components and more complex imaging mechanisms generally increase cost, limiting large-scale deployment and thus data growth.

Hardware can improve precision—for example, in the final part of the data pyramid—but is unlikely to solve the fundamental problem of scaling data acquisition.

23. Future Directions in 3D Vision and Manipulation

I may next focus more on improving VLMs’ spatial understanding. Models such as Gemini already perform well on many tasks, but their understanding of 3D space remains limited.

Another direction is improving embodied training: how can 3D information help more, and how can we use simulation better? Simulators remain limited both in overall capability and in the physical phenomena they reproduce. More 3D data capturing real physical processes could enable more realistic simulation. Those are the directions I am mainly considering.

Many current vision-language models have not truly incorporated a 3D modality. Their inputs are mostly RGB images. Even at that level, questions involving spatial understanding expose limited capability. There are therefore two gaps: no explicit 3D information, and persistent limitations in spatial reasoning from a single RGB image even before considering multiple views.

References:

Project website: https://nvlabs.github.io/FoundationStereo/

https://nvlabs.github.io/FoundationStereo/

https://arxiv.org/pdf/2603.09971

https://nvlabs.github.io/FoundationStereo/

https://www.youtube.com/watch?v=WAr8ZY3mYyw

https://bingjietang718.github.io/pdfs/rss2024.pdf

https://developer.nvidia.com/blog/making-industrial-robots-more-nimble-with-nvidia-isaac-manipulator-and-vention-machinemotion-ai/

Smart Note · 3D Vision