Back to Robot Learning & Foundation Models

Robot Learning & Foundation Models

Interview with Yilun Du: Embodied Intelligence Through EBMs and Video Generation

Yilun Du discusses composable energy landscapes, video planning, generalization and the research challenges facing general-purpose robots.

Zha Dami2025-07-3026 min readWeChat original
Contents

Longtime Smart Note readers may remember three articles especially well: Diffusion Policy—Generating Robot Actions with Diffusion Models; the first Robot Data interview, EP1: Yang Gao and Wenchuan on Training Robot Foundation Models from Internet Human Videos (ATM, an RSS 2024 Paper with Perfect Review Scores); and the series’ final interview, EP9: Lirui Wang on Highly Heterogeneous Robot Datasets and Tasks, about POCO. They may seem unrelated, but all connect to the same person: today’s guest, Yilun Du.

He was the first researcher to apply diffusion models to robot action generation, through Planning with Diffusion, and the first to propose video prediction for robot trajectory prediction, through UniPi. All three works we previously covered connect, in different ways, to the research direction he has consistently pursued: energy-based models (EBMs). [Editorial note: these are the author’s priority claims about collaborative research. The cited project pages establish the contributions and coauthors, but do not independently establish an absolute “first” across the entire history of either field.]

I learned a great deal from speaking with Dr. Du. I am also grateful that Smart Note has given me the opportunity to discuss academic journeys and ideas with so many outstanding researchers. If, like me, you have recently been wondering whether VLAs can really make general-purpose robotics work, please listen to this episode and read the edited interview.

Podcast: Interview with Yilun Du: Embodied Intelligence Through EBMs and Video Generation (listen in the original WeChat article).

Beyond end-to-end imitation learning, many imaginative and promising approaches deserve attention: Yilun’s EBMs and video generation; the learned-dynamics-plus-MPC approach in In Conversation with Yunzhu Li: General Manipulation Through Learned Dynamics Models; and the general-controller approaches in Interview with Xue Bin (Jason) Peng: General Controllers for Humanoid Whole-Body Motion and Interview with Zhengyi Luo: PHC, a General Humanoid Controller. Each deserves to be seen by more people in the industry.

At the time of this July 2025 interview, Yilun Du is introduced as an assistant professor at Harvard’s Kempner Institute and Department of Computer Science, and a senior research scientist at Google DeepMind. He earned his PhD in MIT’s Department of Electrical Engineering and Computer Science, advised by Leslie Kaelbling, Tomás Lozano-Pérez and Joshua B. Tenenbaum. He also earned his undergraduate degree at MIT, worked as a researcher at OpenAI, held internship and visiting-research roles at FAIR and Google DeepMind, and won a gold medal at the International Biology Olympiad. These affiliations describe the original publication period.

His research spans generative models, decision-making, robot learning, embodied agents and scientific applications of these tools. He aims to develop intelligent embodied agents that act autonomously in the physical world. In particular, he uses generative AI to build world models and introduce systematic planning and iterative reasoning into learned agents. Key challenges are limited modeling data and generalization to unseen situations. He addresses them through composable generative models, using learned energy landscapes—energy-based models—to generalize beyond limited datasets. His early EBM work also helped advance the development of diffusion models in 2020.

Original article outline

Contents

👇

Academic background

1. Important research projects

Embodied intelligence through composable energy landscapes

2. Generalizable systems through composable energy landscapes

3. What is an energy landscape?

4. A closer explanation of energy-based models

5. Diffusion models and energy landscapes

6. Combining diffusion models with robotics

7. Why continue working on EBMs?

8. Why compositionality and generalization?

9. What does compositionality mean, and what is composed?

10. How much attention does this approach receive?

Embodied intelligence through video generation

11. Promising directions in embodied AI

12. Current video-based approaches

Understanding intelligence

13. What does intelligence mean to you?

14. Have language and visual intelligence been achieved?

15. How far away are general-purpose robots as an AGI goal?

Discussing approaches to embodied intelligence

16. Which approaches can be deployed soon?

17. Why favor video-generation-based planning?

18. Views on end-to-end VLAs

19. Why do VLAs attract attention despite disappointing results?

20. Reservations about reinforcement learning

21. The biggest challenges for general-purpose robots

22. Task memory versus 3D spatial memory

23. Intelligence from language versus physical experience

24. Where might future breakthroughs occur?

Research directions for general-purpose robots

25. Why has embodied-AI research become less varied?

26. Advice for newcomers choosing research directions

Academic background

I have just started as an assistant professor at Harvard. Before that, I spent a year at DeepMind, and earlier earned my PhD at MIT. My research mainly combines generative models with robotics.

Early on, I did a lot of work on energy-based models. Around 2018, I spent a year at OpenAI, where we began exploring early compositional models. From 2018 or 2019, we pursued larger-scale modeling of dynamics and gradually developed toward score-based and diffusion-based methods. I therefore began exploring diffusion models early and applied them to robot tasks, bringing EBM and diffusion ideas into robotics. [Editor’s note: The transcript’s labels “Early compositional models (ECM),” “large-volume dynamics” and “square-based” appear imprecise. They are descriptive transcript wording, not verified formal method names; “score-based” is the relevant established term discussed below.]

Robotics is our lab’s core direction, so I have consistently explored applying generative-modeling ideas to intelligent robots. The goal is a genuinely intelligent robot that performs varied tasks in varied environments—for example, chopping vegetables, cooking and washing dishes at home.

Alongside general intelligent robots, I think about making generative models more scalable, using them to accomplish complex tasks better, and even developing entirely new generative-model paradigms.

1. Important research projects

First, one of my early papers on EBMs was fairly innovative. Most researchers were working on GANs, while we were among those exploring a denoising-like process for image modeling early on.

Later, we studied compositional generation: combining different energies to generate samples. I find that valuable. We also did vision work on using EBMs to handle images with different styles or content.

Then came Planning with Diffusion and the subsequent Diffusion Policy work.

We also did several video projects, including video generation in UniPi and video–language planning, combining video, language and action models to plan complex tasks.

Another project, Multi-Agent LLMs, studied how multiple large language models could collaborate to complete tasks.

Finally, our recent work explores reasoning with energy-based models, a direction I pursued toward the end of my PhD.

Those are the main papers, exploring architecture innovation, efficiency and multimodal collaboration. Most center on EBMs, though not all. Multi-Agent LLMs, for example, combines different language models into a multi-agent system rather than relying entirely on EBMs. Overall, EBM ideas run through most of my doctoral research.

Embodied intelligence through composable energy landscapes

2. Generalizable systems through composable energy landscapes

My PhD thesis is titled Learning Generalizable Systems by Learning Composable Energy Landscapes. Its three key ideas are generalization, energy landscapes and composability.

I think robot learning’s biggest problem is insufficient large-scale data. If I want to train an LLM for natural-language processing, there is already abundant online text. But to teach a robot to wash dishes or cook, the available data are limited. Even collecting some human demonstrations cannot cover all environments and task configurations.

My central question is therefore: with limited data, how do we design a system that performs well on unseen scenes or tasks? That motivates what I call a compositional structured system, or CSS in this discussion.

The core idea is compositional modeling. Rather than one enormous model, decompose the problem into smaller models focused on different subtasks. Recombine them as needed for new task settings. This can improve generalization, particularly on entirely unseen tasks.

It resembles programming. To build a complex system such as a web server or search engine, you combine small functions rather than write one giant function from scratch. Likewise, we want a library of skills or modules in robotics that can combine flexibly to execute complex tasks.

A single large end-to-end policy network is extremely sensitive to changes: a slightly smaller robot, a higher table or brighter lighting may cause the system to fail completely. Compositional modeling decomposes the task into more stable, reusable subproblems, improving adaptation to new environments.

We use EBMs to compose these modules. Energy functions allow composition driven by optimization and suited to continuous spaces, rather than simply making one model depend on another’s output.

Each submodel can be an energy term representing a constraint: one model handles the first constraint, another the second, and so on. Optimizing their combined energy yields the final control strategy. Modules need not be called sequentially; composition becomes more flexible and structured.

As generative AI rapidly develops—LLMs, video models and decision models—an energy-based composition framework can integrate models with different training methods and functions. You might have a language model, a video model and an action model. The energy-based formulation provides a mechanism to integrate them into one system. That is its strength.

3. What is an energy landscape?

An EBM is a relatively longstanding type of generative model. It learns an energy function that assigns lower energy to real data, such as real photographs, and higher energy to samples inconsistent with the distribution. Informally, energy distinguishes good samples from poor ones.

An EBM defines a probability distribution through an energy function. Each input—an image or state, for example—has an energy value: lower energy means a more likely or plausible input, while higher energy means a less likely one. Unlike the direct generation mechanisms of GANs or VAEs, an EBM learns to evaluate whether a sample fits the distribution. Sampling is a separate procedure rather than a direct output of the energy function itself.

We explored generating images with EBMs in early 2019 and published an OpenAI blog demonstrating realistic image generation. In several respects, that resembles later diffusion models, especially the sampling dynamics. We sampled images using gradients and a method similar to Langevin dynamics. This dynamics-based sampling later became one of the foundations of diffusion models.

Langevin dynamics is a mathematical description of particles moving in a thermal bath, widely used in statistical physics, sampling algorithms such as MCMC, and generative modeling. Its central idea is movement guided by potential energy while also influenced by thermal noise and friction.

Our later score-based and diffusion-based generation methods can be understood as continuations or evolutions of that early EBM work. In that sense, our work was an important precursor to these developments, rather than the sole origin of diffusion models.

4. A closer explanation of energy-based models

The concept connects to graphical models studied roughly two or three decades earlier. Directed and undirected graphical models describe complex systems through products of factors. Each factor models a local constraint or substructure, and their combination represents the whole system’s joint distribution.

An EBM is essentially a form of that undirected graphical-model formulation. Each component can be expressed as an energy function, and their combination determines the overall behavior. (Author’s aside: I do not yet understand EBMs thoroughly enough to explain them in depth here. I hope to develop a better understanding after encountering more work in this area.)

We brought this energy-composition idea into neural networks and extended it to embodied AI. Rather than only learning one end-to-end model, we decompose tasks into modules representing local structures or subtasks, express them as neural-network energy functions, and optimize their total energy to form a complete strategy. It is a neural-network version of ideas from early graphical models.

Relatively few researchers had systematically applied this kind of energy composition in neural networks. We are trying to carry the classical idea forward into modern machine learning, especially embodied intelligence, and explore applications beyond robotics.

5. Diffusion models and energy landscapes

I began studying diffusion models very early, around 2018–2019, when we were working on EBMs. We found that EBMs and score-based models were almost identical. [Editorial note: this is the interviewee’s account of their conceptual connection; the qualifications on mathematical equivalence are explained below.]

A score-based generative model generates samples by estimating the gradient of the data distribution’s log probability density—the score function. The central idea is to learn the direction toward higher data probability, then progressively move from noise toward realistic data.

In an EBM, you learn an energy function and generate images using an energy-based sampler such as Langevin dynamics. A score-based model does not explicitly learn energy; it directly learns the gradient instead, the score function. Either way, both use the same Langevin sampling process, so their mechanisms are consistent. [Editorial note: for a density proportional to exp(−E), the score is the negative gradient of energy. The interview describes their connection intuitively; not all score-based samplers are ordinary Langevin MCMC.]

Many papers say these models are essentially the same, only expressed differently. One could even argue that there is no need for a separate name, “score-based model”: it can be regarded as a form of energy-based model. From that perspective, a diffusion model can also be viewed as a special score-based model, so diffusion and EBMs are very closely connected. [Editorial note: not every parameterized score field is the gradient of a scalar energy, and not every diffusion sampler is identical to ordinary Langevin MCMC. These are the qualifications to this intuitive explanation of their connection.]

Imagine an energy landscape where some photographs have low energy and others high energy. To generate a sample, start from noise and optimize along the energy function: take a gradient-descent step toward lower energy, add some noise, take another step, add noise again. That is Langevin sampling. Repeating it brings you to a low-energy region and a plausible image—the generated sample.

The whole Langevin sampling process is essentially “gradient plus noise,” exactly like a diffusion model’s reverse process. In that sense, the two models are very similar. [Editorial note: “exactly like” is the interviewee’s intuitive comparison. Their sampling mechanisms have important mathematical connections, but are not universally identical, as explained above.]

6. Combining diffusion models with robotics

My early work focused on EBMs, around 2019. As we recognized their close relationship with diffusion models, we reframed ideas from earlier EBM papers in diffusion terminology and wrote related work.

One paper was Model-Based Planning with Energy-Based Models. We later transferred ideas from that 2019 framework into diffusion modeling. I think this was among the earliest attempts to apply diffusion models to robotics.

Then, with Shuran Song and Cheng Chi, we tried applying the method from Planning with Diffusion for Flexible Behavior Synthesis

to real robot tasks. The experiments worked well, forming the background to Diffusion Policy—Generating Robot Actions with Diffusion Models.

7. Why continue working on EBMs?

I find these models fascinating. Generative modeling was very exciting then, particularly ideas involving editing and decision-making—things unlike what I had seen before. At OpenAI, my colleague Ilya Sutskever was enthusiastic about EBMs. He often told me he believed generative models over the next ten or twenty years would be based on them. We discussed this every day, and I came to believe it too.

When I started my PhD, diffusion models were gaining popularity, and I already liked EBM ideas. We wrote several papers. Although these models were not yet especially popular and there were relatively few papers, I saw how close they were to later diffusion models.

I thought we could rewrite earlier work in diffusion terminology so more people would read and accept it. Those papers had received little attention because few researchers worked on the area or understood it. Using the language of diffusion could make the ideas more visible.

8. Why compositionality and generalization?

If you have energy functions—energy landscapes—combining two models is straightforward: add their energies. Each defines what samples it favors. With the energies added, sampling favors solutions compatible with both, informally “between” their requirements.

That is an advantage of energy models: multiple models compose naturally through addition. Expressing a model in terms of energy is therefore well suited to combining models.

The second point is generalization. Composition can let a system perform well on unseen tasks or environments by combining more capabilities and constraints to address diverse unknown tasks.

A persistent core question in robotics is: with little data, how can robots still achieve goals in unseen settings? Much current imitation learning and Diffusion Policy training relies on limited demonstrations, and the models can perform well only in similar environments.

That is not yet a truly smart robot. Many simulation-RL projects repeatedly train in simulation, then test in an almost identical real environment. That is limited generalization.

A genuinely intelligent robot must perform well on environments and tasks it has not seen. That is robotics’ most fundamental challenge.

9. What does compositionality mean, and what is composed?

Composition can occur at different levels. For example, an LLM can perform high-level reasoning: given “I want to cook,” it generates steps such as chopping vegetables or finding ingredients.

A second level, such as a video model, translates those instructions into visual plans or representations of the actual scene and object states.

An action model at the third level turns visual information into execution, such as robot-arm trajectories.

There are many ways to combine models, and the components need not only be policies. You can combine a vision model and an action model, for example.

You can also combine constraints. In grasping, the arm must avoid other objects, successfully grasp the target, and place it smoothly and continuously rather than jumping abruptly. Each constraint can be modeled by an energy function and combined into a policy or plan.

Dynamics and policy models can also be jointly optimized for better control.

There are many forms of composition, involving models or constraints. These resemble ideas in classical robotics algorithms. We can borrow those ideas and translate them into modern compositional-model frameworks.

10. How much attention does this approach receive?

Not many people currently use it. The prevailing question is whether, given enough data, end-to-end training will directly learn effective capabilities.

I think in the coming years we may find that even enormous datasets do not necessarily make simply training a huge model work especially well. People may then consider adding more structured design.

I believe structural composition will increasingly be adopted. It remains early: most researchers are first trying large end-to-end models to see what happens. My estimate, at this interview, is that wider recognition and adoption may come in two or three—or three or four—years.

Embodied intelligence through video generation

11. What other directions in embodied AI look promising?

Recently, I have mainly worked on video models, using them to drive robot policies directly or for end-to-end learning.

LLMs are powerful, but I think their physical-world understanding and low-level control capabilities are limited. Video models have an advantage in learning physical knowledge. Seeing doors opened or cups grasped does not give perfectly precise physics, but it provides substantial physical information and an understanding of concrete movements.

A video model can show what chopping vegetables looks like, which helps execute the task. An LLM has less direct understanding of the physical process. Asked how to move between places or cook, it often omits physical details and gives a shallow account of reality.

Video models directly use visual information and therefore have a more intuitive connection to the physical world. Combining them with robotics is interesting and promising. Although research is growing, it still receives less attention than LLMs or large multimodal models. I think it is more meaningful and suited to robots’ perception and action requirements.

12. What video-based approaches exist, and which do you use?

One common approach uses a video model to generate data, then feeds those data into a VLA model to improve downstream performance.

Our approach emphasizes using the video model itself as a planner, as in UniPi. Given the current image, it generates a future video sequence. We infer the action to take from that predicted sequence. Knowing the desired position at each future frame makes determining the current action comparatively straightforward, using a controller.

Our pipeline uses video for high-level planning, executes the plan through a closed-loop controller, and replans with the video model after several steps. We think that structure is more reasonable and promising.

Other work either does video-based planning directly or generates video data to train another model. In our observations, current VLAs have limited generalization and control quality. Inverse-dynamics labeling of generated video is often inaccurate too, so generating data and then training does not work as well as hoped. We think directly using the video model as a high-level planner is a more reasonable and effective path.

Generating data with video models means synthesizing images of robot trajectories. First generate video or images of different tasks; then use an inverse-dynamics model (IDM) to label actions. The resulting visual–action pairs expand the training dataset for stronger visual–action models. [Editor’s note: The source gives “VLM” as an example, but action-predicting models are more accurately described here as visual–action models or VLAs.]

Understanding intelligence

13. How does the intersection of generative models and robotics serve your understanding of intelligence?

I think intelligence, or AI, ultimately needs robotics as a physical carrier. Truly capable AI must complete varied tasks in real environments. Vision, natural-language processing and machine learning solve smaller problems that are parts of this larger one.

The central question is how to build AI with human-like general capability—AGI. I see that as essentially the same question as building a general-purpose robot.

For me, general-purpose robotics is AI’s ultimate problem. We are still far away, and many methods and details remain unclear, but I regard it as the eventual goal. Many other research directions are subproblems within it.

14. Have language and visual intelligence been achieved?

I do not think language understanding is completely solved. If I treat a model as an agent and speak to it extensively, its behavior is not ideal. It may answer a single question well, but multi-step interaction requiring a response at each stage noticeably reduces performance.

Vision is more complicated. Models often struggle with detailed descriptions such as “there is a table at the far right of the chair, a cup of water at the table’s upper-right corner, and a bowl above the cup.” Giving a photo and asking exactly what it contains can also produce disappointing results.

So although some language and vision models look impressive, I think they remain far from flexible, accurate understanding and interaction.

15. How far away are general-purpose robots as an AGI goal?

I think it is early to pursue general-purpose robots. There can be a misleading impression that collecting enough data and training a task model guarantees success. We ran many experiments, including a Gemini Robotics project while I was at Google. I felt our VLA was substantially stronger than others.

Yet performance in real environments fell far short of expectations. Good-looking metrics do not mean a usable robot system is close. I therefore think many startups pursuing general-purpose robots may be starting too early. These judgments reflect my experience at the time of this interview.

Discussing approaches to embodied intelligence

16. Which embodied-intelligence approaches can be deployed in the short term?

I personally see promise in video-model planning for relatively simple factory tasks: inserting a component into a hole, assembling a few pieces, or placing one object onto another. Video models may work well there. Imitation learning, such as Diffusion Policy, can also train decision policies. Both could produce practical short-term results on simpler tasks.

I am less optimistic about RL for these problems. In our earlier attempts, hard-coding an entire simulated environment was inferior to using demonstrations for the same task, and generalization was limited. My experience with VLAs is that they offered little help for practical deployment, so I am also skeptical of that route in the short term.

17. Why favor video models as planners?

Traditional robotics has many motion-planning methods. Planning is central to more general, generalizable robots. I have participated in several diffusion-based planning projects and consider this an important, promising direction.

18. Views on end-to-end VLAs

The idea sounds reasonable, but I have a question: action data and language data are very different. Mixing them in training may lose some reasoning and causal information. I have not yet seen a clear benefit from directly applying LLMs to action learning.

I think that path has problems. Many alternative directions exist, and a startup might do much better by trying them rather than simply reproducing the VLA pattern. In my experience, these VLAs perform poorly in practice; actually trying them makes that apparent.

19. Why do VLAs attract attention despite disappointing results?

I think people strongly trust LLMs’ capabilities. Although their visual performance is less impressive, their language performance leads people to believe combining vision with language should be very useful.

Adding actions makes the story seem complete: a model understands language and vision and can act. It is persuasive, so many teams choose it. My view is that actual results may be less satisfactory and remain some distance from usable systems.

20. Reservations about reinforcement learning

I see several key challenges for RL in robot control.

First, reward design is often unsatisfactory. In locomotion, I can specify what the model should do at each step, making rewards clear and RL effective. In manipulation, most intermediate steps lack an obvious reward, making good policies difficult to learn.

Another issue is the state space. In locomotion, the observed state is low-dimensional and the sim-to-real difference is small, so RL-based policies can still work. In whole-body control or manipulation, physical discrepancies are larger and the state space is very high-dimensional. It is difficult to guarantee good results when directly transferring RL training in simulation to the real world.

Third, horizons are longer. Locomotion can receive rewards at each step, while manipulation may require thousands of steps before feedback, making RL learning difficult.

Those are, in my view, the biggest challenges for RL in robot control, especially complex manipulation.

People still attempt RL manipulation, but I think most attempts have not yet achieved the desired results. They often address very simple tasks, such as grasping and rotating a cube. Rewards are clearer, states simpler, and the task may involve local movements rather than complex motion of an entire arm.

At this interview, I therefore see much RL manipulation work as still addressing comparatively simple problems. These are my assessments of the field, not a claim that no successful manipulation-RL systems exist.

21. The biggest challenges for general-purpose robots

Several issues matter for general intelligence, and I think some larger challenges remain.

First, most current policy models have no memory. They predict actions from the current frame without using history. If a robot moves and an object temporarily leaves view, it forgets where the object was. Fixed scenes with unchanging object positions hide this problem. Genuine intelligence needs memory, a crucial capability receiving too little attention.

Second is fast learning. Humans quickly learn new environments or tools. Existing models often lack continual-learning mechanisms to absorb and adapt to new knowledge.

Third, online learning matters. Humans learn extensively while interacting with their environment. Many models separate training from deployment and lack continuous online adaptation.

Finally, perceptual diversity matters. Humans manipulate using touch and force as well as vision. Tactile and force signals are especially important for dexterous manipulation, but are used relatively little. This is an overlooked and significant direction.

22. Is task memory different from 3D spatial memory?

The memory I mean differs from the spatial memory used in many localization and navigation systems. An autonomous vehicle may remember a 3D map; I am talking about dynamic, task-related memory.

Suppose a robot opens a cupboard, searches it and closes the door. How does it remember which items it saw? A 3D representation alone struggles with that. It describes spatial structure, but does not necessarily record “I searched this cupboard, and what I need is not there.”

So 3D spatial memory alone is often insufficient. An intelligent agent needs more flexible, semantic memory of specific events during its actions.

23. Intelligence from language versus physical experience

Looking at human development, I feel we spent a great deal of time learning motor-control skills and physical regularities through our environment. This learning long preceded language, which suddenly developed only in the last ten thousand years of human history. Only with language did what we later called “intelligence” emerge. [Editorial note: the claim that spoken language arose in the last ten thousand years is not an established historical date. Research on language evolution places its origins much earlier, although the exact chronology is debated. The interview’s broader point is the distinction between physical experience and linguistic knowledge.]

Historically, human skills developed through interaction with the real world. Understanding human intelligence may require building a smart robot first. In my view, intelligence fundamentally comes from interaction with the world, rather than language.

Language models attract much attention, but I do not think they truly understand the physical world. Given several objects and asked to reason about moving them between places, they often struggle with physical causality. Given photos and asked “how should I throw this object over there?”, they may also fail to answer.

I therefore wonder whether language alone, without physical-world experience, can teach intelligence. Current models particularly encounter problems on long, complex tasks requiring a dozen or twenty-odd consecutive reasoning and action steps.

Such capabilities may emerge gradually only through genuine environmental interaction. Biological and historical perspectives suggest returning to the physical world is hard to avoid. I cannot clearly see whether language models alone lead to the AI we imagine. Many people are trying, but that route to “real AI” is not clear to me.

24. Where might future breakthroughs occur?

I cannot say what the next biggest breakthrough will be. If I knew, I would probably already be working on it! (Laughs.) My impression is that too much work looks alike. About five years ago, conferences featured papers with diverse ideas and directions. Now almost everything seems connected to LLMs or diffusion, and much of it is similar.

We should think about different directions; that is most interesting to me. Many researchers seem to agree that attaching an LLM or vision model to a robot will solve intelligence. I do not believe the route is so direct.

It reminds me of the optimism around 2015–2016: “We have perception and manipulation; add some RL and we can solve general intelligence.” It was not that simple. Today’s thinking sometimes similarly stacks existing modules and assumes every task will work. A little reflection suggests it may be harder.

Overall, I think research perspectives are too short-term, focused on existing frameworks rather than looking further ahead.

Research directions for general-purpose robots

25. Why was embodied-AI research more varied before its recent popularity?

I see several reasons for its increasing homogeneity.

First, many companies pursue this area, particularly an LLM-plus-vision-plus-action pipeline. When companies use it for demos, papers and fundraising, others naturally follow. People around you are doing it, and you may want to work for those companies. That is a practical incentive.

Second, publishing has become more competitive. If you do not pursue fashionable directions—stacking LLMs and VLMs to get flashy results—people may overlook your work. The field and information volume are so large that attention goes to visible work matching the dominant paradigm. Less fashionable but deeper work can disappear from view.

A deeper issue is the field’s rapid expansion and influx of newcomers. Lessons learned are not passed on effectively. Researchers with several years of experience may know a direction is difficult or limited, while newcomers choose seemingly simple, direct routes: add an LLM and more data to get a result.

That creates a cycle: more people repeat apparently reliable approaches, while the field becomes stuck and lacks real progress.

The problem is complex: industry trends, academic incentives and rapid expansion all contribute. When everyone rushes to catch up, homogeneity becomes likely.

I consider it the biggest current problem, especially in the last two years. Sometimes every month’s ideas feel identical, even repeating after three years. Many newcomers lack historical grounding and produce apparently new ideas that are actually old.

26. Advice for newcomers choosing research directions

I have always thought researchers can choose less fashionable areas. I do not rush papers or worry much about someone publishing an identical one first. If a direction is sufficiently independent and novel, the paper remains valuable and interesting even if submitted a year after completion. In crowded areas, such as popular neural-network topics, differentiation is harder.

Try choosing a direction farther from the crowd. One test is: if I ignore current papers and trends, explore my own ideas for one or two years, and still produce an interesting paper, the direction is probably sufficiently independent.

Nor do I think reading must focus on the latest papers. Work from forty or fifty years ago, including classical books on AI and model-based approaches, contains ideas that remain inspiring. I like reading those first to understand the essentials, then looking at some newer work. Many traditional AI ideas are more interesting to me.

For example, Marvin Minsky wrote Steps Toward Artificial Intelligence around 1960. We still use many ideas discussed there: learning, now often through neural networks; reinforcement learning; planning; search; and updating networks. They resemble today’s prominent research directions. [Editor’s note: The paper was published in 1961.]

Many things that look new were systematically proposed decades ago. For example, a year ago RL was not popular, and now it is fashionable again—but people were working on it long before. Ultimately, reading some older papers shows that many of the core ideas are already there.

I think meaningful innovation often comes from people who understand that history deeply. New papers can sometimes obscure a sense of direction, whereas revisiting early work teaches a great deal.

References

Yilun Du’s personal website:

https://yilundu.github.io/

UniPi: Learning Universal Policies via Text-Guided Video Generation

https://research.google/blog/unipi-learning-universal-policies-via-text-guided-video-generation/

Learning Generalizable Systems by Learning Composable Energy Landscapes

https://yilundu.github.io/thesis.pdf

Model-Based Planning with Energy-Based Models

https://arxiv.org/abs/1909.06878

Planning with Diffusion for Flexible Behavior Synthesis

https://diffusion-planning.github.io/

Diffusion Policy

https://diffusion-policy.cs.columbia.edu/

Steps Toward Artificial Intelligence

https://courses.csail.mit.edu/6.803/pdf/steps.pdf

Smart Note · Robot Learning & Foundation Models