More than two months have passed since I first conceived the Robot Data series, and seven episodes have been published. For episode eight, we explore RoboGen, a particularly interesting project. Unlike collecting demonstrations through human teleoperation or motion capture, RoboGen generates robot learning experiences in simulation using generative models and differentiable simulation. Its authors come from CMU, Tsinghua, MIT, UMass Amherst and the MIT–IBM AI Lab: Yufei Wang*, Zhou Xian*, Feng Chen*, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held and Chuang Gan. The asterisks denote equal contribution.
I was delighted to interview two of the authors, Professor Chuang Gan and Dr. Zhou Xian.
In this August 2024 introduction, Chuang Gan was a faculty member at UMass Amherst and a research manager at the MIT–IBM Watson AI Lab. He earned his PhD at Tsinghua under Andrew Chi-Chih Yao and was a postdoctoral researcher at MIT, working with Antonio Torralba, Daniela Rus and Joshua Tenenbaum. His research connects computer vision, AI, cognitive science and robotics, with the overall goal of building a human-like autonomous agent able to perceive, reason and act in the physical world.
Zhou Xian is introduced here as a PhD graduate of Carnegie Mellon University's Robotics Institute, advised by Katerina Fragkiadaki. His interests span robotics, computer vision and world-model learning. Before CMU, he completed his undergraduate studies at Nanyang Technological University in Singapore, working with Quang-Cuong Pham and I-Ming Chen. His research focuses on unified neural policies and data engines for robotics and beyond. These introductions retain the original interview's timeframe.
Join me for Robot Data episode eight as we explore automated generation of robot demonstrations and an introduction to differentiable physics engines.
Original article outline
Contents
↓
Part I: Zhou Xian on generating robot demonstrations at scale with RoboGen
1. Introducing RoboGen
2. RoboGen's core idea
3. How RoboGen works
Task proposal
Scene generation
Training-supervision generation
Learning skills using the generated information
Part II: Chuang Gan on building world models with differentiable simulation
1. How differentiable physics engines work
2. Differentiable simulation for robot learning
3. Differentiable simulation and the sim-to-real gap
4. Differentiable simulation and robot data
Part III: A review of differentiable physics research
Some reflections
Part I: Zhou Xian on generating robot demonstrations at scale with RoboGen
1. Introducing RoboGen
RoboGen is a robotic agent that automatically learns diverse skills through generative simulation. Rather than using a language model or generative AI directly to produce policies or low-level actions, it takes a generative approach to creating tasks, scenes and training supervision. The aim is to scale up robot skill learning with minimal human intervention.
RoboGen overview
Several important capabilities in RoboGen come from Genesis, including scene generation and rendering and fast skill learning through differentiable physics. Genesis is presented here as a generative, differentiable simulation platform for general-purpose robot learning, unifying the simulation of different materials and a broad range of tasks. Its ambition is to combine generative AI with physical simulation to create abundant, diverse data and let robots learn many skills in varied environments. At the time of this August 2024 interview, the team said development was nearly complete and expected a public release the following month. Editorial clarification: that was a forecast, not a current release promise; the public project dates its launch to December 2024. 'Fully differentiable' describes the goal, not blanket support across every solver and interaction. Official documentation identifies differentiable MPM and tool-based interaction while other solver capabilities differ.
2. RoboGen's core idea
The idea behind RoboGen is to extract relevant information and use it to build a robot-training environment. That includes the skills a robot should learn, the setting for each skill, the environment's appearance and how the robot interacts with it. We also need clear completion criteria, usually expressed as rewards or losses. Large models, such as language models, can generate the code for those functions automatically. We want to automate every preparation step and connect the pipeline—from defining rewards to generating code—so the training process becomes highly automated from beginning to end. This is the goal and core idea behind the fully automated training pipeline we first proposed.
Reinforcement learning works well for many rigid-body interactions, both locomotion and manipulation. Deformable objects complicate matters, making differentiable components important. Fluids and clothing differ from rigid objects, and RL needs an effective representation of their state. If we move beyond rigid-body interaction, the complexity of that state space becomes a challenge. Simulators often use particle systems; a piece of cloth might contain hundreds of thousands of particles. Writing a conventional reward function that assesses those particle states can be difficult and expensive. First, the particle count makes describing the state and checking goal completion costly. Second, it is hard to write an explicit function for a deformation such as folding clothing and its corresponding particle configuration. These difficulties motivate differentiable components, which can make learning soft-object interaction more efficient. This explains their role in our approach, rather than establishing that every deformable-object task requires differentiable simulation.
Gradient information can accelerate policy learning and optimization compared with relying on reinforcement learning alone. Combining the approaches effectively is still an active research topic, without a mature framework that resolves all these issues. Gradients are not always ideal. Contact can make the optimization landscape nonsmooth or discontinuous, introducing noise and harming training efficiency. Gradients also convey local information, which may be insufficient for global optimization in complicated nonconvex problems. These are challenges the research community is still addressing.
3. How RoboGen works
RoboGen is an automated pipeline for acquiring diverse skills repeatedly. It has four stages: A, task proposal; B, scene generation; C, training-supervision generation; and D, learning skills with that information. It follows a self-guided propose–generate–learn cycle. The agent first proposes tasks and skills of interest, then populates a simulated environment with relevant objects and assets in appropriate spatial arrangements. It decomposes the high-level task into subtasks, chooses a suitable method—reinforcement learning, motion planning or trajectory optimization—generates the supervision and learns policies for the proposed skills. Repeating this loop can yield many demonstrations; 'endless' describes repeatable generation rather than a guarantee of unlimited successful skills.
How RoboGen works
Task proposal
This stage lets the language agent keep proposing new tasks autonomously. We use conventional in-context learning, providing examples that explain the requirements and allow it to propose further tasks. Controlling diversity is important. Our strategies include automatically sampling objects from large datasets and using their semantic information to generate meaningful interactions. Given a microwave, for example, the agent might propose heating soup or another food.
Tasks can also be generated from an object's affordances—the actions it makes possible—or from a scene, such as asking a robot to perform activities in a living room. The agent generalizes from examples to propose more meaningful tasks. Examples are not the only method, but they help stimulate useful ideas. In principle, we could simply ask which tasks are meaningful; in practice, concrete examples help initiate and guide the agent's reasoning.
Scene generation
A scene broadly has two parts: the background, such as the room's appearance, and the foreground of interactive objects. Each object is represented by a 3D asset or mesh, obtained using techniques such as text-to-3D or image-to-3D generation. These areas are advancing quickly, with new methods producing better assets, and serve as upstream modules for us. We ask how to arrange those assets meaningfully in space. Our initial approach was fairly rough: propose a task, identify the required objects and place them reasonably in a scene. There were usually fewer than ten objects.
We add distractors during training to increase visual diversity and make training more comprehensive. We are also working on generating scenes such as kitchens in image or video space, extracting information from them and bringing it into a virtual 3D world. Individual objects can be retrieved from large datasets or created with 3D generative models. We are developing methods for creating articulated objects beyond a fixed set of categories and expect to release them soon, as of this interview. 3D Gaussian Splatting, or 3DGS, is a representation and rendering technique, not a generative method by itself. Its advantage is faster rendering than some earlier methods. We can represent videos of real scenes using 3DGS and train generative methods in that representation space.
Using 3DGS to expand real-world data through real-to-sim reconstruction is also becoming popular; we discussed this in Hang Zhao and Ziwen Zhuang on Robot Parkour and Autonomous Driving: Where Robot Data Comes From. But a standard 3DGS representation has no physical properties of its own, making it naturally suited to noninteractive backgrounds. If interaction is needed, we can reconstruct surfaces as meshes, extract foreground objects and assign physical properties, such as mass and friction, for rigid or deformable simulation. The point is to use real-world diversity to enrich interactive virtual scenes. Alternatively, generative methods can start from noise to create new content. Generation and 3DGS representations can be combined and coupled at higher levels. Editorial clarification: mesh reconstruction is one implementation route. Gaussian representations can also be coupled to separate physical models; they are not inherently impossible to use in an interactive system.
Training-supervision generation
Reinforcement learning optimizes an objective defined by a reward function. If you want a robot to run at three meters per second, you can give higher rewards as its velocity approaches that target. Without such an objective, the agent does not know what behavior is desired. A task such as running forward must therefore be translated into an appropriate reward—for example, penalizing the difference between current and target velocity. This turns an abstract task into executable code. The language model helps design that reward, and RL then trains a policy to pursue the objective it defines.
Learning skills using the generated information
The final stage produces varied robot demonstrations in simulation. One form comes from an RL policy: a closed-loop controller choosing different actions for different states, appropriate for complex tasks that require adaptation. Another comes from motion planning: a trajectory whose actions are predetermined for a given setup. Simpler tasks, such as moving a hand from one position to another, may need spatial planning rather than RL. Generated data still needs validation, and failed demonstrations must be filtered out.
To distinguish a policy from a trajectory, consider recording one dishwashing attempt: that is a trajectory. If you record a hundred, learning can combine them into a policy able to respond to differences such as a bowl's position or appearance. One policy may still cover only a particular kitchen and dishwashing task. The eventual goal is a general policy handling, say, a thousand different tasks. Through imitation learning, demonstrations from trajectories or smaller policies can be distilled into a larger policy that brings multiple tasks together.
Part II: Chuang Gan on building world models with differentiable simulation
1. How differentiable physics engines work
Our topic, differentiable physics engines, can feel like one of those phrases whose individual words make sense but whose combination does not. I searched popular explainers on Zhihu and Bilibili. Apart from material about Yuanming Hu's Taichi and DiffTaichi, I found little systematic discussion of differentiable physics in robotics. With no better explainer at hand, I decided to write an introductory account myself.
Explanation: a differentiable physics engine simulates a physical system while allowing derivatives of its outputs with respect to relevant inputs or parameters to be computed. Conventional engines use mathematical models and physical laws to approximate motion and interaction, but often do not expose gradients suitable for direct use in learning or optimization. A differentiable engine incorporates differentiable models or numerical techniques to make that gradient information available. This concerns the represented numerical system; contacts and other nonsmooth phenomena can still make derivatives difficult.
What is simulation? Start with an object's state, let a robot take an action and predict how the state changes. That is forward dynamics or forward simulation: state, action, next state. For a robot task, we can then measure progress toward a goal and assign a reward, as in RL. If the simulator does not supply gradients, searching for an effective action sequence may be inefficient and require tens or hundreds of millions of samples. Editorial clarification: the original calls forward simulation itself a search problem. Forward simulation evaluates dynamics; searching or optimizing actions is the separate process that calls it.
The advantage of differentiable simulation is access to gradients, allowing gradient-based optimization of actions and parameters to increase reward or reduce loss. Derivatives through state–action–state transitions can guide that optimization. Without them, one can use search or approximate gradients. I see directly available gradients as a route to better and faster solutions. Editorial clarification: the original describes every gradient as mathematically exact and the result as necessarily better and faster. Gradients are of a discretized model and depend on numerical and contact approximations; gradient descent does not guarantee a global optimum or universal speed advantage.
Many existing physics engines do not provide the differentiability needed by our workflow, which limits convenient access to gradient-based solutions and leads researchers toward RL or motion planning. Trajectory optimization can be very efficient—sometimes more so than RL—but implementing suitable differentiable models is technically demanding, and progress has been slow. I believe solving this could transform robot learning. RL is commonly used to learn skills in simulation, yet many researchers find it sample-inefficient and would welcome alternatives. We want to build an efficient differentiable system. The road is long and the engineering effort substantial, but we believe it is a necessary path. Editorial clarification: trajectory optimization is still possible with analytic models, finite differences or approximate derivatives; it is not categorically unavailable without a differentiable engine. The efficiency comparison expresses our experience and research position, not a consensus ranking of every method.
2. Differentiable simulation for robot learning
Our work on differentiable physics began in collaboration with Yuanming Hu's Taichi project. Differentiable simulation is an important branch of computer graphics, where the emphasis may be making simulation more accurate and efficient. In robotics, we focus on learning skills more efficiently, using gradients to guide the process.
For example, in our 2021 collaboration on PlasticineLab, PPO and other RL approaches learned inefficiently, while our gradient-based approach was orders of magnitude more efficient on the evaluated tasks. That strengthened our conviction to build a differentiable simulation system for robot skill learning. With an efficient learning method, we can collect many demonstrations in simulation and work on real-world skill transfer. Our overall logic is to build powerful differentiable physical simulation, generate and collect abundant manipulation data, train a foundation model for skill learning, and then learn an adaptive policy to transfer those capabilities to reality. The reported speed advantage is tied to the benchmark and comparison, not every robot task.
RoboNinja, our later collaboration with Shuran Song's group, follows this logic too. It cuts varied objects, including soft objects with rigid cores—situations that are difficult to cover through teleoperation alone. We construct and collect large quantities of simulation data, train a policy and transfer it to reality. FluidLab, SoftZoo and this year's ICLR project, ThinShellLab, follow a similar line of thought. 'Unlimited data' describes repeatable simulation generation, subject to computational resources and the model's coverage.
RoboNinja demonstration
Some tasks are frustrating with RL: train for a long time and you still cannot find a useful policy. In our experience, a differentiable engine can sometimes find one in half an hour. That is why I see differentiable physics as an effective route to robot skill policies. But most simulators do not expose the needed gradients, which is a pain point. Building such an engine requires substantial mathematical and programming expertise; I estimate there are fewer than a hundred people worldwide who can write one. This motivated Genesis: bring together Chinese researchers with that expertise to tackle it. Even in graphics, differentiable simulation can be demanding work with limited recognition. Editorial clarification: the half-hour figure and expert count are the interviewee's examples and estimates. Lack of built-in differentiability does not rule out trajectory optimization through other derivative methods, despite the original's categorical wording.
Our broader aim is to build a world model for robots. Why do humans learn skills so quickly? They have a model of the world. I have long believed robot intelligence should come from world models: the ability to predict state–action–state transitions and efficiently plan action sequences. I believe a powerful differentiable physics engine can act as a robot's brain, supporting the acquisition of many skills. A world model can also be learned from real-world experience. The shared goal is more than one particular skill: it is a broad understanding and representation of the physical world. I think that kind of model is essential to meaningful generalization.
3. Differentiable simulation and the sim-to-real gap
Robot simulation faces two major challenges. One is learning skills faster; differentiable simulation can quickly find useful policies for some tasks. The other is narrowing the sim-to-real gap. Differentiability can help align simulation with reality, which distinguishes our workflow from using an engine such as PyBullet without accessible gradients. Domain randomization is commonly used there, but its parameter choices can be difficult to control. Differentiable simulation offers another option: calibrating the model with real data. Given a real-world video, for example, we can optimize simulator parameters so the simulated outcome matches what the video shows, using gradient descent to bring them closer. Editorial clarification: the transcript calls this 'Digital Tune'; the process described is parameter calibration or system identification. Domain randomization is not the only calibration method available to a nondifferentiable simulator.
4. Differentiable simulation and robot data
Let us distinguish a few concepts. RL, differentiable physics and motion planning can help find policies or action sequences in simulation. Diffusion Policy, diffusion Transformers, imitation learning and offline RL can then learn skills from available data. Demonstrations may come from real-world teleoperation or from a simulator—two collection routes. Either source can feed those later learning methods. These are distinctions between stages of a workflow, not mutually exclusive algorithm families; RL and learned models can also be combined within either stage.
We hope to build a differentiable physics platform broadly covering rigid bodies, deformable bodies and other specialized materials, together with good rendering. That returns us to the question of how to create a virtual world where robots learn and then apply their learning in reality. Turning virtual experience into real capability is our mission. A major bottleneck in robotics is data. Teleoperation is popular but difficult to scale, so I place more faith in simulation-generated data. The difficulty is that writing a physics engine has a much higher entry barrier than building a teleoperation system.
Many robot labs prefer teleoperation because it is easier to get started. Whether it is the ultimate route is another question. Simulation today feels to me a little like deep learning in the 1980s: the barrier looks high and relatively few people want to invest the effort. We are willing to work patiently on something we believe could transform the field, even if it takes three to five years or longer. We still believe simulation and differentiable simulation can address the fundamental question of where robot data will come from.
Part III: A review of differentiable physics research
General-purpose robot simulators and physics engines have largely emphasized rigid bodies, and some now have differentiable branches. Nimble Physics, developed through Stanford research, is an analytically differentiable fork of DART. I introduced DART in Let's Talk About Sim, Part I: The Robot Simulator Ecosystem. MuJoCo XLA, MJX, includes a JAX reimplementation of MuJoCo with automatic-differentiation support. Here, differentiability refers to the JAX implementation; it should not be assumed for every later MJX backend or every nonsmooth contact configuration.
Compared with general-purpose simulators, differentiable simulation is often developed by individual academic teams. The ecosystem is fragmented, without one platform dominating the applications discussed here. Simulately, introduced in Let's Talk About Sim, Part II: Interviewing the Simulately Authors, organizes these tools. Examples include DiffSim, FluidLab, DiSECt, PlasticineLab, DiffArticulated, ThinShellLab, DaXBench, Nimble Physics, MuJoCo XLA and SoftMAC. Below, I review their main characteristics.
DiffSim
DiffSim is a differentiable physics framework supporting interactions among many rigid and deformable objects. Earlier frameworks often focused on limited object classes, such as spheres or two-dimensional polygons. DiffSim instead seeks scalable support for broader geometry and topology, representing objects as meshes and exploiting sparse contacts for differentiable collision handling. Its authors report up to two orders of magnitude less memory use and computation than the evaluated particle-based methods. The original says 'at least' two orders of magnitude; the paper says 'up to,' an important distinction. The implementation uses C++.
PlasticineLab
PlasticineLab is a differentiable physics benchmark for soft-body manipulation. Many existing robot simulators focus on rigid bodies, whereas deformable dynamics are more difficult to simulate, control and analyze. A major challenge is the continuum's effectively infinite degrees of freedom and corresponding high-dimensional equations after discretization. PlasticineLab supplies varied deformable-object manipulation tasks, using DiffTaichi for differentiable elastic and plastic deformation.


DiffArticulated
DiffArticulated presents an efficient method for differentiable articulated-body simulation. It derives gradients using spatial algebra and the adjoint method, with little additional computational overhead. In the reported comparisons, it runs about ten times faster than differentiation through general autodiff tools and reduces memory requirements by roughly two orders of magnitude. Autodiff through many simulation steps can consume large amounts of memory, forcing shorter horizons or larger time steps. That can restrict learned behaviors and compromise simulation stability. These improvements are benchmark-dependent.


FluidLab
FluidLab is a differentiable simulation environment for manipulation involving complex fluid dynamics, including interactions between fluids and between fluids and solids. At its core is FluidEngine, a differentiable simulator covering multiple materials: solids, liquids and gases; elastic, plastic and rigid bodies; Newtonian and non-Newtonian liquids; and smoke and air. It is implemented using Taichi, a programming system suited to numerical and physics computation, and supports large-scale parallel GPU computation. Its gradient information supports downstream optimization tasks.


DaXBench
DaXBench is a differentiable simulation framework for deformable object manipulation, DOM. Unlike approaches limited to one type of deformable object, it covers materials including fluids, ropes and cloth. It provides a common benchmark for comparing approaches such as planning, imitation learning and reinforcement learning.

ThinShellLab
ThinShellLab is a differentiable platform for robotic interaction with thin-shell materials, such as paper and cloth, with different physical properties. Earlier work often used heuristic strategies or learned from real-world demonstrations and focused on limited material types and tasks, such as unfolding cloth. Extending those methods to more varied thin shells and tasks is challenging. ThinShellLab provides a suite of manipulation tasks and demonstrates real-to-sim system identification and sim-to-real skill transfer in several physical experiments, addressing the gap between simulation and reality.

To reconstruct a real scenario in simulation, ThinShellLab first collects physical data using vision-based tactile and force sensors.

SoftMAC
SoftMAC is a differentiable framework coupling soft bodies with articulated rigid bodies and cloth. Bringing different materials into a unified simulation is a central challenge for robot manipulation. SoftMAC models soft bodies using the continuum-mechanics-based Material Point Method, MPM. With simulators and contact models for each material, it introduces a differentiable coupling mechanism to model the interactions of soft bodies with the other two material classes.
Jade
Jade is a differentiable physics engine for articulated rigid bodies that builds on Nimble Physics and addresses interpenetration. It models contact as a linear complementarity problem, LCP, providing intersection-free collision simulation and stable solutions for multiple frictional contacts. Continuous collision detection identifies impact times, and a backtracking strategy prevents intersections between bodies with complicated geometry.

Some reflections
In the week of this August 2024 article, I went to Beijing for the World Robot Conference, WRC. It was packed. I saw many companies' robots, and plenty were brought out to walk around. The atmosphere was lively, but the products' forms were very similar. I cannot necessarily come up with ingenious new forms myself, yet I still hope to see some robots that look and work differently.
[An expressive reaction]
I also met many groups of children on study trips. Start robotics education young: tinkering with robots is joyful. Building something from nothing, then seeing it move, gives a tremendous sense of accomplishment. I hope next year's conference brings more products and technologies that make us say 'wow.'
References
RoboGen
Nimble Physics: https://nimblephysics.org/
DiffSim: Yi-Ling Qiao, Junbang Liang, Vladlen Koltun and Ming C. Lin. Scalable Differentiable Physics for Learning and Control, 2020.
PlasticineLab: Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B. Tenenbaum and Chuang Gan. PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics, ICLR 2021.
DiffArticulated: Yi-Ling Qiao, Junbang Liang, Vladlen Koltun and Ming C. Lin. Efficient Differentiable Simulation of Articulated Bodies, ICML 2021.
FluidLab: Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao-Yu Tung, Antonio Torralba, Katerina Fragkiadaki and Chuang Gan. FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulation, ICLR 2023.
DiSECt: Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg and Fabio Ramos. DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting, 2021.
DaXBench: Siwei Chen, Yiqing Xu, Cunjun Yu, Linfeng Li, Xiao Ma, Zhongwen Xu and David Hsu. DaXBench: Benchmarking Deformable Object Manipulation with Differentiable Physics, 2023.
SoftMAC: Min Liu, Gang Yang, Siyuan Luo, Chen Yu and Lin Shao. SoftMAC: Differentiable Soft Body Simulation with Forecast-based Contact Model and Two-way Coupling with Articulated Rigid Bodies and Clothes, 2023 preprint author listing from the original article. The current project page's conference citation lists Min Liu, Gang Yang, Siyuan Luo and Lin Shao, IROS 2024.
Jade: Gang Yang, Siyuan Luo, Yunhai Feng, Zhixin Sun, Chenrui Tie and Lin Shao. Jade: A Differentiable Physics Engine for Articulated Rigid Bodies with Intersection-Free Frictional Contact.
ThinShellLab: a differentiable simulator for manipulating thin-shell materials such as cloth and paper. Historical project link: https://thinshelllab.github.io/ Verified official code: https://github.com/Genesis-Embodied-AI/ThinShellLab