Back to Reinforcement Learning & Simulation

Reinforcement Learning & Simulation

Talking Simulation, Part 2: An Interview with the Simulately Contributors

Yuzhe Qin and Haoran Geng discuss the robotics simulation ecosystem, sim-to-real transfer, reinforcement and imitation learning, and the challenges of building useful simulators.

Zha Dami2024-01-2414 min readWeChat original
Contents
Simulately illustration: a robotic arm, binary digits and the simulately.wiki project name

While researching robot simulators, I discovered Simulately, an open-source project initiated by young researchers in AI robotics to share information about robotics simulation. I was fortunate to interview two contributors, Yuzhe Qin and Haoran Geng. They answered many of my questions about the ecosystem, development challenges, the sim-to-real gap, and the differences and possible combinations of reinforcement learning and imitation learning. I have organized several topics from our conversation here, hoping to make robot simulation easier to understand. The researchers’ affiliations and descriptions of the ecosystem below reflect January 2024. Before the discussion, let me introduce our guests.

At the time of the interview, Yuzhe Qin was a fourth-year PhD student at UC San Diego, jointly advised by Xiaolong Wang and Hao Su. He earned his bachelor’s degree from the School of Mechanical Engineering at Shanghai Jiao Tong University. His research includes dexterous manipulation with five-finger hands, 3D perception, reinforcement learning and imitation learning. He is also interested in realistic robot simulation and transferring policies trained in simulation to physical hardware.

His representative projects include the tactile manipulation work Robot Synesthesia and Rotating without Seeing; teleoperation and imitation-learning projects AnyTeleop, DexMV and From One Hand to Multiple Hands; and the reinforcement-learning-based dexterous manipulation project DexPoint. He is also a major contributor to SAPIEN. I discussed several of his projects in Bringing Robot Hands Closer to Human Hands: A Review of Dexterous Manipulation Research.

At the time, Haoran Geng was an undergraduate in Peking University’s Turing Class, entering in 2020, and had been named one of the university’s 2023 Persons of the Year. He began exploring research after his first undergraduate year, then joined He Wang’s Embodied Perception and Interaction Lab at the Center on Frontiers of Computing Studies. His research record included eight papers in leading venues such as CVPR, ICCV and ICRA. Two representative works received unanimously top review scores and were recognized as an ICCV Best Paper Finalist and a CVPR Highlight. He was also a visiting research student at Stanford and a research intern at the Beijing Institute for General Artificial Intelligence, BIGAI.

His projects include UniDexGrasp++, an ICCV 2023 Best Paper Finalist, and GAPartNet, a CVPR 2023 Highlight. He is also one of Simulately’s initiators. Reading his background, I could not help being impressed by how much he had achieved as an undergraduate.

Our conversation lasted two hours. Haoran joined only the first half because he had another meeting. As an outsider, I asked many basic questions, and both researchers explained them patiently. I had planned to turn the interview into a podcast, but inexperience and rushed preparation meant that the first 30 minutes were not recorded. The connection also stuttered occasionally. Even so, the conversation offers useful material for understanding robotics simulation, and I hope readers find it helpful.

Podcast: Simulately interview. Listen in the original WeChat article.

To listen directly, follow the link or search for SOTA! on Xiaoyuzhou.

Original article outline

Contents

👇

Why Start Simulately?

What Does the Simulation Ecosystem Look Like?

The Biggest Challenges in Building a Good Simulator

Choosing Simulators for RL and IL

Bridging the Sim-to-Real Gap

Simulation with RL or Real Data with IL?

Challenges in Combining the Two Routes

Why Does AnyTeleop Map Hand Motion into Simulation?

Safety When Deploying Imitation Policies Directly

Testing Real-Data Policies in Simulation

Industrial Control and Embodied AI Simulators

The Characteristics of Common Simulation Systems

3D Generation and Simulation

What Makes SAPIEN Distinctive?

Choosing Simulators for Manipulation and Locomotion

Some Further Thoughts

1. Why Start Simulately?

Haoran: Different simulators have different ecosystems, APIs, users and supported features. Their underlying architectures, performance characteristics and behavior on particular tasks also differ. Many problems we encounter during development are shared by others. We wanted an open-source project that presents those common concerns and provides a place to share information and experience, helping developers solve problems faster, especially difficulties faced by beginners.

Yuzhe: At academic conferences, I often discuss these issues with other researchers. The question is frequently not how to use a simulator, but which simulator to choose for a particular problem. Each one has detailed documentation, yet cross-platform documentation is lacking: something that explains their relative strengths and weaknesses and shares knowledge applicable across simulators.

2. What Does the Simulation Ecosystem Look Like?

Haoran: There are many simulators, with different teams having different preferences and sometimes developing their own. The ecosystem is flourishing, but choosing among diverse options is complicated. We can broadly distinguish general-purpose systems such as Isaac Sim and Isaac Gym, which have strong ecosystems and use the PhysX engine. Their extensive interfaces support tasks such as reinforcement learning and model-based learning. Another category specializes in particular materials, such as fluids, soft bodies and cloth. So one useful division is between general-purpose and domain-specific simulators.

Yuzhe: We are specifically discussing simulators for embodied AI. Other simulators existed long before this community emerged, including Ansys for finite-element analysis and COMSOL for coupled multiphysics. Their speed and intended applications kept them from being adopted directly by the embodied AI community. They often prioritize very high accuracy at the expense of speed.

Another useful distinction is between simulators supporting reinforcement-learning algorithms, such as Isaac Sim and Isaac Gym, and systems supporting model-based control. Drake, developed by Russ Tedrake’s group at MIT, is particularly suited to optimal control and model predictive control, or MPC.

Many fast simulators used in embodied AI originated in game technology. Isaac Sim, Isaac Gym and SAPIEN use PhysX, which is also used in Unity. The game industry has contributed substantially because early high-speed simulation technology was concentrated there. Traditional mechanical and electronics simulation tended to be slower and more focused on accuracy, making it less suited to the needs of much contemporary embodied AI research.

3. The Biggest Challenges in Building a Good Simulator

Haoran: More accurate simulation and more realistic rendering usually come at the cost of speed and sampling throughput. Sampling throughput may be crucial for reinforcement learning, while rendering quality can matter more for vision and some imitation-learning applications. Different research projects prioritize different features. Complexity and usability matter too. A lightweight, easy-to-use simulator may be preferable for a simple task, whereas deeper functionality can require lower-level interfaces and more complex frameworks. Terminology note: the transcript’s “sampling efficiency” here concerns how quickly the simulator produces experience, rather than how many samples a learning algorithm needs.

Robotics simulation is still developing, and different groups focus on different tasks. A good existing codebase for the same task makes development easier, so people may favor simulators with stronger ecosystems and reusable code. That can bias their choices.

Yuzhe: The field has many competing approaches and no unified standard. One reason is that simulation is strategically important, with major technology companies backing leading systems. NVIDIA backs the Isaac family, while Google backs MuJoCo. These companies want simulation closely coupled with their hardware or compilers so it runs well on their computing platforms, such as NVIDIA GPUs or Google TPUs. Some underlying differences are smaller than they appear, but companies pursue their own approaches partly to avoid depending on a competitor’s monopoly.

Rigid-body simulation is relatively mature. Soft-body simulation remains much more open, with no clear consensus on the best underlying methods. Simulation specialists in graphics may prefer to build their own systems because existing algorithms do not meet their needs. That is another reason the field lacks a single standard.

4. Choosing Simulators for RL and IL

Haoran: Researchers often have similar preferences for reinforcement learning and imitation learning, choosing the same simulator for the same task. In my own work on RL and IL, the choice is usually consistent, making development and algorithm implementation more convenient.

The algorithms do have different demands. RL, especially online RL, learns through repeated interaction and large amounts of experience, requiring high sampling throughput for rapid policy learning. It therefore needs fast simulation. For imitation learning, both dataset collection speed and data quality matter, including rendering accuracy. Large-scale dataset collection also benefits from fast simulators.

Yuzhe: First, neither imitation learning nor reinforcement learning inherently requires a simulator. Both can be used directly with a physical robot. RL, however, has high sample complexity, and collecting that much real-world data is difficult. That is why researchers turn to simulation.

Training RL in simulation then requires addressing the sim-to-real gap. That is difficult, leading some academics to be cautious about simulation and prefer real robot data. Given RL’s data requirements, some researchers instead favor approaches such as imitation learning.

For relatively simple tasks in my own research, I prefer imitation learning because people can collect demonstrations easily through teleoperation. For complex, highly dynamic tasks, real-world collection can be difficult, so starting in simulation and transferring to the real world may be the best option, often using reinforcement learning.

5. Bridging the Sim-to-Real Gap

Haoran: This is a major research concern. Broadly, the gap has two components: visual appearance and physics. Both present substantial challenges.

On the visual side, much depends on the reliability of perception. Robots need sensors such as cameras to obtain information about the world. Simulated rendering may differ substantially from real images, particularly in complex scenes or with unusual materials. There are both 2D and 3D issues, including point sampling and depth-sensor behavior. Simulation can provide clean depth information, whereas real depth sensors have noise and inaccuracies, especially on transparent or reflective materials, where distortions can occur.

We can improve the hardware and optimize its performance. At the algorithm level, techniques such as randomization help address the differences. We can also avoid particularly problematic representations, choose more reliable features and incorporate improvements early in the algorithm design.

For control and physics, simulated behavior differs from real physics. We may need to avoid especially difficult physical interactions where possible. Policies may also need properties such as adaptability or the ability to use sensor feedback for more stable control.

Yuzhe: Everyone working with simulators faces this problem. If we cannot address it, simulation’s value is limited. There are three broad approaches: build better simulators, augment the data, and find a representation or formulation where the gap is smaller.

First, improve simulation at its source. For depth cameras, conventional simulation simply uses rendered object depths as sensor readings. Other work simulates the camera’s actual operating principles more closely. The resulting depth may look less clean, but better matches real sensor behavior and reduces the gap.

Second, use data augmentation to narrow the gap. Train a model to grasp cups with many different properties—red or green, heavy or light, high or low friction—and it becomes more likely to handle a real cup successfully. Physical and optical augmentation can both help. Domain randomization is the main representative approach.

Third, find an aspect of the problem where simulation and reality already agree more closely. Simplifying the formulation can help. In Rotating without Seeing, for example, tactile sensing was simplified to binary contact information—contact or no contact—to reduce the gap. Another possibility is using derivative information, where differences can be smaller. The goal is to identify a useful point of agreement between simulation and the real world.

Video demonstration from the original article

Rotating without Seeing.

6. Simulation with RL or Real Data with IL?

Yuzhe: My main approach is to combine simulation and real data. For some relatively simple simulated tasks, such as grasping, policy transfer is fairly mature. Complex contact tasks, especially those involving changing contact points, are harder to simulate well and may require real data. I therefore favor initializing a robot foundation model in simulation and then fine-tuning it in the real world to obtain the final policy. Simulation has limitations, while real-data approaches face high collection costs. Their strengths and weaknesses differ, making a combination promising for addressing a broader range of robotics problems.

7. Challenges in Combining the Two Routes

Yuzhe: A common approach is to choose a deep-learning model, train it in simulation and fine-tune its parameters on real data. The underlying assumption treats simulation as a substitute for reality, ideally enabling zero-shot transfer before any real-world adaptation. I think this framing has limitations, particularly when considering simulation’s role in imitation learning. Simulation may offer other benefits beyond simply serving as a source environment for transfer. In a recent paper, we collected data in both the real world and simulation, using simulation for large-scale augmentation to obtain more generalizable and robust policies. Finding effective ways to integrate simulation into imitation learning remains difficult, and there is relatively little work on it. Most practitioners do imitation learning directly on physical robots. I expect researchers, including those around me, to explore this direction more in the coming years. Terminology note: the transcript calls the simulation-plus-real-data-fine-tuning process “zero shot.” Strictly, zero-shot transfer excludes real-world adaptation; fine-tuning is a separate step. The interview’s main proposal is to use simulation for data augmentation as well as transfer.

8. Why Does AnyTeleop Map Hand Motion into Simulation?

Yuzhe: Collecting demonstrations in simulation is still relatively uncommon. One reason I included it in AnyTeleop was that I was interning at NVIDIA, a company that emphasizes simulation. More importantly, I wanted to combine simulated and real data for imitation learning. The system therefore needed to work in both environments, allowing us to collect both kinds of data, compare and coordinate them, and develop stronger policies. Most teleoperation researchers instead collect data directly in the real world without using simulation.

9. Safety When Deploying Imitation Policies Directly

Yuzhe: This is a widespread issue. Deploying a robot without prior simulation validation can carry risks. Hardware-level safeguards are commonly used—for example, overcurrent protection or fuses if an arm draws too much current. At present, most such safeguards are implemented by hardware manufacturers rather than guaranteed by the learning algorithm.

10. Testing Real-Data Policies in Simulation

Yuzhe: Imitation learning from real data typically does not involve simulation. Over the past year, however, some companies and laboratories, including ours, have explored whether models trained on real robot data can run in simulation. This is called real-to-sim. Interest is growing because it enables faster evaluation, especially when a lab lacks the physical robot. Google released data from its RT-1 robots, for example, but without that hardware I cannot test a trained policy on the actual robot. Researchers therefore seek ways to deploy real-data-trained models in simulation to evaluate and understand their performance.

11. Industrial Control and Embodied AI Simulators

Yuzhe: Traditional control systems and embodied AI simulators each have strengths for different settings. The embodied AI systems we discussed emphasize speed. Contact modeling may be less detailed, but they suit individuals or labs that want to train with one graphics card. Drake, developed by Russ Tedrake’s group, can be slower while providing more detailed contact modeling, which is useful for optimal control and related applications.

More broadly, robotics simulators and fundamental mechanics tools—for example, systems that analyze whether a material deforms under force—all have value. They occupy different positions in the speed–accuracy tradeoff rather than competing in exactly the same way. As computing improves, they may become more integrated or converge, but that depends on hardware, particularly computing hardware and cloud technology.

12. The Characteristics of Common Simulation Systems

Yuzhe: Gazebo is closely integrated with ROS and primarily serves people familiar with that ecosystem. Different target users help explain why many simulators coexist. Computer-vision researchers may prefer systems such as Isaac Sim, which I describe in the interview as naturally designed for Torch users. Drake was originally oriented toward optimal control, so control researchers may find its programming model more natural. Technical clarification: Isaac Sim is an Omniverse-based robotics platform with integration into learning workflows; it is not exclusively a PyTorch simulator. Isaac Gym’s direct tensor-based RL interface is a distinct example.

13. 3D Generation and Simulation

Yuzhe: 3D generation does not inherently depend on simulation; it can supply assets to a simulator, which is downstream. Simulators need many 3D objects and models. Historically, artists built them manually, an expensive process that also requires training skilled artists. AI-generated models offer a way to enrich simulation content and build more capable environments.

Researchers are generating 3D objects and scenes and integrating them into robot simulators to construct complete environments. Most generated assets use common formats, such as meshes, which mainstream simulators can import. That generally avoids the need for custom compatibility between every generator and simulator.

14. What Makes SAPIEN Distinctive?

Yuzhe: SAPIEN has two main advantages. First, it offers a fully open-source ecosystem. Compared with commercial systems such as Isaac Sim at the time of this January 2024 interview, users can inspect the implementation, consult the code when debugging, and modify it to add missing functionality or meet their own needs. Second, it is relatively lightweight and easy to deploy, with a straightforward installation. It does not restrict users to computers with NVIDIA graphics hardware; compatible alternatives, including AMD hardware, provide more flexibility and convenience.

15. Choosing Simulators for Manipulation and Locomotion

Yuzhe: Broadly, simulators divide into rigid-body and soft-body systems. For quadrupeds, bipeds and hand manipulation, there is not necessarily a large difference in simulator requirements when the main interactions are rigid-body ones. A system can support different robot types. Choices often reflect existing code and the experience of earlier researchers rather than an independent selection. Early bipedal work favored Isaac Gym, for example, so others in the community followed it. Much of our group’s work is manipulation, so researchers building on our projects may use SAPIEN. Starting from scratch can be difficult for beginners, making existing environments attractive. The entry barrier is high, and more tutorials could help lower it.

We also discussed the difference between Isaac Sim and Isaac Gym, and whether tactile sensing is necessary for dexterous manipulation. For the full conversation, listen to my SOTA! podcast on Xiaoyuzhou.

References

https://yzqin.github.io/#research

https://touchdexterity.github.io/

https://geng-haoran.github.io/

https://simulately.wiki/docs/

https://pku-epic.github.io/UniDexGrasp++/

Smart Note · Reinforcement Learning & Simulation