All writing

Smart Note · Series

Robot Learning & Foundation Models

The RFM series and interviews on manipulation, VLA systems, generative models, and planning.

13 articles

RFM, Episode 1: Physical Intelligence, π0 and the Robot Foundation Model Recipe
Robot Learning & Foundation Models · EP01

RFM, Episode 1: Physical Intelligence, π0 and the Robot Foundation Model Recipe

Complete notes from Sergey Levine’s CoRL 2024 talks on cross-embodiment data, π0, pre-training, post-training and reinforcement learning on real robots.

2024-11-18 · 28 min read
RFM, Episode 2: From Octo to π0—How to Train a Generalist Robot Policy
Robot Learning & Foundation Models · EP02

RFM, Episode 2: From Octo to π0—How to Train a Generalist Robot Policy

Kevin Black’s CoRL 2024 account of Octo’s successes and failures, and the architectural and training lessons that led to π0.

2024-11-28 · 14 min read
RFM, Episode 3: Tsinghua’s RDT and the Architectures of Robot Foundation Models
Robot Learning & Foundation Models · EP03

RFM, Episode 3: Tsinghua’s RDT and the Architectures of Robot Foundation Models

An interview with Songming Liu on RDT-1B, bimanual manipulation, unified action spaces and architectural comparisons with Octo, HPT, ACT, Diffusion Policy and π0.

2024-12-16 · 18 min read
RFM, Episode 4: Comparing the VLA Architectures of π0, Helix and GO-1
Robot Learning & Foundation Models · EP04

RFM, Episode 4: Comparing the VLA Architectures of π0, Helix and GO-1

An interview-informed guide to VLA architectures, imitation learning, action chunks, asynchronous control and latent action planning, with Chiming Liu of AgiBot.

2025-03-24 · 22 min read
RFM, Episode 5: Lucy Xiaoyang Shi on Hi Robot—Teaching Robots to Listen and Think
Robot Learning & Foundation Models · EP05

RFM, Episode 5: Lucy Xiaoyang Shi on Hi Robot—Teaching Robots to Listen and Think

Lucy Xiaoyang Shi explains hierarchical instruction following, high- and low-level policies, grounding, synthetic language data and the data challenges behind π0.

2025-04-08 · 16 min read
Can Foundation Models Bring Us General-Purpose Robots?
Robot Learning & Foundation Models

Can Foundation Models Bring Us General-Purpose Robots?

A full account of the CoRL 2023 scaling debate: data, embodiment, reliability, long-horizon control, human intervention and an investor’s reflections.

2023-11-30 · 17 min read
Diffusion Policy: Generating Robot Actions with Diffusion Models
Robot Learning & Foundation Models

Diffusion Policy: Generating Robot Actions with Diffusion Models

An interview with Cheng Chi explaining Diffusion Policy, multimodal action distributions, coherent action sequences and the data needed for general-purpose robots.

2023-12-04 · 14 min read
Inside RFM-1: An Interview with Covariant Co-founder Peter Chen
Robot Learning & Foundation Models

Inside RFM-1: An Interview with Covariant Co-founder Peter Chen

A March 2024 interview with Peter Chen on RFM-1, robot foundation models, real-world data, world models and commercial deployment.

2024-03-25 · 16 min read
DP3 and iDP3: Improving Diffusion Policy Generalization with 3D Inputs
Robot Learning & Foundation Models

DP3 and iDP3: Improving Diffusion Policy Generalization with 3D Inputs

Yanjie Ze and Huazhe Xu discuss 3D representations, data efficiency, dexterous manipulation and generalization from a single training environment.

2024-11-25 · 13 min read
GenDP: Improving Diffusion Policy Generalization with Dynamic 3D Semantics
Robot Learning & Foundation Models

GenDP: Improving Diffusion Policy Generalization with Dynamic 3D Semantics

A conversation with Yixuan Wang about category-level generalization in Diffusion Policy, dynamic 3D representations and learned dynamics with model predictive control.

2025-01-02 · 14 min read
Conversation with Yunzhu Li: Learning Dynamics for General-Purpose Manipulation
Robot Learning & Foundation Models

Conversation with Yunzhu Li: Learning Dynamics for General-Purpose Manipulation

A full conversation on learned physical models, particle representations, graph networks, MPC, deformable objects and the value of modular robot systems.

2025-01-20 · 17 min read
VoxPoser and ReKep: Task Representations for Zero-Shot Robot Generalization
Robot Learning & Foundation Models

VoxPoser and ReKep: Task Representations for Zero-Shot Robot Generalization

A full interview with Wenlong Huang on spatial and temporal task representations, relational keypoints, world models, planning and zero-shot manipulation.

2025-02-16 · 24 min read
Interview with Yilun Du: Embodied Intelligence Through EBMs and Video Generation
Robot Learning & Foundation Models

Interview with Yilun Du: Embodied Intelligence Through EBMs and Video Generation

Yilun Du discusses composable energy landscapes, video planning, generalization and the research challenges facing general-purpose robots.

2025-07-30 · 26 min read