Smart Note · Series
Robot Learning & Foundation Models
The RFM series and interviews on manipulation, VLA systems, generative models, and planning.
13 articles

RFM, Episode 1: Physical Intelligence, π0 and the Robot Foundation Model Recipe
Complete notes from Sergey Levine’s CoRL 2024 talks on cross-embodiment data, π0, pre-training, post-training and reinforcement learning on real robots.

RFM, Episode 2: From Octo to π0—How to Train a Generalist Robot Policy
Kevin Black’s CoRL 2024 account of Octo’s successes and failures, and the architectural and training lessons that led to π0.

RFM, Episode 3: Tsinghua’s RDT and the Architectures of Robot Foundation Models
An interview with Songming Liu on RDT-1B, bimanual manipulation, unified action spaces and architectural comparisons with Octo, HPT, ACT, Diffusion Policy and π0.

RFM, Episode 4: Comparing the VLA Architectures of π0, Helix and GO-1
An interview-informed guide to VLA architectures, imitation learning, action chunks, asynchronous control and latent action planning, with Chiming Liu of AgiBot.

RFM, Episode 5: Lucy Xiaoyang Shi on Hi Robot—Teaching Robots to Listen and Think
Lucy Xiaoyang Shi explains hierarchical instruction following, high- and low-level policies, grounding, synthetic language data and the data challenges behind π0.

Can Foundation Models Bring Us General-Purpose Robots?
A full account of the CoRL 2023 scaling debate: data, embodiment, reliability, long-horizon control, human intervention and an investor’s reflections.

Diffusion Policy: Generating Robot Actions with Diffusion Models
An interview with Cheng Chi explaining Diffusion Policy, multimodal action distributions, coherent action sequences and the data needed for general-purpose robots.

Inside RFM-1: An Interview with Covariant Co-founder Peter Chen
A March 2024 interview with Peter Chen on RFM-1, robot foundation models, real-world data, world models and commercial deployment.

DP3 and iDP3: Improving Diffusion Policy Generalization with 3D Inputs
Yanjie Ze and Huazhe Xu discuss 3D representations, data efficiency, dexterous manipulation and generalization from a single training environment.

GenDP: Improving Diffusion Policy Generalization with Dynamic 3D Semantics
A conversation with Yixuan Wang about category-level generalization in Diffusion Policy, dynamic 3D representations and learned dynamics with model predictive control.

Conversation with Yunzhu Li: Learning Dynamics for General-Purpose Manipulation
A full conversation on learned physical models, particle representations, graph networks, MPC, deformable objects and the value of modular robot systems.

VoxPoser and ReKep: Task Representations for Zero-Shot Robot Generalization
A full interview with Wenlong Huang on spatial and temporal task representations, relational keypoints, world models, planning and zero-shot manipulation.

Interview with Yilun Du: Embodied Intelligence Through EBMs and Video Generation
Yilun Du discusses composable energy landscapes, video planning, generalization and the research challenges facing general-purpose robots.
No matching articles.