In our Robot Foundation Model (RFM) series, we have already introduced π0 in RFM, Episode 1: Physical Intelligence and the π0 Embodied Foundation Model, and Octo in RFM, Episode 2: Octo. Today we introduce a robot foundation model developed in China: Robotics Diffusion Transformer, or RDT-1B, presented in the paper RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation. It is a diffusion model for dual-arm manipulation built on Diffusion Transformers, or DiTs.
This article is primarily based on my interview with Songming Liu, a co-first author of RDT-1B. In addition to discussing RDT-1B itself, I asked him about architectural differences among several leading robot foundation models. RDT's co-first authors, Songming Liu and Lingxuan Wu, and their coauthors are researchers in Professor Jun Zhu's group in Tsinghua University's Department of Computer Science. RDT benefits from the group's extensive experience developing large models. Embodied intelligence needs this expertise and research to intersect with robotics, and I look forward to that integration accelerating. Editorial note: the comparisons below preserve the interviewee's assessments at the time of the interview. Octo and HPT are not simply autoregressive action models; HPT supports several kinds of action heads and compresses inputs through its stems. π0 uses flow matching rather than a conventional diffusion denoising objective. Here, the discussion groups these continuous generative approaches together when contrasting them with token classification or small policy heads.
Original article outline
Contents
↓
1. Introducing RDT-1B
2. The RDT Architecture
3. Architectural Differences from Octo and HPT
4. Architectural Differences from ACT and Diffusion Policy
5. Comparing RDT with π0
6. Transformer Backbones with Diffusion Heads vs. Diffusion Transformers
7. Further Questions
1. Introducing RDT-1B
RDT-1B is a robot foundation model with 1.2 billion parameters. At the time of its release, it was the largest diffusion-based foundation model for robotic manipulation. We pretrained it on what was then the largest collection of multi-robot datasets: forty-six datasets containing more than one million manipulation episodes.
To strengthen its bimanual manipulation capabilities, we collected more than six thousand episodes on the ALOHA dual-arm robot—one of the largest bimanual datasets available at the time—and used them for fine-tuning. RDT-1B established a new benchmark for dexterity, zero-shot generalization and few-shot learning in our evaluations. It supports control of almost all modern robot arms, including dual-arm systems, joint commands and end-effector commands, and even wheeled locomotion.
One significant advantage of our work is its focus on bimanual tasks. This is a very difficult area, and few teams had made breakthroughs on these tasks. Most robot foundation models at the time, including Google's RT series and ByteDance's GR series, focused on single-arm tasks. We saw ourselves as the only team explicitly targeting bimanual manipulation and obtaining practical results in this setting. That is our core strength. A dual-arm action space has twice the dimensionality, making its distribution more complex. We designed an architecture suited to that complexity, which allows our model to handle these tasks successfully. Actions are also constrained by nonlinear physical laws. Under those conditions, bimanual tasks are even more challenging, so it is understandable that other teams chose not to focus on them. This is our assessment of RDT's positioning among the foundation models being compared, rather than a claim that earlier bimanual projects such as ALOHA did not exist.
RDT's task success rates improved substantially over earlier baselines such as OpenVLA and Octo. We did not test HPT because it was concurrent work: it appeared around September 30, by which point we had already submitted our paper. We therefore did not have an opportunity to compare it, and we do not know its specific performance. Octo and HPT share several implementation ideas; from this perspective, they are different representatives of a similar family of methods. I would expect HPT to encounter some of the same issues as Octo, although that is an assessment rather than a tested result. In our experiments, Octo's performance was not particularly good.
Success rate can be evaluated in many different ways. If we combine difficult and simple tasks, our average success rate reaches roughly 60–70%. In our evaluation, it was the best-performing approach.
2. The RDT Architecture
Our work mainly asks how to train more effectively on the data already available. We therefore chose an architecture combining diffusion models with Transformers. This combination is widely used in text-to-video generation, particularly in DiT and U-ViT, a Vision Transformer backbone for diffusion models. We adapted these models to embodied tasks, hoping to obtain scalability similar to that seen in text-to-image and text-to-video generation. We made architectural and training adjustments so that the model could learn more efficiently from data collected on different robots. Our work concentrates on model design and training on heterogeneous data.

3. Architectural Differences from Octo and HPT
There are similarities, but there are also differences. HPT and Octo generally use a Transformer backbone followed by a diffusion head or a regression head. This differs from our approach in two respects. First, in a backbone-plus-head design, diffusion takes place only in a relatively small head, whereas our entire backbone performs denoising. We think a small diffusion head can substantially restrict the model's ability to represent multimodal action distributions. Professor Sergey Levine also mentioned this in a recent talk, describing it as something they had not done well in Octo. In some situations, it cannot adequately represent complex distributions in the action space, particularly distributions with multiple modes. We therefore prefer a diffusion architecture throughout, as in video and image-generation models, to maximize the model's generative capacity.
Some people worry that diffusion sampling may be slow, or that slow sampling could affect generation quality. That concern did not emerge as a significant problem in our experiments. We used inference-acceleration techniques that made the model quite fast. In the future, diffusion-distillation techniques could accelerate it further, potentially enabling one-step sampling and resolving the sampling-speed issue. Under our current conditions, inference speed is not a problem; at least it is not worse than autoregressive models or conventional Transformer models. One advantage of diffusion, in my view, is that loss computation is efficient and sampling takes place in a continuous space. This can model complex distributions more effectively than some alternatives.
Another difference from Octo and HPT is how they handle different robot embodiments. They aim to train separate encoders or decoders for different robots. HPT, for example, can train one shared backbone while using different encoders and decoders for different action spaces. I see some issues with this. In my comparison, processing image and other observation tokens through Transformer self-attention can create substantial computational overhead, potentially making it much slower than a diffusion model. Image tokens can greatly outnumber action tokens. Multiple sensors may produce hundreds or even thousands of tokens, and self-attention has quadratic complexity in their number. By contrast, our DiT uses cross-attention with action tokens as its main sequence. There are usually only a few dozen action tokens, so attending to video or image information can require much less computation. Editorial clarification: as noted above, HPT's actual implementation includes stems that compress observations, so this comparison should not be read as a claim that it always self-attends over all raw image patches.
Training different encoders for different action spaces means different robots are represented in different spaces. To make good use of their data and learn shared physical regularities, those spaces need to be aligned. There is not yet a clear alignment strategy. One possibility is paired data in which different robots perform the same task in the same scene, but that kind of data is obviously scarce. In our model, by contrast, the encoder, decoder and every network layer are shared across robots. Learning naturally takes place in one unified space, which makes it easier to learn shared regularities.
Consider again HPT's use of different encoders for different embodiments and structures. This approach needs enough data to learn an encoding for each embodiment. With insufficient data, it can struggle. Language models succeed partly because they have enough data to learn flexible formats. In embodied tasks, however, data volumes can vary greatly between robots, and some robots may not have enough data to learn an appropriate encoding.
We chose a more unified data format that different robots can all use. We initially thought this would be difficult, but in practice it was less complicated than expected. Robot-arm action spaces are not as complex as we had imagined. Many action dimensions are fixed and have clear physical meanings. With a general format, the model can learn those meanings directly, rather than having to infer what each dimension represents. This reduces uncertainty in learning and improves efficiency.
We also do not normalize each robot separately. We process all robots' action spaces in a unified way to avoid creating incompatible spaces. Most physical quantities have clear meanings, and their scales are not very different across robots. For example, the spatial position of an end gripper is unlikely to extend beyond roughly ±3 meters. This makes unified treatment reasonably practical, and the physical meaning of data from each dataset can remain explicit. For velocity, zero means stationary and one means one meter per second; for position, zero is the origin and one means one meter. These meanings are consistent across robots. Once appropriate units are chosen, their data can work together to improve the model. By contrast, separate normalization can obscure those physical meanings. When data is scarce, that can even be counterproductive.
4. Architectural Differences from ACT and Diffusion Policy
The architectural difference is that the original ACT and Diffusion Policy setups we are discussing do not take language as input, mix data from different robots or pursue multi-task learning in the same way. For Diffusion Policy, simply thinking of using diffusion to generate actions is not the difficult part. The achievement was making it work well and showing that this route was viable. Its exact architecture is therefore less important than its demonstration that a VAE or diffusion model can be used for this kind of action prediction. Architectures can be changed. Subsequent researchers can make them more scalable, add richer data modalities and extend them to more robotic tasks.
Diffusion models are very good at sampling in high-dimensional continuous spaces, and actions fit that description. Their dimensionality is lower than that of images, but it is still substantial. Diffusion can therefore model the probability distribution in that space in considerable detail. Its disadvantage is slower sampling, but that is less pronounced for action generation because actions are not as high-dimensional. Sampling actions need not be extremely slow, while the model still has a strong ability to capture both diversity and quality. That is also why diffusion has been so successful in text-to-image and text-to-video generation. I see its results there as clearly better than those of other probabilistic generative models. It is therefore not particularly surprising that it can work well for actions too.
ACT takes another approach, using a variational autoencoder, or VAE, for modeling. VAEs are also powerful models, but they have drawbacks. Their sample quality often falls short of diffusion models, in my view, partly because of their assumptions. For example, assuming Gaussian latent variables can impose limitations when modeling continuous distributions. There is substantial theoretical discussion of why VAEs can produce blur when modeling distributions and why their diversity may be lower than that of diffusion models. Technical note: This is a comparison of the formulations being discussed. A Gaussian latent prior alone does not imply that every VAE must generate blurred or less diverse outputs.
These issues already arise in image modeling and can appear in action tasks as well. Insufficient precision can make performance worse than that of diffusion. The generative-modeling literature contains many discussions comparing VAEs with diffusion models. I also explained in our paper why we did not choose a VAE, for similar reasons. A theoretical comparison specifically focused on VAEs and diffusion for embodied tasks would be a very interesting research topic.
5. Comparing RDT with π0
Octo and HPT place a relatively large Transformer backbone before the action head. I compare that backbone to the kind of Transformer used in autoregressive models such as GPT, although this does not mean their complete action-generation systems are themselves autoregressive. The backbone outputs a token representation that a diffusion head then decodes. Another way to understand it is that a small diffusion network is conditioned on this representation. Such a network may contain only tens of millions of parameters.
Another family, including RDT and π0, differs mainly in where the core computation takes place. Much of the action-generation computation in RDT and π0 takes place in a larger continuous generative network. In the terminology of this discussion, π0's diffusion-like action network has a little over six hundred million parameters, while RDT's network is on the billion-parameter scale. Increasing the size of that network significantly improves the model's expressive capacity. Technically, π0 uses a flow-matching action expert rather than a conventional diffusion denoiser. Parameter-count correction: The original interview quotes a little over 600 million for π0's action network, but the published π0 paper specifies an approximately 300-million-parameter action expert, alongside a 3-billion-parameter PaliGemma backbone.
By comparison, the diffusion heads in the Octo and HPT configurations under discussion are small. This simplifies the diffusion component, but we found that it can limit expressive capacity. In these designs, the main computation sits in the Transformer backbone and the diffusion network is a relatively small module. We see this as a limitation in modeling capacity. RDT and π0 devote substantial action-generation capacity to the continuous generative network itself, making it central to action generation. That is the main distinction between these two design families.
RDT and π0 also differ in their conditioning mechanisms. π0 interacts with the keys and values of a vision–language model. That is distinctive, but I do not see it as the fundamental difference here. The central question remains where the computation is concentrated: in a separate Transformer backbone or in the generative action network.
To summarize my comparison, Octo and HPT exemplify a design in which computation is concentrated in a Transformer backbone and the diffusion network is small. I believe this can limit modeling capacity. RDT and π0 instead devote substantial capacity to the generative action network, making it the main component in action generation. That is the core distinction; other implementation details are less fundamental to this comparison.
6. Transformer Backbones with Diffusion Heads vs. Diffusion Transformers
Diffusion is a modeling approach. You can think of it in terms of an objective: train a model with a diffusion loss and generate samples with a diffusion sampling procedure. At its core, a network performs denoising, but there is no strict requirement for a particular network architecture. Many studies use convolutional neural networks rather than insisting on a specific structure. Diffusion Policy, for example, mainly used a CNN and found it performed better than its Transformer variant in the experiments. There may have been issues in the Transformer design, such as unsuitable normalization or insufficiently refined modality conditioning. But that is not a criticism of the paper's main contribution, because network-architecture design was not its primary focus.
When we say RDT uses a Transformer, the question is how to design a Transformer within a diffusion model so that it handles multimodal data efficiently and scales better. We want to feed it more data, learn from that data more efficiently and train on data from multiple robots. Those are things Transformers are especially good at. That is one characteristic of combining diffusion with a Transformer.
There is another approach based on transferring language-model or vision–language-model architectures into robotics. Here I am mainly talking about autoregressive generation. Its backbone performs next-token prediction, which is very different from diffusion. Representative examples include the RT series and OpenVLA, which use Transformer backbones to generate action tokens. Although their data formats differ, the basic idea resembles a language model: treat actions and images as tokenized inputs, pass them through self-attention layers and ultimately generate tokens.
The idea is to process the data as if it were language and solve the task in a language-model-like way: treat all inputs as a token sequence, then generate output tokens. Decoding may use a linear layer or an MLP to obtain the prediction. Some models discussed here, including discrete-action approaches in the RT family, model discrete probabilities. For example, a continuous velocity between −1 and 1 meters per second can be divided into 256 bins, each represented by a token. This turns continuous velocity values into discrete categories. The original interview also groups the GR series into this discussion, although variants within a model family can use different output designs.
The core issue with classification is that discretizing continuous action-space data, such as velocity or position, loses information. It turns a continuous variable into a set of labels, but a plain categorical objective does not directly express the numerical ordering between bins. Each bin is treated as a different category, making complex action spaces difficult to model precisely. In our experiments, discrete probability modeling worked reasonably well for simple tasks, but accuracy dropped significantly as tasks became more complex.
Direct MSE regression has related problems. In complex multidimensional action spaces, it may fail to capture the joint distribution across dimensions, affecting prediction accuracy. Some researchers therefore try to add diffusion's strengths to a Transformer-based model. For example, they attach a small diffusion module after the Transformer's final layer and perform another sampling stage to improve prediction precision and generation quality. More precisely, deterministic MSE regression tends to learn a conditional mean rather than sample a multimodal joint action distribution; this is not a claim that every MSE-trained network ignores all correlations.
I hope this approach can compensate, at least partly, for some limitations of the backbone-based approach. It may alleviate certain problems, but I think its effect remains limited. A small diffusion model has limited expressive power. It can provide a useful addition, but does not fully address the underlying limitations. My interpretation is that the diffusion component is simply too small. It is still somewhat like a regression-style model augmented with diffusion capabilities, rather than a strong diffusion model throughout. In our comparison, its performance is worse than that of the latter. I do not think a Transformer backbone used in this way can match a backbone trained as the core of the diffusion model.
The motivation for constructing this architecture with Transformers is to borrow language models' ability to handle many data formats. They can automatically learn the characteristics of different formats without manual intervention. But this shortcut has problems too: important modeling issues can be overlooked. Flexible formats also require an enormous amount of data, which is relevant to approaches such as HPT and Octo. Language models have enough data to learn this flexibility. Grammar can vary; conclusions can appear early or late; reasons can precede or follow a conclusion; images and other content can be interleaved. Robotics has nowhere near as much data as language modeling, so we cannot expect it to handle every format with the same flexibility.
Under those conditions, I would rather choose a general format—a common denominator—and manually fill it in. In practice, this worked far better than we expected. Robot data formats vary, but they are not as complex as language. Manually choosing an appropriate space and constructing a format can meet our needs and work very well. I prefer that common-denominator approach. In my view, rather than introducing harmful bias, it gives the model useful prior knowledge and helps avoid bugs. The result is more stable and performs very well.
Based on Songming Liu's explanations and materials, I put together the comparison diagram below. ByteDance developed the GR series, Google developed the RT series, and OpenVLA and Octo came from academic collaborations rather than simply being Google releases. HPT is a robot foundation-model architecture from MIT. These approaches all use Transformers, but their action-generation designs vary. RDT and π0 also use Transformers, while placing much greater emphasis on a continuous generative action network: diffusion for RDT and flow matching for π0. As a non-specialist, my simple reading of the interview is that all of these models use Transformers; the distinction emphasized here is how much of action generation the generative network itself handles. RDT and π0 devote more capacity to that component, and the interview argues that this leads to better results in the comparisons being discussed.

7. Further Questions
What do you mean by a common denominator?
The common denominator is a unified action space. Our central idea was that although robots express their action spaces differently—some use velocity, some force and some joint angles—we can concatenate all possible physical quantities into one representation. If a robot provides a particular quantity, we place it in the corresponding slot. If it does not, that slot is left unused and padded. We encode these quantities in a unified 128-dimensional space.
This approach might not work as well for language or images, where formats are much more flexible and dimensionality can be enormous. Language, in particular, has a vast vocabulary and would require a very large space. In robotics, action spaces vary, but not nearly as much as language. Even mapping an action space with a dozen or so dimensions into 128 dimensions is not especially large. A language-model Transformer's hidden representations commonly have thousands of dimensions. For our Transformer, 128 dimensions is relatively small. It can handle this representation directly during training without needing dimensionality reduction.
How should we understand dimensionality here? What exactly does it refer to?
Here it mainly means the dimensionality of the robot's action-command vector: quantities such as velocity, angular velocity, acceleration or direction. A velocity vector in space, for example, has three dimensions. Robots have different functions, so their command vectors differ too. But if we list all possible command dimensions, the total is only a little over one hundred. Language has far more kinds of variation. Standardizing different grammatical and syntactic structures would require categorizing them into a common format. Early language models did attempt approaches like that, but language is too complex: syntax and grammar are almost impossible to enumerate. There may be thousands of types, and combining them makes the possibilities larger still. Editorial clarification: this comparison concerns representational complexity; grammatical types are not literally action-vector dimensions.
What does padding mean?
Some robots do not need particular dimensions, so we pad them. Suppose our data format includes several dimensions such as velocity, acceleration and position. In a particular scenario, we may need to control a robot using only position, without velocity or acceleration commands. What should we do with those unused dimensions?
We can fill the velocity and acceleration slots with a preset padding value. That fills out the extra dimensions and embeds the original action space in a larger one. The format becomes unified, and the system can still process it normally. That is the central idea of padding. In RDT's actual implementation, a binary availability vector is also concatenated with the 128-dimensional action representation. This distinguishes an unused padded slot from a genuine zero-valued physical command.
Why make it open source?
We do not require people to contribute datasets in exchange for using our model. It is open source, and we think flexibility and freedom are essential. We invested substantial resources in training a robot foundation model and want it to be useful across downstream tasks, rather than remaining only a research result. Otherwise, an investment of this scale would lose its purpose.
We encourage people to fine-tune the foundation model on their own data to build different applications. Our tests showed very strong performance: it was the best-performing model in our evaluation, and its architecture has considerable modeling capacity. We want that effort to lead to widespread practical use.
Feedback from real applications can also help us improve it: where is the model insufficiently flexible, and what problems remain? We will use that feedback to refine the next iteration and produce a better version.
References
RDT-1B
https://rdt-robotics.github.io/rdt-robotics/
Songming Liu's Profile
DiT
https://arxiv.org/pdf/2212.09748
U-ViT
https://arxiv.org/pdf/2209.12152