In human development, intelligent brains and dexterous hands have been crucial. Our five-fingered hands use tools and help make the world richer. On the path toward general-purpose robots, I think the biggest limitation beyond intelligence is whether robots can acquire hands as dexterous as ours. What can robotic hands actually do in 2023?
Curious about that question, I reviewed more than a hundred papers from the preceding two or three years on hand-based manipulation with grippers or multi-fingered hands. I organized the work of Chinese researchers active in this area and selected the best results I could find on familiar tasks.
The focus is robot learning for dexterous manipulation, such as training an RL policy in simulation and transferring it to a physical robot, offering broader generalization in the comparison I am making than traditional methods. I hoped for comprehensive coverage, but inevitably missed excellent work and researchers. Corrections and suggestions are welcome. This review is a snapshot as of October 2023.
Special thanks to Yuanpei Chen, Binghao Huang, Yixuan Wang and Man Xie for their expert suggestions.
Original article outline
Contents
👇
1. A map of Chinese researchers in dexterous manipulation
2. Robot hands
3. Rigid-object manipulation
4. Deformable-object manipulation
5. Imitation learning
6. Teleoperation
7. Multimodal sensing
8. Reflections
1. A map of Chinese researchers in dexterous manipulation
This article builds on A Map of Chinese Researchers in AI and Robotics, focusing more closely on hand manipulation. Future articles may cover arm manipulation, LLMs plus robotics, locomotion or 3D objects.

Purple marks faculty based overseas, yellow marks faculty in China, and green marks PhD students in the original chart. The chart reflects the article’s 2023 perspective rather than current appointments; some green entries are explicitly labeled postdoctoral researchers.
David Held’s CMU lab has extensive dexterous-manipulation work, largely with grippers. Much concerns unfamiliar objects, including deformable materials such as cloth, liquids and dough, articulated objects, and multimodal learning with depth, RGB and touch. This article draws on work by his Chinese doctoral students Wenxuan Zhou, Yufei Wang and Xingyu Lin, with their statuses understood in the source’s historical context.
Many researchers active in multi-fingered manipulation studied with CMU’s Abhinav Gupta. They include UC San Diego’s Xiaolong Wang and NYU’s Lerrel Pinto, both his PhD students; MIT doctoral student Tao Chen, who completed a master’s with him; and CMU doctoral student Wenxuan Zhou, another former master’s student. [Editor’s note: The source calls Wang and Pinto associate professors; those ranks should not be assumed to describe their appointments in October 2023.]
Pieter Abbeel’s UC Berkeley academic network remains influential. His former students Pulkit Agrawal at MIT and Deepak Pathak at CMU are well known in dexterous manipulation. Agrawal’s student Tao Chen has produced excellent work. Pathak’s team introduced LEAP Hand as a lower-cost alternative to Korea’s Allegro Hand.
Researchers connected with Stanford’s Fei-Fei Li and SVL include C. Karen Liu, Yuke Zhu, Yunzhu Li and Jiajun Wu. Huazhe Xu at Tsinghua IIIS and Cewu Lu at Shanghai Jiao Tong University also worked with SVL and have substantial hand-manipulation research. These connections are a research network, not necessarily a list of direct doctoral advisorships.
Xiaolong Wang’s UC San Diego group has produced many results. We cite work by his PhD student Yuzhe Qin and former master’s student Binghao Huang, then pursuing a PhD with Yunzhu Li at UIUC. NVIDIA and Dieter Fox’s University of Washington group also have strong dexterous-manipulation work.
Fuchun Sun at Tsinghua and Jianwei Zhang at the University of Hamburg have extensive five-fingered manipulation research. Bristol has significant work on dexterous hands and manipulation, including Dandan Zhang’s research. EPFL’s LASA also focuses on manipulation. [Editor’s note: The source incorrectly names Humboldt University for Jianwei Zhang; his TAMS group is at the University of Hamburg. Affiliations are presented in the original review’s historical context.]
2. Robot hands
Dexterous manipulation has long challenged robotics. Machine learning shows promise in simulation, but many results remain simulated, mainly because suitable hardware is lacking. Common end effectors are two-finger grippers and multi-fingered dexterous hands. Here is a brief introduction to research hardware.
Grippers: common end effectors for basic manipulation, including simple linear motions, with a relatively low hardware barrier.

Multi-fingered hands: common research choices include the three-finger D’Claw, four-finger Allegro and five-finger Shadow Hand. “Shandow” in the source is a spelling error.

Some laboratories build their own systems. Deepak Pathak’s CMU group introduced LEAP Hand, reported at roughly one-eighth the cost of Allegro, and the compliant DASH Hand. The source reports costs under US$2,000 for these low-cost designs; these are historical hardware estimates, not current quotations.

Other options include DLR Hand from the German Aerospace Center, the bebionic prosthetic hand, SCHUNK SVH and Vincent hands.

Chinese examples include Hong Liu’s team at Harbin Institute of Technology, Peng Wang’s team at the Chinese Academy of Sciences’ Institute of Automation, and Inspire Robots.
3. Rigid-object manipulation
Common objects include rigid items such as Lego and cans, deformable materials such as dough and cloth, and liquids. Rigid objects are relatively easier, so many results can run on physical robots. Research areas include in-hand reorientation, non-prehensile manipulation through pushing or poking, extrinsic dexterity using the environment—for example, a wall to stand a box upright—and long-horizon manipulation that chains movements to complete a task. These academic terms describe everyday human actions. Let us see what robots can achieve.
In-hand reorientation is necessary for many dexterous tasks. Tao Chen and colleagues developed a general controller trained with RL in simulation and evaluated on new objects in the real world. It generalizes to object shapes absent from training and handles objects suspended in the air, counteracting gravity while reorienting them. The hardware uses open-source components costing under US$5,000. [Editor’s note: The source describes the hand as facing upward; the especially challenging demonstrated setup uses a downward-facing hand.]
“Visual Dexterity: In-Hand Dexterous Manipulation from Depth,” by Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson and Pulkit Agrawal. The published Science Robotics title is “Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes.”
Another project by Tao Chen and Jie Xu uses a simple model-free framework to learn reorientation with upward- and downward-facing hands in simulation. The policies reorient more than 2,000 geometrically varied objects and transfer zero-shot to new ones. This work remained in simulation rather than physical deployment and received the CoRL 2021 Best Paper Award.

“A System for General In-Hand Object Re-Orientation,” by Tao Chen, Jie Xu and Pulkit Agrawal, CoRL 2021, Best Paper Award.
Recently, NVIDIA, the University of Pennsylvania, Caltech and UT Austin introduced Eureka, combining LLMs and RL. GPT-4 improves reward functions, and RL trains controllers. In simulation, this supports complex five-fingered skills such as pen spinning and rotating objects in the hand, likened in the source to rolling walnuts.

“Eureka: Human-Level Reward Design via Coding Large Language Models,” by Jason Ma et al. https://eureka-research.github.io/
Non-prehensile manipulation and extrinsic dexterity address objects difficult to grasp, such as a book or a large flat object lying on a table. Non-prehensile manipulation moves objects without grasping, for example by pushing or poking. Extrinsic dexterity uses the environment: push a flat box against a wall to stand it upright, then grasp its narrow edge. Wenxuan Zhou in David Held’s group has extensive gripper-based work in both areas. Fuchun Sun’s Tsinghua group and Jianwei Zhang’s Hamburg group also study them; see the references. Zhou’s video below illustrates the ideas.
“Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity,” by Wenxuan Zhou and David Held, CoRL 2022.
Long-horizon dexterous manipulation of rigid objects
Throwing and catching is familiar to humans but challenging for robots, requiring high-speed movement, coordination and varied object interactions. This project uses arms and five-fingered hands, training through multi-agent RL in simulation and transferring to real robots. To overcome the sim-to-real gap, it introduces several algorithmic designs, including an object-trajectory predictor that helps the catching robot estimate motion and respond in real time. Co-first authors Binghao Huang and Yuanpei Chen have extensive dexterous-manipulation work.

“Dynamic Handover: Throw and Catch with Bimanual Hands,” by Binghao Huang*, Yuanpei Chen*, Tianyu Wang, Yuzhe Qin, Yaodong Yang, Nikolay Atanasov and Xiaolong Wang, CoRL 2023. *Equal contribution.
Assembling Lego: dexterous hands face high-dimensional actions and complex compositional dynamics over long tasks. C. Karen Liu’s Stanford group developed an RL system that chains dexterous policies to achieve longer-term goals, with strong sim-to-real transfer.

“Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation,” by Yuanpei Chen, Chen Wang, Fei-Fei Li and C. Karen Liu, CoRL 2023.
Bidan Huang’s Tencent Robotics X team also has excellent long-horizon work. Its April 2023 demonstration of cocktail preparation using an arm and three-fingered hand was striking. Yuke Zhu’s UT Austin group built an interesting instant-noodle-making robot using grippers. Videos of both appear in A Map of Chinese Researchers in AI and Robotics.
4. Deformable-object manipulation
Manipulating deformable objects has many everyday applications, such as kneading dough and folding clothes, but is particularly difficult for robots. Elasto-plastic dough lacks a compact state representation and requires rich contact; folds in cloth create occlusion. Physical demonstrations more often use grippers than multi-fingered hands, while much multi-fingered dough work remains simulated. Here are representative results.
Dressing: manipulating cloth in 3D is difficult. Earlier approaches made strong assumptions about fixed garments or arm poses and generalized poorly. Yufei Wang in David Held’s CMU group studies cloth manipulation. A 2023 project dresses participants with varied arm poses and garments.

“One Policy to Dress Them All: Learning to Dress People with Diverse Poses and Garments,” by Yufei Wang, Zhanyi Sun, Zackory Erickson* and David Held*, RSS 2023. https://sites.google.com/view/one-policy-dress
Visual haptic reasoning for deformable cloth: another Yufei Wang project infers where forces act and their magnitude from visual observations and robot movement.

“Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactions,” by Yufei Wang, David Held and Zackory Erickson, IROS 2022.
Unfolding cloth: if I had a robot housekeeper, the chore I would most want it to handle is putting a duvet inside its cover.
[Emoji]
After reading more than ten cloth-manipulation papers, I doubt we will escape doing that ourselves within ten years—a personal prediction made in 2023. Shuran Song’s Columbia group proposed two robot arms working with a movable air blower, one of the most efficient clothes-folding approaches I have seen. DextAIRity is a self-supervised framework that learns to perform tasks through grasping and airflow-based actions. Closed-loop blowing continuously adjusts direction from visual feedback, handling highly stochastic dynamics. First author Zhenjia Xu earned his undergraduate degree at Shanghai Jiao Tong University and was then a PhD student with Song. He also led RoboNinja, the avocado-cutting project discussed below.
“DextAIRity: Deformable Manipulation Can Be a Breeze,” by Zhenjia Xu, Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng and Shuran Song, RSS 2022. https://dextairity.cs.columbia.edu/
Dough manipulation
Perhaps out of a love for flour-based food, many researchers try robotic kneading, cutting noodles and even making dumplings. Physical systems usually combine grippers with multiple tools, while some simulated systems use multi-fingered hands. The most elaborate dough demonstration I had seen was making dumplings. What happens after they go into the pot, I cannot say!
[Emoji]
.
Flattening dough: shaping dough from RGB-D observations is simple for people but difficult for robots without a fixed setup and preprogrammed trajectories. Xingyu Lin in David Held’s CMU group has many dough projects. A 2022 project takes partial point clouds as input and transfers easily from simulation to the real world, flattening real dough into a target shape without a prespecified external setup or preprogrammed trajectories.

“Learning Closed-Loop Dough Manipulation Using a Differentiable Reset Module,” by Carl Qi, Xingyu Lin and David Held, IROS 2022. https://sites.google.com/view/dough-manipulation
Long-horizon deformable manipulation: another Xingyu Lin project combines tool-use skills, such as cutting with a knife, pushing with a rod and rolling dough with a roller.

“Planning with Spatial-Temporal Abstraction from Point Clouds for Deformable Object Manipulation,” by Xingyu Lin*, Carl Qi*, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan and David Held, CoRL 2022. https://sites.google.com/view/pasta-plan
Making dumplings: Haochen Shi in Jiajun Wu’s Stanford SVL group collaborated with Huazhe Xu, then an assistant professor at Tsinghua IIIS, Zhiao Huang, and Yunzhu Li, then an assistant professor at UIUC on RoboCraft and RoboCook, showing possibilities for long-horizon dough manipulation with multiple tools. RoboCraft uses raw RGB-D observations, converts them to particles, and learns particle-based dynamics with graph neural networks to capture system structure. Coupled with MPC, the learned model plans actions. With only ten minutes of real robot interaction data, it learns dynamics for shaping elasto-plastic objects into varied target forms, including unseen ones. [Editor’s note: The dumpling-making demonstration belongs to RoboCook; it should not be attributed entirely to RoboCraft.]
“RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks,” by Haochen Shi*, Huazhe Xu*, Zhiao Huang, Yunzhu Li and Jiajun Wu, RSS 2022.
Five-fingered dough shaping: DexDeform, from Chuang Gan’s UMass Amherst team and collaborators, uses human demonstrations to abstract skilled multi-fingered manipulation of deformable objects. Contributors include MIT’s Sizhe Li and UC San Diego’s Zhiao Huang, both active in simulated manipulation of materials such as modeling clay. Gan and UC San Diego’s Hao Su, two supervisors on the project, also work extensively on 3D simulation.

“DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics,” by Sizhe Li, Zhiao Huang, Tao Chen, Tao Du, Hao Su, Joshua B. Tenenbaum and Chuang Gan, ICLR 2023.
Cutting multi-material objects: RoboNinja, from Shuran Song, Chuang Gan and collaborators, learns to cut objects with soft exteriors and rigid cores, such as avocados or mangoes. Unlike cutting a single-material cucumber, it aims to remove the soft material while preserving the hard core and maximizing yield.
“RoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects,” by Zhenjia Xu, Zhou Xian, Xingyu Lin, Cheng Chi, Zhiao Huang, Chuang Gan and Shuran Song, RSS 2023. [Editor’s note: The original incorrectly gives RSS 2022; the project is RSS 2023.]
5. Imitation learning
Imitation learning is common in dexterous manipulation. Just as children and adults imitate actions, expressions and language, robots can learn human behavior on particular tasks. These examples illustrate imitation learning for detailed hand manipulation.
Dexterous imitation without physical deployment: Xiaolong Wang’s UC San Diego group introduced DexMV, combining a simulated multi-fingered hand for complex tasks with a computer-vision system recording large-scale human demonstrations of those tasks. Retargeting human movements into robot demonstrations substantially improves learning and supports tasks that RL alone could not solve in their experiments.

“DexMV: Imitation Learning for Dexterous Manipulation from Human Videos,” by Yuzhe Qin*, Yueh-Hua Wu*, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu and Xiaolong Wang, ECCV 2022.
Dexterous imitation with physical deployment: a single-camera teleoperation system efficiently records 3D hand demonstrations using an iPad and a computer. A key contribution is a personalized simulated robot hand with the operator’s kinematics and shape. That avoids unstable retargeting during collection and offers an intuitive, personalized interface for different users. Users first teleoperate their customized hands in simulation. Demonstrations are then retargeted to Allegro, used to train policies, and deployed on real hardware. This work is led by UC San Diego’s Yuzhe Qin.

“From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation,” by Yuzhe Qin, Hao Su* and Xiaolong Wang*, IROS 2022.
6. Teleoperation
Teleoperation means a human remotely controls a robotic hand or arm, using approaches that may involve vision, force or tactile sensing. It is common in dexterous manipulation. Here are representative recent projects.
Vision-based teleoperation: many systems are built for a specific robot and environment, making scaling difficult as platforms and settings diversify. AnyTeleop is a unified system supporting different arms, hands, real-world environments and camera configurations. It is a collaboration between NVIDIA’s Dieter Fox and Yu-Wei Chao and UC San Diego’s Xiaolong Wang and Hao Su. First author Yuzhe Qin has extensive dexterous-manipulation research, earned his undergraduate degree at Shanghai Jiao Tong University, and was then a UC San Diego PhD student jointly advised by Wang and Su.

“AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System,” by Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao and Dieter Fox, Robotics: Science and Systems, 2023.
Jianwei Zhang’s group also developed a vision-based hand–arm teleoperation system. An actively controlled camera observes the operator’s hand from the best viewing angle and an appropriate distance. It combines an end-to-end hand-pose regression network with active vision, using a Shadow Hand.

“A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision,” by Shuang Li, Norman Hendrich, Hongzhuo Liang, Philipp Ruppel, Changshui Zhang and Jianwei Zhang.
Low-cost physical teleoperation: Pieter Abbeel’s team introduced GELLO, a general, intuitive framework lowering the barrier to collecting large-scale, high-quality demonstrations. A controller with the target arm’s kinematic structure uses 3D-printed parts and off-the-shelf motors. Easy to build and use, it collects demonstrations more reliably and efficiently than VR controllers or 3D spacemice in the reported study, including complex bimanual and contact-rich tasks. Designs support Franka, UR5 and xArm, with open-source hardware and software. [Editor’s note: The source calls this force-control-based teleoperation; GELLO principally provides kinematically matched joint-space control, not necessarily full force-feedback teleoperation.]

“GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators,” by Philipp Wu, Yide Shentu, Zhongke Yi, Xingyu Lin and Pieter Abbeel. https://wuphilipp.github.io/gello_site/
Teleoperation plus imitation: Chelsea Finn’s Stanford and Sergey Levine’s Berkeley groups developed low-cost ALOHA hardware and a customized interface to collect real demonstrations for end-to-end imitation. Their Action Chunking with Transformers (ACT) algorithm predicts action chunks, reducing the effective horizon. With ten minutes of demonstrations, it learned challenging tasks such as opening a translucent condiment cup and inserting a battery with reported 80–90% success in those experiments.

“Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” by Tony Z. Zhao, Vikash Kumar, Sergey Levine and Chelsea Finn, RSS 2023.
7. Multimodal sensing
Touch is essential to human hands. Beyond vision, information about temperature, texture, material and moisture guides daily manipulation. While vision and force-based approaches are comparatively established, tactile feedback remains the biggest bottleneck for dexterous robots in my assessment. Here are recent projects addressing it.
Touch-based dexterity: Xiaolong Wang’s group introduced Touch Dexterity to achieve coordinated manipulation from simple tactile data. It rotates objects in an Allegro hand using touch rather than vision. Sixteen force-sensing resistors mounted on the palm and fingers cost approximately US$12 each in the reported setup, with their signals reduced to contact versus no contact.

Rotating without seeing: Touch Dexterity covers the palm, finger links and fingertips with relatively dense binary contact sensing, rather than highly precise sensing over a small area. This is inexpensive. An in-hand rotation policy trained with RL in simulation transfers to a real hand and rotates unfamiliar objects without visual observation.
“Rotating without Seeing: Towards In-Hand Dexterity through Touch,” by Zhao-Heng Yin*, Binghao Huang*, Yuzhe Qin, Qifeng Chen and Xiaolong Wang, RSS 2023.
Finding objects by touch: Tao Chen and collaborators study retrieval without vision, prior knowledge of shape or fixed objects. A camera sees a scene globally, but tactile sensors observe only the local contact. Gathering touch data also applies forces and may disturb the scene. Their system uses sparse fingertip feedback to locate, identify and grasp new objects without visual feedback.
“TactoFind: A Tactile Only System for Object Retrieval,” by Sameer Pai*, Tao Chen*, Megha Tippur*, Edward Adelson, Abhishek Gupta† and Pulkit Agrawal†, ICRA 2023. *Equal contribution; †equal advising.
Multimodal RL for multi-fingered grasping: Jianwei Zhang’s Hamburg group and Fuchun Sun’s Tsinghua group used Shadow Hand with torque, tactile and visual information. Experiments in simulation and on real robots showed improved grasping of unfamiliar objects relative to strictly programmed grasping or RL using fewer modalities, under the tested conditions.
“Multifingered Grasping Based on Multimodal Reinforcement Learning,” by Hongzhuo Liang, Lin Cong, Norman Hendrich, Shuang Li, Fuchun Sun and Jianwei Zhang, IEEE Robotics and Automation Letters, 7(2), 2022; also selected for presentation at ICRA 2023. https://lianghongzhuo.github.io/multifingered_rl_grasp [Editorial clarification: the original lists ICRA 2023, which is a valid conference presentation attribution; the journal publication appeared in RA-L in 2022.]
The multimodal pouring network, MP-Net, robustly estimates liquid levels from sound and haptic input. It trains on a self-collected dataset of 300 robot pouring trials containing audio and force/torque measurements for three types of receiving container.

“Robust Robotic Pouring Using Audition and Haptics,” by Hongzhuo Liang, Chuangchuang Zhou, Shuang Li, Xiaojian Ma, Norman Hendrich, Timo Gerkmann, Fuchun Sun, Marcus Stoffel and Jianwei Zhang, IROS 2020.
8. Reflections
I spent nearly three months considering an article on dexterous hands and manipulation without finding the right organizing perspective. After more than half a year of excitement about embodied intelligence and humanoids, I think it is time to pause and ask whether this wave really leads to the general-purpose robots people imagine. Beyond large models and Tesla Optimus, what drives it? Have substantive software or hardware breakthroughs occurred?
Those questions led me to show what robots can do in the final centimeter: interacting with objects. The many videos and animations aim to convey the capabilities of leading laboratories. Looking at these frontier results may leave readers with the same impression I have: on many manipulation tasks, robots are still far from practical use.
To entrepreneurs and researchers in AI robotics, I want to say this: an investor without a formal robotics background is willing to spend time tracking the latest progress. I will not rush for immediate results. In a field I believe in for the long term, I want to do the work thoroughly and avoid following the crowd.
References
https://www.cs.cmu.edu/~abhinavg/
https://r-pad.github.io/areas/
https://sites.google.com/view/vcd-cloth
https://binghao-huang.github.io/dynamic_handover/
https://www.lerrelpinto.com/#publications
https://cap.csail.mit.edu/improbable-ai-lab-lab-tours
https://fang-haoshu.github.io/
https://www.eng.yale.edu/grablab/research.html
https://tams.informatik.uni-hamburg.de/people/liang/
https://tams.informatik.uni-hamburg.de/
https://www.agile-robots.com/en/robotic-solutions/agile-hand
https://yufeiwang63.github.io/
“HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation” by Wenxuan Zhou, Bowen Jiang, Fan Yang, Chris Paxton*, David Held*, CoRL 2023
“Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity” by Wenxuan Zhou, David Held, CoRL 2022
“Reinforcement Learning Based Pushing and Grasping Objects from Ungraspable Poses,” by Hao Zhang, Hongzhuo Liang, Lin Cong, Jianzhi Lyu, Long Zeng, Pingfa Feng and Jianwei Zhang.