SPD robot training pairs virtual practice with real demos

SPD robot training uses simulated demonstrations to prepare a dexterous bimanual policy for real tasks including Jenga, according to a LinkedIn post by robotics commentator Léo Morillon. He describes SPD, short for Pre-training Visual Dexterity in Simulation, as research from Stanford, MIT and Scale AI. The approach combines extensive virtual teleoperation with one to two hours of real teleoperated demonstrations per task, targeting the difficulty of collecting training data for two arms and two dexterous hands.

According to Morillon’s account, five people collected 75 hours of demonstrations over one week by controlling a virtual robot in VR. The simulated system had 56 degrees of freedom, and collection covered six scenes and hundreds of objects. Those demonstrations were used to pretrain a 222 million parameter diffusion transformer policy before task specific adaptation on real data. The deployed policy operates autonomously at 30 Hz, using action chunks of eight steps and observations from multiple camera views plus proprioception. The five demonstrated tasks were plate racking, cup stacking, Jenga, mug hanging and bottle tossing.

Morillon reports that simulation pretraining outperformed behavior cloning from scratch across all five tasks. His account describes relative task progress rather than per task success percentages. That makes the reported result useful for comparing training approaches, rather than establishing operational reliability. He argues that virtual teleoperation offers easier resets and parallel data collection without risking physical hardware. He also reports that lower pretraining loss correlated with better real world performance, offering a practical signal for evaluating further simulation training.

Source: Léo Morillon on LinkedIn