NVIDIA and Berkeley open source T-Rex tactile robot model
In a post on X, Jim Fan said NVIDIA and Berkeley are open sourcing T-Rex, a tactile robot model and training methodology that gives touch a dedicated control loop alongside visual motion planning. The work targets contact events that cameras struggle to capture, such as a magnetic part snapping into place, a paper cup separating from a stack or a USB connector settling into a port.
T-Rex uses what Fan described as a mixture of transformer architecture running two asynchronous clocks. A slower visuomotor expert plans motion, while a faster tactile expert makes high frequency corrections in real time. The tactile component processes four touch updates for each vision update, reflecting the faster rate at which contact forces can change.
The release also includes 50 hours of synchronized robot play data across roughly 5,500 episodes, collected with tactile hand hardware offering 22 degrees of freedom. Fan called it the largest released tactile dataset to date, to the collaborators’ knowledge, and said it was available on HuggingFace.
The training recipe extends the earlier EgoScale work. Human egocentric video is used for pretraining, followed by tactile robot play during mid training. Fan said the experiments showed this approach can connect contact free pretraining with contact rich manipulation, although the post did not provide benchmark scores or comparisons with other tactile control systems.
Source: @DrJimFan on X
