suddhu/

tactile_transformer

While vision-based touch sensors interpret contact geometry as images, they remain out-of-distribution from natural images. The embedded camera directly perceives the illuminated gelpad, and contact depth is either obtained via photometric stereo, or supervised learning. Existing touch-to-depth relies on convolution, however recent work has shown the benefit of a ViT for dense depth prediction in natural images. We present a tactile transformer for predicting contact depth from vision-based touch, trained entirely in simulation to generalize across multiple real-world DIGIT sensors. For use, refer to the NeuralFeels repository.

Our code is based on FocusOnDepth, a re-implementation of the popular DPT vision transformer. We make necessary modification to work for tactile images, and share the weights online. The models data/tactile_transformer/dpt_real.p and data/tactile_transformer/dpt_sim.p are trained on TACTO data from simulated interaction with YCB objects. The models differ slightly in the augmentations used during data generation.

First download tactile data from YCB objects:

Run the test script

If you find NeuralFeels useful in your research, please consider citing our paper: