You can deploy this model and start earning money today!

AllenBi21/
HTT

representation-learning

feature-extraction

cc-by-nc-4.0

Views

No views yet

HTT — Heterogeneous Tactile Transformer

A multimodal tactile representation model. One shared transformer backbone encodes four different tactile sensors into a common 192-dimensional embedding space, pretrained with masked-autoencoder reconstruction and cross-modal alignment. Feed a raw sensor reading, get a feature vector for any downstream head (classification, force / slip estimation, policy learning).

Modality	Type	Raw input
`gsmini`	vision (GelSight Mini)	uint8 image `[224, 224, 3]`
`9dtact`	vision (9DTact)	uint8 image `[224, 224, 3]`
`xela`	taxel array	float `[T, 72]`
`tac02`	taxel array	float `[T, 66]`

Checkpoint


File	`htt_4sensors_best.pth` (~69 MB)
Contents	`model_state_dict` = 4 encoders + shared 9-layer trunk + 4 decoders (17.1 M params)
Embedding dim	192
SHA-256	`024f4c3a067168197d0a6996bbca5c03e744ed5abd1d35a666dbf78e7ac673f0`

Slim inference/finetune checkpoint (optimizer / predictor states dropped).

Usage

Use it with the HTT package (contains the architecture, preprocessing, and examples). Download the weights into checkpoints/:

hf download AllenBi21/HTT htt_4sensors_best.pth --local-dir checkpoints

python

import numpy as np
from htt import HTT

model = HTT(modality="gsmini")                       # loads checkpoints/htt_4sensors_best.pth
frame = np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8)   # your sensor frame
emb = model(frame)                                   # -> torch.Tensor [1, 192]

Read the raw-input contract before feeding your own data — the model returns garbage on out-of-distribution inputs.

License

Released under CC BY-NC 4.0 (non-commercial). Portions are derived from Meta's V-JEPA / DINOv2 (Apache-2.0 and CC-BY-NC-4.0).