A large-scale, tactile-reactive bimanual manipulation dataset, collected via teleoperation on a
Dexmate Vega-1 robot with two Sharpa Wave dexterous hands. Stored as a
LeRobotDataset v3.0.
One episode from each of 20 motor primitives (head-camera view, cropped to the workspace), each with a different object.
Teleoperation setup: Manus gloves + VIVE trackers drive the bimanual Dexmate Vega-1 with two Sharpa Wave hands; observations come from a head-mounted ZED X Mini camera and two wide-view ZED X One S wrist cameras.
5,370 language-annotated trajectories · 22 motor primitives · 207 objects
Format
LeRobotDataset v3.0 (codebase_version: v3.0)
Composition
Object categories, episodes per motor primitive, and per-object episode counts.
Collection setup
The full teleoperation/hardware stack used to collect the dataset is open-sourced in
hardware_code/ in the T-Rex repo.
Robot. The Dexmate Vega-1 is a dual-arm mobile robot with 7 actuated joints per arm, here
equipped with two Sharpa Wave dexterous hands. During collection the wheels, torso, and head
joints are fixed; only the 14 arm joints and the two hands are actuated.
Cameras. A head-mounted ZED X Mini stereo camera (its left monocular RGB stream is recorded)
plus two wide-view ZED X One S monocular RGB cameras mounted on the wrists, posed so the head
camera covers the full reachable workspace while the wrist cameras keep the fingers visible
without significant palm occlusion. All three streams are recorded at 640×360.
Tactile. Each hand carries five fingertip tactile sensors. Per sensor, the dataset stores the
raw sensor image and the estimated deformation map (both as video), and the estimated 6-axis net
wrench (observation.tactile_force).
Teleoperation. Manus gloves capture fingertip positions relative to the hand base, retargeted
to the Sharpa Wave hands with the manufacturer's differential-inverse-kinematics package
(Pinocchio + CasADi). Two VIVE trackers provide SE(3) wrist poses, converted to arm joint
commands via differential inverse kinematics (Pink),
low-pass filtered, and tracked by the manufacturer's low-level cascade PID controller. A 30 Hz
high-level thread records observations and joint-space targets while asynchronously updating a
300 Hz low-level control thread — the dataset's 30 fps matches the high-level loop, and action
holds its 30 Hz joint-space targets.
Per-frame features
feature
shape
description
observation.state
(58,)
joint positions, laid out [L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22]
Finger order is thumb, index, middle, ring, pinky. Full per-dimension joint names are in
meta/info.json (features[*].names).
Per-episode metadata
meta/episodes/*.parquet carries language and task labels per episode, in addition to the standard
LeRobot fields (episode_index, tasks, length, per-feature stats, video pointers):
canonical target/receptacle (only set for lift_and_place; null otherwise)
Tactile video encoding (read before decoding)
The raw and deformation tactile videos are stored losslessly (libx264 -qp 0) because their
pixel values are physically meaningful (raw sensor images and deformation maps). They are grayscale —
the signal lives entirely in the luma (Y) plane — and use full-rangeyuvj420p, so values
span the full 0–255 with no range conversion. Decode the luma plane to recover the original uint8
images exactly, e.g. frame.to_ndarray(format="gray") in PyAV.
No web thumbnails for tactile. Lossless H.264 forces the High 4:4:4 Predictive profile,
which most browsers and the Hugging Face preview cannot decode — so the tactile videos do not show
thumbnails on the dataset page. This is expected; decode them locally (ffmpeg / PyAV / torchcodec).
The RGB cameras use standard yuv420p (limited range, BT.709) and preview normally.
The T-Rex Quick Start
companion repo browses, inspects, and replays single episodes without downloading the full dataset,
and includes a notebook you can
open directly in Colab.
Dataset viewer
The Hugging Face table viewer is manually configured (the configs block above) with two views:
frames (default) — the raw per-frame observation.state, action, and
observation.tactile_force arrays (plus index columns). This is the actual data, and what
datasets.load_dataset("zekaiwang/trex_dataset") returns by default.
episodes — one row per episode with its language caption, motor_primitive, object, and
target: a readable table of contents. Switch to it with the config dropdown, or load it with
datasets.load_dataset("zekaiwang/trex_dataset", "episodes").
Videos are stored as separate .mp4 files referenced by timestamp pointers, so they are not shown in
the table viewer — load them via LeRobot or the quick-start tools. For interactive video playback, use
the LeRobot dataset visualizer.
Layout
data/chunk-000/file-*.parquet per-frame state / action / tactile_force (+ index columns)
videos/<key>/chunk-*/file-*.mp4 23 video streams (3 RGB + 20 tactile)
episodes_preview.parquet curated per-episode labels for the web viewer (see "Dataset viewer")
meta/info.json features, shapes, fps, codebase_version
meta/episodes/*.parquet per-episode metadata + stats + video pointers
meta/tasks.parquet task (caption) table
meta/stats.json global feature statistics
Citation
If you find the T-Rex Dataset useful, please cite:
bibtex
1@misc{trex2026,
2 title={T-Rex: Tactile-Reactive Dexterous Manipulation},
3 author={Dantong Niu and Zhuoyang Liu and Zekai Wang and Boning Shao and Zhao-Heng Yin and Anirudh Pai and Yuvan Sharma and Stefano Saravalle and Ruijie Zheng and Jing Wang and Ryan Punamiya and Mengda Xu and Yuqi Xie and Yunfan Jiang and Letian Fu and Konstantinos Kallidromitis and Matteo Gioia and Junyi Zhang and Jiaxin Ge and Haiwen Feng and Fabio Galasso and Wei Zhan and David M. Chan and Yutong Bai and Roei Herzig and Jiahui Lei and Fei-Fei Li and Ken Goldberg and Jitendra Malik and Pieter Abbeel and Yuke Zhu and Danfei Xu and Jim Fan and Trevor Darrell},
4 year={2026},
5 eprint={2606.17055},
6 archivePrefix={arXiv},
7 primaryClass={cs.RO},
8 url={https://arxiv.org/abs/2606.17055},
9}
