This project is a research PoC for vision-driven teleoperation of the MuJoCo Menagerie Shadow Hand, with the longer-term goal of building a richer bio-inspired robotics stack around it.
The current system already closes a minimal perception-to-action loop:
human hand -> MediaPipe hand pose estimation -> retargeting -> Shadow Hand actuator commands -> MuJoCo simulation
The system uses the MuJoCo Menagerie Shadow Hand model and runs it in MuJoCo simulation.
The broader purpose is to turn this into a foundation for future work on:
tactile sensing
self-calibration
signal fusion and perception
action models
reinforcement learning
bio-inspired robot manipulation
In other words, this repository is not only a teleoperation demo. It is the beginning of a modular experimental platform for studying how perception, control, touch, and learned policies can be combined in one reproducible MuJoCo system.
Add tactile/contact sensing and interpretable diagnostics.
Introduce self-calibration and pose-alignment optimization.
Build signal fusion modules across vision, touch, and future modalities.
Use the resulting state/action interface for action models and RL.
Longer-term direction:
text
1multimodal sensing -> fused state estimate -> calibrated hand model
2-> action prior / action model -> closed-loop control -> manipulation skill learning
A useful robot-hand research platform should support more than just kinematic imitation. It should let us study how multiple signals can be combined into a richer control system.
This repo is meant to become a small but serious foundation for that direction:
That makes it relevant not only as a demo, but as infrastructure for future work in bio-inspired robotics, sensor fusion, contact-rich manipulation, and learning-based control.
