The model controls a Franka Panda robot performing ultrasound probe manipulation tasks
(placement, transverse scanning, anatomical navigation) on abdominal, thyroid, and arm phantoms.
Demo
Unified demo — GR00T-H-N1.7 on TUM SonATA
Open-loop inference: probe positioning → transverse-plane traversal. One frame shows the live
cameras (third-person / wrist / ultrasound), predicted vs. ground-truth action curves, position
and orientation tracking error, the commanded Franka motion (GT green-ghost vs. predicted
orange), and the EEF path. Full video + reproducible pipeline:
Hemanth21k/world-models.
Model Details
Property
Value
Base model
nvidia/GR00T-H-N1.7
Backbone (frozen)
NVIDIA Cosmos-Reason2-2B (Qwen3-VL); features @ layer 16, dim 2048
Projector + diffusion action head (backbone frozen)
Framework
DeepSpeed ZeRO-2, PyTorch 2.7
Final loss
~0.025 at step 16,000 (from 1.57; ~0.039 → 0.025 over the last 6k steps)
Training Notes
A gradient spike (loss ≈ 55, grad norm ≈ 155) occurred at approximately step 2,500 when
the learning rate reached its peak. Training recovered automatically via gradient clipping.
For future runs at this batch size, a peak learning rate of 4e-4 or lower is recommended.
Training was stopped at step 16,000 (≈0.8 epoch). The loss was still decreasing but
decelerating, and test-split error was already strong (see Evaluation); a longer / lower-LR /
multi-epoch run is left as future work.
Evaluation
Held-out test split (482 episodes). Per-step error of the predicted vs. ground-truth EEF
action, swept over the re-inference horizon H. Position = XYZ L2; orientation = geodesic
angle; baseline = zero-motion (hold last observed pose). Seeded for reproducibility.
Mode
H
Pos (cm)
Rot (°)
Baseline pos (cm)
Baseline rot (°)
Open-loop
1
0.09
1.07
0.19
0.53
Open-loop
8
0.37
1.47
0.84
2.37
Open-loop
16
0.64
2.15
1.55
4.39
Open-loop
50
1.61
4.92
4.24
12.70
Rollout
16
4.72
14.14
1.55
4.39
Open-loop (true state each step): sub-cm to ~6 mm, beating the zero-motion baseline
~2–2.6× on position; degrades gracefully as re-inference gets sparser.
Rollout (predicted EEF pose fed back as the reference state): errors compound (~4 cm /
~14°). Note this rollout is hybrid — only the EEF pose is fed back; cameras and the rest of
the state come from the dataset (a true closed loop needs a world model).
Full sweep (all horizons, both modes, std/median/max) and the evaluation code are in the
GitHub repo.
The SonATA dataset is a robotic sonography collection from TUM's Computer Aided Medical
Procedures (CAMP) Lab, containing synchronized ultrasound imaging, external RGB cameras,
contact force/torque measurements, robot joint state, and natural language instructions
collected from abdominal, thyroid, and arm phantoms.
Subset
Episodes
Tasks
SonATA_abdomen
1,533
287
SonATA_arm
~1,107
—
SonATA_thyroid
~780
—
Acknowledgements
This work was conducted at the Quantitative Bio Imaging Lab (QBIL)
at The University of Texas at Dallas.
Research reported in this publication was supported in part by the National Cancer Institute
of the National Institutes of Health under Award Numbers R01CA288379 and R01CA204254,
and by the Cancer Prevention and Research Institute of Texas (CPRIT) under Award Number
RP240289. The content is solely the responsibility of the authors and does not necessarily
represent the official views of the National Institutes of Health.
Computing resources were provided by the QBIL Lab GPU cluster at UT Dallas.
1@software{pasupuleti2026worldmodels,
2 author = {Pasupuleti, Hemanth},
3 title = {world-models: Unified interface for testing and extending
4 world model architectures for Physical AI},
5 year = {2026},
6 url = {https://github.com/Hemanth21k/world-models},
7 note = {Quantitative Bio Imaging Lab (QBIL), The University of Texas at Dallas.
8 Supported by NIH R01CA288379, R01CA204254 and CPRIT RP240289.}
9}
