Cosmos 3 Nano-0H Surgical dVRK is a domain-specific post-trained Cosmos 3 Nano checkpoint for short-horizon surgical kinematics prediction and dVRK motion simulation. It adapts the Nano-scale Cosmos world model to dVRK/da Vinci endoscopic scenes and uses a native dVRK action representation that is compatible with GR00T-H-style dual-arm end-effector control.

The model is intended as a research world model for surgical robotics: predicting and comparing near-future tool motion, recovering action-like kinematic structure from short video clips, validating action interfaces and studying low-latency motion simulation behaviour in visually constrained robotic manipulation settings.

Video is a secondary diagnostic channel in this limited NanoH run, not the main product. The video objective remains much harder than the action/kinematic objectives: generated video quality is substantially weaker than the kinematics outputs even when the rollout is conditioned. Use this checkpoint as a compact surgical motion model, not as a surgical video generator.

It is not intended for clinical decision-making, autonomous surgery, medical diagnosis or patient-facing deployment.

Model details

Base model: nvidia/Cosmos3-Nano
Adaptation: supervised post-training on dVRK surgical windows
Primary task: surgical kinematics prediction and short-horizon dVRK motion simulation
Secondary task: action-conditioned visual rollout for context and diagnostics
Prediction horizon: short-horizon 16-frame chunks
Visual input: single endoscopic camera stream
Action input: dual-PSM dVRK tensor, relative EEF motion with rot6D
Training objective: mixed forward dynamics, inverse dynamics and policy
Precision: bfloat16 training
Distributed strategy: single-node FSDP
Checkpoint format: Cosmos DCP checkpoints plus optional HF export

Action space

The model registers dVRK as a first-class Cosmos embodiment rather than mapping it onto an existing robot id.

text

embodiment: dvrk
domain_id: 22
raw_action_dim: 20
chunk_length: 16

Native dVRK tensor:

text

actions_dvrk: (16, 20)

left_xyz_rel(3), left_rot6d_rel(6), left_jaw_abs(1),
right_xyz_rel(3), right_rot6d_rel(6), right_jaw_abs(1)

Surgical simulator bridge:

text

actions_unified44: (16, 44)
actions_unified44[..., :20] == actions_dvrk
actions_unified44[..., 20:44] == 0

Cosmos action head:

text

actions_cosmos3: (16, 64)
raw_action_dim: 20

The 20D native tensor is padded to the Cosmos 3 64D action head while preserving the raw dimension for action-channel masking. A 44D zero-padded bridge is included for surgical simulator interoperability.

Training data

Training uses chrisvoncsefalvay/openembodiment-dvrk-subset, a convenience subset of nvidia/PhysicalAI-Robotics-Open-H-Embodiment.

The subset mirrors dVRK/da Vinci records and associated metadata so training machines can stream from one stable HF/Xet-backed repository. The underlying source data and credit remain with NVIDIA and the Open-H Embodiment contributors.

Training recipe

The release recipe is designed for a single 8xH100 node:

FSDP over 8 GPUs
bfloat16 precision
activation checkpointing
16-frame action chunks
single-camera video stream
all-objective sampling across forward dynamics, inverse dynamics and policy
checkpoint save interval of 100 iterations
W&B metrics and qualitative visual logging
interrupt-tolerant DCP checkpoints
final artefact upload to a private Hugging Face model repository

The recipe demonstrates that useful domain adaptation of Cosmos 3 Nano does not require enterprise-scale cluster choreography. Larger runs remain valuable, but the single-node path is already strong enough to reveal meaningful surgical domain improvement.

Evaluation

Evaluation emphasises short-horizon kinematic consistency and action-space validity. Video metrics are retained as diagnostics, but kinematic and toolpath checks are the primary success criteria for this run:

tensor validation for dVRK, 44D simulator bridge and Cosmos 64D padded action
held-out episode splits
FDS-L1 over short rollouts
toolpath and end-effector trajectory comparison against ground truth
inverse-kinematics consistency on held-out motion snippets
action-space normalization and scale audits
qualitative W&B panels with context, prediction, actual, error strip and action norms
GR00T-H interop samples in native, 44D and 64D action formats
PSNR, lightweight global SSIM and temporal frame-difference L1 as secondary video diagnostics

Latency and smoothness are first-class evaluation concerns. A short rollout that keeps motion coherent and fast is more valuable for this use case than a long rollout that looks impressive only until the simulator has to respond.

Intended use

Appropriate use:

research on surgical robotics world models
short-horizon dVRK kinematics and toolpath prediction
action-space validation and interop work
simulation prototyping and qualitative model comparison
non-clinical robotics research demos

Out-of-scope use:

clinical decision support
autonomous surgical control
diagnosis, treatment planning or patient triage
safety-critical deployment
replacing a validated simulator or physical robot test

Limitations

The model is specialised to dVRK-like endoscopic surgical scenes.
It is optimised for short-horizon consistency, not indefinite rollout length.
It may hallucinate tool geometry, tissue response, lighting or contact effects.
Video rollouts are weaker than the kinematics outputs and should be treated as qualitative diagnostics rather than photorealistic surgical video.
It should not be treated as a calibrated dynamics oracle.
Single-camera training limits viewpoint disentanglement.
Dataset coverage reflects the dVRK subset available in Open-H Embodiment.

Safety

This checkpoint is a research artefact. Predicted toolpaths, actions and generated video should be interpreted as learned simulation outputs, not as measured surgical reality. Any downstream robotics use requires independent validation, simulator checks, hardware safety systems and domain-expert review.

Citation

Please cite the upstream Open-H Embodiment dataset and NVIDIA Cosmos materials when using this model or the associated dataset subset. Cite this checkpoint repository for the dVRK action mapping, kinematics-focused training recipe and released weights.

bibtex

@misc{voncsefalvay2026cosmos3nanohsurgicaldvrk,
  title = {Cosmos 3 NanoH surgical dVRK},
  author = {von Csefalvay, Chris},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face model repository},
  url = {https://huggingface.co/chrisvoncsefalvay/cosmos3-nanoh-surgical-dvrk}
}

Artefacts

Expected repository contents after release:

text

model/                 exported HF checkpoint
checkpoints/           resumable Cosmos DCP checkpoints
eval/                  quantitative reports and qualitative references
interop/               dVRK, 44D simulator and 64D Cosmos action samples
README.md              this model card

License

Released model materials are licensed under OpenMDW-1.1. Model code and adapter utilities in the source repository remain Apache-2.0. Dataset files retain the terms of the upstream Open-H Embodiment release. Base model usage is subject to the terms of the upstream Cosmos 3 Nano release.

introvoyz041/cosmos3-nanoh-surgical-dvrk

Cosmos 3 Nano-H surgical dVRK

Model details

Action space

Training data

Training recipe

Evaluation

Intended use

Limitations

Safety

Citation

Artefacts

License

introvoyz041/
cosmos3-nanoh-surgical-dvrk