Cosmos3-Nano DROID Forward-Dynamics World Model
Action-conditioned forward-dynamics (FD) world model for the DROID manipulation setup,
post-trained from Cosmos3-Nano. Given a single conditioning frame plus a chunk of robot
actions, it predicts the resulting multi-view video — usable as a rollout environment for
policy evaluation / dreaming.
- Base model: Cosmos3-Nano (Qwen3-VL-8B MoT backbone + Wan2.2 VAE tokenizer)
- Mode:
forward_dynamics — condition on frame 0 + all actions → predict video
- Action space:
joint_pos (8D absolute joint targets), use_state on
- Views: multi-view concat (wrist + 2 exterior cameras)
- Chunk length: 32 · operating point: resolution 480, shift 5
- Precision: bf16
Checkpoints
Each subfolder is a self-contained HF safetensors export (7 shards + config.json + checkpoint.json).
| subfolder | iter | held-out motion-PSNR | PSNR | SSIM | note |
|---|
best-iter5600/ | 5,600 | 14.63 dB | 14.03 | 0.391 | best by held-out motion-PSNR |
latest-iter23800/ | 23,800 | 14.57 dB | 14.01 | 0.391 | latest; metric plateaued (see below) |
Held-out eval = seeded 1% split (never trained on), n=12 episodes, motion-PSNR masked to moving regions.
The single-chunk metric plateaus ~5k iters; the extra training in latest does not improve it and does
not add long-horizon/off-manifold robustness — best is the recommended default; latest is provided for
completeness.
Training
- g512: 16× H200 (2 nodes), global batch 512, DROID (92,233 episodes, multi-view).
- Post-trained from the base Cosmos3-Nano DCP checkpoint via the
cosmos-framework action SFT recipe.
Usage
These are cosmos-framework model exports (not transformers-native). Load with the
cosmos-framework inference stack (the exported config.json / checkpoint.json describe the
OmniMoTModel). Point the DROID multiview FD rollout tooling at a subfolder as the checkpoint path.
Caveats
Research checkpoints. FD conditions on a single history frame, so autoregressive rollouts drift over long
horizons and can collapse off-manifold (many re-conditioning steps) — coherent window ≈ 30–60 frames at
keep-10 cadence, longer at chunk-32 cadence. Best for short-horizon / qualitative policy checks.
