MolmoBot-Pi0 — DROID absolute-joint, reasoning-init full fine-tune
MolmoBot-Pi0 (a PaliGemma-decoder VLA with a flow-matching action expert),
fully fine-tuned on equad DROID with absolute joint-pose action targets,
starting from a reasoning-initialized backbone.
Checkpoint: step 18000 — best held-out validation loss (val_action_loss = 0.0041).
What "reasoning-init" means
Standard full fine-tuning starts from the base VLA weights. Here we first bake a
frozen reasoning LoRA into the PaliGemma decoder, then full-fine-tune from that
point. The reasoning adapter is not kept as a separate module — its delta is folded
directly into the base weights:
W <- W + (alpha / r) * (B @ A) # applied in-place, per target module
with r = 64, alpha = 128 (scaling 2.0), folded across 207 modules. The
adapter came from a reasoning-only training run on the same DROID data
(Gemini-annotated chain-of-thought, PaliGemma prompt format).
So reasoning enters as a weight-space prior on the initialization, not as an
auxiliary loss and not as a runtime module. After the fold the model is a plain flat
checkpoint — inference is identical to any other pi0 checkpoint, with no reasoning
code path and no extra parameters.
On resume the fold is deliberately skipped (the delta is already inside the saved
weights), so it is applied exactly once, at step 0.
Action space
Absolute joint pose, 8-dim per step:
dims
meaning
0:7
absolute future joint positions, arm[t] = q[t+1] (7 arm joints)
7
gripper command (0–1)
Action horizon 16. At execution the prediction is used directly as the target
joint pose — no current-position offset (contrast with the joint_delta variant,
where target = current_q + pred).
Normalization statistics are bundled in assets/droid_equad/norm_stats.json; arm
stats follow the observation.state distribution, gripper from the action gripper dim.
Training setup
Base checkpoint
MolmoBot-Pi0-DROID
Reasoning adapter
reasoning-only LoRA, r=64 / α=128, same DROID data
Dataset
droid-equad-stage-1 (LeRobot v3.0)
Cameras
exterior_1_left, wrist_left
Config
molmobot_pi0_lerobot_droid_absjoint
Trainable
entire model (full FT, no LoRA, no freezing)
Hardware
4 × NVIDIA L40 (46 GB), DDP
Batch
64 × grad-accum 8 → effective 512
Optimizer
AdamW, weight_decay 1e-10, grad clip 1.0
LR
peak 5e-5, warmup 1000 steps, constant thereafter
Precision
bf16
Throughput
~30 s / step
Steps
20000 (early-stopped; 30000 was the cap)
Checkpoint selection
Validation runs every 1000 steps on 8 fixed held-out batches (flow noise reseeded
so the metric is deterministic and comparable across steps). Retention keeps the
latest checkpoint (for resume) plus the 2 best by val_action_loss.
Early stopping
Training was supervised by a self-chaining job that, each time a training job ended,
read the cumulative validation series and either resumed training (if val was still
improving) or stopped. It resumed 3 times and stopped at step 20000 once no new best
appeared for 2 consecutive validation points — leaving 10000 of the budgeted steps
unspent.
Validation curve
val_action_loss, every 1000 steps:
step
val
step
val
step
val
step
val
1000
0.0346
6000
0.0074
11000
0.0057
16000
0.0060
2000
0.0199
7000
0.0067
12000
0.0063
17000
0.0046
3000
0.0128
8000
0.0071
13000
0.0089
18000
0.0041
4000
0.0112
9000
0.0064
14000
0.0057
19000
0.0059
5000
0.0096
10000
0.0082
15000
0.0054
20000
0.0045
Roughly an 8× reduction from the first validation point, with most of the gain by
step ~6000 and a slow, noisy tail after ~11000.
Contents
model.safetensors # flat full model (7.0 GB), reasoning delta folded in
metadata.pt # global_step and training bookkeeping
assets/droid_equad/norm_stats.json # state + action normalization statistics
assets/train_config.pkl # serialized openpi TrainConfig
Optimizer state is not included (10.9 GB) — this checkpoint is for inference and
evaluation, not for resuming training.
Related
shrg7/MolmoBot-Pi0-DROID-absjoint — same data and action space, without reasoning-init (the control)
shrg7/MolmoAct2-DROID-absjoint — MolmoAct2 on the same absolute-joint DROID targets
