GR00T N1.7 — Unitree G1 + Dex1, IKEA table assembly (30 Hz, relative arms)
Action-expert fine-tune of
nvidia/GR00T-N1.7-3B on
carroll511/IKEA_table_assembly,
recorded with
teleop_ikea.py on a Unitree G1 with Dex1 grippers. Stationary
bimanual assembly: pick a table leg, insert it, tighten it.
Selected by an open-loop scan over 10 checkpoints, not by eval_loss — see
Checkpoint selection.
Inputs
| Modality | Keys | Notes |
|---|
| video (3) | cam_left_high, cam_left_wrist, cam_right_wrist | cam_left_high is the verified left eye of the head stereo pair; the right eye is dropped (a BCT ablation measured a second head view at 0.00% change in arm accuracy) |
| state (46) | legs(12) waist(3) left_arm(7) right_arm(7) left_gripper(1) right_gripper(1) base_gravity(3) left_eef(6) right_eef(6) | see State conventions |
| language | annotation.human.task_description | 3 subtasks |
Outputs
Chunk of 40 steps (1.33 s at 30 Hz) over 16 dims:
| Key | Dims | Representation |
|---|
left_arm, right_arm | 7 + 7 | RELATIVE (delta from current state) |
left_gripper, right_gripper | 1 + 1 | ABSOLUTE, range 0–5.4 |
No waist and no base velocity. teleop_ikea.py commands the arms only and holds
the waist at its startup pose (0.8° of drift within an episode), and the source
recording's base_cmd_vel is exactly zero on all 66,600 frames — this was
stationary work, so both were left out of the action space rather than trained
as constants.
State conventions (must be reproduced exactly at deployment)
base_gravity — gravity direction in the pelvis frame from the IMU quaternion
(w,x,y,z): -[2(xz - wy), 2(yz + wx), 1 - 2(x² + y²)]. Yaw-invariant, unlike
the raw quaternion.left_eef / right_eef — wrist pose by forward kinematics from the arm joints
in the same row. Convention: g1_body29_hand14.urdf
(sha256 8bbf0066…c238735), waist joints zeroed, wrist_yaw origin translated
+0.05 m along local x, orientation as extrinsic xyz Euler. Verified to
reproduce the dataset's *_eef blocks to 1.5e-08 m / 5.9e-08 rad, at lag 0.- Root position/orientation are deliberately absent.
A mismatch in these conventions silently corrupts 15 of 46 input dims.
state_dropout=0.2 makes the model robust to missing state, not wrong state.
Open-loop accuracy (26 held-out episodes, 614 windows, unnormalized units)
Held-out split is 4 whole recording sessions, not random episodes:
consecutive episodes share a camera pose and foot placement, so a random split
leaves near-duplicates of the val set in training.
| Executed steps | Re-inference period | Arm MAE | Wrist position error |
|---|
| 1 | 0.03 s | 0.80° | 7.99 mm |
| 5 | 0.17 s | 1.39° | 11.06 mm |
| 8 | 0.27 s | 1.80° | 13.14 mm |
| 16 | 0.53 s | 2.67° | 17.65 mm |
| 40 (full chunk) | 1.33 s | 4.33° | 25.85 mm |
Error grows with horizon, so the full-chunk figure is not the deployment figure —
execute the head of the chunk and re-infer. Chunk-averaged over the first 8
steps: arm 1.311°, wrist 10.62 mm. Wrist error is computed by running the same
FK on the predicted and ground-truth arm joints.
Per task (full chunk):
| Task | windows | Arm | Wrist | Gripper | Wrist travel over chunk |
|---|
pick table leg | 142 | 2.585° | 16.57 mm | 0.1845 | 44.2 mm |
insert table leg to table base | 179 | 2.447° | 16.59 mm | 0.1065 | 32.0 mm |
rotate leg to tighten | 293 | 3.318° | 21.05 mm | 0.1158 | 50.6 mm |
insert has the worst error-to-motion ratio (0.52), which matches it being the
precision step. Note pick is the least represented task in training (22.0% of
windows vs 49.8% for rotate) and still scores best on arm and wrist error, so
the spread is task difficulty rather than data imbalance.
Training
| |
|---|
| Data | 250 train episodes / 59,589 frames @ 30 fps (26 episodes / 7,011 frames held out) |
| Trainable | action expert only (1.62B of 3.14B); Cosmos-Reason2 backbone frozen |
| Schedule | 20,000 steps, effective batch 64 (global 16 × accum 4), cosine, lr 1e-4, warmup 0.05, wd 1e-5 |
| Regularization | state_dropout 0.2, colour jitter 0.3/0.4/0.5/0.08, random crop 89.8% |
| Hardware | 2× A100 80GB, DDP with bf16 gradient communication, 8 h 10 m |
Effective batch 64 rather than the 192 used on a 500k-frame dataset: at 60k
frames one epoch is 779 optimizer steps at 64 but only 260 at 192.
Two GPUs, not four: at a fixed effective batch, more ranks split the same 64
samples further while the 1.62B-parameter all-reduce stays once per step and
grows with peer count. Measured here (s/step): 1 GPU 2.46, 2 GPUs 1.46,
4 GPUs 2.24 — the host has no NVLink.
Checkpoint selection
eval_loss is the flow-matching regression objective at random noise levels;
across five earlier runs on a sibling dataset it rose after ~7.5k steps while
open-loop action accuracy kept improving. It is not a model-selection signal
here. Checkpoints were scanned with a fixed-seed open-loop evaluation on the
held-out split instead, at a stride of 10 frames rather than one window per
chunk, because 164 non-overlapping windows cannot separate checkpoints that
differ by 1–2%.
Arm and wrist error plateau by ~16k steps (16k→20k: arm −0.35%, wrist −0.17%);
the remaining gain is gripper timing (−4.64% over the same span).
Ablations that did not pay off
| Variant | Result |
|---|
torso_gravity added to the state (46 → 49) | Worse on every metric (MSE +5.4%, wrist +1.0%). It is a near-exact linear function of base_gravity + waist, both already in the state (R² 0.9985), so it adds redundancy, not information. |
| Colour jitter halved | Within noise (+0.3% to +2.4%). The frozen encoder's features move 64% of a scene-change distance under the default jitter, yet halving that changed nothing measurable — the action expert is not limited by this. |
| Warm-start extension to 30k total | MSE degraded 10.5% at +2k steps and took 6k steps to recover, ending +0.1% on the deployment metric. Checkpoints here are saved with --save-only-model, so optimizer and scheduler state are absent and a completed cosine has annealed the LR to ~1e-12; resuming restarts Adam from zero moments and jumps the LR back up. |
