Galahad C3-jfs (rebuild) — dual-branch grounded VLA
Paper:
Instruction Blindness in Vision–Language–Action Policies: Diagnosis and a Low-Rank Data Cure —
PDF ·
Project page ·
Code + battery
Single-weight MolmoAct2 policy: LoRA folded into the base, foresight head (_gf.*, 13 tensors) carried in the
same model.safetensors. One forward, dual output (action + predicted object region). Not a base+adapter bundle.
Status: RECIPE ORIGINAL · CODE PART-RECONSTRUCTED · NUMBERS RE-MEASURED · BATTERY PENDING
Rebuilds the weight published as galahad-release-v2, deleted 2026-07-21.
- Recipe is original, not a reconstruction. Trained on the intended fork pin
80633827176a0203064cb141383664fba024e050 (exposes enable_lora_vlm/enable_lora_action_expert
natively; no compat patch). The pin is on no branch of allenai/lerobot (rebased past) — it survives on
refs/pull/2/head.
- Reconstructed:
galahad_foresight.py — a provably pre-rename snapshot; other silent drifts possible.
- 🔴 Prior numbers do NOT transfer. TASK 94.0% n=200, SWAP2 200/200, C4-i 58.3%, C6 0.3-0.5 cm were on the
DELETED weight. Not repeated here.
Recipe (frozen)
| |
|---|
| base | allenai/MolmoAct2-Think-LIBERO |
| fork pin | 80633827176a0203064cb141383664fba024e050 (old API, native) |
| LoRA | VLM + action_expert, r32 / α16 / dropout0.05; trainable 109069184 (expert LoRA'd, not full-trained) |
| schedule | joint-from-start (GALAHAD_FS_OBS_FIRST=0), 3000 steps, DDP-8, batch 1/GPU |
| foresight | M=8 tokens, G=16, α W=0.05 (frozen — original never rebalanced), BCE pos_weight 15 |
| data | phi-monster/arena-deconf-c1v2-jul2026 (900 ep / 95,658 frames, baked 16×16 region GT) |
| merge | peft merge_and_unload; self-check A/B/C PASS (fold Δ 0.012, tol 0.05); lora=0 _gf=13 base_prefix=0 |
| repo | phi-monster/research @ d09e4df |
Contract gates (this run)
| gate | result |
|---|
| token append (train) | seq 472 -> 480 (+8 foresight tokens) |
| region GT | binary min0/max1/n_unique2 (identity stats; quantile normalizer did not touch it) |
| region GT positive-fraction | 0.0508 — compact object blob, not the ~0.5 arm-sweep |
| action expert | LoRA'd, not full-trained |
| foresight regionIoU (train) | 0.572 (median of last 40 per-rank samples, n=954; per-rank single-sample stat) |
Evaluation — un-fakeable battery on libero_object_task (LIBERO-PRO, the B63 bench)
Re-measured on THIS weight (n=40/face = 8 non-round-bottle scenes x 5; round bottles dropped via --drop,
a known-0%-oracle-grasp motor wall, not a grounding wall). Suite: zhouxueyang/LIBERO-Pro bddl+init (paired;
derangement + satisfiability re-verified), assets from an on-box LIBERO-PRO checkout. Every cell gated on the
inference token-append (seq 470 -> 478 [contract MATCHED]), 0 tripwires.
| face | result | reading |
|---|
| TASK (perturbed instruction, full vision) | 38/40 = 95.0% | follows the substituted instruction |
| SWAP2 (in-scene decoy) | true-target 0/40 = 0.0%, OBEYED_NAME 38/40 | EEF goes to the named decoy (d_named<<d_true), not the memorized target |
| OCC (both cameras blanked) | 0/40 = 0.0% | collapses -> vision is required |
| NONSENSE ("xxx" instruction) | 8/40 = 20.0% | mostly collapses (5/8 scenes 0%); 72.5pp below TASK -> instruction required; residual per-scene visual prior on ~3 scenes |
| UNPERTURBED_COMPLETION (stock libero_object) | 34/40 = 85.0% | positive control: the weight is capable |
Cold-verify — honest read (NOT all controls collapse):
- OCC collapses to 0% -> vision is required (rules out solving from proprio/memory).
- SWAP2 is the un-fakeable core: true-target ~0 WITH OBEYED_NAME high (EEF goes to the named decoy,
d_named << d_true). A visual-prior / instruction-ignoring policy would grab the CANONICAL object, not the
decoy, so SWAP2 cannot be a visual-prior artifact -> instruction-following is real.
- NONSENSE mostly collapses (8/40 = 20.0%): 5 of 8 scenes go to 0% under a meaningless instruction; 3 scenes
retain a per-scene visual-prior success. The instruction-grounding excess is large (TASK 38/40 = 95.0% vs
NONSENSE 8/40 = 20.0% = ~72pp), so the instruction is required; a small residual visual prior remains on a few
scenes. Reported as-is.
- Merge action-equivalence 1.19e-2 (cf. B63 1.17e-2) = faithful fold.
🔴 These are the rebuilt weight's OWN numbers. B63's TASK 94.0% / SWAP2 200/200 were on the deleted weight.
Evaluation — un-fakeable battery on libero_object_task (LIBERO-PRO, the B63 bench)
Re-measured on THIS weight (n=40/face = 8 non-round-bottle scenes x 5; round bottles dropped via --drop,
a known-0%-oracle-grasp motor wall, not a grounding wall). Suite: zhouxueyang/LIBERO-Pro bddl+init (paired;
derangement + satisfiability re-verified), assets from an on-box LIBERO-PRO checkout. Every cell gated on the
inference token-append (seq 470 -> 478 [contract MATCHED]), 0 tripwires.
| face | result | reading |
|---|
| TASK (perturbed instruction, full vision) | 38/40 = 95.0% | follows the substituted instruction |
| SWAP2 (in-scene decoy) | true-target 0/40 = 0.0%, OBEYED_NAME 38/40 | EEF goes to the named decoy (d_named<<d_true), not the memorized target |
| OCC (both cameras blanked) | 0/40 = 0.0% | collapses -> vision is required |
| NONSENSE ("xxx" instruction) | 8/40 = 20.0% | mostly collapses (5/8 scenes 0%); 72.5pp below TASK -> instruction required; residual per-scene visual prior on ~3 scenes |
| UNPERTURBED_COMPLETION (stock libero_object) | 34/40 = 85.0% | positive control: the weight is capable |
Cold-verify — honest read (NOT all controls collapse):
- OCC collapses to 0% -> vision is required (rules out solving from proprio/memory).
- SWAP2 is the un-fakeable core: true-target ~0 WITH OBEYED_NAME high (EEF goes to the named decoy,
d_named << d_true). A visual-prior / instruction-ignoring policy would grab the CANONICAL object, not the
decoy, so SWAP2 cannot be a visual-prior artifact -> instruction-following is real.
- NONSENSE mostly collapses (8/40 = 20.0%): 5 of 8 scenes go to 0% under a meaningless instruction; 3 scenes
retain a per-scene visual-prior success. The instruction-grounding excess is large (TASK 38/40 = 95.0% vs
NONSENSE 8/40 = 20.0% = ~72pp), so the instruction is required; a small residual visual prior remains on a few
scenes. Reported as-is.
- Merge action-equivalence 1.19e-2 (cf. B63 1.17e-2) = faithful fold.
🔴 These are the rebuilt weight's OWN numbers. B63's TASK 94.0% / SWAP2 200/200 were on the deleted weight.
Evaluation — un-fakeable battery on libero_object_task (LIBERO-PRO, the B63 bench)
Re-measured on THIS weight (n=40/face = 8 non-round-bottle scenes x 5; round bottles dropped via --drop,
a known-0%-oracle-grasp motor wall, not a grounding wall). Suite: zhouxueyang/LIBERO-Pro bddl+init (paired;
derangement + satisfiability re-verified), assets from an on-box LIBERO-PRO checkout. Every cell gated on the
inference token-append (seq 470 -> 478 [contract MATCHED]), 0 tripwires.
| face | result | reading |
|---|
| TASK (perturbed instruction, full vision) | 38/40 = 95.0% | follows the substituted instruction |
| SWAP2 (in-scene decoy) | true-target 0/40 = 0.0%, OBEYED_NAME 38/40 | EEF goes to the named decoy (d_named<<d_true), not the memorized target |
| OCC (both cameras blanked) | 0/40 = 0.0% | collapses -> vision is required |
| NONSENSE ("xxx" instruction) | 9/40 = 22.5% | mostly collapses (5/8 scenes 0%); 72.5pp below TASK -> instruction required; residual per-scene visual prior on ~3 scenes |
| UNPERTURBED_COMPLETION (stock libero_object) | 34/40 = 85.0% | positive control: the weight is capable |
Cold-verify — honest read (NOT all controls collapse):
- OCC collapses to 0% -> vision is required (rules out solving from proprio/memory).
- SWAP2 is the un-fakeable core: true-target ~0 WITH OBEYED_NAME high (EEF goes to the named decoy,
d_named << d_true). A visual-prior / instruction-ignoring policy would grab the CANONICAL object, not the
decoy, so SWAP2 cannot be a visual-prior artifact -> instruction-following is real.
- NONSENSE mostly collapses (9/40 = 22.5%): 5 of 8 scenes go to 0% under a meaningless instruction; 3 scenes
retain a per-scene visual-prior success. The instruction-grounding excess is large (TASK 38/40 = 95.0% vs
NONSENSE 9/40 = 22.5% = ~72pp), so the instruction is required; a small residual visual prior remains on a few
scenes. 🔴 9/40 = 22.5% is HIGHER than B63's 6.0% (12/200) — at n=40 vs B63's n=200 this is likely small-n,
but the garbage-instruction control is a bit less clean on this reconstruction. Stated, not smoothed.
(A second independent n=40 nonsense run gave 20.0%, consistent.)
- Merge action-equivalence 1.19e-2 (cf. B63 1.17e-2) = faithful fold.
Reproduction verdict: these n=40 numbers REPRODUCE the deleted weight's B63 grounding pattern
(TASK ~94% -> 95.0%; SWAP2 OBEYED high 38/40 + true-target 0; OCC 0). The n=40 re-measurement
is the release target for the reconstructed weight; we do NOT extend to n=200.
🔴 The n=200 main table was measured on the now-DELETED weight and is NOT transferred here (TASK 94.0%,
SWAP2 200/200). This release ships the reconstructed weight with its own n=40 re-measurement above + the
verdict that it reproduces B63. The recipe is original (real fork pin 80633827, native LoRA API), but
galahad_foresight.py is a recovered pre-rename snapshot => RECONSTRUCTION, not a byte reproduction.
Bench: libero_object_task from zhouxueyang/LIBERO-Pro.
C6 chain-demo figure: pending — handed to the writing team; not rendered on this weight here (so its absence is intentional, not missing).
