Re-run at 5,000 steps with the ego half filtered to clips that fill measured coverage gaps in the sim pool. Both gap mixtures now beat sim-only on the same 5-episode holdout: sim10 87.19 / gapmix10 84.09 (−3.6%) / gapmix20 79.76 (−8.5%) deg² clean MSE. sim10 itself degraded from 82.10 (2,400 steps) to 87.19 (5,000) — it overshot its knee at 21 epochs of 9 minutes of data — while the mixtures improved, so the durable claim is that ego co-training makes the recipe robust to the training budget. Full analysis, evals (LLM-paraphrase perturbation, CI-MSE on gripper-derived and VLM-annotated intervals), caveats and the filter-attribution question: REPORT_ROUND2.md.

Round-2 artifacts: sim10/4999 (fresh restart, NOT the same run as sim10/2399), gapmix10/4999, gapmix20/4999, metrics/round2/, eval/round2/, configs/round2/, data/round2_gap_pools/. Everything below this line is the original round-1 card.

π₀.₅ on SO-101: Does Retargeted Ego Video Buy Anything On Top of 9 Minutes of Sim?

Status: All three arms trained to completion 2026-08-15. Offline-evaluated 2026-08-15 on the 5-episode sim holdout with the fd-studio suite (offline proxies + perturbation robustness + CI-MSE). Runs: mm_pi05_sim10/sim10 · mm_pi05_mix10/mix10 · mm_pi05_mix20/mix20 — W&B uc9odsyk / r5y0x9eb / tunaarse, project mm-so101-sim-vs-ego · 2,400 steps each · 59–61 min each · 2× H100 80GB

1. Executive Summary

Objective. Ask whether EgoDex human video, retargeted to SO-101, is worth anything on top of a very small sim corpus (9 min 15 s, one task). Three π₀.₅ arms, one embodiment, one holdout, matched step budget; the only thing that moves is what is in the batch.

arm	total data	per-batch draw (of 64)	what it asks
`mm_pi05_sim10`	~9 min sim	sim 64	the baseline
`mm_pi05_mix10`	~5 + ~5 min	sim 32 / ego 32	replace half the sim with ego at fixed data budget
`mm_pi05_mix20`	~10 + ~20 min	sim 32 / ego 32	add ego on top at fixed compute

Outcome (fact). All three ran to completion, exit 0, no divergence, no interruption, ~~1 h each (~~$12 total). Final-200-step mean flow_loss: sim10 0.001750, mix10 0.001665, mix20 0.002930.

Outcome (eval, 2026-08-15). On the 5 held-out sim episodes (1,671 frames, unseen by all three arms), scored in degrees:

arm	clean MSE (deg²)	RMSE	MAE	vs sim10	perturbation ratio	CI-MSE mean / median	gripper acc
`pi05_base` (zero-shot control)	1634.96	40.43°	25.38°	19.9×	1.040×	746.47 / 619.67	0.433
sim10	82.10	9.06°	4.33°	—	0.991×	16.84 / 11.67	0.907
mix10	85.12	9.23°	4.58°	+3.7%	1.045×	16.56 / 13.79	0.904
mix20	91.80	9.58°	4.65°	+11.8%	1.039×	19.98 / 14.91	0.897

Fine-tuning works; composition barely matters. The un-fine-tuned base is 19.9× worse than the baseline on clean MSE and 44× worse on CI-MSE, with gripper accuracy at 0.433 — i.e. no idea when to open or close. Against that 1,553 deg² gap, the entire spread between the three trained arms is 9.7 deg², or 0.6% of the distance the fine-tuning covered. Every arm learned the task; which data they learned it from is a second-order effect.

Retargeted ego video did not help. The sim-only baseline wins on clean error, is the only arm the perturbation does not degrade, and has the best CI-MSE median. Adding ego on top (mix20) is worse than substituting it (mix10). Honest resolution: with 5 episodes, treat gaps under ~10% as a tie, so sim10 vs mix10 is a tie and mix20's 11.8% only just clears the threshold. The safe conclusion is "ego is not worth including at this scale", not "ego is actively harmful".

The predicted failure mode appeared in the predicted place. wrist_roll is where the mixtures lose most (1.14× and 1.19× the baseline's per-dim MAE), and wrist_roll is the dimension the pre-launch preflight identified as most compressed by blended quantile norm — sim occupying just 0.17 of the mix arms' normalised band. shoulder_lift and elbow (compressed only 1.06–1.24×) show far smaller penalties. That correspondence is the report's strongest single piece of evidence, and it means the measured mixture penalty is at least partly a normalisation cost rather than a statement about ego data itself.

Training loss ranked the arms wrongly, twice. Raw final loss says mix10 is best (0.001665 vs sim10's 0.001750); the holdout says sim10. A first-order norm correction did recover the right direction but overstated the gap by ~20× (predicted 1.83×/2.36×, measured 1.04×/1.12×) because it treated a flow-matching velocity loss as if it were action error. Neither raw nor corrected training loss is a substitute for the holdout.

Biggest success. Three perfectly conditioned, fully matched, cheap runs with a shared schedule object that made step-count drift structurally impossible — plus a preflight that predicted, before any GPU time was spent, exactly which dimension would cost the mixtures accuracy.

Biggest limitation. Five episodes of one task is a thin eval, and every arm is undertrained at 2,400 steps. The ranking is a statement about this data scale and this step budget, not a general one — and the differences between trained arms are small enough (0.6% of the fine-tuning effect) that a larger holdout could reorder them.

Did it achieve its objective? Yes. The three arms are matched, trustworthy, and now scored on a shared leak-free holdout in physical units. The answer is negative for the mixture hypothesis at this scale.

Top three lessons.

The preflight found the schema trap in the wrong place, and the eval confirmed it mattered. The design doc predicted the gripper. The gripper was fine (59% range IoU, 0.90 accuracy on every arm); wrist_roll and wrist_flex were compressed to 0.17 and 0.28 of the band — and wrist_roll is exactly where the mixtures lost.
chunk_last/chunk_first is a free task-diversity readout, monotone in ego content. 2.09 → 2.64 → 3.86 across 0 / 81 / 162 ego clips.
2,400 steps is not enough. Every arm's loss was still falling 7.7–8.7% in its final 200-step window, grad-norm never came within 4× of the clip threshold, and parameter norm moved +0.010%. All three are undertrained, equally — so this is a small-budget result.

2. Experiment Overview

Motivation. The client (mm / makermods) has 9 min 15 s of ManiSkill SO-101 sim for one pick-and-place task, and 324 EgoDex human clips retargeted to the same arm. The question is whether the ego corpus is worth including at all, and whether it is better used to replace scarce sim or to augment it.

Research question. At a fixed 2,400-step budget on one embodiment and one eval task, does substituting or adding retargeted ego video change sim-task performance?

Methodology. Three arms differing only in mixture composition. samples_per_batch is a hard per-batch count, so each source's gradient share is exactly samples_per_batch / batch_size regardless of how much of it sits on disk — both mixture arms therefore sit at exactly 50/50 gradient share by construction, and "5 min + 5 min" vs "10 min + 20 min" are statements about the pools (revisit rate), not the draw.

Variable	sim10	mix10	mix20
Sources	1	2	2
Sim frames trained on	14,987	7,756	14,987
Ego frames trained on	0	8,322	17,957
Sim episodes excluded	5 (holdout)	27 (holdout + 22 dropped)	5 (holdout)
Ego holdout fraction	—	0.75 (→ 81 clips)	0.50 (→ 162 clips)
Draw	64 / 0	32 / 32	32 / 32
Sim epochs	10.25	9.90	5.12
Ego epochs	—	8.44	4.28
Norm-stats asset	`mm_sim10`	`mm_mix10`	`mm_mix20`

Held constant. Architecture, LoRA variant/rank, batch 64, peak LR 3.5e-5 with 250-step warmup and cosine to 3.5e-6, decay_steps = num_train_steps = 2400, bf16, EMA off, quantile norm, delta-joint/absolute-gripper actions, action_dim=32, action_horizon=50, max_token_len=200, augmentation on, num_workers=16, seed 42.

Why all three run the same number of steps. Unequal step counts are a double confound: they change exposure and schedule position, because decay_steps tracks num_train_steps, so a longer run spends more absolute steps at floor LR and reports a flattered final loss. A single _SCHEDULE object is shared by all three arms in the config so they cannot drift apart. This was a direct correction of a defect in the earlier YAM mixture study, which ran 11.4k/17.7k/23.6k/32k and could not then use training loss to rank its arms at all.

The epoch spread is deliberate and known. Against the usual guidance (1–3 epochs of a scarce pool, memorisation risk past ~5), mix20 sits at the edge and the other two are ~2× over. This is not fixable by shortening the run: one epoch of the entire sim dataset is 234 steps, so obeying the epoch band literally would mean a ~1,200-step run that has barely left warmup. The two things the rule conflates — has the optimiser converged, and am I memorising — come apart completely at nine minutes of data.

3. Dataset Analysis

3.1 Summary

Property	`sim_v21`	`ego_v21`
Provenance	`makermods/maniskill_50ep_so101_blue_cube_orange_tray_20260812_131142`, LeRobot v3.0 → v2.1	`angkul07/ego-data` (EgoDex) via DT-pipeline stage 6 run F (`--arm dominant`, `IK_FREE_ROLL`, approach-aware BPP)
Version	LeRobot v2.1 (asserted at launch, no migration)	LeRobot v2.1
Episodes / frames	50 / 16,658	324 / 36,442
Duration	9 min 15 s @ 30 fps	20 min 15 s @ 30 fps
Episode length	237–447 f, median 331	30–354 f, median 96
Cameras	`observation.images.front`, `.wrist` — rendered sim viewpoints	same keys — synthesised, 2 crops from 1 ego camera
Video	640×480 h264 yuv420p	640×480 h264 yuv420p
State / action	float32[6], degrees	float32[6], degrees
Joint order	`[pan, lift, elbow, wrist_flex, wrist_roll, gripper]`	same (remapped from the URDF's reversed order)
Action convention	PD setpoint, leads state by 5 frames	absolute next-frame target, exact
Distinct tasks	1	89

Both halves carry a byte-identical feature schema — format, rate, DOF, joint order, units and gripper polarity all agree. Timestamp jitter is 2.6e-07 s (sim) and 1.1e-07 s (ego): exact 1/30 grids, not resampled approximations. Camera frames equal state rows on 100/100 and 648/648 videos respectively, so the frame-surplus defect flagged during dataset construction was fixed before these runs.

3.2 The preflight, and what it actually found

The config docstring mandates a norm-stats preflight before launching, and predicts the gripper as the likely failure: "quantile norm is computed over the blended mixture, so a pool whose gripper lives on a different scale gets crushed under the other's range and flow matching averages the two into a half-open gripper." This is what pi05_ax91_mix9010 shipped.

The gripper is fine. Range IoU 59%, and sim occupies 0.07–0.73 of the blended band. Normalised to the joint's physical travel (URDF −10°…100°), sim spans [0.10, 0.58] and ego [0.00, 0.86]; the band [0.15, 0.45] holds 61.5% of sim frames and 74.8% of ego frames. More to the point the signal agrees: the typical per-episode close→open swing is 0.107→0.389 in sim and 0.154→0.447 in ego — same direction, ≈0.29 of travel each. Polarity was settled from video, not assumed: at each dataset's gripper minimum the sim jaws are pinched shut and the ego hand is closed around the object.

The rotational joints are not fine. Measured off the actual computed norm stats (which are over delta actions, so ego's faster per-step motion stretches them beyond what raw-position quantiles suggest), sim's own q01–q99 occupies this fraction of each arm's normalised band:

dim	sim10	mix10	mix20	half-range widening vs sim10 (mix10)
wrist_roll	1.00	0.17	0.17	5.98×
wrist_flex	1.00	0.28	0.28	3.57×
shoulder_pan	1.00	0.40	0.46	2.49×
gripper	1.00	0.67	0.68	1.49×
elbow	1.00	0.80	0.84	1.24×
shoulder_lift	1.00	0.93	0.94	1.08×

The cause is real and not a bug: ego's median episode is 96 frames against sim's 331, so retargeted human motion moves much faster per step; and across 89 tasks the arm visits the whole workspace (ego pan median −54° vs sim's +4.9°, ego wrist_roll spanning ±160°). The consequence is that the baseline resolves the sim signal at full normalised resolution and the mix arms resolve it at roughly a sixth on the worst channel. This is a cost the mix arms pay and the baseline does not, and it is a confound in the "composition is the only moving part" claim. It is inherent to blending under shared quantile norm, not fixable by config.

3.3 The 89-task discrepancy

configs/mm/datasets.py states TASK = "pick up the cube and place it in the tray" # the only task string in either pool. This comment is wrong. Sim has one task string; ego has 89 distinct ones ("pick up a tv remote from the table and place it in a tub.", etc.), ~11 clips each. With prompt_from_task=True, half of every mix batch carries a sentence that is never the eval prompt.

This was surfaced before launch and the decision was to leave it as-is. The reasoning: forcing sim's sentence onto ego would pair one prompt with 89 different behaviours, which is contradictory supervision on the language channel — strictly worse than letting language correctly identify the behaviour. The consequence for interpretation is that the mix arms measure "does diverse pick-place ego data help", not the docstring's "one task, one embodiment". Any write-up must say so.

3.4 Holdout integrity

SIM_HOLDOUT_EPISODES = (4, 14, 24, 34, 44) — picked by stride, 1,671 frames, 10.03% of the sim pool. Verified withheld from all three arms at load time from the training logs: sim10 holding out 5/50, mix10 holding out 27/50 (the 5 plus 22 dropped for the half-size pool, merged in the manifest so a config passes one list and cannot forget the holdout), mix20 holding out 5/50.

No eval holdout on the ego side by design: the headline metric is sim-task performance on sim frames, so ego is a training ingredient and never something scored.

Note the ego subsets are independent draws, not nested — select_holdout_episodes calls rng.choice(..., size=n), and choice at n=162 is not a prefix of choice at n=243. mix10's 81-clip pool is therefore not a subset of mix20's 162-clip pool. Both are uniform samples of the same corpus and the experimental variable is how much ego there is, so this is acceptable; strict nesting would need pre-built subsets with their own roots.

4. Model & Training Configuration

Field	Value
Base checkpoint	`gs://openpi-assets/checkpoints/pi05_base/params` (12.5 GiB)
Model	`Pi0Config(pi05=True)`, flow matching
`action_dim` / `action_horizon` / `max_token_len`	32 (6 real + 26 pad) / 50 (1.67 s at 30 Hz) / 200
Total / trainable / frozen	3,381.3 M / 872.8 M / 2,508.5 M (measured via `nnx.state` split on the config's freeze filter)
PEFT	LoRA `gemma_2b_lora` rank 16; action expert full-rank; SigLIP trainable
Precision	bfloat16; fp32 param-norm tracking
Optimizer	AdamW β₁ 0.9, β₂ 0.95, ε 1e-8, wd 1e-10, `clip_by_global_norm(1.0)` — never triggered on any arm
LR schedule	Cosine: warmup 250 (10.4% of run) → 3.5e-5 → 3.5e-6, `decay_steps=2400`
Batch	64 (32/GPU data parallel, `--fsdp-devices 1`)
EMA / grad accum	None / None
GPUs	2× H100 80GB, ~74 GB/GPU allocated at `XLA_PYTHON_CLIENT_MEM_FRACTION=0.9`
Steps	2,400 (final step index 2399)
Augmentation	On — brightness 0.15, contrast 0.15, saturation 0.1, hue 0.03, crop_scale (0.95, 1.0), geometric on `base_0_rgb`
Validation	None in-run; openpi has no in-training validation metric
Code rev	`f19dd1c` (`configs/mm/so101/sim_vs_ego.py` modified for checkpoint retention, see §9-A)

On the 10.4% warmup. High against the usual 2–5%, kept deliberately: the bound that matters is absolute, not fractional. The risky region is the first few hundred steps whatever the run length — freshly initialised LoRA B-matrices produce their largest gradients there. Scaling to 5% would put warmup at ~120 steps, below where this family's runs stop spiking. The data supports the choice: peak grad-norm on all three arms lands at step 1–10 (0.25 / 0.18 / 0.25) and there is no discontinuity at the warmup boundary (§ Appendix B).

5. Training Analysis

5.1 Progress

Three single-shot runs, no interruptions, no divergence, no NaNs, no restarts. Every 200-step window improved on every arm. No arm reached a plateau; the best window is the last window in all three cases.

Phase	Steps	Behaviour (all arms)
Initial collapse	0 – ~100	loss 0.036–0.044 → 0.012–0.014; grad-norm 0.14–0.18 → 0.044–0.054
Warmup completion	250	No discontinuity (see Appendix B)
Log-linear descent	~400 – 2,000	−10% to −20% per 200 steps
Tail	2,000 – 2,399	−7.7% to −8.7% per 200 steps — schedule expired mid-descent

5.2 Loss analysis — the comparison, and why it does not work

Final-200-step statistics:

arm	mean	median	std	CV%	min	max	lag-1	chunk_first	chunk_last	ratio
sim10	0.001750	0.001673	0.000486	27.8	0.000903	0.003673	+0.036	0.001393	0.002918	2.09
mix10	0.001665	0.001634	0.000448	26.9	0.000847	0.003205	−0.053	0.000950	0.002507	2.64
mix20	0.002930	0.002868	0.000613	20.9	0.001684	0.005378	−0.075	0.001165	0.004492	3.86

Lag-1 autocorrelation is |r| < 0.08 on all three — the step-to-step noise is white, so 200-step means resolve differences of a few percent. The measurement is precise. It is the interpretation that fails.

Read naively, mix10 beats the baseline by 4.9% and mix20 loses by 67%, suggesting "replacing sim with ego helps slightly; adding more ego hurts a lot". Neither claim survives contact with the two corrections below.

Correction 1 — different norm stats

Each arm's loss is MSE in its own normalised units. Converting to a common physical scale, under the stated assumption that normalised error is distributed equally across the six real dimensions (so MSE_phys ≈ MSE_norm · mean(scale_d²), scale_d = (q99−q01)/2 in degrees):

arm	MSE_norm	mean(scale²) deg²	RMSE-equivalent	vs sim10	chunk_last RMSE-equiv	vs sim10
sim10	0.001750	4,261.8	2.73°	1.00×	3.53°	1.00×
mix10	0.001665	14,978.4	4.99°	1.83×	6.13°	1.74×
mix20	0.002930	14,175.7	6.45°	2.36×	7.98°	2.26×

The ranking inverts — and §5.5 confirms the inverted order is the correct one. But the magnitude here is badly wrong: measured holdout RMSE is 9.06 / 9.23 / 9.58°, i.e. ratios of 1.00 / 1.02 / 1.06, against the 1.00 / 1.83 / 2.36 predicted above. Two reasons, both worth recording: flow_loss is a flow-matching velocity regression loss, not an action error, so rescaling it by an action range is a category error; and the equal-error-across-dims assumption is false (per-dim MAE in §5.5 varies 2–7° across dimensions). Treat this table as a diagnostic that the raw-loss reading was unsafe, not as a prediction of holdout error. It got the direction right by luck as much as construction.

Correction 2 — different task diversity

arm	ego clips in batch pool	distinct task strings	final `chunk_last/chunk_first`
sim10	0	1	2.09
mix10	81	90	2.64
mix20	162	90	3.86

chunk_first is near-trivial under delta actions (the first action of a chunk is approximately "keep doing what you are doing"), so the ratio is carried by chunk_last — 50-step-ahead prediction. Within one repeated behaviour the future is barely ambiguous; across 90 tasks it is. The ratio rises monotonically with ego content, and mix20 — which has twice mix10's ego pool and therefore visits each ego clip half as often — shows it most strongly.

This means part of mix20's higher loss is a harder objective, not a worse policy. It pushes the opposite way from Correction 1. The two cannot be disentangled from training loss alone, which is precisely why §7-1 exists.

5.3 Learning-rate analysis

Same over-damped signature on all three arms:

Clipping never engaged. Max grad-norm 0.2548 (sim10, step 1), 0.1831 (mix10, step 10), 0.2536 (mix20, step 7) — all inside warmup, all a factor ~4–5 below the 1.0 clip. Steady-state final-200 grad-norm 0.0202 / 0.0178 / 0.0217, a factor ~50 below the clip.
Parameter norm moved +0.0097% / +0.0106% / +0.0109% over the run (1803.236 → ~1803.41–1803.43). For calibration, the 11.4k-step pi05_abcego_sd arm moved +0.054%. These checkpoints are pi05_base plus a very small correction.
No plateau. Every window improved; the final window still fell 7.7–8.7%.

Undertraining vs overtraining. All three are undertrained at 2,400 steps on the optimisation axis, identically so — which at least preserves the comparison. But note the tension the design already flagged: at 5–10 epochs of a nine-minute pool, memorisation is the other risk, and training loss cannot distinguish "still learning" from "memorising harder". The declining-grad-norm memorisation signature that flagged mix9010 in the YAM study is not present here — grad-norm declines smoothly and proportionally with loss on all three, which is the ordinary pattern, not the pathological one. Resolving this properly needs the holdout.

Stability. Under the rolling robust test (median + 6·MAD, 201-step window, steps ≥ 200): sim10 10 spikes worst 2.72×, mix10 10 spikes worst 2.48×, mix20 14 spikes worst 2.34×. All isolated single-step events that self-correct. For calibration, the YAM 50/50 arm had 1 and the pathological 90/10 console arm had 103 with excursions to 3.1×. These are benign, and notably the mix arms are not noisier than the baseline despite drawing from two sources.

5.4 W&B

Item	Value
Project / entity	`mm-so101-sim-vs-ego` / `kavinrajkr60-dsfsd`
Runs	`sim10` `uc9odsyk`, `mix10` `r5y0x9eb`, `mix20` `tunaarse`
Logged	`flow_loss`, `chunk_first/last`, `grad_norm`, `param_norm` @ 100-step interval; per-step in `train_metrics.log` (2,401 lines each, published)
Final / best checkpoint	2399 on each arm (final = lowest-training-loss; no validation metric existed to define "best")
Media / tables	None logged

5.5 Offline evaluation (2026-08-15) — the arms' capability evidence

Protocol. fd-studio eval/ suite, adapted to SO-101 (see §12). Sim holdout episodes 4, 14, 24, 34, 44 — 1,671 frames, stride 1, withheld from all three arms at training time. Open-loop: at each frame the policy's predicted 50-step action chunk is compared against the logged chunk. Errors are in degrees — RobotOutputs un-normalises before comparison, so all arms are compared in one physical unit despite their different norm stats. A pi05_base zero-shot control was run on the identical protocol. 2× H100, 2–3 concurrent jobs per pass.

Three metric families were run:

Offline proxies — action MSE/MAE, per-dim MAE, gripper accuracy.
Perturbation robustness — visual aug (ColorJitter both cameras + random resized crop on front only, per-episode deterministic seed) plus a paraphrased instruction; ground truth unchanged.
CI-MSE — DTW-aligned chunk error restricted to grasp/release windows.

5.5.1 Clean offline proxies

arm	frames	MSE (deg²)	RMSE (deg)	MAE (deg)	gripper acc (dim 5)
`pi05_base`	1,671	1634.9641	40.4347	25.3833	0.4325
sim10	1,671	82.0965	9.0607	4.3267	0.9072
mix10	1,671	85.1190	9.2260	4.5817	0.9037
mix20	1,671	91.8037	9.5814	4.6498	0.8970

sim10 wins on every aggregate among the trained arms. Gripper accuracy is flat across them (0.897–0.907), i.e. grasp/release timing is not what separates them — joint precision is. The base's 0.4325 is the reference point for what "no idea" looks like: below the 0.5 a coin would get, because its gripper output is not merely uninformed but anti-correlated with the truth on this task.

Reading the control. 19.9× on MSE, 5.9× on MAE. The gap between the best and worst trained arm (9.7 deg²) is 0.6% of the gap between the base and the best trained arm (1,552.9 deg²). Two consequences: the fine-tuning is unambiguously doing the work, and any narrative that treats the sim-vs-mix difference as a major result is out of proportion to what was measured.

5.5.2 Per-dimension MAE (degrees), and the norm-compression signature

arm	pan	lift	elbow	w_flex	w_roll	grip
`pi05_base`	10.031	53.017	54.149	8.755	12.842	13.506
sim10	2.146	6.597	6.889	3.458	3.365	3.504
mix10	2.258	7.065	7.235	3.533	3.836	3.563
mix20	2.134	7.139	7.155	3.637	4.015	3.818

Ratio vs sim10:

arm	pan	lift	elbow	w_flex	w_roll	grip
mix10	1.05	1.07	1.05	1.02	1.14	1.02
mix20	0.99	1.08	1.04	1.05	1.19	1.09

wrist_roll carries the mixture penalty, and it is the dimension the §3.2 preflight singled out as most compressed under blended quantile norm (sim occupying 0.17 of the band, a 5.98× range widening). The two least-compressed dimensions, shoulder_lift (1.08×) and elbow (1.24×), show the smallest relative penalties. shoulder_pan is the one imperfect fit — compressed 2.49× but essentially tied (1.05 / 0.99), so compression is a contributing mechanism rather than the whole story.

5.5.3 Perturbation robustness (paired)

Two full passes over the same 5 episodes rather than the suite's default 50/50 split: at n=5 a split leaves 2 vs 3 episodes per subset and the gap would be mostly episode-difficulty noise. Paired passes make this a within-episode comparison.

arm	clean MSE	perturbed MSE	ratio	absolute delta
`pi05_base`	1634.9641	1700.2081	1.040	+65.24
sim10	82.0965	81.3386	0.991	−0.76
mix10	85.1190	88.9575	1.045	+3.84
mix20	91.8037	95.3633	1.039	+3.56

Note the base's ratio (1.040) sits in the same band as the mixtures' — a good illustration of why perturbation ratios must never be read as robustness on their own. A model with no task competence degrades by the same relative amount as a good one; only the absolute delta (+65.24 vs +3.84) separates them.

sim10 is unaffected — the ratio below 1.0 is within noise, not a real improvement. Both mixtures degrade by ~4%. This runs against the usual expectation that more diverse training data buys robustness, but the perturbation here is mild (it mirrors the augmentation all three arms trained with) so this measures in-distribution stability, not OOD robustness.

Caveat on the text half. No Gemini key was available, so paraphrasing fell back to the rule-based path, which produced "Place it in the tray, after grab the cube" — semantically recognisable but ungrammatical, i.e. a harsher perturbation than an LLM paraphrase. It is identical across arms, so the comparison is fair, but the absolute ratios are pessimistic.

5.5.4 CI-MSE

320 critical timesteps (~19% of frames), DTW-aligned along the horizon axis, interpolation factor 5.

arm	mean	median	mean (1–99%)	per-interval means
`pi05_base`	746.465	619.667	730.649	692.29 / 955.44 / 544.98 / 793.15
sim10	16.845	11.674	16.276	17.74 / 19.33 / 23.65 / 6.66
mix10	16.564	13.793	16.030	19.21 / 13.03 / 27.15 / 6.86
mix20	19.982	14.905	19.248	33.87 / 13.31 / 26.30 / 6.44

The base is 44× worse than sim10 here against 19.9× on the whole-episode metric — the critical grasp/release windows discriminate roughly twice as sharply as aggregate error, which is the entire argument for the metric.

mix10 edges the mean by 1.7% while losing the median by 18% — sim10 is better on the typical critical frame but has a heavier tail (Q99 64.95). At n=320 from 5 episodes neither difference is resolvable; the honest reading is sim10 ≈ mix10 on critical intervals, mix20 clearly worse.

On the annotator. The suite's default critical-interval labeller is a Gemini VLM. With no key available, intervals were derived deterministically from the ground-truth gripper channel: binarise action[:,5] at its per-episode midpoint, take ±8 frames around each state flip, merge overlaps. For a single-task pick-and-place with a clean gripper signal this is arguably a better witness than a VLM — a grasp is definitionally the frame the jaws close — but it is a different metric from the published YAM CI-MSE numbers and must not be compared against them. All 5 episodes yielded exactly 4 transitions, which is the expected grasp/release structure.

6. Resource & Infrastructure Analysis

Metric	sim10	mix10	mix20
Wall clock (train stage)	1 h 05 m 21 s	1 h 02 m 00 s	1 h 01 m 22 s
tqdm elapsed (steps only)	1:01:07	59:35	58:57
Steady-state step time	~1.5 s/step	~1.4 s/step	~1.4 s/step
Norm-stats stage	2 min (computed)	skipped (precomputed)	skipped (precomputed)
GPU util	100% both cards	100% both cards	100% both cards
Host load avg (61-CPU quota)	~8	~40	~40

Total campaign: ~3 h 05 m of training plus ~~10 min of setup, on 2× H100 80GB (~~$12).

The loader is not the bottleneck, despite appearances. The mix arms run at 4–5× the host CPU load of the baseline — 324 short ego clips (median 96 frames) mean far more video opens and seeks per batch than sim's 50 long ones — but step time is the same or slightly better than the baseline's, and both GPUs stay pinned at 100%. num_workers=16 absorbs it within the 61-CPU cgroup quota.

sim10 being marginally the slowest despite the lightest decode is a checkpointing artefact: it ran under save_interval=300 and wrote 8 checkpoints of 13 GB, against 4 for the mix arms. Its tqdm rate shows transient excursions to 3.0 s/it around save boundaries.

Storage was the binding constraint, not compute. A π₀.₅ checkpoint here is 13 GB (6.7 GB params + 5.9 GB optimizer state). The config's original retention (save_interval=300, max_to_keep=4, keep_period=600) keeps six = 78 GB per arm; three arms would need 234 GB against a 150 GB disk, and mix20 would have died mid-run. See §9-A.

7. Failure & Root Cause Analysis

No run-level failures. Every issue is methodological.

Issue 1 — Training loss ranked the arms wrongly (RESOLVED by §5.5)

Field	Detail
Observation	Raw training loss ranks mix10 < sim10 < mix20. The holdout ranks sim10 < mix10 < mix20. Raw loss picked the wrong winner.
Evidence	§5.2 vs §5.5.1.
Root cause	Two compounding effects: per-arm norm stats rescale the objective, and lower task diversity deflates the loss (§5.2, Correction 2). openpi has no in-training validation metric — `holdout_fraction` and `exclude_episodes` only remove episodes from the training stream — so nothing in-run could have caught this.
Confidence	High.
Resolution	The 5-episode holdout was scored in degrees on 2026-08-15. Quote §5.5, never §5.2, for any cross-arm claim.

Issue 1b — The eval is thin, and there is no zero-shot control

Field	Detail
Observation	5 episodes / 1,671 frames of a single task; the sim10-vs-mix10 gap (3.7%) sits inside the stated ~10% tie band.
Evidence	§5.5.
Confidence	High (structural).
Consequence	The campaign can say "ego does not help at this scale". It cannot rank sim10 against mix10.
Resolved part	The `pi05_base` control HAS now been run: 1634.96 deg², 19.9× the baseline, gripper accuracy 0.4325. The fine-tuning is unambiguously responsible for the arms' competence, and — independently — the YAM sibling study's ~20× zero-shot figure reproduces here on a different embodiment.
Follow-up	If the sim10/mix10 distinction ever matters, enlarge the holdout, which costs sim training data.

Issue 2 — Checkpoint selection was traded away for disk

Field	Detail
Observation	Only step 2399 survives on each arm. The design explicitly wanted 6 checkpoints per arm so the holdout could pick the knee.
Evidence	§9-A; config comment now records the trade.
Root cause	13 GB × 6 × 3 arms = 234 GB against a 150 GB disk.
Confidence	High.
Mitigation taken	`save_interval=600, max_to_keep=1` — a rolling crash-resume point that is overwritten, with `train.py:334` always saving the final step regardless. Retention is a disk policy and does not touch the optimizer, so the arms remain matched even though sim10 ran under the old setting.
Residual risk	If the holdout later shows the arms peaked before 2,400, the remedy is a shorter re-run of all three, not a mid-curve checkpoint — none is retained. §5.3 suggests peaking early is unlikely (all three still descending 7.7–8.7% at the end), but it is no longer observable.

Issue 3 — Blended quantile norm compresses the sim signal on the mix arms

Field	Detail
Observation	Sim's q01–q99 occupies 0.17 of the normalised band on `wrist_roll`, 0.28 on `wrist_flex`, for both mix arms.
Evidence	§3.2, computed from the arms' own `norm_stats.json`.
Root cause	Ego's per-step deltas are much larger (median episode 96 vs 331 frames) and its workspace coverage much wider (89 tasks). Quantile norm is computed over the blend.
Confidence	High on the measurement; Medium on the downstream impact, which is unmeasured.
Follow-up	Not fixable by config — it is what blending means under shared norm. Two real options: (a) accept and report it, as done here; (b) per-source normalisation, which is a change to openpi's transform stack and would itself become the experimental variable. Do not do (b) inside this study.

Issue 4 — Ego language does not match the eval prompt

Field	Detail
Observation	89 ego task strings vs sim's 1; half of every mix batch carries a prompt never seen at eval. `datasets.py`'s comment asserting one task string across both pools is wrong.
Evidence	§3.3, `meta/tasks.jsonl`.
Confidence	High (structural fact).
Decision	Left as-is, deliberately. Forcing sim's sentence onto ego would pair one prompt with 89 behaviours.
Follow-up	Fix the comment in `datasets.py`. State in any write-up that the mix arms test diverse ego pick-place data, not same-task ego data.

Issue 5 — Documented run instructions did not match the runner

Field	Detail
Observation	`run_yam.sh` derives its checkpoint subdirectory by stripping known family prefixes; `mm_` matches none, so the arm name must be passed explicitly as `$2`.
Evidence	The config docstring anticipated this and documented the workaround; it was followed.
Confidence	High.
Follow-up	Cosmetic. Either add `mm_pi05_` to the prefix list or leave the docstring note.

8. Key Scientific Insights

Insight 1 — Training loss picked the wrong winner; the holdout corrected it. Measured, not argued. Raw final loss ranks mix10 best (0.001665 vs sim10's 0.001750, a 4.9% edge). The 5-episode holdout ranks sim10 best (82.10 vs 85.12 deg², a 3.7% edge the other way). This is the same failure mode recorded in the YAM study (pi05_abcego_sd had the lowest training loss of four arms and the 3rd-best holdout score, 43% worse than the winner), reproduced here in a cleaner setting where step counts are matched and composition is the only difference. Operational rule: never sort a mixture table by training loss. A mixture study without a shared holdout is not a study.

Insight 1b — But do not over-trust the correction either. The first-order norm rescaling in §5.2 recovered the right ordering and then overstated the gap by ~20× (1.83× predicted vs 1.02× measured on RMSE). Rescaling a flow-matching velocity loss by an action range is a category error that happens to preserve sign. A correction that gets direction right and magnitude wrong by an order of magnitude is a warning flag, not a metric.

Insight 1c — The compression confound is real and localised, which makes the headline result partly mechanical. wrist_roll was the dimension the preflight flagged (compressed to 0.17 of the normalised band) and it is where the mixtures lose most (1.14×, 1.19×), while the barely-compressed shoulder_lift and elbow lose least. So an unknown but non-trivial share of the measured mixture penalty is the cost of sharing a quantile normaliser with a wider-ranging pool, not evidence about ego data's content. A per-source-normalisation arm would separate the two — and that, not more ego data, is the experiment this result actually motivates.

Insight 1d — Measure the floor before arguing about the ceiling. The base control reframes the whole campaign: fine-tuning buys 19.9× and the choice between sim-only, half-ego and double-ego moves 0.6% of that. Both facts are worth reporting, but only one is worth spending another GPU-day on. The control cost 5 minutes and is the single highest information-per-second measurement in this report — it should be the FIRST eval run in any future campaign, not the last, because it calibrates how much the rest of the comparison can possibly matter.

Insight 2 — chunk_last/chunk_first is a free task-diversity readout and it is monotone in mixture composition. 2.09 (1 task) → 2.64 (90 tasks, 81 ego clips) → 3.86 (90 tasks, 162 ego clips). Compare 3.2× on YAM single-task teleop and 7.7× on the 261-task YAM mixture. The metric is already logged on every run at zero cost. Two uses: it fingerprints effective diversity without touching the data, and it warns that a low aggregate loss on low-diversity data is expected rather than impressive.

Insight 3 — Preflights pay off by finding the trap somewhere other than predicted. The design predicted a gripper-scale collapse (the pi05_ax91_mix9010 failure). The gripper was clean; wrist_roll and wrist_flex were not, at 0.17 and 0.28 of the normalised band. Had the preflight only checked the predicted channel it would have passed and the confound would have gone unrecorded. Check every dimension, not the one the last failure taught you to fear.

Insight 4 — Two sources cost host CPU, not GPU time. 4–5× the host load, identical step time, GPUs pinned at 100% throughout. The intuition that a mixture of many short clips will bottleneck the loader is wrong at this scale with num_workers=16 inside a 61-CPU quota. Budget CPU, not wall clock, when adding a fragmented source.

Insight 5 — At 2,400 steps a π₀.₅ LoRA fine-tune is an extremely light touch. +0.010% parameter drift, grad-norm ~50× below the clip, no plateau. This bounds catastrophic-forgetting risk and makes the arms cheap to re-run, but it also means most of each checkpoint's behaviour is inherited from pi05_base. Any capability claim must be measured against a pi05_base zero-shot control, which this campaign does not yet have.

Insight 6 — Matching the schedule across arms is worth the config plumbing. One shared _SCHEDULE object made step-count drift structurally impossible. The predecessor study ran four arms at four different step counts and consequently could not use training loss to rank them at all — a defect that cost it its headline comparison. Here the arms are genuinely matched on everything except composition; the remaining confounds (norm stats, task diversity) are properties of the data, not accidents of the runner.

9. Engineering & Process Takeaways

A — Cost out checkpoint retention before launch; it is the binding constraint, not GPU time. 13 GB per π₀.₅ checkpoint (6.7 params + 5.9 optimizer state). The config's original policy would have needed 234 GB on a 150 GB disk, and would have failed mid-third-arm — the worst possible time. Caught by projecting from the first arm's step-300 checkpoint rather than by hitting ENOSPC. The fix (save_interval=600, max_to_keep=1, keep_period=None) exploits train.py:334, which always saves the final step on top of the interval, so a rolling overwritten checkpoint still yields crash-resume and a guaranteed final artifact. Retention does not touch the optimizer, so arms stay matched across a mid-campaign change.

B — Precompute norm stats for later arms while an earlier one trains. compute_norm_stats.py --skip-videos is CPU-only and took ~2 min per arm against a 61-CPU quota, invisible to the running GPU job. Both mix arms then skipped stage [1/3] entirely. More valuable than the time saved: it validated that the ego pool loads, that both mixtures resolve, and that draws sum to the batch size — before those arms were on the critical path.

C — Fingerprint norm stats and check them on load. norm_stats_fingerprint.json records sources, roots, draws and holdout fractions, and is verified at load. Three overlapping mixtures over two pools is exactly the setup where a cp -r from a neighbouring arm silently normalises against the wrong distribution — the pi05_ax91_mix9010 failure. Here it raises instead.

D — Guard destructive helpers on process state, not on session state. The first version of the checkpoint-prune helper guarded on tmux session contents and did not stop it running against a live training job. Rewritten to pgrep -f "train.py <config>" with a dry-run default, and both paths tested against the running arm before use. A prune script that can race a live writer is a data-loss incident waiting for a distraction.

E — Publish the per-step log with the checkpoints. train_metrics.log (2,401 lines per arm) is what makes §5 of this report independently checkable after the box is released. Everything in §5.2–5.3 — windowed means, autocorrelation, spike counts, param drift — was computed from those three files alone. Zero marginal cost; standing policy.

F — Cap thread pools before fanning out; nproc lies inside a cgroup. The first eval fan-out ran 6 processes, each of which sized its JAX/pyav/torch pools off nproc (128) rather than the cgroup quota (61) — 711 threads per process, ~4,300 total. Load average hit 272, the box stopped answering SSH within 60 s, GPU1 sat at 0%, and the three perturbed jobs did not complete a single episode in 8 minutes. Setting OMP_NUM_THREADS=6 (plus MKL/OPENBLAS/NUMEXPR) and re-running the same three jobs took 170–178 s each against 820–957 s for the thread-thrashed clean pass — 5× faster while doing strictly more work per frame (the perturbed pass adds torch ColorJitter on every camera). Process-level fan-out is the right shape for this suite, as its own comments say; it just has to be paired with per-process thread caps or the processes fight each other for a quota none of them can see.

G — Chain dependent runs with a correctness gate, not a timer. The mix20 launcher waited on train.py exit, then verified mix10 had actually reached step 2399 before starting, and aborted otherwise. Cost: 20 lines. It converted "watch a terminal for an hour" into an unattended handoff without risking a second run stacked on top of a broken first.

10. Prioritised Recommendations

High priority

Recommendation	Source	Expected benefit	Risk	Effort	Confidence
Run a per-source-normalisation arm	I-1c, §5.5.2	Separates "ego content does not help" from "sharing a quantile normaliser with ego costs wrist_roll accuracy" — currently confounded, and the wrist_roll signature says the second term is non-zero	Becomes its own experimental variable; needs an openpi transform change	~1 day + 1 GPU-h	Medium-High
Do not adopt ego co-training for this client at this scale	§5.5	The baseline wins on clean error, robustness and CI-MSE median; ego costs data-prep effort for no measured gain	The result is budget-specific — revisit at higher step counts	None (policy)	Medium-High

Medium priority

Recommendation	Source	Expected benefit	Risk	Effort	Confidence
Log per-source loss on mixture arms	I-1	Separates "ego helps sim" from "ego is just harder"; the single most valuable missing metric	None	~4 h	High
Fix the `TASK` comment in `configs/mm/datasets.py`	§7-4	Prevents the next reader inheriting a false premise	None	Trivial	High
Extend all three arms to ~7,200 steps and re-eval	§5.3	Every arm was still descending 7.7–8.7%; the ranking may not be converged	~~3× cost (~~$36); must re-run all three to stay matched	~9 GPU-h	Medium-High

Low priority

Recommendation	Source	Expected benefit	Risk	Effort	Confidence
Add `mm_pi05_` to `run_yam.sh`'s prefix-stripping list	§7-5	Removes a documented footgun	None	Trivial	High
Per-source normalisation experiment	§7-3	Would remove the compression confound structurally	Becomes its own experimental variable; do not mix into this study	~2 days	Low-Medium

11. Proposed Next Experiments

A — Per-source normalisation arm (Priority: high; ~1 day + 1 GPU-h) Objective: Decide whether the mixture penalty is about ego's content or about shared quantile norm. Hypothesis: Normalising each source against its own quantiles removes most of the wrist_roll gap (1.14–1.19× → ~1.0×) and moves mix10 level with or ahead of sim10. Why it matters: If true, "ego does not help" is wrong — the correct statement becomes "ego helps only if you do not let it dominate the normaliser", which is a recipe change rather than a dead end.

(The pi05_base zero-shot control that used to head this list was run on 2026-08-15; see §5.5.1.)

B — Per-source loss attribution (Priority: high; ~1 GPU-h) Log the sim-half and ego-half loss separately on the mixture arms. The sim half can then be compared directly against sim10 under the caveat of differing norm stats — and that comparison is what actually answers "does adding ego make the sim frames easier or harder to fit".

C — Step-extension at matched budget (Priority: medium; ~9 GPU-h) Objective: Test whether the ranking is converged. Hypothesis: At ~7,200 steps (3× current, ~15–30 epochs) the arms separate more clearly, and the high-epoch baseline begins to show memorisation on the holdout while the ego-augmented arms do not. Variables: num_train_steps and proportional warmup only; all three arms together. Note: This is also the cleanest test of the mixture's regularisation value, which at 2,400 steps is likely invisible.

D — Nested ego subsets (Priority: low) mix10's 81 clips are not a subset of mix20's 162 (independent rng.choice draws). Pre-building nested subsets with their own roots would make the "how much ego" axis strictly monotone. Only worth doing if A shows a real mix10-vs-mix20 difference.

12. Reproducibility

Code

configs/mm/so101/sim_vs_ego.py and configs/mm/datasets.py at rev f19dd1c, with the checkpoint-retention change described in §9-A applied to all three arms (save_interval=600, max_to_keep=1, keep_period=None).
Arms registered as mm_pi05_sim10, mm_pi05_mix10, mm_pi05_mix20.

Data

Sim: makermods/maniskill_50ep_so101_blue_cube_orange_tray_20260812_131142 → converted to LeRobot v2.1, 50 eps / 16,658 frames, at $MM_SIM_ROOT (default /workspace/mm/sim_v21)
Ego: EgoDex retargeted via DT-pipeline stage 6 run F, LeRobot v2.1, 324 clips / 36,442 frames, at $MM_EGO_ROOT (default /workspace/mm/ego_v21)
Both read as-is; version asserted at launch, no migration.

Commands

bash

# norm stats (mix arms precomputed while the baseline trained)
uv run scripts/compute_norm_stats.py --config-name mm_pi05_mix10 --max-frames 200000 --skip-videos
uv run scripts/compute_norm_stats.py --config-name mm_pi05_mix20 --max-frames 200000 --skip-videos

# training — the arm name must be passed explicitly (run_yam.sh does not strip `mm_`)
WANDB_PROJECT=mm-so101-sim-vs-ego ./vast_run/run_yam.sh mm_pi05_sim10 sim10
WANDB_PROJECT=mm-so101-sim-vs-ego ./vast_run/run_yam.sh mm_pi05_mix10 mix10
WANDB_PROJECT=mm-so101-sim-vs-ego ./vast_run/run_yam.sh mm_pi05_mix20 mix20

Environment. Python 3.11, JAX 0.5.3, LeRobot @ 0cf8648, orbax 0.11.13, bf16 on 2× H100 80GB (driver 555.58.02); seed 42; full bitwise determinism not guaranteed (XLA autotuning unpinned).

Evaluation

The fd-studio suite (eval/offline_eval, eval/perturbation, eval/ci_mse) at rev d4bf6a8, with three SO-101 adaptations:

eval_all.py gained a --cameras <slot>=<dataset_cam> flag. It previously hardcoded YAM's three cameras and their bespoke observation/<slot>_image obs keys; SO-101 needs two cameras on the observation/image/<slot> convention that RobotInputs actually reads. The YAM default is unchanged, so existing callers are unaffected.
Critical intervals were derived from the GT gripper rather than a Gemini VLM (§5.5.4). Note the trap: validate.py's load_critical_intervals treats the JSON's "episode" field as a positional index into the prediction file (0..n−1), not a dataset episode id — keying it 4/14/24/34/44 silently drops four of five and misaligns the fifth. The generator emits positional indices and carries the dataset id alongside as dataset_episode.
Paraphrasing used the offline rule-based fallback (no Gemini key).
A fourth config, mm_pi05_base_control, was registered to make the zero-shot base loadable. It is not an experimental arm and is never trained. create_trained_policy restores params by exact structural match, and pi05_base ships no LoRA matrices, so the trained arms' gemma_2b_lora trunk rejects it outright (at keypath ['PaliGemma']['llm']['layers']['mlp']: expected 6 children, got 2). The control is identical to mm_pi05_sim10 except paligemma_variant="gemma_2b". At initialisation LoRA is a functional no-op — B is zero — but the parameter tree still differs, and restore is structural, so the non-LoRA variant is required. Its checkpoint dir was staged as a symlink to the base params plus a copy of sim10's assets/mm_sim10, since the base has no norm stats of its own; absolute un-normalisation is therefore arbitrary but lands its outputs on the scale of the arm it is compared against.

bash

# offline proxies + perturbation, per arm and pass (3 concurrent, thread-capped — see §9-F)
OMP_NUM_THREADS=6 CUDA_VISIBLE_DEVICES=0 uv run python eval/offline_eval/scripts/eval_all.py \
  --policy pi05 --config mm_pi05_sim10 --ckpt checkpoints/mm_pi05_sim10/sim10/2399 \
  --root /workspace/mm/sim_v21 --val_repo <sim repo id> \
  --cameras "base_0_rgb=front,left_wrist_0_rgb=wrist" --gripper_dims 5 \
  --episodes 4,14,24,34,44 --stride 1 --horizon 50 --fps 30 \
  --perturbation eval_mm/perturbation_clean.json \
  --dump-predictions eval_mm/predictions_sim10.h5 --out eval_mm/proxies_sim10_clean.json

# CI-MSE
python eval/ci_mse/val_metrics/validate.py eval_mm/predictions_sim10.h5 \
  --intervals eval_mm/critical_intervals.json --output eval_mm/ci_sim10.hdf5

Artifacts in this repository

sim10/2399/, mix10/2399/, mix20/2399/ — full checkpoints (params/, train_state/, assets/), 13 GB each
<arm>/2399/assets/mm_<arm>/norm_stats.json — the norm stats each arm trained against, plus the fingerprint
metrics/train_metrics_<arm>.log — per-step metrics, 2,401 lines per arm
logs/ — the complete training logs from the box: per-arm pipeline.log, train.log, norm_stats.log, plus prep_norm.log, chain_mix20.log, uv_sync.log
eval/ — everything behind §5.5: proxies_<arm>_{clean,perturbed}.json, predictions_<arm>.h5 ([1671, 50, 6] chunks + targets + proprio + episode ids), ci_<arm>.{txt,hdf5}, critical_intervals.json, both perturbation manifests, the adapted scripts, and the per-job logs
README.md — this report

W&B. Project mm-so101-sim-vs-ego, entity kavinrajkr60-dsfsd; runs uc9odsyk (sim10), r5y0x9eb (mix10), tunaarse (mix20).

Appendix A — Loss trajectory (per 200 steps)

`mm_pi05_sim10`

Window	flow_loss	Δ	chunk_first	chunk_last	ratio	grad_norm	param_norm
0–199	0.01727	—	0.01504	0.02796	1.86	0.07077	1803.238
200–399	0.00827	−52.1%	0.00414	0.01403	3.39	0.03644	1803.257
400–599	0.00689	−16.6%	0.00361	0.01158	3.21	0.03273	1803.288
600–799	0.00563	−18.3%	0.00307	0.00932	3.04	0.03057	1803.319
800–999	0.00454	−19.3%	0.00255	0.00750	2.94	0.02784	1803.346
1000–1199	0.00389	−14.3%	0.00242	0.00665	2.75	0.02650	1803.369
1200–1399	0.00330	−15.3%	0.00207	0.00557	2.70	0.02579	1803.385
1400–1599	0.00286	−13.1%	0.00190	0.00475	2.50	0.02416	1803.397
1600–1799	0.00237	−17.3%	0.00161	0.00394	2.45	0.02230	1803.403
1800–1999	0.00215	−9.2%	0.00160	0.00371	2.32	0.02285	1803.407
2000–2199	0.00190	−11.7%	0.00150	0.00325	2.17	0.02165	1803.410
2200–2399	0.00175	−7.7%	0.00139	0.00292	2.09	0.02016	1803.411

`mm_pi05_mix10`

Window	flow_loss	Δ	chunk_first	chunk_last	ratio	grad_norm	param_norm
0–199	0.01759	—	0.01258	0.02684	2.13	0.06067	1803.240
200–399	0.00868	−50.6%	0.00273	0.01361	4.98	0.03747	1803.264
400–599	0.00647	−25.4%	0.00240	0.00969	4.04	0.03233	1803.302
600–799	0.00522	−19.4%	0.00196	0.00795	4.06	0.02833	1803.336
800–999	0.00416	−20.3%	0.00170	0.00616	3.62	0.02565	1803.364
1000–1199	0.00361	−13.1%	0.00158	0.00556	3.52	0.02397	1803.387
1200–1399	0.00304	−15.8%	0.00144	0.00452	3.15	0.02254	1803.402
1400–1599	0.00268	−12.0%	0.00128	0.00404	3.16	0.02176	1803.413
1600–1799	0.00223	−16.8%	0.00107	0.00345	3.22	0.01987	1803.420
1800–1999	0.00210	−5.8%	0.00122	0.00321	2.64	0.02078	1803.424
2000–2199	0.00182	−13.1%	0.00102	0.00277	2.72	0.01917	1803.426
2200–2399	0.00167	−8.7%	0.00095	0.00251	2.64	0.01777	1803.427

`mm_pi05_mix20`

Window	flow_loss	Δ	chunk_first	chunk_last	ratio	grad_norm	param_norm
0–199	0.01853	—	0.01224	0.02926	2.39	0.06024	1803.239
200–399	0.01100	−40.6%	0.00267	0.01798	6.74	0.03792	1803.263
400–599	0.00887	−19.4%	0.00244	0.01410	5.77	0.03692	1803.302
600–799	0.00761	−14.2%	0.00212	0.01195	5.64	0.03400	1803.337
800–999	0.00643	−15.5%	0.00180	0.01010	5.61	0.03186	1803.367
1000–1199	0.00554	−13.8%	0.00172	0.00860	5.00	0.02915	1803.390
1200–1399	0.00486	−12.2%	0.00155	0.00730	4.72	0.02690	1803.407
1400–1599	0.00437	−10.2%	0.00160	0.00667	4.16	0.02606	1803.418
1600–1799	0.00377	−13.6%	0.00127	0.00572	4.52	0.02456	1803.425
1800–1999	0.00353	−6.5%	0.00135	0.00536	3.97	0.02447	1803.429
2000–2199	0.00321	−9.0%	0.00124	0.00483	3.89	0.02312	1803.431
2200–2399	0.00293	−8.7%	0.00116	0.00449	3.86	0.02173	1803.432

Appendix B — Warmup detail (20-step block means)

Block	sim10 loss	sim10 gn	mix10 loss	mix10 gn	mix20 loss	mix20 gn
0–19	0.04395	0.18383	0.03635	0.13610	0.03723	0.14444
40–59	0.02020	0.06812	0.02050	0.06141	0.02121	0.06472
100–119	0.01177	0.05404	0.01341	0.04771	0.01382	0.04391
160–179	0.00988	0.04246	0.01206	0.04730	0.01323	0.03866
240–259	0.00869	0.03765	0.00973	0.04187	0.01099	0.03574
249–268 (warmup boundary)	0.00873	0.03916	0.00990	0.04439	0.01155	0.03799
260–279	0.00876	0.03824	0.00976	0.04103	0.01258	0.04108
400–419	0.00790	0.03441	0.00735	0.03420	0.00979	0.03877
600–619	0.00625	0.03139	0.00600	0.03112	0.00813	0.03432

No discontinuity at step 250 on any arm — warmup length is adequate.

Appendix C — Norm-stats comparison (action half-ranges, degrees)

scale_d = (q99 − q01) / 2, over delta actions for dims 0–4 and absolute for the gripper.

dim	sim10	mix10	mix20	mix10/sim10	mix20/sim10
shoulder_pan	28.6	71.1	61.5	2.49×	2.15×
shoulder_lift	109.1	117.6	115.7	1.08×	1.06×
elbow_flex	103.7	129.1	123.0	1.24×	1.19×
wrist_flex	19.3	68.8	69.2	3.57×	3.59×
wrist_roll	37.0	220.9	217.2	5.98×	5.88×
gripper	19.1	28.5	28.0	1.49×	1.46×

Appendix D — Stability

arm	max grad_norm	@ step	final-200 grad_norm	param_norm drift	spikes (>6·MAD, step ≥ 200)	worst spike
sim10	0.2548	1	0.02016	+0.0097%	10	2.72×
mix10	0.1831	10	0.01777	+0.0106%	10	2.48×
mix20	0.2536	7	0.02173	+0.0109%	14	2.34×

Clip threshold 1.0 — never approached on any arm.

Appendix E — Evaluation run detail

Holdout: sim episodes 4, 14, 24, 34, 44 — 1,671 frames, stride 1, verified withheld from all three arms at load time (§3.4).

arm / pass	frames	wall clock	threads/proc	notes
sim10 clean	1,671	819.9 s	~711	thread-thrashed fan-out (6 procs)
mix10 clean	1,671	952.5 s	~711	"
mix20 clean	1,671	956.8 s	~711	"
base clean	1,671	~144 s	~538	`OMP_NUM_THREADS=6`, 2 procs
base perturbed	1,671	~150 s	~538	"
sim10 perturbed	1,671	177.7 s	~538	`OMP_NUM_THREADS=6`, 3 procs
mix10 perturbed	1,671	170.2 s	~538	"
mix20 perturbed	1,671	174.5 s	~538	"

The perturbed pass does strictly more work per frame (torch ColorJitter on both cameras) and ran 5× faster. The difference is thread caps, not workload — see §9-F.

Critical intervals per episode (±8 frames around each GT gripper state flip, merged):

dataset episode	frames	gripper range (deg)	threshold	transitions	intervals	critical frames
4	378	1.7 – 32.5	17.1	4	4	64 (16.9%)
14	332	1.1 – 33.1	17.1	4	4	64 (19.3%)
24	309	1.4 – 32.9	17.2	4	4	64 (20.7%)
34	299	0.8 – 29.7	15.3	4	4	64 (21.4%)
44	353	1.5 – 31.5	16.5	4	4	64 (18.1%)

Every episode has exactly 4 gripper transitions, the expected grasp/release structure for a single pick-and-place, which is a useful sanity check on the derived annotator.

angkul07/pi05-mm-so101-sim-vs-ego

⚡ Round 2 (2026-08-17): 5,000 steps + capability-gap-filtered ego — the ranking flips