longnav-objectnav-flow-nopose-cotrain-2p5hz-rtc
The real-time-chunking (RTC) variant of
longnav-objectnav-flow-nopose-cotrain-2p5hz
(cotrain-v3): the same 2.5 Hz ObjectNav/PointNav policy, fine-tuned with
training-time action conditioning (
arXiv:2512.05964)
so each action chunk can be generated
while the robot executes committed
actions from the previous chunk — masking up to 400 ms of inference latency
without shifting the decision clock. Training mixture (unchanged from v3):
human MP3D ObjectNav with and without pose injection plus generated HM3D
PointNav at 1:1:2. Headline:
no regression at d = 0 (0.683 oracle success
vs v3's 0.663 on the pinned
sample101 set) and
0.624 oracle success with
200 ms of masked latency (d = 5).
What this is
A Qwen3-VL-2B backbone with a LoRA adapter (r=128, alpha=256) and a
flow-matching action head that emits a timed trajectory — a chunk of 20
cumulative relative planar poses at 0.04 s spacing, PID-tracked on a holonomic
base. Observations at 2.5 Hz; each decision executes 10 of the 20 poses.
The RTC addition: the flow head accepts a committed-action prefix — the
d per-tick body-frame differentials the robot will execute while this chunk
is being generated. Prefix rows enter the denoising loop clean at per-tick
flow time 0 and are pinned through integration; the generated postfix is
consistent with them by construction. Rows [0, d) of a conditioned chunk
equal the commitment exactly. d = 0 reproduces the unconditioned model bit
for bit. Whether a checkpoint understands a prefix is carried in
turn_vector_head_config.json → fm_config.rtc_delay_max (10 here); the
eval harness refuses to pass a prefix to a checkpoint without it.
Training
Resumed from cotrain-v3's step 9000 (75% of its 12,000-step schedule) and
trained the final 3,000 steps with prefix conditioning — the recipe of
arXiv:2512.05964, compute- and schedule-matched to the v3 release (the same
cosine continues; optimizer, RNG and data order restored). Per example the
commitment length is drawn d ~ exp(0.8^d) over [0, 10] (P(0) ≈ 0.22,
E[d] ≈ 3, P(d ≥ 5) ≈ 26%): gentler than the paper's halving, chosen to keep
early-row supervision coverage (~95%) while still training the 200 ms regime.
Loss on the non-committed rows only. No new parameters relative to v3.
Results (sample101, HM3D ObjectNav val, n=101, oracle-stop convention)
Same pinned 101 episodes and per-episode seeds for every row; scenes are held
out from all training components (ObjectNav demos are MP3D; PointNav demos are
HM3D train). d is the assumed inference delay in 40 ms control ticks: the
first d ticks of each decision interval execute the previous chunk's
committed rows and the model is conditioned on them. oSPL is the corrected
oracle SPL (start_m / max(start_m, path_at_min_m) if oracle success else 0);
the harness's reported oracle_spl column is known-broken.
| policy | oracle success | oSPL (corrected) | npwpl | success |
|---|
| cotrain-v3 (baseline, d=0 only) | 0.663 | 0.348 | 0.420 | — |
| this, d = 0 | 0.683 | 0.336 | 0.409 | 0.584 |
| this, d = 5 (200 ms masked) | 0.624 | 0.295 | 0.372 | 0.495 |
d = 0 vs v3 is within n=101 noise in both directions — the conditioning
fine-tune did not cost the base policy. d = 5 vs d = 0, paired on episodes:
10 improved / 16 worsened / 75 unchanged (sign test p ≈ 0.16). Note d = 0 is
the zero-latency ideal, not the deployment alternative; a naive-async
baseline (stale chunks, no conditioning) has not been measured here.
Training mixture
| component | ratio | rows | what it is |
|---|
| objectnav_nopose | 1 | 39,061 | human MP3D ObjectNav, no pose markers |
| objectnav_pose | 1 | 39,061 | the same episodes with pose injection |
| pointnav | 2 | 7,971 | generated HM3D-train PointNav, stratified long goals |
Sampled at ratios (not concatenated); realized split 0.249 / 0.253 / 0.497.
Steps 0–9000 are literally cotrain-v3's; steps 9000–12000 add conditioning.
Files
| file | why |
|---|
adapter/ | LoRA (adapter_config.json + adapter_model.safetensors) |
turn_vector_head.pt | action head (flow velocity field + readout) + pose encoder |
turn_vector_head_config.json | head config, incl. fm_config.rtc_* (the prefix-capability flag) and the <pose> modality spec |
trainer_state.json | the training curve; global_step 12000 identifies the checkpoint |
| tokenizer / preprocessor files | as in the base repo |
Optimizer and RNG state are deliberately excluded: this is an inference
checkpoint, not a resume point.
What this depends on that is NOT public yet
Everything the v3 card lists (--no-pose-injection, the mixed-modality
collator fix), plus the RTC branches of both project repos: the flow
head's prefix-conditioned decode (longnav.utils.flow_matching_head /
vector_rollout.step(prefix=...)) and the eval harness's latency-masking
scheduler (objectnav_eval.schedule, the --delay-source family). The
commands below will not run without them.
Running / evaluating it
Zero-delay (comparable to every prior sample101 number — note there are NO
delay flags; this path is bit-identical to the pre-RTC harness):
1python scripts/eval_objectnav_policy.py \
2 --episodes <hm3d objectnav val split> \
3 --scene-root <scene_datasets> \
4 --output-dir <out> \
5 --episode-ids "$(cat sample101_ids.txt)" \
6 --ckpt <this checkpoint> \
7 --policy-python <env with transformers> \
8 --policy-sys-path <longnav src, rtc branch> \
9 --policy-backend flow_rollout --policy-seed 1000 \
10 --max-steps 175 --dt 0.04 --gap 10 \
11 --success-distance 1.0 --distance-to VIEW_POINTS \
12 --seed 0 --navmesh dataset --fresh-sim-per-episode \
13 --auto-stop --auto-stop-delay 15 \
14 --no-pose-injection \
15 --record-video --pid-preset baseline --keep-tracking-series
With masked latency (here 5 ticks = 200 ms; any d <= fm_config.rtc_delay_max
and d <= H - gap = 10):
1 ... same flags ... \
2 --delay-source fixed --delay 5
The delay is always the assumed delay — a scheduling quantity, never a
measured latency — and each per-episode record carries the realized
(d_assumed, d_actual, tossed) trace in its shard-level output. Design docs:
docs/RTC_TRAINING.md (spatial_training) and docs/LATENCY_MASKING.md
(habitat_physical_nav).