A closed-loop driving policy that reasons through unfamiliar long-tail scenarios without fine-tuning on any AV dataset.

A minimal-shot autonomy prototype for Waymo's WOD-E2E long-tail benchmark, built entirely from scratch. Two independent tracks:

Spotlight Reflex (simulation track): A closed-loop policy built on six geometric world-state scalars — obstacle pressure, route blockage, lateral clearances — and nine maneuver candidates scored against trust-region references. The decision is a function of geometry only, not object identity. Runs in a custom 2D simulator and in Waymo's sensor-realistic AlpaSim with the same code.

WOD-E2E harness (benchmark track): A trajectory selector trained on Waymo's 479 preference-labeled validation frames using segment-grouped 5-fold cross-validation. Candidates come from kinematic physics, ridge regression, and temporal ego-history models; the selector picks among them using ~137 features plus visual embeddings (NVIDIA Cosmos, 64d) fed to a GPU MLP direct policy. Best result: 7.845 RFS.

The two tracks are deliberately isolated — simulator metrics are never used for WOD model selection.

The custom simulator is the controlled lab. I built it to make long-tail AV scenes cheap, randomized, and exactly measurable: the simulator knows the route, obstacles, actor motion, clearance, progress, and collision state at every step. That makes it possible to test whether a policy is using route/occupancy geometry rather than memorizing labels or passing one hand-picked demo.

AlpaSim is the transfer test. After the mechanism worked internally, I kept the same token-policy API and ran it through a second execution stack built from WOD-E2E front-camera scenes. The AlpaSim adapter reconstructs the same proxy geometry from camera, route, and ego-dynamics inputs, then calls the same policies: raw DAgger, clamped DAgger, hard-veto hybrid, source-decayed DAgger, and Spotlight Reflex.

That split is intentional: the simulator answers why a policy decision is made under known geometry; AlpaSim answers what survives when the same decision interface is forced through sensor-realistic proxy-state reconstruction.

Recent systems such as LaST-VLA pursue a large-model route: replace textual chain-of-thought with physically grounded latent spatio-temporal reasoning, then train for trajectory generation at scale. This repo does not try to reproduce that stack.

The complementary contribution here is narrower and more inspectable: keep the action interface explicit, run the same token API in both the custom simulator and AlpaSim, and measure transfer per axis — collision, offroad, lane adherence, and progress — instead of hiding everything inside one aggregate score. The advantage is not only auditability; it is failure localization. The experiments identify when grounded selection transfers, when learned token selection breaks, and which metric axis caused the break.

COMPASS: 9.137 / 10 (700 ranked runs) · 95% CI collision rate [0.0, 0.0053]

This is the primary Spotlight Reflex simulator benchmark: 350 deterministic rollouts over seeds 1–10. The gauntlet row here is the 60-run primary-suite slice.

This is a separate larger matched comparison against the baseline policy (420 scenarios, seeds 1–80), not the same sample as the 60-run gauntlet row above.

Oracle gap: 1.407 RFS for the best tracked 5-fold selector. The right candidate exists in the pool; the remaining problem is selector grounding, not candidate generation. Grounding ablations are summarized in docs/corl2027/results/wod_grounding_ablation_table.md.

The external result is intentionally reported per axis, not as a single winner: clamping improves route geometry, axis-constrained clamping further improves offroad and distance-to-ground-truth, hybrid veto improves progress/offroad behavior, and source decay improves wrong-lane rate. Full paired tests are in docs/corl2027/results/alpasim_matrix10_with_axis_analysis.md. The proxy-visibility audit is the key interpretation check: the axis-constrained and hard-veto AlpaSim runs both saw 0/1990 structured-hazard frames, so collision failures are upstream of token ranking and require actor-aware proxy reconstruction rather than a stronger veto threshold (audit). The repo now includes the oracle-proxy probe for that diagnosis: scripts/build_alpasim_oracle_actor_proxy.py extracts privileged world-frame actor poses from AlpaSim ASL logs, and the token_dagger_iter2_axis_constrained_oracle_actor_clamped preset injects them into the same learned selector for a controlled oracle-proxy ablation (commands). Do not conflate the actor-aware numbers: the 0.70 -> 0.40 collision reduction is a score-cutoff world-frame oracle diagnostic, and the historical 0.70 -> 0.60 raw result was a 10-clip diagnostic. The canonical matched raw diagnostic is the 30/30 world-frame oracle rerun in artifacts/alpasim_actor_blindness_30scene_raw_analysis.md: collision stays 0.60 -> 0.60 with better=0, worse=0, offroad worsens 0.207 -> 0.310, wrong-lane improves 0.241 -> 0.172, and progress improves 0.170 -> 0.184. That makes the actor-complete probe informative but still inconclusive for the collision-causality probe. The follow-up collision-surface audit (Markdown, JSON) shows why: the same 18/30 clips collide under baseline and oracle. At first impact, baseline logs show maintain as top candidate and 0/18 structured-hazard frames, while oracle logs show proxy hits, actors, and non-maintain selected actions on 17/17 logged collision clips. The remaining collision surface therefore points beyond selector ranking alone, toward candidate-set coverage or controller/traffic execution. The selector-side candidate counterfactual (Markdown, JSON) injects the same world-frame actor proxy into the baseline timeline and reconstructs candidate feasibility without rerunning AlpaSim. It finds 0/18 actor-axis-safe first-impact frames, only 5/25 missed actor-axis-safe actionable frames, and 12/18 scenes where actor-axis-safe selected tokens still collide later. That narrows selector miss to a minority explanation; the remaining proof step is true candidate/controller replay. The selector-free direct grid replay now covers that next step (cost-ranked, max-clearance): both runs improve only 2/18 collision clips (1.000 -> 0.889, McNemar p=0.5000). The max-clearance replay also improves distance-to-ground-truth (1.41 m -> 0.74 m), but it still collides on 16/18 clips, so the residual failure is not explained by the learned selector or direct-grid cost ranking alone. The partial bridge analysis (Markdown, JSON) connects this external failure to the controlled proxy perturbation: at first impact, baseline adapter logs contain 0/18 structured hazards while the world-frame oracle proxy has positive actor hazards in 18/18; actionable collision windows have median 22 oracle hazards versus 4 in terminal-matched non-collision controls. This is a hazard-dropout bridge in the structured-hazard interface, not yet a full residual-density bridge for route, heading, lane, or feature-noise errors.

This table evaluates learned-policy transfer variants. It should be read as the transfer-diagnostic extension rather than a replacement for the original Spotlight readiness check (collision_at_fault: 0.0, dist_to_gt: 0.42 m).

Two primary references — each focused on one track with no repeated content:

If you are browsing the stable main branch, switch to CoRL-2027 for the more technical experiment logs, AlpaSim transfer harness, and ongoing paper work. On this branch specifically, start here:

Available clusters: construction · intersection · pedestrian · cyclist · cut-in · foreign object debris · special vehicle · spotlight · others

For AlpaSim specifically, keep the checkout path explicit:

bootstrap_alpasim_checkout.sh clones the pinned upstream AlpaSim checkout and builds a minimal $ALPASIM_ROOT/.venv with uv. The launcher then invokes $ALPASIM_ROOT/.venv/bin/python and $ALPASIM_ROOT/.venv/bin/alpasim_wizard directly, so it does not depend on AlpaSim's full workspace lockfile. During setup, repo-tracked AlpaSim patches in third_party/alpasim_overrides/ are applied first; this includes the route-waypoint bridge required by the actor-axis proxy.

Published paper checkpoints live in the public Hugging Face repo amtellezfernandez/minimal-shot-av-corl2027-checkpoints. The tracked manifest at artifacts/models_manifest.json defines the exact files, checksums, and local target paths used by scripts/fetch_checkpoints.py.

Important: ALPASIM_ROOT must point at a real nested AlpaSim git checkout, not a copied folder. If you copied alpasim/ between machines and it lost its .git metadata, the wizard will resolve configs against the wrong repo root and fail. ./scripts/bootstrap_alpasim_checkout.sh now detects that case, moves the invalid tree aside, and reclones the pinned upstream checkout automatically.

Before any real launch, ./scripts/check_alpasim_readiness.py verifies the four failure points that caused most of the Spark bring-up issues: checkout shape, Docker access, presence of the local alpasim-base:0.66.0 image, and gated USDZ scene artifacts for the requested preset.

For the CoRL evidence loop, the repo now exposes three explicit gates:

Switch --mode print to --mode both --continue-on-error on an x86_64 AlpaSim host after readiness passes. The Best-Paper-level blocker is explicit in artifacts/corl_evidence_strength_audit.md: the current local artifacts support a strong diagnostic paper, but not yet a positive external method claim.

For a one-command repo audit on this branch, use:

That wrapper:

Current limitation: real AlpaSim rollouts are only supported on x86_64 hosts. The NVIDIA NRE sensorsim image used by the wizard is amd64-only; on ARM hosts such as DGX Spark we observed emulated sensorsim hangs and crashes before the gRPC port became ready. The readiness check now fails fast on ARM unless MSA_ALLOW_UNSUPPORTED_ALPASIM_ARM=1 is set explicitly.

If the repo contains a folder-local Hugging Face token at .env.alpasim_hf, you can trigger direct gated AlpaSim downloads and the learned-policy matrix with:

That wrapper now does four repo-local setup steps automatically before the run:

For an explicit preflight without launching anything: