You can deploy this model and start earning money today!

nacefmbarek/
simlingo-predictive-fusion

Name: simlingo-predictive-fusion
Availability: InStock

autonomous-driving

carla

vision-language-action

internvl2

robotics

mit

Views

No views yet

SimLingo — Predictive Fusion (fdrop03)

InternVL2-1B (InternViT-300M + Qwen2-0.5B, LoRA r=32 α=64) camera-VLA driving policy for CARLA, with a two-stage cross-attention fusion where InternViT vision tokens are the query:

Stage 1 — KV = 360° LiDAR → BEV occupancy histogram (current scene).
Stage 2 — KV = DeltaTok-forecast future embedding — a frozen DINOv3 ViT-B/16 tokenizer plus a trained Predictor that forecasts the next delta token without seeing the future frame.

Enriched tokens replace <IMG_CONTEXT> in place. Training code, config, and the data-collection pipeline: github.com/nacef112/kaut-project. Built on carla_garage and SimLingo.

Recipe


`route_as`	`target_point_command` — GPS target point and the language command
`predict_route_as_wps`	True
`speed_wps_mode`	`2d`
`fusion_dropout_prob`	0.3 — withholds BEV and forecast TOGETHER, per sample, train-only. Forces the target-point/command path to carry gradient instead of only the fusion shortcut.
forecast	single horizon (not multi-horizon)
`num_context_frames` / `delta_frame_gap`	4 / 1
DeltaTok tokenizer	included in this repo (`deltatok_best_b2dheldout.ckpt`)

Files

file	size	notes
`fdrop03_step050000_weights.ckpt`	3.2 GB	epoch=2, step=50000 — the warm-start every downstream checkpoint in this project line continued from. Use this one unless you specifically want the later epoch.
`fdrop03_final_epoch003_weights.ckpt`	3.2 GB	epoch=3, step=82840 — further trained, same recipe
`deltatok_best_b2dheldout.ckpt`	2.0 GB	DeltaTok forecast tokenizer (single-horizon) that Stage-2 was trained against
`SHA256SUMS.txt`	—	integrity check

Both weight files load state_dict only (optimizer/scheduler state stripped), verified to load strict=True into a freshly constructed model — 0 missing, 0 unexpected keys, 1317.6M params.

Load

python

import torch
sd = torch.load("fdrop03_step050000_weights.ckpt", map_location="cpu", weights_only=False)
state_dict = sd.get("state_dict", sd)
# model = hydra.utils.instantiate(cfg.model, ...)  -- see the training repo for cfg construction
model.load_state_dict(state_dict, strict=True)

The model class (DrivingModel) and its Hydra config live in the training repo linked above — this is weights only, not a standalone inference package. DINOV3_BACKBONE env var overrides the (gated) facebook/dinov3-vitb16-pretrain-lvd1689m backbone path if you have a local copy; see the training repo's README for requesting access.

What this is not

Not a multi-horizon, trimodal-fusion, LoRA-only-SFT, or CoT/adaptive-CoT checkpoint — those are separate lines built on top of this same architecture, not included here.

nacefmbarek/simlingo-predictive-fusion