savi-finetuned-30slots
StoSAVi encoder from savi-pretrained-30slots, additionally fine-tuned on LMDrive with auxiliary heads (5 waypoints, traffic-light state, stop-sign state). Stage 1b of the object-centric LMDrive pipeline.
Overview
This is one of four model artefacts released alongside a thesis on replacing the multimodal sensor-fusion encoder in
LMDrive with an object-centric encoder based on
Stochastic Slot Attention from Video (SAVi), evaluated on the CARLA LangAuto benchmark. The four artefacts are:
ni-eminen/savi-pretrained-30slots -- Stage 1a encoderni-eminen/savi-finetuned-30slots -- Stage 1b encoderni-eminen/object-centric-lmdrive-4qt -- Q-Former + LLM (4 query tokens)ni-eminen/object-centric-lmdrive-8qt -- Q-Former + LLM (8 query tokens)
Architecture
- architecture:
StoSAVi + waypoint / traffic-light / stop-sign heads
- num slots:
30
- slot layout:
24 general + 1 traffic-light/stop-sign role + 5 waypoint role
- slot size:
256
- input resolution:
[192, 192]
- clip length:
6
- kld method:
var-0.01
- waypoints head:
GRU predictor over 5 role slots -> 5x(dx,dy)
- traffic light head:
Linear(256, 2)
- stop sign head:
Linear(256, 2)
Inputs
Same as savi-pretrained-30slots: 192x192 RGB BEV, [0, 1] normalisation,
temporal clips of 6 frames. Additionally accepts a scalar velocity for
velocity-conditioning of the role slot initial latents.
Training
Fine-tuned from savi-pretrained-30slots on LMDrive using an
L1 waypoint loss (weight 0.5), CE traffic-light loss (0.1),
CE stop-sign loss (0.01), and MSE reconstruction loss (1.0).
Optimizer AdamW with initial LR 7.5e-4, cosine schedule, weight decay
0.05, gradient clipping 5. Batch size 6, 4 A100-40GB GPUs, 2 epochs
(approximately 35 hours).
Evaluation
Evaluated on the same 72 held-out BEV frames used for
savi-pretrained-30slots.
| Metric | Value |
|---|
| Reconstruction MSE (per-pixel, [0, 1] scale) | 0.00163 ± 0.00053 |
| Reconstruction SSIM | 0.8796 ± 0.024 |
| Reconstruction MSE (foreground pixels only) | 0.00579 ± 0.00365 |
| Reconstruction SSIM (foreground pixels only) | 0.7095 ± 0.183 |
| FG-ARI (%) | 41.21 ± 21.99 |
| mIoU (%) | 5.68 ± 2.67 |
Fine-tuning increases whole-image MSE by 41% and foreground MSE by
107% relative to the pretrained encoder while leaving object-partition
quality (FG-ARI, mIoU) essentially unchanged.
Citation
1@mastersthesis{nieminen2026objcentric,
2 title = {Object-centric world models for language-instructed autonomous driving},
3 author = {Nieminen, Matias},
4 year = {2026},
5 school = {Aalto University}
6}
Related repositories
