The adapter teaches the base model DriveLM's answer-format conventions (short declarative sentences, terse Yes/No for perception, multi-clause None, no, none. for stacked prediction questions). It does not add temporal or grounded-vision reasoning beyond what the base model already has.

Highlights

Trained on 1,024 samples from the nuScenes-mini locally-resolvable subset of DriveLM (~20 min on a single RTX 2070 SUPER, 8 GB).
~3.4× ROUGE-L lift on a 3,770-sample evaluation vs the zero-shot baseline (0.157 → 0.541).
−26% mean latency vs baseline (1,420 → 1,046 ms through vLLM at concurrency 4) — LoRA outputs are shorter.
Adapter file is 12.8 MB — lora_r=8, lora_alpha=16, 3.19 M trainable params (0.37% of base).

Usage

vLLM (recommended)

vLLM auto-attaches the adapter when launched with --enable-lora. The base model and the adapter are served from one process under different model_ids:

bash

vllm serve Qwen/Qwen3.5-0.8B \
  --enable-lora \
  --max-lora-rank 8 \
  --lora-modules '{"name":"drivelm-lora","path":"./qwen-lora","base_model_name":"Qwen/Qwen3.5-0.8B"}' \
  --attention-backend TRITON_ATTN \
  --dtype float16 \
  --max-model-len 1024

Then request the adapter by name:

bash

curl http://127.0.0.1:8001/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "drivelm-lora",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "Camera view: CAM_FRONT"},
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
    {"type": "text", "text": "Question: Are there moving cars in front of the ego car?\nAnswer in one short sentence."}
  ]}]
}'

Transformers + PEFT

python

from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText

base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3.5-0.8B", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-0.8B", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "pranavthombare/qwen3.5-0.8b-drivelm-lora").eval()

Training Details


Base model	`Qwen/Qwen3.5-0.8B`
Adapter type	LoRA
LoRA rank / alpha	8 / 16
LoRA target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Vision tower	Frozen (catastrophic-forgetting risk on 38 unique camera frames)
Base precision during training	4-bit NF4 (bitsandbytes) + fp16 compute
Training samples	1,024 (natural distribution: 492 perception / 311 prediction / 211 planning / 10 behavior)
Camera mode	`front-arc` (CAM_FRONT_LEFT + CAM_FRONT + CAM_FRONT_RIGHT, ≤448 px long edge)
Epochs	1
Optimizer	AdamW, lr 2e-4, no schedule
Effective batch size	1 × gradient_accumulation_steps 2
Label masking	Loss only on assistant tokens (prompt masked to `-100`)
Hardware	Single NVIDIA RTX 2070 SUPER (8 GB)
Training wall clock	~20 minutes
Final epoch-average loss	0.44

Full training pipeline and ablations live at https://github.com/pranavthombare/bosch-drivelm-qwen (TBD).

Evaluation

Evaluated on the same 3,770 DriveLM samples that overlap with the training set — see "Limitations" below.

Overall

Metric	Baseline	LoRA	Δ
ROUGE-1	0.166	0.550	+0.384
ROUGE-2	0.069	0.188	+0.119
ROUGE-L	0.157	0.541	+0.384
Token-F1	0.117	0.510	+0.393
Exact match	0.37%	39.26%	+38.89 pp
Mean latency	1,420 ms	1,046 ms	−374 ms

Per question category (ROUGE-L)

Category	N	Baseline	LoRA	Δ
perception	1,738	0.217	0.489	+0.272 ↑
prediction	1,181	0.097	0.659	+0.562 ↑
planning	813	0.107	0.502	+0.395 ↑
behavior	38	0.305	0.036	−0.269 ↓

Limitations

Train/eval set overlap. The first 1,024 samples in DriveLM's natural ordering were used for training and are also part of the 3,770-sample evaluation set. The headline +0.384 ROUGE-L lift therefore overstates generalization. Held-out evaluation on disjoint frames would lower the perception/prediction/planning gains by an unknown amount. The behavior regression is unaffected (no behavior samples appeared in training).
Behavior category regression. Only 10 of 1,024 training samples were behavior questions, none in the multi-clause format DriveLM uses. The adapter collapsed to a terse Turn left. default for all behavior inputs. ROUGE-L dropped from 0.305 → 0.036 on this category.
Yes-bias overcorrection. Training perception had a 3.2:1 Yes/No skew. The adapter flips some baseline-correct No. answers to Yes. on the evaluation set.
No referent-token grounding. DriveLM questions reference objects via <c1,CAM_FRONT,x,y> tokens that require bbox-grounded vision. The base model has no grounding head; LoRA does not fix this.
No CAN-bus signal access. DriveLM behavior answers include ego-velocity attributes ("driving fast", "not moving") that cannot be inferred from a single camera frame.
nuScenes-mini scope. Training and eval cover 38 frames across 6 scenes (Singapore + Boston, daylight bias). Generalization to weather, night, or other geographies is untested.

A future stratified retrain (250 perception + 250 prediction + 250 planning + all 38 behavior × 4 upsampling) is documented as the next experiment.

Reproducibility

The full pipeline — data loader, training script, eval client, vLLM launcher, Docker setup — is in the project repository. The config used for this adapter:

bash

DRIVELM_TRAIN__NUM_SAMPLES=1024
DRIVELM_TRAIN__EPOCHS=1
DRIVELM_TRAIN__LR=2e-4
DRIVELM_TRAIN__LORA_R=8
DRIVELM_TRAIN__LORA_ALPHA=16
DRIVELM_TRAIN__GRADIENT_ACCUMULATION_STEPS=2
DRIVELM_DATA__CAMERA_MODE=front-arc
DRIVELM_DATA__IMAGE_LONG_EDGE=448
DRIVELM_MODEL__QUANTIZATION=auto

License

Apache-2.0, matching the base model. nuScenes images and DriveLM annotations retain their respective dataset licenses (see nuScenes terms and DriveLM) — this adapter does not redistribute either.

Framework versions

PEFT 0.19.1
transformers (HuggingFace main as of training date)
bitsandbytes 0.49.2