A LoRA adapter that fine-tunes Qwen/Qwen2-VL-7B-Instruct on DriveLM-nuScenes v1.1 while giving the model a numeric ego-speed hint at inference time. Compared with the previous v1 (visual-only) LoRA, v3 breaks the long-standing behavior-accuracy ceiling on this dataset (44% → 60%+).

TL;DR

Metric (100 held-out val QAs, seed=42)	Zero-shot	v1 (visual only, 154k train)	v3 (visual + speed, 154k train)	Δ
Exact match	9	62	66	+4
Fuzzy substring	11	65	70	+5
Verbose (pred > 3× GT)	14	0	0	0

Per DriveLM level (25 QAs each):

Level	v1 exact	v3 exact	Δ
Perception	16	16	0
Prediction	22	22	0
Planning	13	13	0
Behavior	11 (44%)	15 (60%)	+4 (+16pp)

All gains land on the behavior level — v3 matches v1 on perception / prediction / planning and adds +4 exact points where the label class is directly speed-derived ("driving fast / normal / slowly / not moving").

A companion v3-20k variant (trained on 20k QAs instead of 154k) trades total exact for even stronger behavior — see the v3-20k branch — 17/25 behavior (68%) at 64 total exact.

What changed vs v1

One thing only: at inference time, prepend "Ego speed: X.Y m/s.\n" to the user prompt (X.Y = ego vehicle's speed magnitude in m/s). Everything else is identical to v1 — same LoRA rank, same base, same DriveLM QA schema.

Where to get the speed value:

From nuScenes: differentiate ego_pose.translation between consecutive samples (script below).
From a running vehicle: read wheel-speed / CAN-bus / GPS-derived speed.
From estimation: any monocular / visual-odometry estimate is fine — models is tolerant of noise (v3 was trained on pose-diff speeds which have ~0.1-0.3 m/s noise).

Why it works: behavior labels in DriveLM are a discretization of ego speed. Numeric grounding lets the LLM directly threshold-classify instead of guessing from static visual cues. See W17 negative log for the ablation that ruled out visual-temporal approaches.

Intended use

Same as v1 (front-view dashcam → DriveLM-style Q&A), plus:

Robotics / research prototypes where you already have odometry (wheel encoders, IMU, GPS) and want a driving-focused VLM to condition on it.
Behavior classification as a substitute for hand-tuned speed thresholds.

Non-intended use

Same limits as v1 (single front camera, English only, DriveLM QA style, not a safety-critical policy).
Do not feed adversarial speed values expecting the model to hallucinate — the numeric prefix strongly biases behavior labels.

How to use

python

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
import torch

BASE = "Qwen/Qwen2-VL-7B-Instruct"
ADAPTER = "zhudanburujiandan/Qwen2-VL-7B-DriveLM-LoRA-v3"

model = Qwen2VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda", attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

processor = AutoProcessor.from_pretrained(BASE, min_pixels=100*28*28, max_pixels=512*28*28)

ego_speed_m_s = 5.5  # from odometry / pose diff
question = "Predict the behavior of the ego vehicle."

user_text = f"Ego speed: {ego_speed_m_s:.1f} m/s.\n{question}"
messages = [
    {"role": "system",
     "content": "You are a driving perception assistant. You see the front camera view of "
                "the ego vehicle. Answer the question concisely based only on what is visible. "
                "You will also be told the current ego speed in m/s at the start of each prompt."},
    {"role": "user", "content": [
        {"type": "image", "image": "path/to/front_cam.jpg"},
        {"type": "text",  "text": user_text},
    ]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=192, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Expected outputs (behavior questions):

Ego speed 0.0 m/s → "The ego vehicle is going straight. The ego vehicle is not moving."
Ego speed 3.1 m/s → "The ego vehicle is going straight. The ego vehicle is driving slowly."
Ego speed 5.2 m/s → "The ego vehicle is going straight. The ego vehicle is driving with normal speed."
Ego speed 7.3 m/s → "The ego vehicle is going straight. The ego vehicle is driving fast."
Ego speed 10+ m/s → "The ego vehicle is going straight. The ego vehicle is driving very fast."

Other levels (perception / prediction / planning) still work; the speed prefix is harmless (and slightly helpful) there.

Deriving speed from nuScenes samples

If you're evaluating on nuScenes / DriveLM directly, differentiate ego pose:

python

from nuscenes.nuscenes import NuScenes
import numpy as np

nusc = NuScenes(version="v1.0-trainval", dataroot="path/to/nuscenes", verbose=False)

def sample_speed_m_s(sample_token: str) -> float:
    s = nusc.get('sample', sample_token)
    scene_samples = [x for x in nusc.sample if x['scene_token'] == s['scene_token']]
    scene_samples.sort(key=lambda x: x['timestamp'])
    idx = next(i for i, x in enumerate(scene_samples) if x['token'] == sample_token)
    lo = max(0, idx - 1); hi = min(len(scene_samples) - 1, idx + 1)
    def pose(x):
        sd = nusc.get('sample_data', x['data']['CAM_FRONT'])
        return np.array(nusc.get('ego_pose', sd['ego_pose_token'])['translation']), sd['timestamp']
    pa, ta = pose(scene_samples[lo]); pb, tb = pose(scene_samples[hi])
    dt = (tb - ta) / 1e6
    return float(np.linalg.norm(pb - pa) / dt) if dt > 0 else 0.0

Central difference (previous + next samples) gives ~0.5-1s smoothing that matches training.

Speed → label empirical distribution (why this works)

On DriveLM's 3,652 behavior QAs (train split), the "driving fast / normal / slowly / not moving" categories map cleanly onto pose-diff speed:

Label	Median speed	p10 – p90
not moving	0.00 m/s	0.00 – 1.14
slow	3.11 m/s	0.68 – 4.85
normal	5.22 m/s	3.69 – 6.59
fast	7.29 m/s	5.77 – 8.81

The p10/p90 bands barely overlap, so thresholds around [1.5, 4.0, 6.0] m/s already recover the labels almost perfectly. What v3 learns is essentially that mapping (with a small residual for tie-breaking via visual scene).

Training details

Same as v1, except:

Prompts prepend "Ego speed: X.Y m/s.\n" (computed from nuScenes ego_pose diffs).
Same 154,363 train QAs, 1 epoch on 1× L40S 48 GB, 19h50min.
eval_loss plateau 0.175 (v1 was 0.19; small but real gain in loss too).
LoRA r=32, α=64, target = q/k/v/o + gate/up/down_proj. Trainable 80.7 M / 8.37 B (0.96%).

Compute delta vs v1: same ~20h. Speed prefix costs ~15 extra tokens per sample; no measurable slowdown.

Related work

v1 (154k, visual only) — 62 exact, behavior 44%. Repo.
W15 (3-cam panorama) — negative (52 exact). Per-camera resolution loss hurts perception.
W16 (4-frame temporal from DriveLM's own keyframes, non-uniform 0.5–13s gaps) — 53 exact, behavior unchanged. First hint that time signal alone isn't enough.
W17 (4-frame temporal from real nuScenes samples, uniform 0.5s gaps) — 54 exact, behavior dropped to 8/25 (32%). Ruled out "denser video → better behavior".
v3 (this) — solved via numeric grounding, not visual data.

Takeaway: for VLMs to classify a physical quantity that has a native numeric label (speed, distance, count), feed the number. Denser video, more cameras, and longer training all failed on this axis; a single float in the prompt lifted 44% → 60%.

Known limitations

Same as v1 (see v1 README) plus:

Requires an odometry input at inference. If you can't provide one, use v1 (this adapter still runs without the prefix — behavior degrades to roughly v1 numbers or worse).
Speed unit locked to m/s. Pass in the wrong unit (km/h, mph) and the model silently mis-classifies. There is no unit safeguard in the prompt.
Value range trained on ≈ 0–13 m/s (nuScenes urban driving). Higher speeds (highway 30+ m/s) are extrapolation; the model has never seen "driving very very fast" in training text.
Same DriveLM QA schema narrowness applies (short answers, English only).

Citation

Same base + dataset citations as v1 (see v1 README):

bibtex

@article{qwen2vl,
  title={{Qwen2-VL}: Enhancing Vision-Language Model's Perception of the World at Any Resolution},
  author={Wang, Peng and Bai, Shuai and others},
  journal={arXiv preprint arXiv:2409.12191},
  year={2024},
}

@inproceedings{drivelm,
  title={{DriveLM}: Driving with Graph Visual Question Answering},
  author={Sima, Chonghao and Renz, Katrin and others},
  booktitle={ECCV},
  year={2024},
}

License

Apache-2.0 (inherits from Qwen2-VL-7B-Instruct and DriveLM-nuScenes v1.1).