A LoRA adaptation of Qwen3-VL-2B-Instruct for front-camera perception on
Bench2Drive. It answers metric, planner-consumable questions about the scene ahead:
where each vehicle is, how far the lane is clear, what the traffic light governing
your lane shows, whether the gap to the lead vehicle is closing.
Merged and standalone — load with from_pretrained, no adapter needed.
Input format
Front camera only: the current frame plus three history frames at 0.5 s intervals.
Front-view frames (oldest to newest):
[t-1.5s]<image>[t-1.0s]<image>[t-0.5s]<image>[t=0 current]<image>
<question>
Ego-centric coordinates: origin (0,0), X = lateral (right +), Y = longitudinal
(forward +), metres. Trained at a vision budget of MAX_PIXELS=131072
(≈512×256); set the same at inference.
Labels are derived from the simulator ground truth in Bench2Drive-Full-Sup
(anno/*.json.gz): 3D boxes are projected through the CAM_FRONT intrinsics and
extrinsics, and only objects that actually resolve in the front view are used.
Every task's answer distribution is balanced by construction, so the language
prior cannot carry it.
13 tasks: object counting, nearest-object localisation, full-neighbourhood
localisation, free distance in own lane, lead-vehicle type/distance, gap dynamics,
traffic-light state, category presence, side occupancy, lead-vehicle motion, ego
speed state, ego steering state, and 3 s ego trajectory.
Evaluation
6,000 held-out items from 120 clips disjoint from training. Each task is
reported against that set's own majority-class baseline, and against a blind
control — the identical prompt with every image replaced by a uniform grey frame.
The sighted-minus-blind margin is the visual contribution; absolute accuracy alone
is not evidence of perception.
task
majority
sighted
blind
visual
free distance
25.0%
97.0%
32.7%
+64.3
category presence
50.0%
96.4%
50.0%
+46.4
side occupancy
50.0%
96.3%
50.0%
+46.3
lead vehicle
50.0%
94.2%
50.0%
+44.2
traffic light
45.2%
92.9%
45.2%
+47.7
lead motion
50.0%
84.2%
50.0%
+34.2
nearest-object loc (±3 m)
0.3%
84.2%
2.3%
+81.9
object count
20.0%
70.9%
20.0%
+50.9
gap dynamics
33.3%
65.3%
33.3%
+32.0
multi-object loc
20.0%
63.3%
20.0%
+43.3
ego speed state
25.0%
60.0%
25.0%
+35.0
ego steering state
33.3%
55.8%
33.3%
+22.5
ego trajectory
1.8%
52.8%
37.5%
+15.3
overall
27.2%
78.2%
32.5%
+45.7
Localisation uses a 3 m tolerance. The multi-object row is all-or-nothing — every
gold object matched by a distinct prediction within 3 m and the same object count.
Per object, which is what a planner actually consumes:
precision
recall
F1
exact count
median error
multi-object localisation
85.3%
81.8%
83.5%
76.2%
0.43 m
blind
0.0%
0.0%
0.0%
20.0%
—
Blinded, it matches zero objects — the cleanest visual evidence in the suite.
Caveat on trajectory. Blind accuracy is 37.5%, far above the 1.8% majority
baseline: a stopped ego yields all-zero waypoints and steady cruising yields a
predictable straight line, both guessable without the image. Only ~15 points of
that task's 52.8% are visual. Do not cite trajectory accuracy as perception
evidence. Every other task's blind score sits on its majority baseline.
Against the prior checkpoint
Same eval set, same protocol:
prior driving-QA LoRA
this model
overall
33.2%
78.2%
traffic light
58.8%
92.9%
lead vehicle
24.2%
94.2%
nearest-object loc
40.3%
84.2%
object count
37.6%
70.9%
Training
LoRA r=64, α=128, all linear layers of the language model plus the vision-language
aligner; ViT frozen. bf16, lr 1e-4 cosine, warmup 0.03, weight decay 0.1, global
batch 120, max length 6144, 2 epochs, 3×A100-40GB, ms-swift.
Freezing the ViT is deliberate: a matched ablation that unfroze it with a 10×-lower
ViT learning rate (1e-5) scored 81.3% vs 81.1% on the comparable task subset —
no benefit for 25M extra parameters.
Limitations
Front camera only; objects outside the front view are invisible to it, and
questions about them are unanswerable rather than merely hard.
Trained and evaluated in CARLA/Bench2Drive. No real-world validation.
Trajectory output is largely prior-driven (see caveat above).
Traffic-light colour is learned from the simulator's affects_ego flag; frames
where a light governs the lane but is not visible were excluded from training
rather than labelled "none".
Pedestrians, trucks and buses appear in under 1% of frames, so the model has
effectively no supervision for them.
