You can deploy this model and start earning money today!

dkhursen/
InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd

Name: InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd
Availability: InStock

OpenGVLab/InternVL2-2B

finetune

Views

No views yet

InternVL2-2B — LoRA-25k + Oracle Visual Annotation (Red Circle + CTags + Background)

Part of the Master's thesis:

Visual Question Answering for Autonomous Driving Dmytro Khursenko, Czech Technical University in Prague, Faculty of Electrical Engineering, 2026. Supervised by Ing. David Hurych, Ph.D. (Valeo) and doc. Georgios Tolias, Ph.D. (CTU FEE).

GitHub · Demo · Pseudo-label dataset

📄 Thesis PDF expected by end of June 2026, following successful defense (CTU FEE).

This checkpoint is the oracle upper-bound experiment: the same LoRA-25k model weights are evaluated on test images where ground-truth 3D bounding boxes have been projected onto the stitched 6-camera views at test time as red circles around each object, with class-name text labels (CTags) overlaid and the background dimmed to highlight annotated objects. The model weights are unchanged — only the test images differ.

The oracle jump from 0.560 → 0.775 Final Score (with Coord improving from 0.015 → 0.758, a 50× increase) shows that visual object localization is the dominant bottleneck for VLMs on the DriveLM benchmark.

This checkpoint contains fully merged weights — the LoRA adapter is already merged into the base model. Load and use it directly with AutoModel.

Note: to reproduce the oracle results, pass annotated test images (GT bounding boxes rendered as red circles + class labels + dimmed background) as input. Using plain images will give the same results as the LoRA-25k baseline (Final 0.560).

All released models

Model	HF repo	Final
LoRA-25k	dkhursen/InternVL2-2b-LoRA-25k-drivelm	0.560
LoRA-300k	dkhursen/InternVL2-2b-LoRA-300k-drivelm	0.493
LoRA-25k + DL-PL 10% ⭐	dkhursen/InternVL2-2b-LoRA-25k_plus_DL-PL-10pct	0.589
LoRA-25k + Oracle annotation (this model)	dkhursen/InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd	0.775

Pseudo-label dataset: dkhursen/drivelm-pseudo-labels

Results on custom test split (oracle inference)

Evaluated on the custom i.i.d. DriveLM-nuScenes test split (3,340 QA pairs) with GT visual annotations rendered on the test images.

Metric	Score
Final	0.775
Accuracy	0.853
ChatGPT	0.725
Language	0.779
BLEU-1	0.908
BLEU-2	0.879
BLEU-3	0.849
BLEU-4	0.819
ROUGE-L	0.891
CIDEr	5.817
Match	0.793
Coord	0.758

Score definitions (all table values normalised to [0, 1])

Accuracy — exact-match on MCQ (A/B/C/D) and Yes/No questions; strict letter-only format required

Language — mean of BLEU-1–4, ROUGE-L, and CIDEr/10

ChatGPT — GPT-3.5-turbo semantic similarity judge (0–100 scale, ÷100 in Final)

Match — (F1_coord × 100 + GPT_match) / 2 on prediction answers (0–100 scale, ÷100 in Final); blends spatial coordinate F1 with a GPT judge

Coord — pure coordinate token F1 at L1 < 16 px; diagnostic only, not included in Final

Final — 0.4 × (GPT/100) + 0.2 × Language + 0.2 × (Match/100) + 0.2 × Accuracy

Full metric breakdown: evaluation/README.md

Note on evaluation scope. All scores are measured on the custom i.i.d. local test split (3,340 QA pairs) using the local DriveLM evaluation script. The official DriveLM evaluation server was not used as the primary reporting source: it returns only aggregate scores without per-metric breakdown, the infrastructure was intermittently unreliable (model output parsing errors could not be diagnosed directly — only via GitHub issues), and ChatGPT/Match metrics depend on OpenAI API calls that fail when the API quota for the billing period is exhausted.

Impact of annotation style (same LoRA-25k model, different test-time visual annotations):

Visual Annotation	Final	Acc	ChatGPT	Lang	Match	Coord
None (raw image)	0.560	0.826	0.589	0.459	0.338	0.015
Red BBox + CTags + Bkgd	0.772	0.848	0.726	0.827	0.778	0.740
Red BBox + CTags	0.655	0.832	0.684	0.589	0.568	0.393
Red BBox	0.634	0.743	0.740	0.741	0.419	0.112
Red Circle + CTags + Bkgd (this checkpoint)	0.775	0.853	0.725	0.779	0.793	0.758
Red Circle + CTags	0.624	0.727	0.728	0.695	0.430	0.141
Red Circle	0.646	0.793	0.729	0.761	0.443	0.158
Red Midpoint + CTags + Bkgd	0.771	0.858	0.716	0.826	0.780	0.736
Red Midpoint + CTags	0.757	0.862	0.724	0.785	0.742	0.677
Red Midpoint	0.653	0.799	0.735	0.746	0.463	0.199

CTags = class-name text labels overlaid on objects. Bkgd = background dimming to highlight annotated objects.

Online model comparison (no oracle annotation):

Model	Final	Acc	ChatGPT	Lang	B1	B2	B3	B4	RL	CIDEr	Match	Coord
Mini-DA†	0.606	0.898	0.668	0.416	0.596	0.564	0.533	0.503	0.651	0.470	0.381	0.000
LoRA-25k (base of this model)	0.560	0.826	0.589	0.459	0.732	0.668	0.606	0.547	0.714	0.230	0.338	0.015
LoRA-25k + DL-PL 10%	0.589	0.836	0.676	0.451	0.719	0.654	0.589	0.525	0.710	0.222	0.304	0.013
LoRA-25k + DL-PL 30%	0.548	0.832	0.605	0.434	0.695	0.625	0.554	0.483	0.692	0.201	0.264	0.008
LoRA-25k + DL-PL 50%	0.532	0.832	0.584	0.433	0.691	0.622	0.551	0.481	0.699	0.171	0.230	0.007
LoRA-25k + DL-PL 100%	0.511	0.805	0.544	0.430	0.687	0.616	0.543	0.470	0.695	0.165	0.232	0.007
LoRA-300k	0.493	0.339	0.706	0.412	0.607	0.552	0.501	0.452	0.676	0.323	0.303	0.006

† Mini-DA = OpenGVLab/Mini-InternVL2-2B-DA-DriveLM, fine-tuned by OpenGVLab on DriveLM. DL-PL 30%, 50%, 100% variants not publicly released (lower performance without quality filtering).

Visual comparison: raw image vs. oracle annotation

Without annotation	With Red Circle + CTags + Background
Raw scene	Oracle-annotated scene

Same nuScenes keyframe. Left: plain stitched 6-camera collage fed to the model online. Right: same image with ground-truth 3D bounding boxes projected as red circles, class-name labels, and dimmed background — the oracle inference input.

Metric comparison for the same scene:

Metric	Raw image (LoRA-25k)	+ Red Circle + CTags + Background
Final	0.560	0.775
Accuracy	0.826	0.853
ChatGPT	0.589	0.725
Language	0.459	0.779
Match	0.338	0.793
Coord	0.015	0.758

The jump from 0.015 → 0.758 on Coord with no model change confirms that visual grounding — localising objects by their bounding-box coordinates — is the dominant bottleneck for current VLMs on this benchmark.

Training configuration

This model uses the same training configuration and weights as dkhursen/InternVL2-2b-LoRA-25k-drivelm. The difference is at inference time only: the test images include ground-truth visual annotations.

Parameter	Value
Base model	OpenGVLab/InternVL2-2B
Fine-tuning method	LoRA (merged into weights in this checkpoint)
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
LoRA target modules	all linear layers in the LLM (backbone and MLP frozen)
Learning rate	4e-5 (cosine, warmup 0.03)
Effective batch size	64 (8 GPU × 4 × grad-acc 2)
Epochs	10
Max sequence length	8192
Precision	bfloat16
Hardware	8 × NVIDIA A100 40 GB

Full configuration: configs/finetune/internvl2_2b_lora.yaml

How to use

python

import torch
from PIL import Image
from transformers import AutoTokenizer, AutoModel
import torchvision.transforms as T
from torchvision.transforms.functional import InterpolationMode

MODEL_ID   = "dkhursen/InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd"
# For oracle results: pass a test image with GT bounding boxes rendered as
# red circles + class-name labels + dimmed background.
# For standard use: any plain driving scene image works (results = LoRA-25k baseline).
IMAGE_PATH = "path/to/annotated_scene.jpg"
QUESTION   = "<image>\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision."

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD  = (0.229, 0.224, 0.225)


def build_transform(input_size):
    return T.Compose([
        T.Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
    ])


def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
    best_ratio_diff = float("inf")
    best_ratio = (1, 1)
    area = width * height
    for ratio in target_ratios:
        ratio_diff = abs(aspect_ratio - ratio[0] / ratio[1])
        if ratio_diff < best_ratio_diff:
            best_ratio_diff = ratio_diff
            best_ratio = ratio
        elif ratio_diff == best_ratio_diff:
            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
                best_ratio = ratio
    return best_ratio


def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
    orig_width, orig_height = image.size
    aspect_ratio = orig_width / orig_height
    target_ratios = sorted(
        {(i, j)
         for n in range(min_num, max_num + 1)
         for i in range(1, n + 1)
         for j in range(1, n + 1)
         if min_num <= i * j <= max_num},
        key=lambda x: x[0] * x[1],
    )
    best_ratio = find_closest_aspect_ratio(
        aspect_ratio, target_ratios, orig_width, orig_height, image_size
    )
    target_w = image_size * best_ratio[0]
    target_h = image_size * best_ratio[1]
    cols = best_ratio[0]
    resized = image.resize((target_w, target_h))
    tiles = []
    for i in range(best_ratio[0] * best_ratio[1]):
        col = i % cols
        row = i // cols
        box = (col * image_size, row * image_size,
               (col + 1) * image_size, (row + 1) * image_size)
        tiles.append(resized.crop(box))
    if use_thumbnail and len(tiles) != 1:
        tiles.append(image.resize((image_size, image_size)))
    return tiles


def load_image(image_path, input_size=448, max_num=12):
    image = Image.open(image_path).convert("RGB")
    transform = build_transform(input_size)
    tiles = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
    return torch.stack([transform(tile) for tile in tiles])


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True, use_fast=False)
model = AutoModel.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
).eval().cuda()

pixel_values = load_image(IMAGE_PATH, max_num=12).to(torch.bfloat16).cuda()
generation_config = {"max_new_tokens": 512, "do_sample": False}

response = model.chat(tokenizer, pixel_values, QUESTION, generation_config)
print("Answer:", response)

Batch inference and evaluation (requires cloning the repo):

bash

git clone https://github.com/dmitrykhursen/VQA-AD-CTU
cd VQA-AD-CTU
bash scripts/setup_env.sh && source vqa-ad-ctu-env/bin/activate

# Run inference (edit MODEL= inside the script; use annotated images path for IMAGES_ROOT)
sbatch scripts/05_inference.sh

# Evaluate
bash scripts/06_evaluate.sh inference/outputs/<MODEL>/local_test.json

Intended use and limitations

Intended use:

Quantifying the upper bound of visual grounding for VLMs on DriveLM
Ablation study: how much performance is lost purely due to the model's inability to localise objects visually
Research on visual grounding improvements for driving VQA

Limitations:

Oracle results require ground-truth 3D annotations at test time — not applicable in real deployment
For standard (non-oracle) use, dkhursen/InternVL2-2b-LoRA-25k_plus_DL-PL-10pct is the recommended checkpoint (Final 0.589)
Not intended for deployment in real autonomous driving systems
Trained and evaluated on nuScenes scenes from Boston and Singapore only

Resources

Resource	Link
Code & scripts	github.com/dmitrykhursen/VQA-AD-CTU
Demo gallery	dmitrykhursen.github.io/VQA-AD-CTU
Pseudo-label dataset	dkhursen/drivelm-pseudo-labels
Best online checkpoint	dkhursen/InternVL2-2b-LoRA-25k_plus_DL-PL-10pct
Base model	OpenGVLab/InternVL2-2B
DriveLM benchmark	github.com/OpenDriveLab/DriveLM
nuScenes dataset	nuscenes.org

Citation

bibtex

@mastersthesis{khursenko2026vqa,
  author     = {Khursenko, Dmytro},
  title      = {Visual Question Answering for Autonomous Driving},
  school     = {Czech Technical University in Prague, Faculty of Electrical Engineering},
  year       = {2026},
  supervisor = {Hurych, David and Tolias, Georgios}
}

bibtex

@inproceedings{sima2024drivelm,
  title     = {DriveLM: Driving with Graph Visual Question Answering},
  author    = {Sima, Chonghao and Renz, Katrin and Chitta, Kashyap and others},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2024}
}