Training procedure
This model is fine-tuned using Supervised Fine-Tuning (SFT) with a LoRA (Low-Rank Adaptation) setup on top of Qwen3-VL-8B-Instruct.
The training is part of the Automingo project, which focuses on safety-critical driving VQA using structured, scenario-based reasoning over short temporal image sequences.
Dataset
Training is performed on the Automingo-VQA dataset, designed for structured reasoning in driving scenarios:
- 6,565 images
- 1,313 annotated events
- 5,792 question–answer pairs
- 5-frame temporal snippets centered around safety-critical events
The dataset emphasizes:
- cut-ins
- traffic light transitions
- vulnerable road users
- leading vehicle braking
- construction and lane changes
- intersections and roundabouts
Fine-tuning setup
LoRA adapters are applied to the following modules:
q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj
Best configuration:
- Learning rate:
2e-4
- LoRA rank:
32
- Gradient accumulation:
8
- Optimizer:
AdamW
Training was conducted using a sweep-based approach to optimize hyperparameters.
Training objective
The model is trained to:
- answer structured driving-related questions
- produce reasoning aligned with safety-critical interpretation
- avoid invalid or non-actionable outputs
Evaluation and results
Evaluation setup
Evaluation is performed using the Automingo benchmark pipeline:
- Multiple-choice question (MCQ) answering
- Post-processing with structured evaluation scripts
Metrics:
- MCQ accuracy
- Invalid attempts
- Semantic score (Lingo-Judge)
Benchmark results
| Model | MCQ Acc. (%) | Invalid Attempts | Lingo-Judge |
|---|
| Qwen3-VL-8B (base) | 81.5 | 9 | 0.556 |
| Automingo-VLM-8B (this model) | 89.3 | 43 | 0.628 |
Key improvements
- +7.8% absolute gain in MCQ accuracy over the base model
- Improved structured reasoning for safety-critical scenarios
- Competitive semantic reasoning performance
Observations
-
Strong performance on:
- cut-in scenarios
- leading vehicle interactions
-
Remaining challenges:
- intersections
- roundabouts
Overall, the fine-tuned model achieves strong performance on the Automingo benchmark and demonstrates specialization for ADAS-style reasoning tasks.
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: peft
model_name: crisp-sweep-3_lwysu1i9
tags:
- base_model:adapter:Qwen/Qwen3-VL-8B-Instruct
- lora
- sft
- transformers
- trl
licence: license
pipeline_tag: text-generation
Model Card for crisp-sweep-3_lwysu1i9
This model is a fine-tuned version of
Qwen/Qwen3-VL-8B-Instruct.
It has been trained using
TRL.
Quick start
1from transformers import pipeline
2
3question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
4generator = pipeline("text-generation", model="None", device="cuda")
5output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
6print(output["generated_text"])
Training procedure
This model was trained with SFT.
Framework versions
- PEFT 0.18.1
- TRL: 0.29.0
- Transformers: 4.57.6
- Pytorch: 2.10.0+cu126
- Datasets: 4.6.0
- Tokenizers: 0.22.2
License
This model is released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
You are free to share, copy, redistribute, adapt, transform, and build upon this model for any purpose, including commercial use, provided that appropriate credit is given to the original authors.
Citations
If you use this dataset in your research, product development, benchmark, model training, evaluation pipeline, or any derived work, please cite the associated paper:
Cite Automingo as:
1@inproceedings{Divis_2026_CVPR,
2 title = {Automingo: Seeing the Unseen - Vision-Language Edge Case Dataset for Detection and Analysis of Autonomous Driving},
3 author = {Diviš, Václav and Barceló Álvarez, Íñigo and Fariñas Nubla, Alejandro and Sánchez, Enrique and Hernández-Ros Briales, Antonio and Valach, Ondřej and Gruber, Ivan and Hruž, Marek},
4 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
5 month = {June},
6 year = {2026},
7 pages = {665-674}
8}
9
10Cite TRL as:
11
12```bibtex
13@software{vonwerra2020trl,
14 title = {{TRL: Transformers Reinforcement Learning}},
15 author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
16 license = {Apache-2.0},
17 url = {https://github.com/huggingface/trl},
18 year = {2020}
19}
