Aloe-Vision is a medical Large Vision–Language Model built on Qwen2-VL-Instruct, released in 7B and 72B sizes. The model is trained on a ~3.5 M samples balanced mixture across medical vs. general and multimodal vs. text-only sources, rebalanced by loss-contributing assistant tokens to avoid long-answer bias. We implement leakage control of evaluation images in the training data via exact 64-bit image-hash matching, removing any duplicates from the training. Quality filtering of the training data combines (1) LVLM-based sample scoring (1–5 scale) for image–question–answer coherence and relevance and (2) answer perplexity checks to flag trivial or noisy annotations. Thresholds are dataset-specific and manually tuned, leading to the removal of low-quality outliers while preserving clinically meaningful diversity. Furthermore, the model is additionally fine-tuned on 17.2 K adversarially perturbed medical samples to enhance robustness against sycophantic and misleading multimodal cues. The model is released for research purposes under CC BY-NC-SA 4.0.
We construct a balanced mixture across two axes: modality (multimodal vs text-only) and domain (medical vs general). All sources are normalized to a unified trl conversation schema. Medical multimodal includes both global understanding and fine-grained region reasoning.
Aloe-Vision targets comprehensive evaluation across medical multimodal, medical text-only, general multimodal, and general text-only tasks. Benchmarks are run with identical settings for Aloe-Vision and baselines to ensure reproducibility.
Multimodal via VLMEvalKit, text-only via lm-evaluation-harness.
Decoding: greedy, accuracy by exact match for MCQ and Y/N.
LLM-as-judge (SLAKE): Qwen2.5-VL-72B with a rubric-based {0.0, 0.5, 1.0} scale.
Results
Model
OmniMedVQA
GMAI-MMBENCH
PathMMU
ProbMed
SLAKE
MMMU
MultiMedQA
MMLU
Qwen2-VL-72B (general)
77.90
51.03
64.71
73.87
68.15
61.22
74.25
81.86
InternVL3.5-30B-A3B (general)
91.60
63.91
72.07
82.21
79.87
60.67
71.21
81.68
Linghsu-32B
80.20
53.54
67.60
80.84
86.08
53.00
72.08
81.32
HuatuoGPT-Vision-34B
68.90
48.31
54.90
71.79
60.03
27.11
60.57
72.80
Aloe-Vision-72B
85.20
55.12
70.32
77.71
69.36
62.00
76.35
82.52
Aloe-Vision-72B-AR
84.00
54.79
71.45
77.29
67.36
62.89
76.24
82.55
Adversarial Robustness
To improve robustness against noisy or misleading inputs, we conducted an additional fine-tuning stage focused on adversarial robustness. This stage aimed to mitigate common LVLM vulnerabilities such as sycophantic behavior or misleading multimodal cues. An adversarial benchmark was first created by applying controlled perturbations to existing medical datasets (distinct from those used in evaluation). These perturbations introduced conflicting or false multimodal signals (e.g., mismatched region annotations or incorrect textual hints).
Using this adversarially transformed dataset, we trained an Aloe-Vision-72B-AR variant through a single-stage post-training SFT consisting of 17.2K adversarial samples. The adversarial fine-tuning employed the same optimization setup as the base model and ran for 1 epoch.
This procedure yielded substantial improvements across all adversarial evaluation categories while preserving performance on standard benchmarks.
The following table reports model accuracy (%) under different adversarial perturbations.
Columns correspond to:
Cap = misleading captions inserted into the image
Pmt = misleading captions in the prompt
Syc = sycophantic prompt bias
Leg = misleading legends inserted into the image
Model
Cls Base
Cap
Pmt
Syc
Det Base
Cap
Pmt
Syc
Leg
Qwen2-VL-72B
61.2
2.1
5.1
35.6
75.9
3.7
3.6
14.7
41.1
InternVL3.5-30B
68.3
7.3
5.1
35.7
73.1
35.1
33.5
27.5
48.1
Lingshu-32B
68.1
4.5
29.7
55.4
80.2
8.5
19.6
35.0
60.1
HuatuoGPT-Vision-34B
59.6
22.2
10.0
16.1
64.1
40.9
5.6
5.2
50.7
Aloe-Vision-72B
64.2
4.7
6.9
73.9
69.7
3.0
1.9
8.6
37.5
Aloe-Vision-72B-AR
69.8
17.0
50.5
62.8
81.8
48.6
64.0
66.3
53.4
Safety, Risks & Limitations
Not a medical device. Do not rely on outputs for diagnosis/treatment.
Failure modes: may hallucinate, misinterpret findings, or over-generalize across modalities and specialties.
Sensitive content: can produce unsafe content if prompted adversarially.
Recommended practice
Keep a qualified clinician in the loop for any medically relevant use.
Clinical safety: Aloe-Vision is a research model. It must not be used for diagnosis, treatment, or clinical decision-making. Always place a qualified human in the loop.
Citation
bibtex
1@InProceedings{pmlr-v315-guasch-marti26a,
2 title = {Aloe-Vision: Robust Vision-Language Models for Healthcare},
3 author = {Guasch-Mart\'i, Jaume and Lopez-Cuena, Enrique and Su\'arez-Fern\'andez, Mart\'in and Bayarri-Planas, Jordi and Arias-Duart, Anna and Garcia-Gasulla, Dario},
4 booktitle = {Proceedings of The 9th International Conference on Medical Imaging with Deep Learning},
5 pages = {2404--2426},
6 year = {2026},
7 volume = {315},
8 series = {Proceedings of Machine Learning Research},
9 month = {08--10 Jul},
10 publisher = {PMLR},
11 url = {https://proceedings.mlr.press/v315/guasch-marti26a.html},
12}
Acknowledgments
Developed by the High Performance Artificial Intelligence (HPAI) group at Barcelona Supercomputing Center (BSC). Contact: hpai@bsc.es.
