This is a Vision-Language Model (VLM) fine-tuned specifically for the classification and analysis of skin lesions using dermoscopy images. It is built on top of the Qwen/Qwen2.5-VL-3B-Instruct base model and fine-tuned using QLoRA (via Unsloth) on the ISIC 2019 dataset.

The model is highly optimized for edge deployment. Despite being a powerful multimodal AI, the base model is quantized to 4-bit precision, and only the language adapters were fine-tuned (leaving the vision encoder frozen). This allows the model to run inference locally on consumer-grade GPUs with as little as 4GB of VRAM.

Developed by: saad9694

Model type: Vision-Language Causal Language Model (LoRA Adapters)

Language(s) (NLP): English

License: Apache 2.0 (Note: The underlying ISIC dataset is intended for non-commercial academic/research purposes).

Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

The model takes a dermoscopy image and a prompt as input, and outputs a diagnostic category for the skin lesion. It is capable of categorizing lesions into the following classes based on the ISIC 2019 standard:

Melanoma (MEL)

Melanocytic nevus (NV)

Basal cell carcinoma (BCC)

Actinic keratosis (AK)

Benign keratosis (BKL)

Dermatofibroma (DF)

Vascular lesion (VASC)

Squamous cell carcinoma (SCC)

Out-of-Scope Use

MEDICAL DISCLAIMER: This model is strictly for educational, research, and hobbyist purposes. It is NOT a medical device, nor is it a substitute for professional medical advice, diagnosis, or treatment. Never delay seeking professional medical advice because of something you have interpreted from this model.

Bias, Risks, and Limitations

Dataset Bias: The model is trained on the ISIC 2019 dataset, which predominantly features certain skin tones and lighting conditions. Its accuracy may degrade significantly on skin tones, lesion types, or image qualities not well-represented in the training data.

Resolution Limits: The model performs best when images are resized to a manageable resolution (e.g., 448x448) to avoid visual noise and CUDA Out-of-Memory errors during 4-bit inference.

Hallucinations: As an LLM-based system, the model may confidently output incorrect medical terminology or misclassify severe lesions.

How to Get Started with the Model

Use the code below to run local inference. Crucial Note: When running the base model in 4-bit precision, image pixel values must be explicitly cast to torch.float16 to prevent visual static/hallucinations.

import torch from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor, BitsAndBytesConfig, AutoTokenizer from peft import PeftModel from PIL import Image from qwen_vl_utils import process_vision_info import gc

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct" MODEL_PATH = "saad9694/Qwen2.5-VL-ISIC-Dermatology" IMAGE_PATH = "./your_test_image.jpg"

1. Load Base Model in 4-bit (Matches training config)

quant_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4" )

base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained( BASE_MODEL, quantization_config=quant_config, device_map="auto" )

2. Apply LoRA Adapters

model = PeftModel.from_pretrained(base_model, MODEL_PATH)

3. Load Base Processor & Finetuned Tokenizer

processor = AutoProcessor.from_pretrained(BASE_MODEL) processor.tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

4. Format Image and Prompt (Downscale to fit 4GB VRAM)

raw_image = Image.open(IMAGE_PATH) raw_image.thumbnail((448, 448)) TEMP_IMAGE_PATH = "./temp_resized.jpg" raw_image.save(TEMP_IMAGE_PATH)

messages = [ {"role": "user", "content": [ {"type": "image", "image": TEMP_IMAGE_PATH, "max_pixels": 200704}, {"type": "text", "text": "Analyze this dermoscopy image. What diagnostic category does this lesion fall into?"} ]} ]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) image_inputs, video_inputs = process_vision_info(messages)

inputs = processor( text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt" ).to("cuda")

CRITICAL FIX for 4-bit VLM: Convert image pixels to float16

if "pixel_values" in inputs: inputs["pixel_values"] = inputs["pixel_values"].to(torch.float16)

5. Generate Diagnosis

gc.collect() torch.cuda.empty_cache()

with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=128, use_cache=True, do_sample=False, # Strict clinical mode repetition_penalty=1.1 # Prevents text loops )

generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, outputs)] response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(f"AI DIAGNOSIS:\n{response}")

Training Details

Training Data

Trained on the ISIC 2019 dataset (MKZuziak/ISIC_2019_224 on Hugging Face), encompassing 25,332 dermoscopy images of various skin lesions. The categorical labels were mapped to descriptive conversational text to fine-tune the assistant's visual-linguistic alignment.

Training Procedure

The model was fine-tuned using the Unsloth library to apply QLoRA, drastically reducing VRAM usage.

Vision Encoder: Frozen

Language Layers: Fine-tuned (Adapters applied)

Hardware: Google Colab T4 GPU (16GB VRAM)

Training Hyperparameters

Training regime: bf16 mixed precision (if supported) / fp16

Rank (r): 16

LoRA Alpha: 16

Batch Size: 2 (with gradient accumulation of 4)

Learning Rate: 2e-4

Max Sequence Length: 2048

Optimization: 4-bit quantization with nf4 and double quantization.

Environmental Impact

Hardware Type: NVIDIA T4 Tensor Core GPU

Cloud Provider: Google Cloud (Colaboratory)