Views
No views yet
.png)
unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit base model, leveraging Unsloth's 4-bit quantization for 60% memory reduction while maintaining high accuracy.| Aspect | Base Model (unsloth/Llama-3.2-11B-Vision-Instruct) | LumiChats Fine-Tuned Model |
|---|---|---|
| Accuracy | ✅ Identifies image type (Panoramic Radiograph) | ✅ Exact identification + precise pathology |
| Specificity | ❌ Hallucinates details (fractures, misalignments) | ✅ Focuses on ground truth (osteolytic lesion) |
| Medical Terminology | ⚠️ General terms, some inaccuracies | ✅ Professional clinical language |
| Output Length | 📝 Long, speculative descriptions | 📝 Concise, actionable reports |
| Clinical Relevance | ❌ Includes irrelevant details | ✅ Pathology-focused analysis |
meta-llama/Llama-3.2-11B-Vision-Instruct1lora_r = 16
2lora_alpha = 16
3lora_dropout = 0.0
4
5# Comprehensive layer fine-tuning
6finetune_vision_layers = True # Vision encoder layers
7finetune_language_layers = True # Language model layers
8finetune_attention_modules = True # Attention mechanisms
9finetune_mlp_modules = True # Feed-forward networks1per_device_train_batch_size = 2
2gradient_accumulation_steps = 4
3max_steps = 30
4learning_rate = 2e-4
5optimizer = "adamw_8bit"
6lr_scheduler = "linear"pip install transformers torch accelerate bitsandbytes1from transformers import AutoModelForCausalLM, AutoProcessor
2import torch
3
4model_id = "lumichats/LumiChats-Llama-3.2-11B-Vision-Instruct-4bit"
5
6# Load with 4-bit quantization
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 torch_dtype=torch.bfloat16,
10 device_map="auto",
11 load_in_4bit=True,
12 bnb_4bit_compute_dtype=torch.bfloat16
13)
14
15processor = AutoProcessor.from_pretrained(model_id)1import requests
2from PIL import Image
3
4# Load medical image
5image_url = "https://example.com/panoramic_radiograph.jpg"
6image = Image.open(requests.get(image_url, stream=True).raw)
7
8# Prepare prompt
9prompt = """You are an expert radiographer. Analyze this medical image and provide a professional clinical description focusing on pathology and anatomical findings."""
10
11# Process
12inputs = processor(text=prompt, images=image, return_tensors="pt")
13
14# Generate
15with torch.no_grad():
16 outputs = model.generate(
17 **inputs,
18 max_new_tokens=150,
19 do_sample=True,
20 temperature=0.1,
21 top_p=0.95,
22 pad_token_id=processor.tokenizer.eos_token_id
23 )
24
25# Decode
26response = processor.decode(outputs[0], skip_special_tokens=True)
27print(response)1@misc{lumichats-llama32-vision-11b-4bit,
2 author = {LumiChats Team},
3 title = {LumiChats-Llama-3.2-11B-Vision-Instruct-4bit: A Specialized Radiology Assistant},
4 year = {2024},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/lumichats/LumiChats-Llama-3.2-11B-Vision-Instruct-4bit}}
7}unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit