Gemma 3 4B-IT WHO-SFT QLoRA Adapter
This repository contains a QLoRA adapter for
unsloth/gemma-3-4b-it-unsloth-bnb-4bit.
It was created as part of the MSc thesis
Construction of Medical Question-Answering Training Collections and Fine-Tuning of Large Language Models for Reliable Medical Question Answering.
Training corpus
The adapter was trained with WHO-SFT, a collection of 4,057 English medical question-answer pairs generated from 237 WHO Fact Sheets. Questions and answers were generated from source text in chunks, structurally validated, deduplicated, and converted to the same supervised fine-tuning format as Drugs-SFT.
Code, data-preparation scripts, experiment definitions, and evaluation outputs are available at:
Training configuration
- Base model: Gemma 3 4B-IT quantized to 4 bits.
- Adaptation method: QLoRA.
- LoRA rank: 8.
- LoRA alpha: 8.
- Learning rate: 5e-5.
- Epochs: 1.
- Maximum sequence length: 1,024 tokens.
Evaluation
Evaluation used the full MIRAGE benchmark with constrained selection among the valid answer options for each question.
| Subset | Base model | WHO-SFT adapter |
|---|
| MedQA | 47.45% | 44.30% |
| MedMCQA | 44.66% | 43.01% |
| PubMedQA | 48.80% | 56.00% |
| BioASQ | 70.71% | 66.83% |
| Medical MMLU | 57.12% | 54.45% |
| Overall | 49.26% | 47.62% |
Loading the adapter
This repository contains adapter weights only. Load the compatible base model first and apply this adapter with PEFT.
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base_model_id = "unsloth/gemma-3-4b-it-unsloth-bnb-4bit"
5adapter_id = "citiusLTL/gemma-3-4b-it-who-sft-qlora"
6
7tokenizer = AutoTokenizer.from_pretrained(adapter_id)
8base_model = AutoModelForCausalLM.from_pretrained(
9 base_model_id,
10 device_map="auto",
11)
12model = PeftModel.from_pretrained(base_model, adapter_id)
Limitations and intended use
This adapter is intended solely for research. It is not a clinical device and must not be used for diagnosis, treatment, or medical decision-making. MIRAGE evaluates constrained answers to closed questions; its results do not constitute clinical validation.
