Model Card for Llama 3.2 3B Instruct — Primary Healthcare Fine-tune
Fine-tuned version of unsloth/Llama-3.2-3B-Instruct on a large medical QA dataset for primary healthcare, developed as part of the CSCI 566 (Deep Learning) course project at USC.
Model Details
Model Description
This model is a LoRA fine-tune of Llama 3.2 3B Instruct, trained on ~146K medical multiple-choice QA examples to improve performance on clinical reasoning tasks relevant to primary healthcare. Both LoRA adapter weights and quantized GGUF files are provided.
Model type: Causal Language Model (LoRA fine-tune)
Answer medical multiple-choice questions and provide evidence-based clinical reasoning for primary healthcare queries.
Downstream Use
Can be further fine-tuned or used as a backbone for medical decision-support tools, clinical education assistants, or healthcare chatbots.
Out-of-Scope Use
This model is not intended for direct clinical diagnosis or as a replacement for qualified medical professionals. It should not be used for emergency medical advice or high-stakes clinical decisions without human oversight.
Bias, Risks, and Limitations
The model may produce plausible-sounding but incorrect medical information.
Performance may degrade on rare conditions, non-English queries, or highly specialized subspecialties.
Recommendations
Always consult a qualified healthcare professional for personal medical advice. Outputs from this model should be treated as informational only and verified against authoritative clinical sources.
~146,354 medical multiple-choice QA examples in ShareGPT chat format, covering a broad range of primary care topics including internal medicine, pharmacology, and clinical reasoning. A validation split of ~5,548 examples was held out during training.
The system prompt used during training:
"You are a knowledgeable medical AI assistant. Provide accurate, evidence-based medical information. Always recommend consulting a qualified healthcare professional for personal medical advice."
Training Procedure
Training Hyperparameters
Training regime: bf16 mixed precision
Fine-tuning method: LoRA via Unsloth + TRL SFTTrainer
Held-out validation split (~5,548 examples) from the same medical QA distribution as training data, plus standard medical benchmarks (MedQA, MedMCQA).
Metrics
Accuracy on multiple-choice medical QA benchmarks
Results
Benchmark
Base Model
Fine-tuned
CareQA
55.20%
53.80%
HEADQA
57.70%
60.36%
MedQA
52.08%
50.20%
MMLU (Medical)
61.95%
59.75%
MMLU-Pro (Medical)
37.65%
32.89%
PubMedQA
73.90%
70.10%
Summary
Fine-tuning Llama 3.2 3B Instruct yielded mixed results. HEADQA improved noticeably (+2.66%), while most other benchmarks saw small decreases. The relatively small performance gaps suggest the fine-tune partially adapted to the training distribution without severely degrading base capabilities. PubMedQA remains the strongest benchmark for this model, indicating good biomedical literature comprehension even after fine-tuning.
Technical Specifications
Model Architecture and Objective
Decoder-only transformer (Llama 3.2 3B Instruct) with LoRA adapters applied to attention and MLP projection layers. Trained with next-token prediction (SFT) on chat-formatted medical QA data.
