LoRA adapters (rank=16) fine-tuned on MedQA-USMLE with QLoRA. Achieves 78.48% accuracy on the MedQA-US test set — within 3 points of GPT-4 (83.97%). Combined with a FAISS RAG pipeline over medical textbooks, reaches 80.99%.

Model	MedQA-US Accuracy
GPT-4 — Xiong et al. (ACL 2024)	83.97%
Llama 3.3 70B + QLoRA + RAG	80.99%
Llama 3.3 70B + QLoRA (this model)	78.48%
Llama 3.3 70B base	74.45%
GPT-3.5 — Xiong et al. (ACL 2024)	65.04%

Full code and RAG pipeline: github.com/MartinLiarte/medqa-llm

How to Get Started

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "martinarra31/medrag-llama33-70b-qlora")
model.eval()

Training Details

Training Data

MedQA-USMLE-4-options — 9,160 train / 1,018 val / 1,273 test (4-option USMLE format).

Training Hyperparameters

Parameter	Value
LoRA rank	16
LoRA alpha	32
Target modules	q, k, v, o, gate, up, down proj
Learning rate	5e-5
LR scheduler	cosine
Effective batch size	64 (1 × 16 grad accum × 4 GPUs)
Epochs	3
Max sequence length	1024
Quantization	4-bit NF4 (QLoRA)

Training Infrastructure

Hardware: 4× NVIDIA L40S 48GB
Multi-GPU: DeepSpeed ZeRO-2
Training time: 4h 18min

Evaluation

Evaluated on the MedQA-US test set (1,273 questions, greedy decoding).

Fine-tuned only: 78.48%
Fine-tuned + RAG (FAISS over 18 medical textbooks): 80.99% Reference: Xiong et al., Benchmarking Retrieval-Augmented Generation for Medicine, ACL 2024. arxiv:2402.13178

Bias, Risks, and Limitations

These adapters are intended for research purposes only. The model should not be used for real clinical decision-making. Performance was evaluated on USMLE-style multiple choice questions and may not generalize to other medical tasks or real patient scenarios.

Model Card Authors

Martín Liarte — CS student at UPV (Valencia), specializing in Computing & AI.

Framework versions

PEFT 0.13.0
TRL 1.7.0
Transformers 4.46.0