Training Configuration
Quantization
- 4-bit NF4 Quantization
- Double Quantization Enabled
- Compute Type: FP16
LoRA Configuration
- Rank (r): 32
- Alpha: 64
- Dropout: 0.05
- Bias: None
Target Modules
- q_proj
- k_proj
- v_proj
- o_proj
- gate_up_proj
- down_proj
Training Hyperparameters
- Epochs: 2
- Learning Rate: 1e-4
- Batch Size: 4
- Gradient Accumulation Steps: 4
- Max Sequence Length: 512
- Warmup Ratio: 0.03
- Optimizer: AdamW
- Evaluation Strategy: Step-based Validation
Evaluation Results
Evaluation was performed on a held-out validation set.
Perplexity
| Model | Perplexity |
|---|
| Base Phi-3 Mini | 4.0056 |
| Fine-Tuned Model | 1.8500 |
Improvement:
- 53.8% reduction in perplexity
ROUGE-L
| Model | ROUGE-L |
|---|
| Base Phi-3 Mini | 0.1731 |
| Fine-Tuned Model | 0.3060 |
Improvement:
- 76.7% increase in ROUGE-L
BERTScore
| Model | BERTScore F1 |
|---|
| Base Phi-3 Mini | 0.8250 |
| Fine-Tuned Model | 0.8436 |
Key Findings
- Significant reduction in perplexity on medical validation samples.
- Improved overlap with reference medical answers measured by ROUGE-L.
- Better adaptation to healthcare-specific terminology and concepts.
- Achieved strong performance gains using parameter-efficient fine-tuning without modifying the full 3.8B model.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE_MODEL = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base_model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL
)
model = PeftModel.from_pretrained(
base_model,
"YOUR_USERNAME/phi3-medical-qlora"
)
prompt = """
Instruction:
What are the symptoms of diabetes?
Input:
Response:
"""
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=150
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Limitations
- This model is intended for educational and research purposes.
- Outputs should not be used as professional medical advice.
- The model may generate incorrect or outdated medical information.
- Human verification by qualified healthcare professionals is recommended.
Disclaimer
This model is not a medical device and should not be used for diagnosis, treatment, or clinical decision-making.
Author
Javvadi Naga Venkata Anjaneya Prasad
Acknowledgements
- Microsoft Phi-3 Team
- Hugging Face
- PEFT
- TRL
- Medical Meadow Dataset Contributors
