The model was fine-tuned using LoRA (Low-Rank Adaptation) on a dataset of 10,000 medical cases with symptom-diagnosis pairs, achieving 99.0% training accuracy at checkpoint-600.

Key Features

🎯 High Accuracy: 99.0% training accuracy
🔬 Medical Specialization: Built on MedAlpaca, already optimized for medical domain
⚡ Efficient: Uses LoRA adapters (~67M parameters) on MedAlpaca-7B base
💾 Memory Efficient: Compatible with 4-bit quantization

Training Details

Base Model

Model: medalpaca/medalpaca-7b
Architecture: LLaMA-based medical language model
Pre-training: Medical domain knowledge

Dataset

Size: 10,000 symptom-diagnosis pairs
Format: Patient symptoms → Medical diagnosis
Train/Validation Split: Standard split with held-out validation set

Training Configuration

LoRA Hyperparameters:

LoRA Rank (r): 16
LoRA Alpha: 32
LoRA Dropout: 0.05
Target Modules: q_proj, v_proj

Training Hyperparameters:

Learning Rate: 2e-4
Batch Size: 4 per device
Gradient Accumulation Steps: 4
Effective Batch Size: 16
Number of Epochs: 3
Warmup Steps: 100
Optimizer: AdamW (8-bit)
Weight Decay: 0.01
Max Gradient Norm: 1.0
LR Scheduler: Linear with warmup

Training Environment:

GPU: NVIDIA A100 40GB
Precision: Mixed FP16
Quantization: 4-bit NF4 with double quantization
Framework: Hugging Face Transformers 4.45.0 + PEFT 0.12.0

Training Results

Final Checkpoint (checkpoint-600):

Training Loss: ~0.05
Training Accuracy: 99.0%
Total Training Steps: 600
Training Time: ~2 hours

Model Architecture

Base Model: medalpaca/medalpaca-7b (~7B parameters)
LoRA Adapters: ~67M trainable parameters
Model Size: ~64 MB (LoRA adapters only)
Architecture Type: LLaMA-based medical language model

Intended Use

Primary Use Cases

✅ Medical diagnosis prediction from symptom lists
✅ Clinical decision support systems (with medical oversight)
✅ Medical education and training
✅ Healthcare AI research
✅ Ensemble medical diagnosis systems

Out of Scope

❌ Direct patient care without medical professional oversight
❌ Emergency medical decisions
❌ Replacement for professional medical judgment

Usage

Installation

pip install transformers peft torch bitsandbytes accelerate

Loading the Model

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

# Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "medalpaca/medalpaca-7b",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Load fine-tuned adapter
model = PeftModel.from_pretrained(
    base_model,
    "Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("medalpaca/medalpaca-7b")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Inference Example

python

# Suppress warnings
import warnings
warnings.filterwarnings('ignore')
import os
os.environ['TRANSFORMERS_VERBOSITY'] = 'error'

# Patient symptoms
symptoms = "fever, cough, fatigue, body aches, headache"

# Format prompt
prompt = f"""Below is a patient case with symptoms. Provide ONLY the most likely diagnosis.

### Symptoms:
{symptoms}

### Diagnosis:
"""

# Generate
inputs = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        temperature=0.1,
        do_sample=False
    )

# Decode
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
diagnosis = result[len(prompt):].strip().split('\n')[0]

print(f"Diagnosis: {diagnosis}")

Performance Metrics

Metric	Value
Training Loss	~0.05
Training Accuracy	99.0%
Training Steps	600
Checkpoint	checkpoint-600

Model Comparison

Part of a 4-model ensemble for medical diagnosis:

Model	Base	Adapter Size	Training Acc	Checkpoint
BioMistral-7B	BioMistral	52 MB	99.1%	700
MedAlpaca-7B	MedAlpaca	64 MB	99.0%	600
MedGemma	Gemma-2B	35 MB	TBD	1100
BioGPT	BioGPT	12 MB	TBD	1100

Limitations

Limited to diagnostic categories in 10K training samples
Performance depends on accurate symptom description
Does not consider patient history, labs, or imaging
May not perform well on rare conditions
English language only
Requires medical professional interpretation

Ethical Considerations

⚠️ Medical AI Ethics:

Should never replace professional medical judgment
Requires appropriate medical oversight in clinical settings
Users must understand model limitations
Clinical validation needed for real-world deployment

⚠️ Bias Considerations:

Training data may reflect diagnostic biases
Performance may vary across demographics
Regular monitoring recommended for production use

Citation

If you use this model, please cite:

bibtex

@misc{chauhan2024medalpaca,
  author = {Sugandha Chauhan},
  title = {MedAlpaca-7B Fine-tuned for Medical Diagnosis},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis}}
}

Disclaimer

⚠️ IMPORTANT MEDICAL DISCLAIMER

This AI model is for educational and research purposes only. It is NOT:

A substitute for professional medical advice, diagnosis, or treatment
Approved for clinical use without medical oversight
Intended for emergency medical situations
A replacement for qualified healthcare providers

Always consult a physician or qualified healthcare provider for medical decisions.

License

Released under the same license as MedAlpaca-7B. See base model for details.

Contact

Author: Sugandha Chauhan
Repository: https://huggingface.co/Sugandha-Chauhan/MedAlpaca-SymptomDiagnosis
Model Portfolio: 4-model ensemble for medical diagnosis

Model Version: 1.0 (Checkpoint-600)
Last Updated: November 2024
Part of: Multi-model Medical Diagnosis System