You can deploy this model and start earning money today!

sabber/
medphi-medical-qa-adapter

Name: medphi-medical-qa-adapter
Availability: InStock

safetensors

Views

No views yet

MediPhi Medical QA Adapter

This is a LoRA adapter fine-tuned on Microsoft's MediPhi-Instruct for medical question-answering. The model is designed to provide comprehensive, accurate answers to questions about medical diseases, conditions, and health-related topics.

Model Description

Model Type: LoRA Adapter for Causal Language Model
Base Model: microsoft/MediPhi-Instruct (3.8B parameters)
Trainable Parameters: 0.328% (12.5M parameters via LoRA)
Language: English
Domain: Medical/Healthcare
Task: Question Answering, Conversational AI
License: Apache 2.0

Model Purpose

This model serves as a medical assistant chatbot capable of answering user queries about medical conditions, diseases, symptoms, treatments, and genetic disorders. It has been fine-tuned on 16,406 medical Q&A pairs covering a wide range of health topics including rare genetic disorders and common medical conditions.

Key Features

Medical Domain Expertise: Trained on diverse medical Q&A covering diseases and conditions
Comprehensive Responses: Generates detailed explanations including definitions, causes, symptoms, and treatments
Step-by-Step Reasoning: Employs structured thinking for medical information delivery
Efficient Fine-tuning: Uses 4-bit quantization with LoRA for memory efficiency
Patient Education Focus: Optimized for explaining complex medical concepts clearly

Training Data

Dataset Statistics

Total Q&A Pairs: 16,406 medical question-answer pairs
Dataset Size: 21 MB
Data Splits:
- Train: 12,304 samples (75%)
- Validation: 2,051 samples (12.5%)
- Test: 2,051 samples (12.5%)

Data Coverage

The dataset covers a wide range of medical topics including:

Rare Genetic Disorders: Tourette syndrome, Denys-Drash syndrome, etc.
Common Conditions: Dry eye syndrome, immunodeficiency disorders
Medical Concepts: Genetic inheritance patterns, diagnostic methods
Treatment Information: Management strategies, preventive care

Data Format

python

{
    "messages": [
        {
            "role": "system",
            "content": "You are a knowledgeable medical assistant. Provide accurate information about medical conditions and diseases. Always think step by step."
        },
        {
            "role": "user",
            "content": "What is [medical condition]?"
        },
        {
            "role": "assistant",
            "content": "[Comprehensive medical explanation]"
        }
    ]
}

Training Details

Training Configuration

Framework: PyTorch with Hugging Face Transformers
Fine-tuning Method: LoRA (Low-Rank Adaptation) with SFT (Supervised Fine-Tuning)
Quantization: 4-bit NF4 with double quantization
Compute: Single RTX 5090 GPU (16 vCPU, 141 GB RAM)
Training Time: ~140 steps to convergence

LoRA Hyperparameters

python

{
    "r": 8,
    "lora_alpha": 32,
    "target_modules": ["o_proj", "qkv_proj", "gate_up_proj", "down_proj"],
    "lora_dropout": 0.05,
    "bias": "none",
    "task_type": "CAUSAL_LM"
}

Training Hyperparameters

python

{
    "num_train_epochs": 3,
    "per_device_train_batch_size": 4,
    "gradient_accumulation_steps": 8,
    "learning_rate": 2e-4,
    "lr_scheduler_type": "cosine",
    "max_seq_length": 1024,
    "optim": "adamw_torch",
    "gradient_checkpointing": True,
    "packing": True
}

Quantization Configuration

python

{
    "load_in_4bit": True,
    "bnb_4bit_quant_type": "nf4",
    "bnb_4bit_use_double_quant": True,
    "bnb_4bit_compute_dtype": "bfloat16"
}

Performance

Training Convergence

Step	Training Loss	Validation Loss
20	1.1799	0.8300
40	0.7834	0.7168
60	0.7185	0.6892
80	0.6838	0.6710
100	0.6641	0.6592
120	0.6638	0.6515
140	0.6441	0.6446

Key Observations:

Rapid convergence within 140 training steps
Training and validation loss converged, indicating good generalization
No significant overfitting observed

Qualitative Improvements

Example 1 - Dry Eye Condition

Original Dataset Response: Citations and contact information only

Fine-tuned Model Response: Comprehensive explanation covering:

Definition and mechanism
Environmental, aging, and medication-related causes
Symptoms (gritty sensation, redness, blurred vision, light sensitivity)
Treatment options (artificial tears, lifestyle modifications, medical interventions)

Example 2 - Genetic Disorders

Original Dataset Response: Basic definition of 3 types

Fine-tuned Model Response: Expanded information including:

Inheritance patterns (autosomal dominant/recessive, X-linked, mitochondrial)
Specific examples (cystic fibrosis, sickle cell disease, Huntington's disease, Down syndrome)
Diagnostic methods and genetic testing
Management strategies and treatment approaches
Prevention through genetic counseling

Usage

Installation

pip install torch transformers peft bitsandbytes accelerate

Basic Usage

python

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Load model and tokenizer
model = AutoPeftModelForCausalLM.from_pretrained(
    "sabber/medphi-medical-qa-adapter",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("sabber/medphi-medical-qa-adapter")

# Prepare conversation
messages = [
    {
        "role": "system",
        "content": "You are a knowledgeable medical assistant. Provide accurate information about medical conditions and diseases. Always think step by step."
    },
    {
        "role": "user",
        "content": "What is Type 2 Diabetes and what are its main symptoms?"
    }
]

# Tokenize and generate
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Pipeline Usage

python

from transformers import pipeline

# Create conversational pipeline
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

# Ask medical question
messages = [
    {"role": "system", "content": "You are a knowledgeable medical assistant. Provide accurate information about medical conditions and diseases. Always think step by step."},
    {"role": "user", "content": "What causes high blood pressure?"}
]

result = pipe(messages)
print(result[0]['generated_text'][-1]['content'])

Multi-Turn Conversation

python

conversation_history = [
    {
        "role": "system",
        "content": "You are a knowledgeable medical assistant. Provide accurate information about medical conditions and diseases. Always think step by step."
    }
]

# First question
conversation_history.append({"role": "user", "content": "What is asthma?"})
response = generate_response(conversation_history)
conversation_history.append({"role": "assistant", "content": response})

# Follow-up question
conversation_history.append({"role": "user", "content": "What triggers asthma attacks?"})
response = generate_response(conversation_history)
print(response)

Merging Adapter with Base Model

python

from peft import AutoPeftModelForCausalLM

# Load and merge
model = AutoPeftModelForCausalLM.from_pretrained(
    "sabber/medphi-medical-qa-adapter",
    torch_dtype="auto",
    device_map="auto"
)
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("medphi-medical-qa-merged")
tokenizer.save_pretrained("medphi-medical-qa-merged")

System Prompt

The model uses the following system prompt for optimal performance:

You are a knowledgeable medical assistant. Provide accurate information about
medical conditions and diseases. Always think step by step.

This prompt encourages:

Structured reasoning: Step-by-step explanations
Accuracy focus: Emphasis on providing correct medical information
Comprehensive coverage: Detailed responses covering multiple aspects

Limitations and Bias

Limitations

Training Data Scope: Model trained on 16,406 Q&A pairs; may not cover all medical conditions
Not a Medical Professional: Cannot replace professional medical advice or diagnosis
Language: English only
Clinical Validation: Outputs should be reviewed by healthcare professionals before clinical application
Rare Conditions: Performance may vary for extremely rare or newly discovered conditions
Quantization Effects: 4-bit quantization may affect precision in certain edge cases

Bias Considerations

Dataset Bias: Training data may reflect biases present in medical literature
Language Bias: Trained exclusively on English medical content
Regional Bias: May reflect medical practices and terminology from specific regions
Completeness: May provide more detailed responses for well-documented conditions

Ethical Considerations

Not for Diagnosis: This model should NOT be used for self-diagnosis or medical decision-making
Professional Review Required: All outputs must be reviewed by qualified healthcare professionals
Patient Safety: Users should always consult with licensed medical professionals for health concerns
Transparency: Users should be informed when AI-generated medical content is provided
Privacy: Do not share personally identifiable health information when using this model

Intended Use

Primary Use Cases

✅ Medical Education: Teaching medical concepts and terminology ✅ Patient Information: Providing general information about conditions and diseases ✅ Research Assistant: Helping researchers understand medical concepts ✅ Content Generation: Creating draft content for medical education materials ✅ Conversational AI: Building medical information chatbots and assistants

Out-of-Scope Use

❌ Clinical Diagnosis: Not validated for diagnostic purposes ❌ Treatment Planning: Not suitable for creating treatment plans ❌ Emergency Response: Not appropriate for emergency medical situations ❌ Prescription Decisions: Cannot be used for medication recommendations ❌ Mental Health Crisis: Not designed for crisis intervention or counseling ❌ Legal/Medical Records: Not validated for official medical documentation

Evaluation Benchmarks

The model has been prepared for evaluation on standard medical benchmarks:

MEDQA: Medical Question Answering benchmark
MEDMCQA: Multiple Choice Medical Questions
PubMedQA: Biomedical literature question answering
MMLU Medical Subsets:
- Anatomy
- Clinical Knowledge
- College Medicine
- Medical Genetics
- Professional Medicine

Note: Comprehensive benchmark results will be added as evaluation completes.

Future Improvements

Suggested enhancements based on current limitations:

Increase LoRA Rank: Higher rank for greater model capacity
Full Precision Training: Use FP32 or FP16 instead of 4-bit quantization
Data Augmentation: Expand training data with more diverse medical sources
Error Analysis: Systematic analysis of model failure cases
Benchmark Evaluation: Complete evaluation on medical QA benchmarks
Multi-lingual Support: Extend to support multiple languages
Clinical Validation: Formal evaluation by medical professionals

Model Architecture

Base Model: MediPhi-Instruct (Phi-3.5-mini-instruct)

Key Components:

Parameters: 3.8 billion
Vocabulary Size: 32,064 tokens
Hidden Dimension: 3,072
Layers: 32 Phi3DecoderLayers
Attention: Multi-head self-attention with rotary positional embeddings
Activation: SiLU (Swish) activation function
Normalization: RMSNorm layer normalization

LoRA Target Modules:

o_proj - Output projection in attention
qkv_proj - Query-Key-Value projection in attention
gate_up_proj - Gate and up projection in MLP
down_proj - Down projection in MLP

Citation

If you use this model in your research, please cite:

bibtex

@misc{medphi-medical-qa-adapter,
  author = {Sabber Ahamed},
  title = {MediPhi Medical QA Adapter: LoRA Fine-tuning for Medical Question Answering},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/sabber/medphi-medical-qa-adapter}},
  note = {Fine-tuned on 16,406 medical Q&A pairs for patient education and medical information retrieval}
}

Please also cite the base MediPhi model:

bibtex

@article{medphi2024,
  title={MediPhi: A Medical Language Model},
  author={Microsoft Research},
  journal={arXiv preprint},
  year={2024}
}

Model Card Authors

Sabber Ahamed

Model Card Contact

For questions, issues, or feedback, please:

Open an issue on the model repository
Contact via Hugging Face profile

Acknowledgments

Base Model: Microsoft MediPhi-Instruct team
Framework: Hugging Face Transformers, PEFT, and TRL libraries
Compute: GPU infrastructure for model training
Community: Open-source ML and medical NLP communities

Additional Resources

Training Code: Available in project repository
Evaluation Scripts: Provided for reproducibility
Documentation: Comprehensive README with implementation details

Medical Disclaimer: This model is provided for educational and research purposes only. It is NOT approved for clinical use, medical diagnosis, or treatment planning. All medical information should be verified by qualified healthcare professionals. In case of medical emergencies, contact emergency services immediately. Always consult with licensed medical professionals for health concerns and treatment decisions.

Technical Disclaimer: This model may generate incorrect or incomplete information. Users should verify all outputs and use appropriate safeguards when deploying in production environments. The model's responses should be reviewed and validated before any public-facing use.