This model is a LoRA fine-tuned version of EleutherAI's GPT-Neo 125M for medical reasoning and clinical QA-style generation.
It was fine-tuned using parameter-efficient training (LoRA) on the OpenMed/Medical-Reasoning-SFT-Mega dataset.
🔹 Only adapter weights are trained (base model not fully fine-tuned)
🔹 Optimized for instruction-style medical reasoning
🔹 Lightweight & efficient to run
📌 Model Details
Base Model: EleutherAI/gpt-neo-125M
Architecture: Causal Language Model
Fine-Tuning Method: LoRA (PEFT)
Task Type: Medical reasoning / QA generation
Training Objective: Next-token prediction (causal LM)
🧠 Training Setup
Dataset
Name: OpenMed/Medical-Reasoning-SFT-Mega
Split: 95% train / 5% validation
Downsampled:
40,000 training samples
5,000 validation samples
Reformatted into structured chat format:
🧠 Training Setup
Hyperparameters
Parameter
Value
Epochs
3
Batch Size
8
Gradient Accumulation
2
Learning Rate
2e-4
Block Size
256
Weight Decay
0.01
FP16
Enabled (if CUDA available)
LoRA Configuration
Parameter
Value
Rank (r)
8
Alpha
16
Dropout
0.05
Target Modules
q_proj, v_proj
Bias
None
Only a small percentage of total parameters were trainable (~<1%), making training efficient.
📊 Evaluation
Evaluation was performed on a held-out validation set.
Metric: Cross-entropy loss
Reported:
Eval Loss: (auto-filled during training)
Perplexity: exp(eval_loss)
Perplexity was calculated as:
ppl = exp(eval_loss)
🚀 Usage
Since this repo contains LoRA adapter weights, you must load it with the base model:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = "EleutherAI/gpt-neo-125M"
adapter = "ahmedrayan/medical_lora"
tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter)
prompt = "Common method by which bacteria can acquire new genetic material?"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
🎯 Intended Use
This model is intended for:
Medical reasoning research
Educational experimentation
Fine-tuning demonstrations
PEFT / LoRA learning projects
⚠️ Not intended for real clinical decision-making.
