Pythia-1B AdaLoRA Fine-Tuned on MIMIC-III

Model Details

• Base model: EleutherAI/pythia-1b
• Fine-tuning method: AdaLoRA (Adaptive Low-Rank Adaptation)
• Dataset: hejazizo/mimic-iii (impressions field)
• Task: Causal Language Modeling (CLM)
• Sequence length: 1024 tokens
• Precision: Mixed (FP16)

Training Setup

• Batch size (per device): 8
• Gradient accumulation steps: 2
• Effective batch size: 16
• Learning rate: 2e-4
• Weight decay: 0.01
• Epochs: 20
• LoRA settings:
  • r=8, init_r=12, target_r=4
  • lora_alpha=64, lora_dropout=0.05
  • Orthogonal regularization weight = 0.5
• Frameworks: Hugging Face Transformers, PEFT, PyTorch
• Hardware: Multi-GPU (CUDA)

Results