You can deploy this model and start earning money today!

mia-project-2025/
pythia-1B-LoRA-MIMIC-III

safetensors

apache-2.0

Views

No views yet

Pythia-1B LoRA Fine-Tuned on MIMIC-III (Impressions)

This repository contains a LoRA fine-tuned version of EleutherAI’s Pythia-1B model, trained on the MIMIC-III dataset.
The goal of this model is to adapt a general-purpose causal language model to the clinical domain for improved text generation on medical discharge summaries.

Model Details

Base model: EleutherAI/pythia-1b
Parameter-efficient fine-tuning: LoRA (r=8, α=64, dropout=0.1)
Dataset: hejazizo/mimic-iii
Task: Causal Language Modeling (CLM)
Max sequence length: 1024 tokens
Precision: Mixed (FP16)

We trained using LoRA adapters, then merged them into the full model. Both adapters and the merged version are provided.

Training Setup

Batch size (per device): 8
Gradient accumulation steps: 2
Effective batch size: 16
Learning rate: 2e-4
Epochs: 10
Weight decay: 0.01
Hardware: Multi-GPU (NVIDIA CUDA)
Framework: PyTorch + Hugging Face Transformers + PEFT

Training & Evaluation Results

Training Summary:

json

{
  "train_runtime": 4671.0191,
  "train_samples_per_second": 63.498,
  "train_steps_per_second": 0.993,
  "train_loss": 2.2328,
  "epoch": 10.0
}

## Final Evaluation Metrics (Epoch 10):

Metric	Value
eval_loss	2.3736
eval_perplexity	10.7356
eval_runtime	46.8561
eval_samples_per_second	139.3200
eval_steps_per_second	4.3540
epoch	10.0

mia-project-2025/pythia-1B-LoRA-MIMIC-III

Pythia-1B LoRA Fine-Tuned on MIMIC-III (Impressions)

Model Details

Training Setup

Training & Evaluation Results

mia-project-2025/
pythia-1B-LoRA-MIMIC-III