You can deploy this model and start earning money today!

mia-project-2025/
pythia-1B-LoRA-writing-prompts

Name: pythia-1B-LoRA-writing-prompts
Availability: InStock

safetensors

2106.09685

apache-2.0

Views

No views yet

Pythia-1B with LoRA on WritingPrompts

This model applies LoRA (Low-Rank Adaptation) fine-tuning to EleutherAI/pythia-1b using the WritingPrompts dataset.
The model is trained for causal language modeling on story prompts with LoRA adapters instead of full fine-tuning, making it parameter-efficient.

Model Details

Base model: EleutherAI/pythia-1b
Fine-tuning method: LoRA
LoRA Config:
- Rank (r): 8
- LoRA Alpha: 64
- Target Modules: query_key_value, dense
- LoRA Dropout: 0.1
Batch size: 8 (gradient accumulation = 2)
Learning rate: 2e-5 with cosine scheduler
Epochs: 10
Optimizer: AdamW
Precision: FP16
Framework: Hugging Face Transformers + PEFT

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from peft import PeftModel

# Load base model and tokenizer
model_name = "EleutherAI/pythia-1b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model = AutoModelForCausalLM.from_pretrained(model_name)

# Load LoRA-adapted model
model = PeftModel.from_pretrained(base_model, "mia-project-2025/pythia-1B-LoRA-writing-prompts")

# Run inference
generator = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
print(generator("Once upon a time", max_length=50))

---

## Training Results

## Final results after 10 epochs:

Train Loss: 2.8867

Eval Loss: 2.8017

Eval Perplexity: 16.47

## Training performance:

Train runtime: 826.76s

Train samples/sec: 96.83

Train steps/sec: 1.52

## Evaluation performance (epoch 10):

Eval runtime: 3.24s

Eval samples/sec: 144.21

Eval steps/sec: 4.63