You can deploy this model and start earning money today!

mia-project-2025/
pythia-1B-LoRA-wikitext-103

Name: pythia-1B-LoRA-wikitext-103
Availability: InStock

safetensors

apache-2.0

Views

No views yet

Pythia-1B + LoRA Fine-Tuning on WikiText-103

This repository contains a LoRA fine-tuned version of EleutherAI/pythia-1b trained on the WikiText-103 dataset.
LoRA adapters were applied to efficiently fine-tune the model for causal language modeling while keeping the majority of parameters frozen.

Model Description

Base Model: EleutherAI/pythia-1b
Fine-tuning Method: Low-Rank Adaptation (LoRA)
Task: Causal Language Modeling
Dataset: WikiText-103 (raw v1)

LoRA introduces trainable rank-decomposition matrices into the attention and dense layers of the model, allowing efficient fine-tuning with reduced compute and memory costs compared to full parameter updates.

Training Setup

Framework: Transformers + PEFT + PyTorch
Adapter Method: LoRA
Target Modules: query_key_value, dense
LoRA Config: r=8, alpha=64, dropout=0.1
Batch size: 8 (gradient accumulation: 2)
Sequence length (block size): 1024
Optimizer: AdamW
Learning rate: 2e-5 with cosine decay
Epochs: 10
Precision: FP16
Callbacks: Early stopping, custom metric logging

Results

Final Training Metrics

Training Loss: 2.4572
Final Step Loss: 2.4434
Gradient Norm: 0.4280
Learning Rate at End: 1.56e-07

Evaluation Metrics (Epoch 10)

Evaluation Loss: 2.4363
Evaluation Perplexity: 11.43
Evaluation Runtime: 2.2418s
Samples per Second: 107.50
Steps per Second: 3.57

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from peft import PeftModel

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("./pythia-wikitext-lora")

# Load base model and LoRA adapters
base_model = AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-1b")
model = PeftModel.from_pretrained(base_model, "./pythia-wikitext-lora")
model.eval()

# Example
input_text = "The history of natural language processing"
inputs = tokenizer(input_text, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_length=50)
    print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))

mia-project-2025/pythia-1B-LoRA-wikitext-103