This repo contains a LoRA adapter for the "EleutherAI/gpt-neo-125m" which has been trained on the karpathy/tinystories-gpt4-clean dataset for text generation.

Model Information

Base model: EleutherAI/gpt-neo-125m
Fine-tuned on: karpathy/tinystories-gpt4-clean
Training Loop: Trained for 12,000 steps on a RTX 3050 (4GB VRAM)

Training Configuration

This adapter was trained with the following settings:

LoRA rank: 16
LoRA alpha: 32
Target modules: c_attn, c_proj
Dropout: 0.05
Bias: none
Task type: CAUSAL_LM
Precision: fp16
Gradient checkpointing: enabled
Learning rate: 2e-5
Batch size: 1
Gradient accumulation: 4
Save strategy: epoch

How to Use

Load the adapter from Hugging Face Hub

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "EleutherAI/gpt-neo-125m"
adapter_repo = "DireDreadlord/TinyStories-GPT-Neo-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_repo)

Notes

This repository stores only the LoRA adapter. The base model weights are not included here.
To use the adapter, you must load the same base model (EleutherAI/gpt-neo-125m).

Example Inference

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "EleutherAI/gpt-neo-125m"
adapter_path = "./gptn125-lora-tinystories"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_path)

text = "Once upon a time"
inputs = tokenizer(text, return_tensors="pt").input_ids
outputs = model.generate(inputs, max_length=150, do_sample=True, top_p=0.95, temperature=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files Included

adapter_model.safetensors — LoRA adapter weights
adapter_config.json — LoRA configuration and metadata
training_args.bin — training arguments saved by Hugging Face Trainer
README.md — model card and usage instructions