You can deploy this model and start earning money today!

DumbsterDrekk/
hindi-gpt-neo-2.7B

Name: hindi-gpt-neo-2.7B
Availability: InStock

safetensors

Views

No views yet

README: LoRA Fine-Tuning and Loading with PEFT

This README provides an overview of the Low-Rank Adaptation (LoRA) method for efficient fine-tuning of large language models, and demonstrates how to load a LoRA-adapted model using the Hugging Face Transformers and PEFT libraries.

What Is LoRA?
Key Benefits
Prerequisites
Installation
Loading a LoRA-Adapted Model
- Code Explanation
- Inference Example

What Is LoRA?

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that injects trainable low-rank matrices into the attention and feed-forward layers of a pre-trained Transformer model. Instead of updating all model parameters during fine-tuning, LoRA freezes the original weights and learns only the low-rank adapters. This approach dramatically reduces the number of trainable parameters and memory footprint while preserving model performance.

LoRA is especially useful for:

Resource Constrained Environments: Fine-tuning large models on consumer-grade GPUs.
Multi-Domain Fine-Tuning: Maintaining separate adapters for different tasks or languages.
Rapid Experimentation: Faster training times and lower storage requirements for adapters.

Key Benefits

Parameter Efficiency: Only adapter weights are trained (often <1% of total parameters).
Memory Savings: Reduced GPU memory usage during training and inference.
Modularity: Multiple adapters can be stacked or swapped without altering the base model.

Prerequisites

Python 3.8+
PyTorch with GPU support (CUDA) or CPU-only
transformers library v4.30.0+
peft library v0.3.0+

Installation

Install the required Python packages using pip:

pip install torch transformers peft

Loading a LoRA-Adapted Model

The following Python script demonstrates how to load a base causal language model and apply a LoRA adapter for inference.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1) Select device (GPU if available)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 2) Define model identifiers
BASE_MODEL = "EleutherAI/gpt-neo-2.7B"
ADAPTER_REPO = "DumbsterDrekk/hindi-gpt-neo-2.7B"

# 3) Load the base model in FP16 and move it to the selected device
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto",
)
base_model.to(device)
base_model.config.use_cache = True

# 4) Load the LoRA adapter from the Hub and merge it with the base model
model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_REPO,
    torch_dtype=torch.float16,
    device_map="auto",
    use_auth_token=True,  # if repository is private
)

# 5) Load the tokenizer (uses adapter repo to pick up special tokens/config)
tokenizer = AutoTokenizer.from_pretrained(
    ADAPTER_REPO,
    trust_remote_code=True
)

# 6) Perform inference with the LoRA-adapted model
prompt = "भारत एक महान राष्ट्र है क्योंकि"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Code Explanation

Device Selection: Automatically detects and uses a GPU if available; otherwise falls back to CPU.
Model Identifiers: BASE_MODEL is the pre-trained checkpoint; ADAPTER_REPO is the Hugging Face repo containing the LoRA adapter.
Base Model Loading:
- Loaded in 16-bit precision (torch.float16) for reduced memory usage.
- device_map="auto" splits the model across GPUs if multiple are present.
- use_cache=True enables faster generation by caching key/value states.
Adapter Loading:
- PeftModel.from_pretrained wraps the base model with the LoRA weights from ADAPTER_REPO.
- use_auth_token=True is needed if the repo is private.
Tokenizer Loading:
- Uses the adapter repo to ensure tokens and any special config are correctly applied.
Inference:
- Tokenizes the input prompt and moves it to the device.
- Generates up to 50 new tokens and decodes them back to text.