Fine-Tuning Language Models with Unsloth and Xformers
This repository contains code for fine-tuning a language model using Unsloth and Xformers. The code is designed to facilitate efficient model training and inference on large datasets, with a focus on optimizing memory usage and training speed.
Overview
This script demonstrates the process of installing necessary packages, loading a pre-trained model from the Unsloth repository, and fine-tuning the model using a specific dataset. The model used in this example is "Meta-Llama-3.1-8B," which has been quantized to 4-bit precision to reduce memory usage and improve training efficiency.
Installation
The following packages are required for the fine-tuning process:
unsloth: A library for fast language model fine-tuning.xformers: A library that provides efficient attention mechanisms, including Flash Attention.trl, peft, accelerate, bitsandbytes: Libraries for training large models.
To install these packages, use the following commands:
1!pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
2!pip install --no-deps "xformers<0.0.27" "trl<0.9.0" peft accelerate bitsandbytes
3!pip uninstall xformers
4!pip install xformers
5!python -m xformers.info
6!pip install triton
Model Configuration
The model configuration includes setting parameters for maximum sequence length, data type detection, and 4-bit quantization. The model is loaded from the Unsloth repository and configured with LoRA (Low-Rank Adaptation) for fine-tuning.
1from unsloth import FastLanguageModel
2import torch
3
4max_seq_length = 2048
5dtype = None # Auto detection
6load_in_4bit = True # Use 4bit quantization
7
8model, tokenizer = FastLanguageModel.from_pretrained(
9 model_name="unsloth/Meta-Llama-3.1-8B",
10 max_seq_length=max_seq_length,
11 dtype=dtype,
12 load_in_4bit=load_in_4bit,
13)
Dataset Preparation
The dataset used for fine-tuning is the Alpaca dataset, which is formatted to include instructions, inputs, and expected responses. The dataset is processed to fit the model’s input requirements.
1from datasets import load_dataset
2
3alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
4
5### Instruction:
6{}
7
8### Input:
9{}
10
11### Response:
12{}"""
13
14dataset = load_dataset("yahma/alpaca-cleaned", split="train")
Fine-Tuning Process
The model is fine-tuned using the SFTTrainer from the trl library. Key training parameters include batch size, learning rate, and the number of training steps.
1from trl import SFTTrainer
2from transformers import TrainingArguments
3
4trainer = SFTTrainer(
5 model=model,
6 tokenizer=tokenizer,
7 train_dataset=dataset,
8 args=TrainingArguments(
9 per_device_train_batch_size=2,
10 gradient_accumulation_steps=4,
11 max_steps=120,
12 learning_rate=2e-4,
13 output_dir="outputs",
14 ),
15)
16trainer.train()
Evaluation
Before and after training, the model is tested using predefined prompts to compare its performance. The outputs demonstrate the model's ability to generate responses based on the given instructions and inputs.
Example Output Before Training
1# Sample prompt for testing before training
2inputs = tokenizer(
3 [
4 alpaca_prompt.format(
5 "Based on the information provided, rewrite the sentence by changing its tense from past to future.",
6 "She played the piano beautifully for hours and then stopped as it was midnight.",
7 "",
8 )
9 ], return_tensors="pt"
10).to("cuda")
11
12outputs = model.generate(**inputs, max_new_tokens=128)
Example Output After Training
1# Sample prompt for testing after training
2outputs = model.generate(**inputs, max_new_tokens=128)
Model Saving
The fine-tuned model can be saved locally or pushed to the Hugging Face Hub for public access.
1model.push_to_hub("HadeelHegazi/lora_model_num_train_epochs1max_steps120", token="hf_your_token")
2tokenizer.push_to_hub("HadeelHegazi/lora_model_num_train_epochs1max_steps120", token="hf_your_token")
Errors
1. CUDA Out of Memory Error:
- Cause: Your GPU doesn't have enough memory to allocate for the current task.
- Solutions:
- Reduce batch size: Lowering the batch size during training or inference can help reduce memory usage.
- Clear unused variables: Ensure you're clearing any unused variables or using
torch.cuda.empty_cache() to free up memory.
- Use
PYTORCH_CUDA_ALLOC_CONF: Set the environment variable PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to manage memory fragmentation. This can be done in your script or in your environment setup.
- Switch to mixed precision: If applicable, you could try using mixed-precision training (e.g., with
torch.cuda.amp) to reduce memory usage.
2. Xformers Installation Error:
- Cause: Xformers was installed for a different version of PyTorch and CUDA than what you have in your environment.
- Solutions:
- Reinstall Xformers: Ensure that you install Xformers compatible with your current PyTorch and CUDA versions. You can do this by running:
1pip uninstall xformers
2pip install xformers --no-cache-dir
- Check your environment: Verify that the CUDA and PyTorch versions are consistent with the requirements of Xformers. For example, make sure that the version of PyTorch matches the one Xformers was built for.
- Confirm installation: After reinstalling, confirm that Xformers is installed correctly by running:
Steps to Resolve:
- Adjust your code to reduce GPU memory usage.
- Reinstall Xformers with the correct versions to match your setup.
- Use environment variables to handle memory more effectively if fragmentation is an issue.
Conclusion
This script provides a comprehensive guide to fine-tuning a large language model using Unsloth and Xformers. The included code can be easily adapted to different datasets and models for various NLP tasks.
base_model: unsloth/meta-llama-3.1-8b-bnb-4bit
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
Uploaded model
- Developed by: HadeelHegazi
- License: apache-2.0
- Finetuned from model : unsloth/meta-llama-3.1-8b-bnb-4bit
This llama model was trained 2x faster with
Unsloth and Huggingface's TRL library.
