Pirate-Language LLM ⚓️
Quantization & Fine-Tuning TinyLlama on a 50k+ Instruction Dataset
📖 Project Overview
This project demonstrates fine-tuning and quantization of the TinyLlama model on a custom dataset of 50k+ instruction-response pairs. The objective was to train a model capable of converting standard English queries into pirate-style responses.
The project highlights how resource-constrained environments (like 4GB GPUs) can still be used effectively to fine-tune large language models using parameter-efficient fine-tuning (LoRA) and quantization (BitsAndBytes).
✨ Key Features
Dataset: 50k+ instruction-response pairs formatted into JSONL for supervised fine-tuning.
Model: TinyLlama fine-tuned with LoRA adapters.
Quantization: 4-bit quantization using BitsAndBytes for reduced memory usage.
Training:
Locally on NVIDIA GTX 1650 (4GB VRAM).
Scaled up to Google Colab T4 (16GB VRAM) for large dataset training.
Optimizations:
Gradient checkpointing
Cosine learning rate scheduler
Mixed precision (bfloat16)
Checkpoint Management: Saved models on Google Drive and Hugging Face Hub to mitigate Colab’s 12-hour GPU session limits.
🚀 Tech Stack
Languages/Frameworks: Python, PyTorch
Libraries: Hugging Face Transformers, TRL, PEFT, LoRA, BitsAndBytes
Compute: CUDA, cuDNN, Google Colab, Local GPU (GTX 1650)
📂 Repository Structure
Pirate-Language-LLM/
│── data/ # Dataset files (JSONL format)
│── notebooks/ # Jupyter notebooks for experiments
│── scripts/ # Training and evaluation scripts
│── checkpoints/ # Saved model checkpoints
│── README.md # Project documentation
│── requirements.txt # Dependencies
🔧 Setup & Installation
Clone the repository:
Install dependencies:
pip install -r requirements.txt
Download dataset (or use your own):
from datasets import load_dataset
dataset = load_dataset("json", data_files="pirate_data_large.jsonl")
Fine-tune the model:
python scripts/train.py
📊 Results
Successfully fine-tuned TinyLlama to translate English into pirate-style text.
Achieved stable training on both local GPU (4GB) and Colab T4 (16GB).
Demonstrated practical quantization + LoRA fine-tuning workflow on limited compute.
🔮 Future Work
Extend the approach to Indian languages, building a voice-to-text and text-to-voice model with Indian accent support.
Pre-quantize larger multilingual models and fine-tune on diverse datasets.
Enable real-time conversational systems with efficient deployment on constrained hardware.
📎 Model & Resources
Hugging Face Model Link
(replace with your link)
🤝 Contribution
Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
📜 License
This project is licensed under the MIT License.
