- Developed by: Tushar Kamthe
- License: apache-2.0
- Finetuned from model : unsloth/gemma-2-2b-it-bnb-4bit
Gemma-2B QLoRA Fine-tuned Model
Model Description
This model is a fine-tuned version of Google's Gemma-2B using the QLoRA (Quantized Low-Rank Adaptation) technique. The model was trained using parameter-efficient fine-tuning (PEFT), where only LoRA adapters were trained while keeping the base model weights frozen.
The model is designed for instruction-following text generation tasks.
Fine-tuning was performed using:
- QLoRA (4-bit quantization)
- LoRA adapters
- HuggingFace Transformers
- PEFT
- Unsloth for faster training
Base Model
Base model used for training:
google/gemma-2b
Training Details
Training Method
The model was trained using QLoRA, which enables efficient training of large language models by:
- Loading the base model in 4-bit quantized format
- Training LoRA adapter weights only
- Keeping base model weights frozen
This significantly reduces GPU memory requirements.
Training Configuration
| Parameter | Value |
|---|
| Method | QLoRA |
| Quantization | 4-bit (NF4) |
| LoRA Rank (r) | 16 |
| LoRA Alpha | 64 |
| LoRA Dropout | 0.05 |
| Optimizer | AdamW |
| Precision | bfloat16 |
| Framework | HuggingFace Transformers |
Hardware
Training was performed on:
- GPU: NVIDIA GPU (Colab / Local GPU)
- Framework: PyTorch
- Libraries:
- transformers
- peft
- datasets
- unsloth
Dataset
The model was fine-tuned on a custom instruction dataset containing prompt-response pairs.
Dataset format:
