Gemma-2-2B LoRA Fine-tuned Model
A LoRA fine-tuned version of Google's Gemma-2-2B-IT model, optimized for specific prompt-response tasks.
Model Details
Model Description
This model is a fine-tuned version of Google's Gemma-2-2B-IT using LoRA (Low-Rank Adaptation). It has been trained on a custom dataset to improve performance on targeted conversational and question-answering tasks.
- Developed by: crobins
- Model type: Causal Language Model (LoRA Fine-tuned)
- Language(s): English
- Finetuned from model: google/gemma-2-2b-it
Uses
Direct Use
- Conversational AI applications
- Question-answering tasks
- Text generation following the Gemma chat format
Out-of-Scope Use
- Real-time factual information without verification
- Generation of harmful, biased, or inappropriate content
- Tasks significantly different from conversational/QA format
Training Details
Training Data
- Source: Custom CSV dataset with prompt-response pairs
- Format: Gemma conversational template format
- Max sequence length: 128 tokens
Training Hyperparameters
- Epochs: 30
- Batch size: 16 (per device)
- Gradient accumulation steps: 4
- Learning rate: 2e-5
- Weight decay: 0.01
- Warmup steps: 10
- LoRA rank (r): 16
- LoRA alpha: 32
- LoRA dropout: 0.05
- Target modules: q_proj, v_proj
- Precision: fp16
Limitations
- Performance limited by the quality and scope of the custom training dataset
- Trained with 128 token limit, may not handle longer contexts optimally
- May inherit biases from the base Gemma-2-2B-IT model
Framework versions
