Gemma2 Fine-Tuned LoRA Model
Overview
This is a LoRA (Low-Rank Adaptation) fine-tuned model based on the unsloth/gemma-7b-bnb-4bit base model. It has been adapted for a tipification analysis task similar to the Llama-3.2-3B-Instruct LoRA fine-tuning, where the model classifies text into categories such as "ESTAFA," "ROBO," "HURTO," and their "TENTATIVA DE" variations.
During fine-tuning, only specific adapter layers were trained (~50 million parameters), while the rest of the base model was frozen. This approach allows parameter-efficient training, significantly reducing computational costs.
Key Features
- Base Model:
unsloth/gemma-7b-bnb-4bit
- Task Type: Causal Language Modeling (
CAUSAL_LM)
- LoRA Parameters:
r: 16lora_alpha: 16lora_dropout: 0.0
- Target Modules:
gate_proj, up_proj, down_proj, k_proj, q_proj, o_proj, v_proj
- Number of Trainable Parameters: 50,003,968
- Training Loss & Validation Loss:
- Observed over 117 steps (1 epoch).
- See table below for detailed step-by-step values.
Dataset Distribution
This model was fine-tuned on the same dataset as the Llama-3.2-3B-Instruct LoRA version, with the following category distribution:
| Category | Count | Percentage |
|---|
| ESTAFA | 4610 | 47.3% |
| ROBO | 2307 | 23.7% |
| HURTO | 2141 | 22.0% |
| TENTATIVA DE ESTAFA | 306 | 3.1% |
| TENTATIVA DE ROBO | 272 | 2.8% |
| TENTATIVA DE HURTO | 113 | 1.2% |
| Total | 9749 | 100% |
Although the dataset has nearly 10K examples in this summary table, the fine-tuning run used an extended version (~15K examples) for this particular training session.
Training Details
- Hardware: Single GPU A100 40Gb
- Num Examples: ~15,000
- Epochs: 1
- Batch Size per Device: 32
- Gradient Accumulation Steps: 4
- Effective Total Batch Size: 128
- Total Steps: 117
- Number of Trainable Parameters: 50,003,968
Training and Validation Loss
Below is a snapshot of how training and validation loss evolved during the single epoch (117 steps):
| Step | Training Loss | Validation Loss |
|---|
| 10 | 2.974900 | 4.242294 |
| 20 | 5.451000 | 4.526450 |
| 30 | 4.150400 | 3.632928 |
| 40 | 3.036100 | 2.615031 |
| 50 | 2.492900 | 2.178700 |
| 60 | 2.095400 | 1.886430 |
| 70 | 2.099200 | 1.548187 |
| 80 | 1.983100 | 2.104600 |
| 90 | 2.020900 | 1.526225 |
| 100 | 1.727700 | 1.699223 |
| 110 | 1.868300 | 1.716561 |
| ... | ... | ... |
Final training concluded at step 117.
We observe a steady decrease in both training and validation losses, indicating the model was converging throughout the single epoch.
Deployment Instructions
You can use this LoRA fine-tuned model with the Hugging Face Transformers library. Below is an example of how to load and run the model for text generation or classification-like tasks:
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3tokenizer = AutoTokenizer.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
4model = AutoModelForCausalLM.from_pretrained("Petermoyano/unsloth-gemma-7b-bnb-4bit-LoRA-Tipification-CausalLM-16R-16Alpha-1Epoch")
5
6input_text = "TENTATIVA DE ESTAFA:"
7inputs = tokenizer(input_text, return_tensors="pt")
8outputs = model.generate(**inputs, max_length=50)
9print(tokenizer.decode(outputs[0], skip_special_tokens=True))
