This model is a fine-tuned version of unsloth/gemma-4-e2b-it-unsloth-bnb-4bit on Occitan language text data using rs-LoRA (Rank-Stabilized LoRA) with the SFTTrainer from the TRL library.

v2 improvements over v1: Larger rank (32 vs 16), lower learning rate with cosine schedule and warmup, weight decay, and more training steps — resulting in significantly lower loss.

Model Description

Base Model: Gemma 4 E2B IT (Instruction-Tuned)
Fine-tuning Method: rs-LoRA (Rank-Stabilized Low-Rank Adaptation)
Training Framework: TRL SFTTrainer
Language: Occitan (Lengadocian dialect)
Model Type: Causal Language Model

Training Details

Hyperparameters (v2)

Parameter	Value
LoRA Rank (r)	32
LoRA Alpha	32
Training Steps	2535
Block Size	384 tokens
Learning Rate	1e-4
LR Scheduler	cosine
Warmup Steps	200
Weight Decay	0.01
Epochs	5
Gradient Accumulation	8
Optimizer	paged_adamw_8bit
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dropout	0
Bias	none

Changes from v1

Parameter	v1	v2
Rank	16	32
Steps	1270	2535
Learning Rate	2e-4	1e-4
Scheduler	linear	cosine
Warmup	0	200
Weight Decay	0	0.01
Gradient Accum.	16	8
Optimizer	AdamW	paged_adamw_8bit

Training Loss Progression

Step	Loss
10	1.571
100	0.539
500	0.415
1000	0.373
1500	0.322
2000	0.275
2530	0.247

Loss decreases steadily throughout training with a smooth cosine decay schedule, converging to ~0.25.

Repository Structure

├── adapter/                    # LoRA adapter weights (for loading with base model)
│   ├── adapter_config.json
│   ├── adapter_model.safetensors
│   ├── chat_template.jinja
│   ├── tokenizer.json
│   └── tokenizer_config.json
├── gguf/                       # GGUF quantized models (for llama.cpp)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q4_K_M.gguf  (3.43 GB)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q5_K_M.gguf  (3.63 GB)
│   ├── occitan-gemma-4-e2b-it-rslora-merged-Q8_0.gguf    (4.97 GB)
│   └── occitan-gemma-4-e2b-it-rslora-merged-f16.gguf    (9.31 GB)
├── config.json               # Model configuration
├── model.safetensors        # Merged 16-bit model (10.2 GB)
├── tokenizer.json
└── README.md

Usage

Option 1: Use the Merged Model (Recommended)

Load the full merged model directly:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer"
)

prompt = "Conta m'en una istueta brèva en occitan"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Option 2: Use with Base Model + LoRA Adapter

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gemma-4-e2b-it-unsloth-bnb-4bit",
    load_in_4bit=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "unsloth/gemma-4-e2b-it-unsloth-bnb-4bit"
)

model = PeftModel.from_pretrained(
    base_model,
    "julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer",
    subfolder="adapter"
)

Option 3: GGUF with llama.cpp

./llama-server -m occitan-gemma-4-e2b-it-rslora-merged-Q4_K_M.gguf -c 512 --port 8081

Quantization Options

File	Size	Description
Q4_K_M	3.43 GB	Recommended: Best quality/size tradeoff
Q5_K_M	3.63 GB	Higher quality, slightly larger
Q8_0	4.97 GB	Near-lossless quality
f16	9.31 GB	Full 16-bit precision (lossless)

Generated Text Examples

Prompt: Lo marin

Generated continuation:

òm auriá vist.
Lo marin òm auriá vist.
Mas la marè n'es tota luènha.
Al mens per far venir l'aiga,
Cal anar a la plaja,
E la marè n'es tota luènha.
La marè n'es tota luènha
Per la far venir,
Per la far venir.
Vèni, marin, vèni,
Porta-

Quality Notes

The model generates high-quality Lengadocian Occitan text with:

✅ Correct Lengadocian vocabulary (marin, auriá, marè, luènha, plaja)
✅ Proper grammar (conditional, imperfect subjunctive)
✅ No French/Catalan interference
✅ Natural narrative style

Training Data

Trained on a corpus of Occitan text in the Lengadocian dialect, including:

Traditional literature
Contemporary texts
Wikipedia articles
Cultural documents

Limitations

The model is specialized for Occitan text generation
Chat template support may vary depending on inference framework
For best results with instruction-following, use the text completion endpoint rather than chat endpoint in some frameworks (e.g., current llama.cpp)

Citation

If you use this model, please cite:

bibtex

@misc{occitan-gemma-4-e2b-it-rslora-sfttrainer-v2,
  author = {Julien},
  title = {Occitan Gemma 4 E2B IT - rs-LoRA SFTTrainer v2},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/julienp79/occitan-gemma-4-e2b-it-rslora-sfttrainer}
}