This repository contains a fine-tuned version of Google's Gemma 4 12B Instruct with a QAT-aware base (gemma-4-12B-it-qat-q4_0-unquantized), optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Adaptation) via SFTTrainer.

The QAT (Quantization-Aware Training) base model was originally quantized to Q4_0 and then dequantized back to bf16, meaning the weights carry QAT-aware characteristics that may improve robustness to post-training quantization.

Repository Structure

Root Directory: Full merged Safetensors weights (bfloat16, compatible with transformers)
adapter/ Folder: The raw RS-LoRA adapter files (for use with PEFT)
gguf/ Folder: Quantized GGUF versions for local inference via llama.cpp, LM Studio, Ollama, etc.

Usage

Using the merged model (Transformers)

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "user", "content": "Escrivètz un cort paragraf en occitan sus la lenga occitana e son importància."},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using the RS-LoRA adapter with PEFT

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12B-it-qat-q4_0-unquantized",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12B-it-qat-q4_0-unquantized")

model = PeftModel.from_pretrained(base_model, "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer", subfolder="adapter")
model.eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Escrivètz un cort paragraf en occitan."}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using GGUF quantizations (llama.cpp)

bash

llama-cli -hf julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer:Q4_K_M \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Or via the GGUF file directly:

bash

llama-cli -m gguf/occitan-gemma-4-12b-it-rslora-qat-sfttrainer-Q4_K_M.gguf \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Adapter Details

Parameter	Value
Adapter Type	RS-LoRA (`use_rslora=True`)
Base Model	`google/gemma-4-12B-it-qat-q4_0-unquantized` (QAT-aware base)
LoRA Rank (r)	32
LoRA Alpha	32
Target Modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Dropout	0
Bias	None
Training Method	SFT (`trl.SFTTrainer`)

Training Details

Data

The model was trained on raw Occitan text from four categories:

Literary — Occitan literature texts
Journalistic — news articles in Occitan
Grammar — grammatical reference texts
Encyclopedic — encyclopedic entries in Occitan

All texts were chunked to 384-token blocks (causal language modeling, no chat templating applied during training).

Hyperparameters

Parameter	Value
Block Size	384 tokens
Learning Rate	5e-5 (cosine schedule)
Optimizer	Paged AdamW 8-bit
Max Gradient Norm	1.0
Warmup Steps	400
Weight Decay	0.01
Epochs	5
Gradient Accumulation Steps	8
Per-device Batch Size	1
Total Steps	~2535
Final Loss	~1.80
Total Tokens Processed	~7.8M

Hardware & Quantisation

GPU: RTX 3060 12GB VRAM
Base Model: QAT-aware q4_0 (dequantized to bf16)
Training Quantisation: 4-bit fp4 (BitsAndBytes) — matches QAT base
Compute dtype: bfloat16
Gradient Checkpointing: enabled (use_reentrant=True)
Vision/Audio Embedders: stripped during training to reclaim VRAM (not needed for text-only fine-tuning)

Framework Versions

PEFT 0.19.1
TRL: 1.4.0
Transformers: 5.10.0.dev0
PyTorch: 2.7.0+cu128
Datasets: 4.8.5
Tokenizers: 0.22.2

GGUF Quantizations

Available in gguf/:

Quant	Size	Description
`f16`	~23 GB	Lossless float16
`Q8_0`	~12 GB	Near-lossless 8-bit
`Q5_K_M`	~8 GB	High quality 5-bit
`Q4_K_M`	~6.9 GB	Recommended 4-bit (best quality/size)
`Q2_K`	~4.5 GB	Aggressive 2-bit (constrained hardware)

Training procedure

This model was trained with SFT (Supervised Fine-Tuning) using SFTTrainer on raw Occitan text. No chat templating was applied during training — the model learns language structure via causal language modeling on chunked text. The RS-LoRA variant helps stabilise training at higher ranks by scaling the LoRA update by alpha / sqrt(r) instead of alpha / r.

The QAT base model (gemma-4-12B-it-qat-q4_0-unquantized) was quantized to Q4_0 and dequantized back to bf16 before training, meaning the base weights carry quantization-aware characteristics. Combined with fp4 training quantization, this approach aims to produce a model that is more robust to post-training quantization (e.g., GGUF Q4_K_M or Q2_K).

Framework versions

PEFT 0.19.1
TRL: 1.4.0
Transformers: 5.10.0.dev0
PyTorch: 2.7.0+cu128
Datasets: 4.8.5
Tokenizers: 0.22.2

Citations

bibtex

@software{geyronneau2024trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {Geyronneau, Quentin and von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2024}
}