The model was trained using LoRA (Low-Rank Adaptation) on a balanced corpus of Occitan texts. This is the largest Occitan Gemma model to date, offering significant improvements in reasoning and linguistic nuance.

🛠️ Training Engineering

Training a 12B model on an RTX 3060 (12GB VRAM) required surgical optimizations:

Memory Management: Utilized paged_adamw_8bit to allow optimizer states to spill into system RAM. Vision and audio embedders were stripped to reclaim VRAM.
Context Window: Block size set to 384 tokens to reduce activation memory overhead.
Quantization: Loaded in 4-bit NormalFloat (NF4) with Double Quantization enabled.
Gradient Checkpointing: Enabled with use_reentrant=True.
Effective Batch Size: 8 (batch 1 × 8 gradient accumulation steps).

Training Details

Parameter	Value
Base model	`google/gemma-4-12B-it`
LoRA rank	8
LoRA alpha	16
LoRA dropout	0
Target modules	`.language_model.(q_proj\|k_proj\|v_proj\|o_proj\|gate_proj\|up_proj\|down_proj)`
Optimizer	`paged_adamw_8bit`
Learning rate	5e-5
Scheduler	Cosine, 400 steps warmup
Epochs	5
Weight decay	0.01
Max grad norm	1.0
Framework	transformers + PEFT (no SFTTrainer, no unsloth)

Dataset

Balanced Occitan corpus across four categories:

Literary — prose and poetry
Journalistic — news and articles
Grammar — grammatical examples and exercises
Encyclopedic — factual and reference texts

Training was performed as raw causal language modeling (no chat wrapping).

📁 Repository Structure

Root Directory: Full merged Safetensors weights (compatible with transformers, accelerate, etc.).
/gguf Folder: Quantized versions for local inference via LM Studio, Ollama, or llama.cpp.
/adapter Folder: The raw LoRA adapter files for researchers who wish to inspect the weights or perform their own merges.

🚀 How to Use

Using Transformers (Python)

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "julienp79/occitan-gemma-4-12b-it-lora"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="bfloat16",
)

messages = [
    {"role": "user", "content": "Pòdes m'ajudar a escriure un pichon tèxt en occitan?"},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

GGUF Quantisation

Quant	Size	Description
Q2_K	~4.5 GB	Aggressive, for very constrained hardware
Q4_K_M	~7 GB	Main recommendation, best quality/size tradeoff
Q5_K_M	~8 GB	Higher quality, for users with more RAM
Q8_0	~11 GB	Near-lossless, for power users
f16	~23 GB	Full precision (source quant)

🤖 About Gemma 4

Gemma 4 is Google's latest family of lightweight open models, built from the same research and technology used to create the Gemini models. This 12B version offers strong reasoning capabilities while remaining suitable for local deployment.