This is a LoRA (Low-Rank Adaptation) adapter for the Gemma 3 12B Instruct model, fine-tuned on the German legal question-answering dataset (GerLayQA paraphrased).

Model Details

Base Model: unsloth/gemma-3-12b-it
Model Type: LoRA Adapter
Language: German (de)
Domain: Legal Question-Answering
Training Dataset: GerLayQA (paraphrased version)

LoRA Configuration

Rank (r): 16
Alpha: 32
Dropout: 0.05
Target Modules: Attention and MLP layers (q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj)
Task Type: Causal Language Modeling

Training Details

Sequence Length: 4096
Batch Size: 32
Learning Rate: 2e-05
Epochs: 7
Checkpoint: 154

Usage

To use this adapter with the base model:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gemma-3-12b-it",
    device_map="auto",
    torch_dtype="auto"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("unsloth/gemma-3-12b-it")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "DomainLLM/gemma-3-12b-it-german-geralayqa-paraphrased-lora")

# Generate text
prompt = "Was ist ein Werkvertrag nach deutschem Recht?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Merging the Adapter (Optional)

If you want to merge the LoRA weights into the base model:

python

from transformers import AutoModelForCausalLM
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/gemma-3-12b-it",
    device_map="auto",
    torch_dtype="auto"
)

# Load and merge LoRA adapter
model = PeftModel.from_pretrained(base_model, "DomainLLM/gemma-3-12b-it-german-geralayqa-paraphrased-lora")
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged-gemma-german-legal")

Applications

This model is designed for:

German legal question answering
Legal document understanding
Legal reasoning and analysis
German civil law (BGB) questions
Legal paragraph interpretation

Limitations

The model is specialized for German legal domain
Performance may vary on non-legal or non-German texts
Should not be used as a replacement for professional legal advice

Citation

If you use this model, please cite:

bibtex

@misc{gemma-german-legal-lora,
  title={Gemma 3 12B IT - German Legal QA LoRA Adapter},
  author={DomainLLM},
  year={2025},
  howpublished={\url{https://huggingface.co/DomainLLM/gemma-3-12b-it-german-geralayqa-paraphrased-lora}}
}

License

This adapter is released under the Apache 2.0 License, consistent with the base Gemma model.