A QLoRA adapter fine-tuned on Kila (Kirjanpitolautakunta) corpus — Finland's official accounting standards board. The adapter specialises Qwen2.5-7B-Instruct on Finnish accounting statements, exemptions, and general guidelines (yleisohjeet) to improve factual accuracy on Finnish GAAP questions.

Model Details

Base model: Qwen/Qwen2.5-7B-Instruct
Adapter type: LoRA (QLoRA 4-bit quantised training)
Language: Finnish (fi)
Domain: Finnish accounting standards (Kila / kirjanpitolaki)
Fine-tuning method: Supervised Fine-Tuning (SFT) on synthetic instruction data generated from Kila corpus chunks
Framework: PEFT 0.19.1 + TRL + Transformers

Training Details

Training Data

Synthetic Finnish Q&A pairs generated from Kila corpus chunks (statements, exemptions, general guidelines). Data pipeline:

Load Kila chunks from sources/kila/{date}/chunks_part-*.parquet
Deduplicate by chunk_hash
Generate 1–2 Finnish instruction/response pairs per chunk via Groq llama-3.3-70b-versatile
Split into train/val by document_id (no document appears in both splits)

Dataset size:

Train: synthetic_kila_train.jsonl (~1,878 KB)
Val: synthetic_kila_val.jsonl (~233 KB)

Training Hyperparameters

Parameter	Value
Training regime	bf16 mixed precision
LoRA rank (`r`)	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Epochs	3
Batch size (per device)	2
Gradient accumulation steps	4
Learning rate	2e-4
LR scheduler	cosine
Warmup steps	10
Max sequence length	2048
Optimizer	paged_adamw_32bit

Hardware

GPU: NVIDIA RTX 5090 (32 GB VRAM) — sm_120, 1x GPU
Blackwell-specific settings: bf16=True, gradient_checkpointing_kwargs={"use_reentrant": False}

Evaluation

Judge: Groq llama-3.3-70b-versatile (temp=0.0)

Rubric (1–5 scale):

correctness — factual accuracy per Kila / kirjanpitolaki
grounding — references authoritative sources
fluency — professional Finnish grammar

Results are reported in eval_summary.csv comparing base model vs fine-tuned adapter on ~12 Finnish accounting questions grounded in specific Kila documents.

How to Get Started

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "rumeshmohan/kila-qwen2.5-7b-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

messages = [
    {"role": "system", "content": "Olet suomalaisen kirjanpidon asiantuntija."},
    {"role": "user", "content": "Miten tutkimusmenot käsitellään kirjanpidossa?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.1,
        top_p=0.9,
        do_sample=True
    )

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Framework Versions

PEFT 0.19.1
Transformers ≥ 4.40.0
TRL ≥ 0.8.6
PyTorch ≥ 2.3.0