Safety note: Treat model output as a proposal only. For children, the final correct answer must come from deterministic Albanian rules / a curated database — not from the LLM alone.

Intended use

Generate short Albanian orthography exercise proposals, for example:

missing_letter — plotëso shkronjën
find_error — gjej gabimin
explain_error — shpjego gabimin (pedagogjikisht)

Usage (Python demo)

1) Install

pip install torch transformers peft accelerate sentencepiece

2) Load the adapter and generate

python

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER_ID = "greta44/albanian-spelling-lora"
BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_ID, use_fast=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(BASE_MODEL)
model = PeftModel.from_pretrained(model, ADAPTER_ID)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

seed_word = "mirë"
grade = 3
exercise_type = "missing_letter"  # or: find_error, explain_error

payload = {
    "seed_word": seed_word,
    "grade": grade,
    "difficulty": "easy",
    "exercise_type": exercise_type,
    "safety": "Kthe vetëm propozim; përgjigjja finale kontrollohet nga rregullat.",
}

prompt = (
    "### Instruksion:\n"
    f"Gjenero një ushtrim të sigurt për drejtshkrimin shqip. Kategoria: {exercise_type}. "
    f"Klasa: {grade}. Vështirësia: easy. Fjala bazë: {seed_word}.\n\n"
    "### Input:\n"
    + json.dumps(payload, ensure_ascii=False)
    + "\n\n### Përgjigje:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=120,
        temperature=0.4,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

text = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(text.strip())

3) Or use the included script

bash

python generate_exercises.py \
  --adapter greta44/albanian-spelling-lora \
  --seed-word mirë \
  --grade 3 \
  --type missing_letter

Training summary

Item	Value
Train / eval split	686 / 77 (from 763 AlbLingo exercises, seed=42)
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Max length	256
Effective batch size	4
Hardware	Apple M1 (MPS, float32)
Protocol target	3 epochs
Reported run	instrumented Phase A + Phase B continuation (~1.4 epoch-equivalent)
Train loss	3.149 → min 0.276
Held-out eval_loss	0.345
Approx. wall-clock	~62 minutes (logged span)

See training_manifest.json in this repository for the full measured summary, and training_loss.png for the loss curve.

Files in this repo

adapter_model.safetensors + adapter_config.json — LoRA weights
tokenizer* / vocab.json / merges.json — tokenizer files
generate_exercises.py — standalone demo script
training_manifest.json — hyperparameters + measured metrics
training_loss.png — training loss curve
instruction_dataset.jsonl — sample instruction pairs

Citation

If you use this model, please cite the related PhD research on Albanian educational NLP / AlbLingo and link this repository.

License

Apache 2.0