SAGA (Symbolic Annotation for Grammar Alignment) uses symbolic NLP parsers as verifiable reward signals for RLHF-style training — we call this Reinforcement Learning from Verifiable Feedback (RLVF).

Generation — vLLM generates multiple completions per Wikipedia NB prompt using the base model at varied temperatures (0.7–1.1)
Scoring — SpaCy nb_core_news_lg scores each completion for grammatical quality
Pair filtering — Completions are paired into (chosen, rejected) by quality delta (threshold ≥ 0.25, min chosen score ≥ 0.2)
DPO training — Standard DPO on the filtered preference pairs

Training details

Hyperparameter	Value
Base model	`norallm/normistral-7b-warm-instruct`
Method	Δ-DPO (no prior SFT)
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q, k, v, o, gate, up, down proj
Trainable params	41.9M / 7.29B (0.58%)
Preference pairs	2,307
Mean chosen score	0.882
Mean rejected score	-0.923
Mean delta	1.805
Training steps	74
Final loss	0.218
Final rewards accuracy	95%
Batch size	8
Gradient accumulation	8
Reward model	SpaCy `nb_core_news_lg`

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "norallm/normistral-7b-warm-instruct",
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("norallm/normistral-7b-warm-instruct")

model = PeftModel.from_pretrained(base_model, "acbueff/normistral-7b-instruct-nb-delta-dpo-nosft")

messages = [{"role": "user", "content": "Skriv en kort tekst om Bergen."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))