nl-nli-klein

At a glance

What to use this for

Primary use cases

1. Groundedness / hallucination check for Dutch RAG. For every sentence in an LLM-generated answer, check whether it entails from the retrieved context. If P(entailment) < threshold → flag or drop the sentence. Cheap enough to run on every generated token, local-only (no context leaves the device).
2. Zero-shot classification via the NLI-as-classifier pattern. Frame any classification task as entailment: premise = the document, hypothesis = "Dit document gaat over boekhouding" (or whatever label you want to check). P(entailment) becomes the class score. Lets you define new label sets without retraining.
3. Summary-faithfulness scoring. For each sentence in a summary, check it against the source document. Contradictions reveal hallucinations; neutrals reveal fabrication.
4. Paraphrase / duplicate detection. Two sentences entailing each other in both directions is a strong paraphrase signal. Useful for deduplicating knowledge bases, matching FAQ entries, or clustering support tickets.
5. Simple contradiction detection in user input. For moderation or QA pipelines where you want to flag when a user's statement contradicts a known fact.

What makes it suitable for these

• Local and private. Runs fully on-device. No data leaves the laptop / VM / cluster. Critical for GDPR-sensitive Dutch workloads (healthcare, legal, HR, government).
• Fast. ~184 pairs/sec on a single M1 CPU core. You can score every sentence of a streaming LLM output without adding perceivable latency.
• Small. 178 MB fits easily alongside an agent runtime, a local vector DB, and a lightweight generation model.
• Cheap. Zero marginal cost per inference. A cloud-based NLI-as-a-service at 1000 RAG queries/day × 10 sentences × 8 chunks = 80k pair-checks/day would cost real money; this model costs zero.

Quick start

Benchmark — SICK-NL test (4,906 pairs)

Head-to-head vs. zero-shot multilingual NLI models

vs. published Dutch NLI baselines (apples-to-apples)

• nl-nli-klein (shipped) — the 80k curated data mix used for this checkpoint. Beats RobBERT v1 at 37% its parameter count and lands within 1.3 points of BERTje at 42% its size.
• nl-nli-klein (strict) — SICK-NL train only (4,439 examples × 3 epochs), directly comparable to the literature baselines. Shows the honest size-performance tradeoff at identical training: 46M loses ~4 points to BERTje (109M) under matched fine-tuning.

Training

Data mix

Method

Note on SICK-NL label conventions

Reproducibility (evaluation protocol)

1. Test set: SICK-NL test split (4,906 examples) of maximedb/sick_nl. No filtering.
2. Canonical label order: {0: entailment, 1: neutral, 2: contradiction}. For any competitor, remap via its config.id2label before scoring (some models, e.g. joeddav/xlm-roberta-large-xnli, invert the order).
3. Per-model tokenizer: AutoTokenizer.from_pretrained(id, revision=<sha>). Never share tokenizers across models.
4. Tokenisation: tokenizer(premise, hypothesis, truncation='only_first', max_length=256, padding='max_length').
5. Precision: fp32 for all models. mDeBERTa does not support fp16.
6. Model loading: AutoModelForSequenceClassification + raw logits + argmax. Never pipeline("zero-shot-classification").
7. Device: CPU for accuracy (MPS has known numeric issues with DeBERTa-v2's disentangled attention).
8. For DeBERTa-v2 competitors, forward token_type_ids alongside input_ids / attention_mask — they're required for pair classification.

Hypothesis-only bias check

Limitations

What the model handles well

• Lexical-overlap entailment — removing or adding modifiers while keeping the main content. "Een hond rent in het park" → "Een hond rent" = entailment ✓
• Direct negation as contradiction — negation of a main verb or subject. "Een hond rent in het park" → "Er is geen hond in het park" = contradiction ✓
• Clearly unrelated pairs as neutral — when premise and hypothesis share no logical connection.
• Quantifier contradictions — "Niemand zit op de bank" → "Er zit iemand op de bank" = contradiction ✓

What the model does poorly

• Temporal / numeric reasoning. "Het contract gaat in op 1 mei 2026" → "Het contract start in mei" — the model predicted neutral (should be entailment). "Het contract start in maart" — the model predicted neutral (should be contradiction). Any task requiring reasoning over dates, numbers, or quantities is out of scope.
• Hyponym / hypernym chains. "Twee mannen spelen voetbal" → "Mensen spelen een sport" — the model predicted contradiction (should be entailment: men → people, football → sport). Category reasoning is brittle.
• Nuanced neutrality. "Een man loopt op straat" → "Een man werkt bij een bank" — the model predicted entailment (should be neutral). When topics are loosely related, the model tends toward entailment or contradiction rather than neutral.
• Semantic opposites without syntactic negation. "Het meisje lacht" / "Het meisje huilt" — the model predicted contradiction confidently, but without niet / geen the gold label is typically neutral in SICK convention.

When to pick klein vs. something else

• You need high-volume Dutch NLI at very low latency (every RAG chunk, every streaming LLM sentence).
• Your inputs look like SICK-style paraphrase pairs or SNLI-style simple entailment — lexical-overlap rewrites, direct negations, straightforward contradictions.

• Your inputs involve temporal, numeric, or categorical reasoning — use a larger Dutch LLM (Fietje / GEITje / Claude) with explicit prompting, not this classifier.
• You need confidence-calibrated predictions on hard-NLI corpora — klein's softmax is not calibrated for ANLI/WANLI-style adversarial pairs.
• You need broader linguistic coverage (Flemish, Frisian, code-mixed). klein is trained Dutch-only.

Other limitations

• SICK-NL is MT + manual-correction of the English SICK, not native Dutch re-annotation. The model and the benchmark inherit some translation-style artefacts, including a preference for lexical-overlap heuristics over deep semantic reasoning.
• e-SNLI-NL is fully machine-translated from English e-SNLI via OPUS-MT. Some hypothesis-only-bias risk is inherent in the training data; we measured it explicitly (0.570, at the SICK-NL majority-class baseline of 0.569 — so no evidence of shortcut learning on the benchmark, but adversarial prompts may exploit bias that the benchmark doesn't surface).
• Max input length is 256 tokens with premise-side truncation. Longer premises get truncated while the hypothesis is preserved.
• Dutch only. No evaluation on Flemish, Frisian, or code-mixed text.
• 3-class task only. If you need "weak entailment" or confidence stratification, use the raw softmax probabilities as an ordinal signal — but see the calibration caveat above.

Ethical considerations

• Outputs are probabilities over 3 classes; no claims about the world, only about logical relationships between two Dutch texts.
• For production hallucination-detection or groundedness decisions, combine with retrieval-quality checks and human-in-the-loop review for consequential outcomes.
• Runs fully locally; no data leaves the device at inference time.

Related models

• LokaalHub/nl-nli-middel — higher-accuracy tier (124M params, RobBERT-2023-base). Coming next.
• LokaalHub/nl-lokaal-klein and nl-lokaal-middel — sibling Dutch PII NER models.

Citation