OpenVINO int8 IR of MoritzLaurer/mDeBERTa-v3-base-mnli-xnli (DeBERTa-v2 NLI cross-encoder), quantized for CPU inference as the NLI entailment scorer (entailment = class index 0) in a single-engine semantic grounding pipeline. The int8 preserves the fp32 max-over-chunks grounding signal at pearson 0.9863 (full-gold 0.9841) - no measurable quality loss.

Model

Base model - MoritzLaurer/mDeBERTa-v3-base-mnli-xnli (DeBERTa-v2 NLI cross-encoder)
Format - OpenVINO IR (openvino_model.xml + openvino_model.bin), int8
Quantization - NNCF SmoothQuant (alpha 0.7) + Fast Bias Correction
int8 parity vs fp32 - pearson 0.9863 (full-gold 0.9841) on the max-over-chunks grounding score
Size - 318 MB int8 (fp32 1.12 GB)
Role - NLI entailment scorer (entailment = class index 0)
Runtime - OpenVINO on CPU (x86-64 Intel/AMD via AVX2 / AVX-512-VNNI; ARM/Graviton via the OpenVINO ARM plugin)

Usage

python

from huggingface_hub import snapshot_download
import numpy as np, openvino as ov
from transformers import AutoTokenizer

d = snapshot_download("stellars/mdeberta-v3-base-mnli-xnli-openvino-int8")
core = ov.Core()
model = core.compile_model(core.read_model(f"{d}/openvino_model.xml"), "CPU")
tok = AutoTokenizer.from_pretrained(d)

enc = tok(["a sentence"], ["another sentence"], return_tensors="np",
          padding=True, truncation=True, max_length=512)
feed = {"input_ids": enc["input_ids"].astype(np.int64),
        "attention_mask": enc["attention_mask"].astype(np.int64)}
logits = model(feed)[model.output(0)]

Quantization details

Method - NNCF SmoothQuant (alpha 0.7) + Fast Bias Correction, via NNCF / optimum-intel
Calibration - 128 (claim, chunk) pairs sampled from 3 selected agentic chat RAG datasets
Why this matters - DeBERTa-v2 disentangled attention is int8-hostile (plain dynamic int8 collapses to pearson 0.35); NNCF SmoothQuant migrates the activation outliers into the per-channel weights, which is the only int8 method that preserves the signal. The standard-attention bge models quantize cleanly with plain NNCF int8

License

Inherits the mit license of the base model.