Dynamic int8-quantized ONNX export of nrl-ai/vn-spell-correction-small. 75 % smaller on disk (530 MB safetensors → 307 MB ONNX int8) and no PyTorch dependency at inference time — runs on plain onnxruntime for CPU / browser / mobile deployment.

Quality on the OOD eval (n=150, hand-curated)

Same 6-slice OOD eval the source model was measured against (nrl-ai/vn-spell-correction-eval-real):

Slice	This (int8)	Source (fp32)	Δ
`forum_25`	64.36 %	64.64 %	-0.28 pp
`mobile_25`	94.18 %	95.29 %	-1.11 pp
`telex_real_25`	15.91 %	16.45 %	-0.54 pp
`ocr_25`	94.21 %	94.19 %	+0.02 pp
`legal_real_25`	93.54 %	93.54 %	+0.00 pp
`news_real_25`	91.34 %	91.34 %	+0.00 pp
Aggregate	77.30 %	77.55 %	-0.25 pp

Quantization cost on aggregate: -0.25 pp word accuracy. Within the bootstrap CI overlap of the source model — no measurable quality loss.

Disk size

Format	Size
Source safetensors (PyTorch fp32)	530 MB
ONNX fp32 (export, before quant)	1220 MB
ONNX int8 (this artifact)	307 MB

The fp32 ONNX export is larger than the safetensors because it unrolls the decoder twice (with-cache and without-cache paths). After int8 weight quantization, the total is comfortably under the PyTorch baseline.

Loading

python

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("nrl-ai/vn-spell-correction-small-onnx-int8")
model = ORTModelForSeq2SeqLM.from_pretrained("nrl-ai/vn-spell-correction-small-onnx-int8")

inp = tok("Toi yeu Viet Nam, dat nuoc tuyet voi", return_tensors="pt")
out = model.generate(**inp, max_length=128, num_beams=1)
print(tok.decode(out[0], skip_special_tokens=True))
# "Tôi yêu Việt Nam, đất nước tuyệt vời"

pip install optimum[onnxruntime]

No PyTorch dependency required at inference time — optimum pulls onnxruntime (and transformers for the tokenizer / config).

When to use this vs the source model

Use this when shipping to CPU-only servers, edge devices, browser (via onnxruntime-web), or mobile (onnxruntime-mobile). The 307 MB / no-PyTorch footprint matters there.
Use nrl-ai/vn-spell-correction-small when running on GPU and PyTorch is already in the deployment. CUDA-accelerated fp16 will out-throughput int8 ONNX on a modern GPU.

Limitations

Same training distribution as the source. All caveats from the source model card apply — in-distribution synthetic eval over-states real-world performance, Vietnamese forum slang and real Telex keystrokes are still the hardest slices.
Dynamic int8 only. Static int8 (with calibration on a held-out set) could squeeze further size at risk of quality. Not done here because the dynamic version already meets the no-quality-loss bar.
Beams = 1 verified. Beam search > 1 should work but isn't benched in this card.

Reproduce

bash

git clone https://github.com/nrl-ai/nom-vn.git
cd nom-vn
pip install -e ".[diacritic-hf]"
pip install optimum[onnxruntime]

# Re-export
python training/onnx_export/export_int8.py \
    --source nrl-ai/vn-spell-correction-small \
    --output training/onnx_export/vn-spell-correction-small-onnx-int8

# Re-bench against the OOD eval
python training/onnx_export/bench_int8.py \
    --model training/onnx_export/vn-spell-correction-small-onnx-int8 \
    --json benchmarks/results/baseline_real_spell_correction_small_onnx_int8.json

License & attribution

Released under Apache 2.0 — same as the source model.

bibtex

@misc{nom_vn_spell_correction_onnx_int8_2026,
  title={Vietnamese Spell Correction — ONNX int8 quantization for edge deployment},
  author={Nguyen, Viet-Anh and {Neural Research Lab}},
  year={2026},
  howpublished={\url{https://huggingface.co/nrl-ai/vn-spell-correction-small-onnx-int8}}
}

nrl-ai/vn-spell-correction-small-onnx-int8

nrl-ai/vn-spell-correction-small-onnx-int8 — ONNX int8 quantization of nrl-ai/vn-spell-correction-small