Dynamic-INT8 ONNX build of BalaRajesh1/mmbert-small-nli, whose upstream repo ships PyTorch weights only. Published so installs pull the quantized weights directly (141 MB instead of a 563 MB fp32 export) instead of re-exporting and re-quantizing per machine.

No retraining, no distillation: the same weights, exported and quantized.

MIT, inherited from both. All credit for the model belongs upstream; this repo only converts it.

The graph takes input_ids + attention_mask only, with no token_type_ids, so it runs on a plain ONNX Runtime session without a tokenizer type-id pass.

Signed INT8 was not portable for this graph. Measured on a 72-pair labeled entailment set (AUC, entailed vs the rest), same fp32 export:

QInt8 was the best variant on one machine and worthless on another. Those two runs are not a controlled comparison (different CPU, and a different exporter/runtime produced each file), so the cause is not pinned down; a host without VNNI running signed INT8 through a saturating path is the prime suspect, and reduce_range=True did lift that host to 0.839. QUInt8 measured the same on both, so that is what ships.

If you quantize this model yourself, re-measure on your target host. An INT8 file that is excellent on one CPU can be a coin flip on another, and nothing in the file will tell you.

Upstream declares tokenizer_class: TokenizersBackend, which only Transformers v5 resolves: older Python Transformers raises, and transformers.js falls back to its base class with a warning. This repo declares PreTrainedTokenizerFast instead. Token ids are unchanged (verified identical on all 72 evaluation pairs); the algorithm lives in tokenizer.json, not in the class name.

Also loads in transformers.js with dtype: 'q8'.

MNLI-matched 85.56 · SNLI test 88.27 · XNLI test (15 languages) 77.72 · WANLI test 69.18. These are the upstream fp32 figures, not re-measured here; the table above is the only measurement made on this build.