Dynamic INT8 quantization of BAAI/bge-reranker-v2-m3 for fast CPU reranking via Hugging Face Text Embeddings Inference (TEI).
Recipe:optimum ORTQuantizer · AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False) (dynamic, no calibration).
Why: ~2.5× faster reranking on AVX-512-VNNI CPUs, ~4× smaller (543 MB vs 2.3 GB fp32), no measured ranking-quality loss on our legal/finance retrieval gold (pravo/cfa/border: zero-question delta vs fp32).
Serving: TEI cpu-1.9.3+ — onnx/model.onnx is the int8 graph; config + tokenizer at root.
Quantized by ЛИИ / csylabs for the «Доверенный ИИ» platform.
