Setting	Value
Tool	🤗 Optimum (`optimum-cli onnxruntime quantize --avx2`)
Type	Dynamic, `IntegerOps` / `QOperator`
Weights	`QUInt8`, symmetric
Activations	`QUInt8`, asymmetric
`per_channel`	`false`
`reduce_range`	`true`
Operators	MatMul, Attention, Gather, Transpose, EmbedLayerNormalization, …
`model.onnx` size	~0.55 GB (weights embedded, no `.onnx_data` file)

Target hardware: quantized with the AVX2 preset. Intended for x86-64 CPUs that have AVX2 but no AVX-512/VNNI (e.g. AMD Ryzen Zen/Zen+/Zen2 such as the 2600X). The model runs on any x86-64 CPU; the AVX2 preset only tunes the quantization parameters for that instruction profile.

Accuracy note: reduce_range=True is enabled. It keeps INT8 weights on 7 bits to avoid u8×s8 saturation (the VPMADDUBSW instruction) on AVX2 CPUs without VNNI, as recommended by ONNX Runtime. This trades a negligible amount of headroom for stable accuracy on non-VNNI hardware; on VNNI-capable CPUs it makes no difference.

Usage with TEI

bash

docker run --rm -p 8080:80 \
  ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \
  --model-id kftof/bge-reranker-v2-m3-onnx-int8-avx2

Call the /rerank endpoint:

bash

curl http://127.0.0.1:8080/rerank \
  -H 'Content-Type: application/json' \
  -d '{
        "query": "What is the capital of France?",
        "texts": ["Paris is the capital of France.",
                  "Chocolate cake recipe"]
      }'

The response ranks the documents by relevance (highest score first).

Reproduce this build

bash

pip install "optimum[exporters,onnxruntime]"
optimum-cli export onnx --model BAAI/bge-reranker-v2-m3 \
  --task text-classification onnx-fp32
optimum-cli onnxruntime quantize --avx2 --onnx_model onnx-fp32 -o onnx-int8
# then rename onnx-int8/model_quantized.onnx -> model.onnx for TEI

License

Inherits the upstream license of BAAI/bge-reranker-v2-m3: Apache-2.0. No re-licensing.