You can deploy this model and start earning money today!

icosahedron10/
qwen3-reranker-0.6b-onnx-fp16

Name: qwen3-reranker-0.6b-onnx-fp16
Availability: InStock

Qwen/Qwen3-Reranker-0.6B

quantized

apache-2.0

Views

No views yet

Qwen3-Reranker-0.6B ONNX FP16

FP16 ONNX Runtime package for Qwen/Qwen3-Reranker-0.6B.

The model expects input_ids, attention_mask, and position_ids as int64 inputs, and returns next-token logits. For reranking, compare the final-position logits for the yes and no tokens.

Files

onnx/model.onnx
onnx/model.onnx_data
tokenizer and model config files at repo root
verify_onnxruntime.py
qwen_onnx_release_tools.py

ONNX Export Creation Commands

The release was built on Windows with Python 3.12 and onnxruntime-gpu==1.26.0.

powershell

optimum-cli export onnx `
  --model Qwen/Qwen3-Reranker-0.6B `
  --library-name transformers `
  --task text-generation `
  --trust-remote-code `
  --opset 18 `
  --device cuda `
  --dtype fp16 `
  --no-post-process `
  qwen3-reranker-0.6b-onnx-fp16

Install And Download

bash

python -m pip install "huggingface_hub[cli]" onnx onnxruntime-gpu transformers numpy
hf download icosahedron10/qwen3-reranker-0.6b-onnx-fp16 --local-dir qwen3-reranker-0.6b-onnx-fp16
cd qwen3-reranker-0.6b-onnx-fp16
python verify_onnxruntime.py

For CPU-only machines, install onnxruntime instead of onnxruntime-gpu and change the provider in your own loader to CPUExecutionProvider.

Quick Start

python

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

model_dir = "qwen3-reranker-0.6b-onnx-fp16"
tokenizer = AutoTokenizer.from_pretrained(
    model_dir,
    local_files_only=True,
    fix_mistral_regex=True,
)

session = ort.InferenceSession(
    f"{model_dir}/onnx/model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

instruction = "Given a web search query, retrieve relevant passages that answer the query"
query = "What is ONNX Runtime used for?"
document = "ONNX Runtime is an inference engine for running ONNX models efficiently."

messages = [
    {"role": "system", "content": instruction},
    {"role": "query", "content": query},
    {"role": "document", "content": document},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False)
encoded = tokenizer(prompt, return_tensors="np", add_special_tokens=False)

input_ids = encoded["input_ids"].astype(np.int64)
attention_mask = encoded["attention_mask"].astype(np.int64)
position_ids = np.arange(input_ids.shape[1], dtype=np.int64)[None, :]

logits = session.run(
    None,
    {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "position_ids": position_ids,
    },
)[0]

yes_id = tokenizer.encode("yes", add_special_tokens=False)[-1]
no_id = tokenizer.encode("no", add_special_tokens=False)[-1]
scores = logits[0, -1, [no_id, yes_id]].astype(np.float64)
scores -= scores.max()
probabilities = np.exp(scores) / np.exp(scores).sum()

print({"no": float(probabilities[0]), "yes": float(probabilities[1])})

Verification

Verified locally with ONNX Runtime GPU:

text

ORT providers used: ['CUDAExecutionProvider', 'CPUExecutionProvider']
relevant: yes=0.999771
irrelevant: yes=0.000018
Reranker CUDA verification passed