You can deploy this model and start earning money today!

icosahedron10/
qwen3-embedding-0.6b-onnx-fp16

Name: qwen3-embedding-0.6b-onnx-fp16
Availability: InStock

Qwen/Qwen3-Embedding-0.6B

quantized

apache-2.0

Views

No views yet

Qwen3-Embedding-0.6B ONNX FP16

FP16 ONNX Runtime package for Qwen/Qwen3-Embedding-0.6B.

The model expects input_ids, attention_mask, and position_ids as int64 inputs, and returns last_hidden_state. For retrieval, use Qwen's query-side instruction format, last-token pooling, and L2 normalization.

Files

onnx/model.onnx
tokenizer and model config files at repo root
verify_onnxruntime.py
qwen_onnx_release_tools.py

ONNX Export Creation Commands

The release was built on Windows with Python 3.12, CUDA-enabled PyTorch, and onnxruntime-gpu==1.26.0.

powershell

optimum-cli export onnx `
  --model Qwen/Qwen3-Embedding-0.6B `
  --library-name transformers `
  --task text-generation `
  --trust-remote-code `
  --opset 18 `
  --device cuda `
  --dtype fp16 `
  --no-post-process `
  qwen3-embedding-0.6b-onnx-fp16

Install And Download

bash

python -m pip install "huggingface_hub[cli]" onnx onnxruntime-gpu transformers numpy
hf download icosahedron10/qwen3-embedding-0.6b-onnx-fp16 --local-dir qwen3-embedding-0.6b-onnx-fp16
cd qwen3-embedding-0.6b-onnx-fp16
python verify_onnxruntime.py

For CPU-only machines, install onnxruntime instead of onnxruntime-gpu and change the provider in your own loader to CPUExecutionProvider.

Quick Start

python

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

model_dir = "qwen3-embedding-0.6b-onnx-fp16"
tokenizer = AutoTokenizer.from_pretrained(
    model_dir,
    local_files_only=True,
    padding_side="left",
    fix_mistral_regex=True,
)

session = ort.InferenceSession(
    f"{model_dir}/onnx/model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

instruction = "Given a web search query, retrieve relevant passages that answer the query"
queries = [
    f"Instruct: {instruction}\nQuery: How can I reset my PostgreSQL password?",
    f"Instruct: {instruction}\nQuery: How do I make sourdough bread?",
    f"Instruct: {instruction}\nQuery: Who wrote Pride and Prejudice?",
]
documents = [
    "To reset a PostgreSQL password, connect as a superuser and run ALTER USER with a new password.",
    "Sourdough bread is made by mixing flour, water, salt, and starter, then fermenting, shaping, proofing, and baking.",
    "Jane Austen wrote the novel Pride and Prejudice.",
]

encoded = tokenizer(
    queries + documents,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="np",
)

input_ids = encoded["input_ids"].astype(np.int64)
attention_mask = encoded["attention_mask"].astype(np.int64)
position_ids = np.repeat(
    np.arange(input_ids.shape[1], dtype=np.int64)[None, :],
    input_ids.shape[0],
    axis=0,
)

last_hidden_state = session.run(
    None,
    {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "position_ids": position_ids,
    },
)[0]

sequence_lengths = attention_mask.sum(axis=1) - 1
rows = np.arange(last_hidden_state.shape[0])
embeddings = last_hidden_state[rows, sequence_lengths]
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)

similarities = embeddings[:3] @ embeddings[3:].T
print(similarities)

Verification

Verified locally with ONNX Runtime GPU:

text

ORT providers used: ['CUDAExecutionProvider', 'CPUExecutionProvider']
similarity matrix:
[[0.5550 0.0997 0.1455]
 [0.1194 0.6390 0.1434]
 [0.1284 0.0251 0.3975]]
Embedding CUDA verification passed