Views
No views yet
| Metric | Value |
|---|---|
| Family | qwen3.5 |
| Quantisation | int8-fp16 |
| Context length | 8,192 tokens |
| Decode rate (GB10, single stream) | ~12 tok/s |
| Source build | github.com/vriveras/qwen-3-5-ort-support @ 38bd75ae |
| Source dir | Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI |
model.onnx — ONNX graph topology. Uses custom ops in domain
ai.ort.contrib.ssm for the SSM linear-attention layers
((via stock ORT MatMulNBits for attention/MoE)).weights.bin — external initializer blob referenced by model.onnx.genai_config.json — ORT GenAI runtime config (input/output names,
KV + SSM cache pair declarations, custom_ops_library path).tokenizer.json + tokenizer_config.json + chat_template.jinja —
HuggingFace-style tokenizer + chat template.qwen3-ssm-cuda-graph, or apply the patches from the source
repo's genai_patches/ against upstream microsoft/onnxruntime-genai.libgated_delta_net_op.{so,dll} built from
github.com/vriveras/qwen-3-5-ort-support
(bash install.sh on Linux, .\build_windows.ps1 on Windows).genai_config.json's
model.decoder.session_options.custom_ops_library to point at where
you installed the custom op library.1git lfs install
2git clone https://huggingface.co/USER/qwen3.5-35b-a3b-int8-onnx-8k /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI
3# Edit genai_config.json's custom_ops_library path to match your install
4cd /path/to/your/clone-of/qwen-3-5-ort-support
5bash start_server.sh --genai --model_dir /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI \
6 --tokenizer /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI1mkdir /path/to/models && \
2 git clone https://huggingface.co/USER/qwen3.5-35b-a3b-int8-onnx-8k /path/to/models/Qwen3.5-35B-A3B-NoRot-GPTQ-Int8-FP16-ONNX-GenAI
3python serve_genai.py --models_dir /path/to/modelsqwen3.5:int8-fp16-8k (or just qwen3.5 for
the default tag).