This repository provides Qwen3-32B-Instruct converted to INT4 ONNX and optimized for DirectML using Microsoft Olive and ONNX Runtime GenAI.

It enables native Windows GPU inference (Intel Arc, AMD RDNA, NVIDIA RTX) without CUDA and without running a Python server, and is intended for use in C# / .NET applications via ONNX Runtime + DirectML.

Model Details

Base model: Qwen/Qwen3-32B
Variant: Instruct
Quantization: INT4 (MatMul NBits, per-channel)
Format: ONNX
Runtime: ONNX Runtime with DmlExecutionProvider
Conversion toolchain: Microsoft Olive + onnxruntime-genai
Target hardware:
- Intel Arc (A770, 130V with large system RAM)
- AMD RDNA2 / RDNA3
- NVIDIA RTX (24 GB recommended, 16 GB possible with paging)

Files

Core inference artifacts:

model.onnx
model.onnx.data ← INT4 weights (≈ 18.6 GB)
genai_config.json
tokenizer.json, vocab.json, merges.txt
chat_template.jinja

Hardware & Memory Notes

Although INT4 quantization greatly reduces VRAM usage, the 32B model still requires:

≥ 16 GB VRAM (with host memory fallback via DirectML)
≥ 64 GB system RAM strongly recommended
Fast NVMe storage for paging

This model is intended for:

Advanced reasoning
Tool orchestration
Structured document analysis
Multi-step planning in local Windows applications

Usage in C# (DirectML)

csharp

using Microsoft.ML.OnnxRuntimeGenAI;

var modelPath = @"Qwen3-32B-Instruct-DirectML-INT4";

using var model = Model.Load(modelPath, new ModelOptions
{
    ExecutionProvider = ExecutionProvider.DirectML
});

using var tokenizer = new Tokenizer(model);
var tokens = tokenizer.Encode("Determine which legal document templates are required for a Dutch mortgage transaction.");

using var generator = new Generator(model, new GeneratorParams
{
    MaxLength = 2048,
    Temperature = 0.6f
});

generator.AppendTokens(tokens);
generator.Generate();

string output = tokenizer.Decode(generator.GetSequence(0));
Console.WriteLine(output);
Prompt Format
The model supports chat-style prompts and function-calling / tool-routing patterns when used with structured system prompts (e.g. Hermes-style schemas).

The provided chat_template.jinja can be used for consistent role formatting.

Performance Characteristics
Much stronger reasoning and instruction following than 14B

Higher latency, but better long-context coherence

Ideal when model must:

Infer document structures

Select templates

Extract structured fields from natural language

License & Attribution
Base model:

Qwen3-32B by Alibaba (see original model card for license)

Conversion:

ONNX + INT4 DirectML optimization performed by Wekkel using Microsoft Olive.

Independent community conversion.

No affiliation with Alibaba or the Qwen team.

Related Models
Smaller & faster:

https://huggingface.co/wekkel/Qwen3-14B-Instruct-DirectML-INT4

wekkel/Qwen3-32B-Instruct-DirectML-INT4

Qwen3-32B-Instruct – DirectML INT4 (ONNX Runtime)

Model Details

Files

Hardware & Memory Notes

Usage in C# (DirectML)

wekkel/
Qwen3-32B-Instruct-DirectML-INT4