You can deploy this model and start earning money today!

stijnvdoornum/
Qwen3-8B-int4-ov

Name: Qwen3-8B-int4-ov
Availability: InStock

Views

No views yet

Qwen3-8B — OpenVINO IR, int4

Qwen/Qwen3-8B converted to OpenVINO IR with int4 weight compression via optimum-intel / NNCF.

Quantization: int4 asymmetric, group size 128, ratio 1.0 (all 252 linear layers)
Size: ~4.9 GB (vs ~16 GB fp16)
Export stack: optimum-intel, OpenVINO 2026.2, NNCF 3.2, transformers 4.57

Usage

python

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "stijnvdoornum/Qwen3-8B-int4-ov"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)  # device="GPU" on Intel iGPU/NPU hosts

messages = [{"role": "user", "content": "What is OpenVINO?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Reproduce

bash

optimum-cli export openvino --model Qwen/Qwen3-8B \
  --task text-generation-with-past --weight-format int4 \
  --group-size 128 --ratio 1.0 <output_dir>

Converted with a small toolkit that also supports int8/fp16 export, AWQ/scale-estimation data-aware int4, and re-quantization of existing IRs.