This repository contains a ready-to-run OpenVINO IR export of p-e-w/Qwen3-4B-Instruct-2507-heretic, prepared for local inference on Intel NPU through OpenVINO Model Server.

This is not a fine-tune. It is an OpenVINO NF4 runtime export of the source model above. In the local benchmark below, this was the best balance of speed and memory among the tested INT4, NF4, and INT8 exports.

Source model

Source model: p-e-w/Qwen3-4B-Instruct-2507-heretic
Original base model: Qwen/Qwen3-4B-Instruct-2507
Architecture: Qwen3ForCausalLM
Task: text generation
License: Apache-2.0, inherited from the source model metadata

OpenVINO export

Compression metadata from this export:

json

{
  "mode": "nf4",
  "nncf_mode": "NF4",
  "group_size": -1,
  "ratio": 1.0,
  "all_layers": true
}

Known local artifact size:

openvino_model.bin: about 1.88 GiB

Tested Intel NPU runtime

Tested locally on Windows with OpenVINO Model Server / OpenVINO GenAI:

Target device: NPU
OVMS task: text_generation
Runtime prompt limit: 16384
Max concurrent sequences: 1
Cache interval multiplier: 64

Example OVMS command:

powershell

ovms.exe `
  --model_path Q:/llm/models/OpenVINO/p-e-w--Qwen3-4B-Instruct-2507-heretic-text-fp16-nf4-cw-ov `
  --model_name p-e-w--Qwen3-4B-Instruct-2507-heretic-nf4-npu `
  --rest_port 8000 `
  --rest_bind_address 0.0.0.0 `
  --task text_generation `
  --target_device NPU `
  --max_prompt_len 16384 `
  --max_num_seqs 1 `
  --cache_interval_multiplier 64 `
  --tool_parser hermes3

Local benchmark

Measured on the local Intel NPU setup above with the OVMS OpenAI-compatible chat completions endpoint.

Quantization	Load time	Avg output speed	Working set after runs	Private memory after runs
NF4	366.8 s	12.34 tok/s	7.42 GiB	2.57 GiB

The benchmark prompt was a short three-sentence OpenVINO explanation request with max_tokens=128.

machine-made-Fibre/Qwen3-4B-Instruct-2507-heretic-OpenVINO-NF4-NPU

Qwen3-4B-Instruct-2507-heretic OpenVINO NF4 for Intel NPU

Source model

OpenVINO export

Tested Intel NPU runtime

Local benchmark

machine-made-Fibre/
Qwen3-4B-Instruct-2507-heretic-OpenVINO-NF4-NPU