Qwen3-14B-Heretic OpenVINO INT4 for Intel NPU
This repository contains a ready-to-run OpenVINO IR export of
0xA50C1A1/Qwen3-14B-Heretic,
prepared for local inference on Intel NPU through OpenVINO Model Server.
It is meant for users who want to skip the local OpenVINO conversion step and
run the model directly with OpenVINO / OVMS on Intel NPU.
Source model
- Source model:
0xA50C1A1/Qwen3-14B-Heretic
- Original base model:
Qwen/Qwen3-14B
- Architecture:
Qwen3ForCausalLM
- Task: text generation
- License: Apache-2.0, inherited from the source model metadata
This is not a fine-tune. It is an OpenVINO INT4 runtime export of the source
model above.
OpenVINO export
The exported directory includes:
openvino_model.xmlopenvino_model.binopenvino_tokenizer.xmlopenvino_tokenizer.binopenvino_detokenizer.xmlopenvino_detokenizer.bin- tokenizer and generation config files
Compression metadata from this export:
1{
2 "mode": "INT4_SYM",
3 "group_size": -1,
4 "ratio": 1.0,
5 "all_layers": true
6}
The model config reports max_position_embeddings: 40960. On the test system
below, OVMS loaded reliably with --max_prompt_len 8192; higher runtime prompt
limits can require more shared memory and may fail or stall on smaller systems.
Tested Intel NPU runtime
Tested locally on Windows with:
- OpenVINO Model Server:
2026.2.1.1122f03bf
- OpenVINO backend:
2026.2.1-21919-ede283a88e3-releases/2026/2
- OpenVINO GenAI backend:
2026.2.1.0-3123-7dea0459b2a
- OVMS task:
text_generation
- Target device:
NPU
- Runtime prompt limit:
8192
- Max concurrent sequences:
1
- Cache interval multiplier:
64
Example OVMS command:
1ovms.exe `
2 --model_path Q:/llm/models/OpenVINO/0xA50C1A1--Qwen3-14B-Heretic-text-fp16-true-int4-sym-cw-ov `
3 --model_name 0xA50C1A1--Qwen3-14B-Heretic-true-int4-npu `
4 --rest_port 8000 `
5 --rest_bind_address 0.0.0.0 `
6 --task text_generation `
7 --target_device NPU `
8 --max_prompt_len 8192 `
9 --max_num_seqs 1 `
10 --cache_interval_multiplier 64 `
11 --reasoning_parser qwen3 `
12 --tool_parser hermes3
Local benchmark
Measured on the local Intel NPU setup above, using the OVMS OpenAI-compatible
chat completions endpoint.
| Run | Prompt tokens | Completion tokens | Elapsed | Output speed |
|---|
| Warm-up | 58 | 32 | 10.61 s | 3.02 tok/s |
| Run 1 | 58 | 128 | 29.43 s | 4.35 tok/s |
| Run 2 | 58 | 128 | 28.75 s | 4.45 tok/s |
Post-warm-up average output speed: about 4.4 tokens/s.
Memory during the same test:
- OVMS process working set: about 9.83 GiB
- OVMS private memory: about 1.93 GiB
Windows Task Manager may report NPU shared memory separately from the process
private memory, so total observed system memory pressure can be higher.
Notes
- This export is intended for Intel NPU use with OpenVINO / OVMS.
- It can also be useful as a reference OpenVINO IR export, but the benchmark
numbers above are only for the tested local NPU configuration.
- Use a smaller
--max_prompt_len if loading stalls or system memory pressure
is too high.
