Note about the missing weights (2026-09-03)
For a long time this repository contained only this model card and none of the
actual model files. That was my own mistake: back in June I created the repo
and pushed the README, but I never actually completed the upload of the OpenVINO
IR weights. I only noticed it now, while uploading other models.
The weights have now been uploaded. The export itself was never lost or corrupted
— it had been sitting intact on my local machine the whole time.
Sorry to anyone who landed here earlier and found an empty repository.
Qwen3 4B Thinking 2507 Heretic CodeFeedback — OpenVINO INT8
This repository contains the OpenVINO INT8 export of:
The source model is a merged code-focused fine-tune based on:
This version was converted from the full merged safetensors model to OpenVINO IR with INT8 weight compression.
Compared with the INT4 version, this INT8 export is expected to be larger and may use more VRAM, but it should preserve more of the original model quality.
Related repositories
Format
| Item | Value |
|---|
| Source model | JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback |
| Format | OpenVINO IR |
| Weight compression | INT8 |
| Intended runtime | OpenVINO / OpenVINO GenAI |
| Recommended device | Intel Arc / Intel GPU with OpenVINO support |
| Main use | Quality comparison against INT4 |
Conversion command
1optimum-cli export openvino \
2 --model ~/models-src/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback \
3 --task text-generation-with-past \
4 --weight-format int8 \
5 --trust-remote-code \
6 ./Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT8
The INT8 export does not need INT4-specific options such as --group-size 128 or --ratio 1.0.
Intended comparison
This INT8 version is mainly intended as a comparison build against:
General expected tradeoff:
| Version | Size | VRAM use | Speed | Quality |
|---|
| OpenVINO INT4 | Smaller | Lower | Faster | Slightly more compressed |
| OpenVINO INT8 | Larger | Higher | Possibly slower | Usually more faithful |
Actual results depend on OpenVINO version, Intel GPU driver, context length, prompt, batch size, and runtime configuration.
OpenArc load example
1curl -X POST http://localhost:8000/openarc/load \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model_path": "/models/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT8",
5 "model_name": "Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-OpenVINO-INT8",
6 "model_type": "llm",
7 "engine": "ovgenai",
8 "device": "GPU.0",
9 "runtime_config": {}
10 }'
Training background
The source model was trained with QLoRA/LoRA on Python and code instruction datasets, then merged back into the base model.
| Dataset | Samples used | Notes |
|---|
iamtarun/python_code_instructions_18k_alpaca | 5,000 | Python instruction/code examples |
m-a-p/CodeFeedback-Filtered-Instruction | 5,000 | Code instruction and feedback examples |
A SWE-smith trajectory experiment was tested separately, but it was not used in the final merged version.
LoRA configuration of source model
| Parameter | Value |
|---|
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Sequence length | 2048 |
| Epochs per stage | 1 |
| Quantized loading during training | 4-bit NF4 |
| Trainable parameters | ~33M |
| Trainable percentage | ~0.81% |
Target modules:
q_projk_projv_projo_projgate_projup_projdown_proj
Intended use
This INT8 OpenVINO version is intended for:
- OpenVINO quality testing
- comparison against INT4
- Intel Arc / Intel GPU inference
- Python code generation
- code explanation
- simple debugging
- instruction-following tests
Hardware notes
| Hardware | Expected status |
|---|
| Intel Arc A750 8 GB | Should be possible, but may use more VRAM than INT4 |
| Intel Arc A770 16 GB | Better target for INT8 |
| Intel Flex / Data Center GPU | May work if OpenVINO sees the GPU |
| CPU-only | Possible but slower |
| NVIDIA CUDA runtimes | Use the original safetensors, AWQ, GPTQ, or GGUF instead |
Important notes
This is not the original F16 model. This is an OpenVINO INT8 compressed export.
For further conversions, use the original merged safetensors repository as the master source:
This is an experimental model. It may produce incorrect code, unsafe suggestions, or hallucinated explanations. Outputs should be reviewed before use in production or security-sensitive environments.
