OpenVINO IR conversion of
ibm-granite/granite-4.1-8b
(dense 40-layer transformer, 128k context, Apache-2.0). Quantized
INT4 symmetric
channel-wise with AWQ + scale-estimation calibration.
The one difference from the sibling
granite-4.1-8b-int4-cw-ov:
the AWQ/scale-estimation calibration set is
real Python source code instead of
wikitext2 prose. The hypothesis is that domain-matched activation statistics protect the
channels that matter for code generation and editing. On this repo's per-task-type
benchmark it scored
20/26 (codegen 9/12) versus the wikitext2-calibrated sibling's
19/26 (codegen 8/12) — a small but real gain on codegen, and slightly faster, with the
other task types unchanged.
Converted with optimum-intel (2.1.0.dev0+314b0c4) / NNCF 3.2.0 / OpenVINO 2026.3 /
transformers 5.10.2, via the
optimum-intel
Python API:
1from optimum.intel import OVModelForCausalLM, OVWeightQuantizationConfig
2
3# corpus = 128 random ~2.4k-char chunks of real Python source
4qcfg = OVWeightQuantizationConfig(
5 bits=4, sym=True, group_size=-1, ratio=1.0,
6 dataset=corpus, awq=True, scale_estimation=True)
7model = OVModelForCausalLM.from_pretrained(
8 "ibm-granite/granite-4.1-8b", export=True,
9 quantization_config=qcfg, compile=False)
1import openvino_genai as ov_genai
2
3pipe = ov_genai.LLMPipeline("granite-4.1-8b-int4-cw-code-ov", "GPU", CACHE_DIR="./.ovcache")
4print(pipe.generate("Write a Python function that merges overlapping intervals.",
5 max_new_tokens=256))
Tested end-to-end as an OpenAI-compatible Continue.dev backend via
core-ultra-llm-server.
