Evaluation uses the same stratified cluster sampling used to construct the calibration imatrix, but with held-out texts not used during calibration.
Evaluation uses token-level perplexity averaged per text, then aggregated by cluster.
granite-4.0-micro-Q4_K_NL
Dutch-calibrated Q4_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix and Unsloth's per-tensor layer promotion map extracted from their Q4_K_XL GGUF. Produces lower perplexity on Dutch text than standard Q4_K_M while fitting in ~2GB VRAM.
This is the primary deployment target in the Dutch Granite quantization series. See the dataset card for full methodology.
What makes this different from standard Q4_K_M
Standard Q4_K_M uses a generic imatrix (or none at all) and applies uniform quantization across all tensor types. This quantization improves on that in two ways:
Dutch imatrix: The importance matrix was built from 3,504 Dutch texts (~295K tokens) drawn from a stratified sample of the Leesplank corpus — 70 semantic clusters × 6 complexity strata (simple/complex × low/mid/high Levenshtein distance). This tells the quantizer which weights matter most for Dutch language modeling specifically, not English or generic multilingual text.
Unsloth layer map: Rather than promoting layers heuristically, the per-tensor quantization types were extracted directly from Unsloth's published granite-4.0-micro-UD-Q4_K_XL GGUF. Unsloth's XL series applies careful per-tensor analysis to identify which layers are sensitive. Extracting that map and applying it with a Dutch imatrix combines their structural insight with Dutch-specific calibration.
Measured on 66 Dutch text clusters (12,600 texts total, same stratified sample — see dataset card). FP16 baseline median PPL: 8.938.
Model
Median PPL
Mean Δ vs FP16
Mean % Δ
FP16 baseline
8.938
—
—
Q4_K_NL (this model)
9.063
+0.131
+1.50%
Q4_K_NL_plain (imatrix only)
9.099
+0.170
+1.96%
Q4_K_XL (Unsloth reference)
9.054
+0.122
+1.39%
IBM Q4_K_M (plain, no imatrix)
9.458
+0.480
+5.47%
The Dutch imatrix + Unsloth layer map narrows the PPL gap relative to vanilla Q4_K_M while landing very close to Unsloth's Q4_K_XL reference (0.131 vs 0.122 — note Q4_K_NL is slightly behind Unsloth Q4 in mean delta, but beats it in median). The plain imatrix-only variant sits 0.039 PPL points above this model, confirming the layer map contributes meaningfully on top of calibration alone.
Note: If VRAM allows, Q5_K_NL achieves dramatically better fidelity (within 0.064% of FP16 on average). Use Q4_K_NL when the 2GB target matters.
KLD (distribution fidelity)
KLD measures how closely the full output probability distribution matches FP16 at each token position — more sensitive than perplexity, which only captures the most likely token. Measured across the same 70 clusters.
Model
Weighted mean KLD
Weighted median KLD
Q4_K_NL (this model)
12.918
11.604
Q4_K_XL (Unsloth)
12.933
11.695
IBM Q4_K_M
12.943
11.626
Q5_K_NL
12.955
11.691
Q6_K_XL (Unsloth)
12.972
11.690
Q6_K_NL
12.993
11.661
Q5_K_XL (Unsloth)
13.000
11.792
Total spread across all 7 models: 0.082 nats. This confirms granite-4.0-micro quantizes robustly regardless of calibration or bit depth — all variants preserve the full distribution similarly well. Q4_K_NL edges the ranking but the margin is within noise.
Usage
bash
1# LM Studio: search for this model name, download, use Granite chat template2# llama.cpp:3llama-cli -m granite-4.0-micro-Q4_K_NL.gguf -ngl 99 -p "Vertel me over ..."
If you use this model, please cite the dataset card and IBM's original Granite 4.0 release.
granite-4.0-micro-Q4_K_NL_plain
Dutch-calibrated Q4_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix without per-tensor layer promotion. Control variant for comparing the contribution of the Unsloth layer map vs calibration alone.
This model uses the same Dutch imatrix as Q4_K_NL but applies standard Q4_K_M quantization uniformly — no tensor-type overrides. It exists to isolate the contribution of the Unsloth layer map. See the dataset card for full methodology.
Model details
Property
Value
Base model
ibm-granite/granite-4.0-micro
Architecture
Granite 4.0 Dense (40 layers, 3B parameters)
Quantization
Q4_K_M with Dutch imatrix, no layer overrides
Calibration tokens
~295K Dutch (Leesplank corpus)
File size
~1.9 GB
Context length
128K
Chat template
Granite instruct
When to use this vs Q4_K_NL
Use Q4_K_NL (the dynamic variant) for deployment — it has the layer map applied and shows lower perplexity. Use this model only if you want to verify the layer map contribution for your specific use case, or if you find Q4_K_NL behaves unexpectedly.
Perplexity
Model
Median PPL
Mean Δ vs FP16
Mean % Δ
FP16 baseline
8.938
—
—
Q4_K_NL (with layer map)
9.063
+0.131
+1.50%
Q4_K_NL_plain (this model)
9.099
+0.170
+1.96%
The layer map reduces the PPL gap by ~23% relative (0.170 → 0.131). Both models beat a vanilla IBM Q4_K_M by a wide margin (+5.47% without any imatrix or layer map).
granite-4.0-micro-Q5_K_NL
Dutch-calibrated Q5_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix and Unsloth's per-tensor layer promotion map extracted from their Q5_K_XL GGUF. Recommended if VRAM allows — achieves perplexity within 0.064% of FP16 on average on Dutch text.
This is the preferred quality target in the Dutch Granite quantization series. The Q5 Unsloth layer map promotes significantly more tensors than Q4 — adding Q, K, and gate/up weights for early and late layers (0,1,2,6,39) on top of the V and ffn_down promotions. See the dataset card for full methodology.
Model details
Property
Value
Base model
ibm-granite/granite-4.0-micro
Architecture
Granite 4.0 Dense (40 layers, 3B parameters)
Quantization
Q5_K_M with Dutch imatrix + Unsloth layer map
Calibration tokens
~295K Dutch (Leesplank corpus)
File size
~2.4 GB
Context length
128K
Chat template
Granite instruct
What Unsloth's Q5 layer map promotes
On top of Q5_K_M base, these tensors are elevated:
All attn_norm and ffn_norm → F32
All attn_v → Q6_K (all 40 layers)
attn_q and attn_k → Q6_K for layers 0,1,2,6,39
ffn_gate and ffn_up → Q6_K for layers 0,1,2,6,39
Selected ffn_down → Q6_K (17 layers) or Q8_0 (layers 6,39)
token_embd → Q6_K, output_norm → F32
Layers 0-2 (early), 6 (attention transition), and 39 (final layer) receive the most promotion — consistent with the known sensitivity of first and last transformer blocks.
Perplexity
Measured on 66 Dutch text clusters (12,600 texts). FP16 baseline median PPL: 8.938.
Model
Median PPL
Mean Δ vs FP16
Mean % Δ
Clusters beating FP16
FP16 baseline
8.938
—
—
—
Q5_K_NL (this model)
8.952
+0.006
+0.064%
24 / 66
Q5_K_NL_plain (imatrix only)
9.027
+0.064
+0.707%
0 / 66
Q5_K_XL (Unsloth reference)
8.981
+0.036
+0.423%
6 / 66
Q6_K_NL
8.962
+0.025
+0.282%
1 / 66
Q5_K_NL achieves a mean PPL delta of only +0.006 vs FP16 — and actually beats FP16's measured perplexity on 36% of clusters. This is not noise: the Dutch calibration data is guiding the quantizer well enough that rounding errors partially cancel across Dutch text sequences. Unsloth's own Q5 sits at +0.036, nearly 6× larger a gap. The plain imatrix-only variant at +0.064 shows the layer map contributes roughly a 10× improvement at Q5 (from 0.064 to 0.006) — a much larger gain than at Q4.
KLD (distribution fidelity)
All Dutch Granite quants show very similar full-distribution fidelity to FP16. The total spread across all 7 tested models is only 0.082 nats. Q5_K_NL sits mid-pack in KLD (12.955 weighted mean vs Q4_K_NL's 12.918 best), which is consistent with the PPL picture: Q5 preserves the most likely token exceptionally well, while Q4 has a marginal advantage in preserving the tail of the distribution. In practice, neither difference is perceptible.
Usage
bash
1# LM Studio: search for this model name, download, use Granite chat template2# llama.cpp:3llama-cli -m granite-4.0-micro-Q5_K_NL.gguf -ngl 99 -p "Vertel me over ..."
If you use this model, please cite the dataset card and IBM's original Granite 4.0 release.
granite-4.0-micro-Q5_K_NL_plain
Dutch-calibrated Q5_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix without per-tensor layer promotion. Control variant for comparing Q5_K_NL vs calibration-only Q5.
Same Dutch imatrix as Q5_K_NL, standard Q5_K_M quantization with no tensor-type overrides. Exists to isolate the Unsloth layer map contribution at Q5 bit width. See the dataset card for full methodology.
Model details
Property
Value
Base model
ibm-granite/granite-4.0-micro
Architecture
Granite 4.0 Dense (40 layers, 3B parameters)
Quantization
Q5_K_M with Dutch imatrix, no layer overrides
Calibration tokens
~295K Dutch (Leesplank corpus)
File size
~2.3 GB
Context length
128K
Chat template
Granite instruct
Perplexity
Model
Median PPL
Mean Δ vs FP16
Mean % Δ
FP16 baseline
8.938
—
—
Q5_K_NL (with layer map)
8.952
+0.006
+0.064%
Q5_K_NL_plain (this model)
9.027
+0.064
+0.707%
At Q5, the Unsloth layer map delivers a 10× reduction in mean PPL gap (0.064 → 0.006). Use Q5_K_NL for deployment.
granite-4.0-micro-Q6_K_NL
Dutch-calibrated Q6_K quantization of IBM Granite 4.0 Micro Dense (3B) with Unsloth's Q6 layer promotion map. Near-FP16 quality ceiling; use this as a reference when measuring quantization degradation, or when quality is critical and VRAM allows.
The Q6 Unsloth layer map is the most aggressive — it promotes all Q, K, V attention weights across all 40 layers to Q8_0, along with selected ffn weights, leaving almost nothing at base Q6_K except the least sensitive feed-forward mid-layers. Combined with the Dutch imatrix this produces a GGUF very close to FP16 quality on Dutch text. See the dataset card for full methodology.
Model details
Property
Value
Base model
ibm-granite/granite-4.0-micro
Architecture
Granite 4.0 Dense (40 layers, 3B parameters)
Quantization
Q6_K with Dutch imatrix + Unsloth layer map
Calibration tokens
~295K Dutch (Leesplank corpus)
File size
~3.3 GB
Context length
128K
Chat template
Granite instruct
What Unsloth's Q6 layer map promotes
On top of Q6_K base, these tensors are elevated to Q8_0:
All attn_q, attn_k, attn_v (all 40 layers) → Q8_0
Selected ffn_down, ffn_gate, ffn_up for layers 0,1,2,6,9,39 → Q8_0
This is effectively a hybrid Q6/Q8 model with F32 norms throughout.
Perplexity
Measured on 66 Dutch text clusters (12,600 texts). FP16 baseline median PPL: 8.938.
Model
Median PPL
Mean Δ vs FP16
Mean % Δ
FP16 baseline
8.938
—
—
Q6_K_NL (this model)
8.962
+0.025
+0.282%
Q6_K_XL (Unsloth reference)
8.971
+0.027
+0.303%
Q5_K_NL
8.952
+0.006
+0.064%
Notably, Q5_K_NL achieves better average perplexity than Q6_K_NL (+0.064% vs +0.282%). This is not a bug: at Q6 the Dutch imatrix and layer map have less room to improve on the base, and the marginal benefits of Dutch calibration are smaller. Q5_K_NL is the sweet spot for Dutch text fidelity in this series.
KLD (distribution fidelity)
Model
Weighted mean KLD
Weighted median KLD
Q4_K_NL
12.918
11.604
Q4_K_XL (Unsloth)
12.933
11.695
IBM Q4_K_M
12.943
11.626
Q5_K_NL
12.955
11.691
Q6_K_XL (Unsloth)
12.972
11.690
Q6_K_NL (this model)
12.993
11.661
Q5_K_XL (Unsloth)
13.000
11.792
Total spread: 0.082 nats across all 7 models. All variants preserve the full output distribution to essentially the same degree — granite-4.0-micro is a very quantization-stable model. Q6_K_NL's slightly higher KLD than Q4_K_NL is a second-decimal artefact without practical significance.
When to use this
As a local reference model when evaluating Q4/Q5 degradation
When running on hardware where 3.3GB fits comfortably and you want maximum Dutch fidelity
Not the recommended deployment target — Q5_K_NL gives better PPL at 0.9GB less
