These quants were produced by an experimental tool, niki-allocator. The tool is under active development and testing, and has not been published yet.

If you use these quants, I strongly recommend:

Using the temperature and sampling settings recommended by Alibaba for Qwen3.5-0.8B
Using this chat template: https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp

Qwen3.5-0.8B quantization quality comparison

Per-tensor type allocation vs Unsloth Dynamic 2.0 quants, evaluated with per-chunk KL divergence against the BF16 model.

Quality vs size

Mean KLD as a function of model size shows the allocator curve consistently below the unsloth curve, with both corpora overlapping closely:

Models

Model: Qwen3.5-0.8B (MTP/NextN, 25 layers: 24 main + 1 NextN head).
Baseline: BF16 GGUF (1485.50 MiB), perplexity 6.055 on kilo-small corpus, 15.508 on wikitext-2 test.
Per-tensor allocator quants (8 sizes): generated by a prototype per-tensor type allocator — each tensor assigned independently to the quantization type (Q5_K, Q6_K, IQ4_XS, etc.) that minimizes its proxy reconstruction error under a hard target size. Quantization performed by llama.cpp's own quantizer; imatrix-guided scale optimization used where the quantizer supports it. Source GGUF, imatrix, and assignment logs preserved.
unsloth quants (8 sizes): unsloth/Qwen3.5-0.8B-MTP-GGUF, produced with Unsloth Dynamic 2.0 quantization (7 of 8 use the UD_ prefix: UD_Q2_K_XL, UD_Q3_K_XL, UD_Q4_K_XL, UD_Q5_K_XL, UD_Q6_K_XL, UD_IQ2_M, UD_IQ3_XXS; IQ4_NL has no prefix and is a more conventional quant), same model, MTP preserved.

Sizes were chosen to match: each allocator quant targets the size of an unsloth quant (e.g. allocator 511 MiB vs unsloth IQ4_NL 510.59 MiB).

Allocator size	Unsloth quant	Unsloth size
390.00 MiB	UD_IQ2_M	390.37 MiB
409.00 MiB	UD_IQ3_XXS	409.23 MiB
434.00 MiB	UD_Q2_K_XL	433.52 MiB
484.99 MiB	UD_Q3_K_XL	484.54 MiB
510.99 MiB	IQ4_NL	510.59 MiB
545.99 MiB	UD_Q4_K_XL	546.44 MiB
585.99 MiB	UD_Q5_K_XL	585.75 MiB
753.97 MiB	UD_Q6_K_XL	754.21 MiB

Allocator type distributions vary by size: smaller targets use cheaper types (IQ4_XS, IQ3_S, Q3_K), larger targets transition to Q5_K/Q6_K/Q8_0. MTP/NextN tensors received types equal to or better than the unsloth quant's base type at the same size; quality gains are not achieved by sacrificing MTP heads.

Evaluation methodology

Metric: KL divergence per evaluation chunk between the quantized model's token probabilities and the BF16 baseline, computed by llama-perplexity with --saved-logits (full probability vectors saved, compared offline). Reported metrics: mean KLD, p95/p99/p99.9 KLD, maximum KLD (worst chunk), RMS Δp, same-top probability, top-flip weight, mean perplexity of the quant.

Corpora:

kilo-small: ~290K tokens, hold-out from the calibration corpus (agent session transcripts: English, Russian, code). Used for the primary comparison (all 16 models evaluated).
wikitext-2 test: ~290K tokens, standard test sample. Used for a subset of models to check generalization outside the calibration domain.

Chunk size: 2304 tokens per chunk (not the default 512). Reasoning models benefit from longer context windows during evaluation; 2304 was the fastest working size on the evaluation hardware. Reported KLD values are per-chunk averages and are not directly comparable to evaluations using 512-token chunks.

KLD bases: BF16 logits saved once per corpus; all quants compared against the same base.

Results: kilo-small corpus (all models)

Model	Size (MiB)	Mean PPL	Mean KLD	p95 KLD	p99 KLD*	p99.9 KLD*	Max KLD	RMS Δp	Same top p	Top flip weight
BF16 baseline	1485.50	6.055	—	—	—	—	—	—	—	—
allocator 390	390.00	6.808	0.1194	0.3741	0.6519	1.3625	11.1072	9.254	84.255	0.021070
unsloth UD_IQ2_M	390.37	8.718	0.3783	1.1583	1.9703	3.7493	10.0537	17.182	73.733	0.062213
allocator 409	409.00	6.590	0.0896	0.2763	0.4807	1.0326	6.7363	8.163	85.817	0.016525
unsloth UD_IQ3_XXS	409.23	7.617	0.2411	0.7498	1.3027	2.5299	7.8096	13.354	78.648	0.040041
allocator 434	434.00	6.716	0.1051	0.3344	0.5902	1.2397	8.3567	8.639	85.431	0.018462
unsloth UD_Q2_K_XL	433.52	8.227	0.3261	1.0307	1.7293	3.2185	9.6882	15.395	75.580	0.053019
allocator 485	484.99	6.282	0.0391	0.1223	0.2104	0.4446	2.4922	5.276	90.614	0.007151
unsloth UD_Q3_K_XL	484.54	6.575	0.0857	0.2693	0.4706	0.9816	4.5137	7.883	86.402	0.015512
allocator 511	510.99	6.209	0.0267	0.0839	0.1472	0.3306	1.9196	4.345	92.246	0.004852
unsloth IQ4_NL	510.59	6.313	0.0453	0.1422	0.2458	0.5262	2.8035	5.607	89.983	0.007999
allocator 546	545.99	6.194	0.0247	0.0786	0.1405	0.3189	1.6694	4.119	92.640	0.004316
unsloth UD_Q4_K_XL	546.44	6.248	0.0300	0.0956	0.1630	0.3332	1.9140	4.545	91.760	0.005340
allocator 586	585.99	6.106	0.0109	0.0343	0.0596	0.1363	0.6685	2.785	94.789	0.001985
unsloth UD_Q5_K_XL	585.75	6.128	0.0119	0.0372	0.0624	0.1221	0.7350	2.833	94.573	0.002085
allocator 754	753.97	6.073	0.0023	0.0072	0.0123	0.0261	0.1577	1.288	97.322	0.000368
unsloth UD_Q6_K_XL	754.21	6.072	0.0027	0.0084	0.0140	0.0284	0.1662	1.388	97.088	0.000465

* p99/p99.9 KLD are per-token percentiles over ~145K values (127 chunks × 1151 tokens per chunk). At 2304-token chunks, within-chunk KLD correlation is high: the effective tail sample for p99.9 is ~1-3 chunks, for p99 ~10-15 chunks. For ranking quants with a large gap (30%+), these metrics are reliable; for close quants (<10% in mean KLD), p99 and p99.9 carry high noise. Mean KLD is the primary metric; p95 is secondary.

At matched sizes, the allocator quant has lower mean KLD in every case. The gap is largest at small sizes (e.g. 390 MiB: 0.119 vs 0.378, −69%) and narrowest at the largest size (754 MiB: 0.0023 vs 0.0027, −15%). At 511 MiB vs unsloth IQ4_NL: mean KLD 0.0267 vs 0.0453 (−41%), max KLD 1.92 vs 2.80 (−32%).

Allocator 434 MiB (mean KLD 0.105) is worse than allocator 409 MiB (0.090), even though the allocator's proxy score was better at 434 MiB (total proxy loss 814 vs 1151 — the assignment was optimal for the proxy). The difference is in the token embedding tensor: at 409 MiB the allocator dropped it to IQ4_XS, freeing budget for more important tensors; at 434 MiB it kept it in Q5_K, spending extra budget where it mattered less. The same pattern appears in the unsloth lineup: unsloth UD_Q2_K_XL (433.52 MiB, KLD 0.326) is worse than the smaller unsloth UD_IQ3_XXS (409.23 MiB, KLD 0.241) — Q2_K is a known weak type, and unsloth also made a similar upgrade mistake further up the line (UD_Q6_K_XL → UD_Q5_K_XL on embeddings at a larger size step). Both dips are real (verifiable by GGUF paths in evaluation logs), not swapped data. They show that the proxy correlates with KLD but is not exact — the allocator's proxy is very cheap compared to real KLD runs, but it is an approximation, not a perfect substitute.

Results: wikitext-2 test (all models)

Same comparison on an unrelated corpus — all 16 models evaluated:

Model	Size (MiB)	Mean PPL	Mean KLD	p95 KLD	p99 KLD*	p99.9 KLD*	Max KLD	RMS Δp	Same top p	Top flip weight
BF16 baseline	1485.50	15.508	—	—	—	—	—	—	—	—
allocator 390	390.00	16.484	0.1317	0.3638	0.6880	1.7232	8.9320	8.640	81.366	0.020899
unsloth UD_IQ2_M	390.37	21.811	0.3751	1.0215	1.9303	4.3372	15.2365	15.452	69.864	0.057970
allocator 409	409.00	15.976	0.0955	0.2621	0.4978	1.2375	10.1608	7.449	83.795	0.015462
unsloth UD_IQ3_XXS	409.23	19.021	0.2349	0.6752	1.3078	3.2066	10.0378	11.987	76.417	0.035573
allocator 434	434.00	16.607	0.1094	0.3123	0.5990	1.5675	9.1137	7.884	83.556	0.017057
unsloth UD_Q2_K_XL	433.52	20.656	0.3073	0.8828	1.6389	3.8417	14.4516	13.264	73.604	0.045124
allocator 485	484.99	16.135	0.0414	0.1164	0.2225	0.5705	4.3733	4.836	89.341	0.006592
unsloth UD_Q3_K_XL	484.54	16.698	0.0860	0.2471	0.4670	1.1557	6.5099	7.053	84.754	0.013869
allocator 511	510.99	15.906	0.0289	0.0806	0.1580	0.4358	2.9061	4.096	90.897	0.004670
unsloth IQ4_NL	510.59	16.186	0.0469	0.1325	0.2603	0.7065	4.0409	5.153	88.837	0.007454
allocator 546	545.99	15.816	0.0266	0.0754	0.1493	0.4091	3.8512	3.873	91.478	0.004207
unsloth UD_Q4_K_XL	546.44	15.610	0.0301	0.0854	0.1581	0.3696	2.2790	3.993	90.831	0.004693
allocator 586	585.99	15.806	0.0121	0.0339	0.0653	0.1846	2.1765	2.655	93.896	0.001941
unsloth UD_Q5_K_XL	585.75	15.722	0.0125	0.0340	0.0632	0.1729	1.7454	2.618	93.738	0.001965
allocator 754	753.97	15.600	0.0026	0.0070	0.0132	0.0369	0.3462	1.249	96.695	0.000399
unsloth UD_Q6_K_XL	754.21	15.622	0.0028	0.0075	0.0137	0.0336	0.2571	1.278	96.739	0.000394

* Same caveat as for kilo-small: ~127 chunks, ~145K per-token KLD values, effective tail sample constrained by within-chunk correlation. Mean KLD is the primary metric.

Relative ranking preserved: allocator quants have lower mean KLD at matched sizes on an unrelated corpus. Absolute KLD values are higher (wikitext-2 is a harder domain for this model), but the allocator vs unsloth gap remains consistent. At 511 MiB vs unsloth IQ4_NL: mean KLD 0.0289 vs 0.0469 (−38%), max KLD 2.91 vs 4.04 (−28%).

Notes

The allocator is a prototype, tested here on a single small model. Results on larger models may differ.
Evaluation uses a single calibration corpus hold-out and one standard corpus; results reflect these specific distributions.
Perplexity values are per-chunk means with 2304-token chunks; they are internally consistent but not directly comparable to PPL reported with different chunk sizes.
Both the allocator and unsloth quants preserve MTP/NextN tensors; quality differences reflect type assignment strategy, not MTP handling.

Acknowledgments

Alibaba and Qwen for the best open models that can run on consumer hardware
Unsloth for quantized models used in comparisons and for setting a high quality bar
michaelw9999 for ideas, inspiration, and advanced-gguf-quantizer

NikiKrutan/Qwen3.5-0.8B-MTP-GGUF

niki-allocator: Qwen3.5-0.8B quantized models