GGUF quantizations of ornith-ai/Ornith-1.5-35B-A3B, a sparse MoE model (256 experts, 8 active per token) with hybrid linear/full attention and an MTP (multi-token-prediction / NextN) head. This repo covers the low-bit IQ ladder from IQ1_S through IQ3_M, quantized with a mixed-corpus importance matrix for better accuracy at the smallest sizes.

Built with a mixed calibration corpus rather than plain wikitext, to better match this model's actual use cases and its MoE expert-coverage needs:

~11M characters, tokenized with the model's own chat template, run through llama-imatrix (-c 1024 -b 1024 -ub 1024 -np 2). Final calibration PPL: 3.9309 +/- 0.00885.

blk.40 is the model's MTP/NextN head. It sits structurally outside the normal forward pass, so no calibration corpus — however good — can ever give it real importance-matrix coverage (llama-imatrix never activates it during a forward pass, confirmed in the load logs of every serving attempt). Quantizing it at the same bit-width as the rest of the ladder fails outright at IQ1_S with "Missing importance matrix for tensor blk.40.attn_k.weight in a very low-bit quantization".

Every quant in this repo pins blk.40.* to Q8_0 via --tensor-type blk.40=q8_0 regardless of the overall quant level, and this is verified programmatically after each quantize (every blk.40.* tensor confirmed Q8_0 or F32 before upload). This adds a small, fixed amount of size to every file but keeps the MTP head numerically sound at every bit-width in the ladder.

(Sizes for IQ3_XS/IQ3_S/IQ3_M will be filled in as each finishes quantizing and uploading.)

Standard llama.cpp builds will load and run these normally; the MTP/NextN tensors are inert unless served with an MTP-aware fork (--spec-type draft-mtp), in which case they can be used for self-speculative decoding.