At 1.07 GB, M matches the BF16 instruction-strict IFEval score within evaluation variation. It uses 47% less disk than L.
Tier
IFEval P / I (%)
MMLU-Pro (%)
BF16 reference
65.43 / 74.70
—
L
65.80 / 74.94
—
M
66.54 / 75.54
—
S
63.22 / 73.38
—
XS
52.68 / 64.15
—
P / I means prompt-strict / instruction-strict. IFEval uses deterministic non-thinking decoding; MMLU-Pro uses sampled long-form reasoning. Only complete model-level scores are reported. A dash means not reported.
Reasoning: XS is intended for non-thinking use. Run it with --reasoning off. Choose S, M, or L for long-form reasoning.
Evaluation protocol
IFEval: 541 prompts, native chat template, enable_thinking=false, temperature 0.
The headline BF16 comparison uses instruction-strict IFEval; the raw scores are shown in the table.
In matched long-form reasoning diagnostics, XS produced longer trajectories and reached the 32,768-token output limit more often than S. For Qwen 0.8B and 2B, a complete comparable MMLU-Pro matrix was not available at release time. The table omits incomplete subject runs.
How the checkpoints are built
All four checkpoints use the same production PTQ pipeline. The precision map is the only tier-specific part.
1. Fit native GGUF codes
Calibration activations produce a curvature objective weighted by true Fisher information for each quantized projection. We adapt the scale and minimum initialization from NeUQI to that objective, then solve integer codes on the target GGUF grid with a guarded cyclic coordinate-descent solver inspired by QuantEase. A final K-quant pass tunes stored scales and minima while keeping packed codes fixed.
2. Reconstruct the deployed trajectory
Layers are calibrated in execution order against activations from the already-quantized prefix. A dense reference path measures accumulated drift. Quantization Error Propagation (QEP) adds that drift to the next reconstruction target, so later layers optimize for the inputs they receive at inference time.
3. Allocate the byte budget
XS and S can choose Q2_K through Q8_0 for each quantizable group. The optimizer trades changes in the teacher distribution against the actual encoded byte cost, including scale and minimum metadata. ANNA provides constrained configuration search. RCO provides an exact-budget route (code). For this model, ANNA selected both the XS and S precision maps. M and L keep the decoder qtypes at Q4_K_M and Q8_0, respectively, and use the same reconstruction and scale-tuning stages.
After schedule selection, PTQ runs again from the source weights. Each layer is then calibrated with the final upstream precision choices.
4. Align the full model
A short affine distillation pass tunes native FP16 scales and minima while qtypes, packed codes, dense weights, and tensor layouts stay fixed. The loss matches the teacher's next-token distribution without changing file size or runtime layout.
We load-test the shipping GGUF and evaluate it on a held-out set of 3,072 sequences with next-token KL. release-manifest.json records its SHA-256, downstream evaluation IDs, and tensor metadata. Final recommendations use complete-model benchmarks.
File details
Tier
Hub selector
GGUF file type
Whole-file BPW
XS
Q3_K_S
MOSTLY_Q2_K
3.140
S
Q4_K_S
MOSTLY_Q2_K
4.109
M
Q4_K_M
MOSTLY_Q4_K_M
4.562
L
Q8_0
MOSTLY_Q8_0
8.558
The Hub selector controls sidebar grouping and download discovery. For XS and S it approximates the whole-file size class; release-manifest.json contains the exact tensor mix. M and L keep their decoder qtypes at Q4_K_M and Q8_0 within the same production PTQ pipeline.
Runtime memory also includes KV cache and buffers, which grow with context length.
TheStageAI deployment stack
These GGUF files target llama.cpp-compatible runtimes. edge-lm runs compressed MLX models on Macs and iPhones. ANNA searches compression configurations under size or compute constraints. The TheStageAI Platform and documentation cover compression, compilation, and serving workflows.
For a specific device, latency target, or memory budget, talk to our team.
