Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic — GGUF
Every quant below was measured, not assumed — perplexity on wikitext-2 and throughput on an RTX 5070 Ti.
Which file should I download?
Short answer: IQ4_XS-imat for the best all-round tradeoff, Q4_K_M-imat for maximum safety margin.
| Priority | Pick | Size |
|---|
| Best overall balance | IQ4_XS-imat | 2.21 GB |
| Highest fidelity at reasonable size | Q4_K_M-imat | 2.40 GB |
| Smallest usable | IQ3_M-imat | 1.94 GB |
| Near-lossless, size no object | Q6_K | 3.19 GB |
Measured results
F16 baseline perplexity: 17.772. Lower is better; "Δ" is degradation versus F16.
With importance matrix (recommended)
The imatrix was computed on the full wikitext-2 train split (4933 chunks, ctx 512) using the F16 weights. Calibration and evaluation use disjoint splits. pp512 / tg128 are prompt-processing and generation throughput (tokens/s) on an RTX 5070 Ti.
| File | Size | PPL | Δ vs F16 | pp512 t/s | tg128 t/s |
|---|
...-Q4_K_M-imat.gguf | 2.40 GB | 18.125 | +2.0% | 7123 | 151 |
...-Q4_K_S-imat.gguf | 2.33 GB | 18.248 | +2.7% | 7325 | 155 |
...-IQ4_XS-imat.gguf | 2.21 GB | 18.746 | +5.5% | 7875 | 162 |
...-Q3_K_L-imat.gguf | 2.15 GB | 18.972 | +6.8% | 6262 | 148 |
...-IQ3_M-imat.gguf | 1.94 GB | 19.227 | +8.2% | 7223 | 171 |
...-Q3_K_M-imat.gguf | 2.02 GB | 19.293 | +8.6% | 6354 | 155 |
...-Q3_K_S-imat.gguf | 1.86 GB | 21.253 | +19.6% | 5996 | 156 |
...-Q2_K-imat.gguf | 1.64 GB | 25.359 | +42.7% | 5548 | 178 |
Without importance matrix
| File | Size | PPL | Δ vs F16 |
|---|
...-Q6_K.gguf | 3.19 GB | 17.710 | −0.3% |
...-Q8_0.gguf | 4.13 GB | 17.900 | +0.7% |
...-Q4_K_S.gguf | 2.33 GB | 17.978 | +1.2% |
...-Q5_K_M.gguf | 2.78 GB | 18.281 | +2.9% |
...-Q4_K_M.gguf | 2.40 GB | 18.298 | +3.0% |
...-Q3_K_L.gguf | 2.15 GB | 20.434 | +15.0% |
...-Q3_K_M.gguf | 2.02 GB | 20.535 | +15.5% |
...-Q3_K_S.gguf | 1.86 GB | 25.490 | +43.4% |
...-Q2_K.gguf | 1.64 GB | 42.863 | +141% |
...-F16.gguf | 7.77 GB | 17.772 | baseline |
Reading these numbers
- The imatrix matters enormously below 4 bits. Q2_K goes from +141% to +42.7%; Q3_K_M from +15.5% to +8.6%. At 4 bits and above the difference is negligible, which is why the high-precision files are provided without one.
- Q2_K is not recommended at any setting. Even with the imatrix it loses 42% — that is a qualitatively different model, not a slightly duller one. It is published for completeness and for extremely memory-constrained experiments only.
- Above Q4_K_S the quants are statistically indistinguishable from each other (differences smaller than the ±0.15 measurement error). Choose by size, not by decimal places.
- IQ beats K-quants on GPU here. IQ4_XS is smaller and faster than Q3_K_L while scoring better. Note this benchmark is CUDA — on pure CPU inference IQ types are typically slower to dequantize, so K-quants may win on CPU-only setups.
Format integrity
The full-precision model was checked (greedy decoding) for damage to output structure versus the original fine-tune. Abliteration left all three intact:
| Check | Original | Abliterated (bf16) |
|---|
<think> block opens and closes correctly | 10/10 | 10/10 |
| Valid parseable JSON on request | 10/10 | 10/10 |
Well-formed <tool_call> given tools | 10/10 | 10/10 |
These checks were run on the bf16 weights. Aggressive quantization can degrade format adherence independently of abliteration — a Q2_K model (+42% perplexity) may drop JSON quotes or malform tool-calls even though the bf16 source does not. If you need reliable structured output, prefer Q4_K_S or above.
Usage
1llama-cli -m Parable-Nanbeige4.2-3B-Claude-Fable-5-heretic-IQ4_XS-imat.gguf \
2 -ngl 99 \
3 -c 8192 \
4 -sys "You are a helpful assistant." \
5 -p "Explain how HTTPS certificate validation works."
Three flags you actually need
-c — set it explicitly. This is not optional.
This model has a 262144-token context and 44 KV-cache layers. Left at default, llama.cpp will try to allocate roughly 45 GB of KV cache, spill it to system RAM, and inference will crawl. Budget approximately 176 MB of cache per 1024 tokens of context. On a 16 GB card, 32k context is comfortable.
-ngl 99 — offload all layers to GPU.
-sys "..." — override the default system prompt, which is in Chinese (你是南北阁…) and inherited from the base model. Without it, English prompts may get Chinese answers.
Optional: shrink the KV cache
Roughly halves cache memory at minimal quality cost — worth it on this architecture given how large the cache is.
Tool calling
The model emits an XML-style <tool_call> block. Front-ends that support llama.cpp's tool/grammar features (llama-server with --jinja, LM Studio, Open WebUI) can parse it directly using the bundled chat template.
Reasoning block
The model emits a <think> block before answering. Compatible front-ends (llama.cpp server, LM Studio, Open WebUI) will collapse it. To skip reasoning for latency-sensitive extraction work, use a chat template with an empty think block.
Compatibility
Requires a llama.cpp build with Nanbeige looped-transformer support — the runtime must honour the num_loops = 2 metadata and execute 44 layers, not 22. Verified working on b10227 and b10236.
If output is incoherent, your build is too old. Mobile apps bundling older llama.cpp (PocketPal, ChatterUI) may be affected; check that generation is coherent before assuming a quant is broken.
An imatrix.dat (the importance matrix used for the -imat quants) is included for reproducibility — you can use it to roll your own quants from the F16 file.
Provenance
Note: refusal behaviour in these weights is intentionally suppressed. Deploy accordingly.
