GGUF quantizations of TinyModels/Atom-350M, for use with llama.cpp and compatible runtimes.
About
Quantizations were produced by using llama.cpp build b10090, via convert_hf_to_gguf.py followed by llama-quantize. Q8_0 and Q6_K are static (non-imatrix) quantizations; Q5_K_M and Q4_K_M use an importance matrix computed with llama-imatrix over 200 chunks of wikitext-2 (wiki.train.raw).
Files
Filename
Quant Type
Size
Description
Atom-350M-Q4_K_M.gguf
Q4_K_M
256 MB
Balanced, recommended default (imatrix)
Atom-350M-Q5_K_M.gguf
Q5_K_M
275 MB
Low quality loss (imatrix)
Atom-350M-Q6_K.gguf
Q6_K
349 MB
Very low quality loss (see note below — many tensors fall back to Q8_0)
Atom-350M-Q8_0.gguf
Q8_0
367 MB
Near-lossless
Atom-350M-F16.gguf
F16
692 MB
Full precision reference
Note on K-quants: this model's tensor dimensions (960, 320, 2560) are not evenly divisible by the 256-element superblock size K-quants require. As a result, 192 of 290 tensors in the Q6_K file fall back to Q8_0 quantization, which is why Q6_K is barely smaller than Q8_0. Expect similar (lesser) fallback behavior in the upcoming Q5_K_M/Q4_K_M builds.
Quantization Comparison
Perplexity and KL-divergence measured on wikitext-2 (wiki.test.raw), 20 chunks, n_ctx=512, using llama-perplexity --kl-divergence against saved F16 reference logits. Tokens/sec measured with llama-bench (pp512/tg128, 4 threads) on Apple Macbook Air M2.
Perplexity & speed
Quant
Size
Perplexity
ΔPPL vs F16
Prompt (t/s)
Gen (t/s)
Notes
F16
692 MB
17.5203
—
1625
51.9
reference
Q8_0
367 MB
17.5818
+0.35%
1664
63.5
near-lossless
Q6_K
349 MB
17.6377
+0.67%
1560
64.0
mostly Q8_0 under the hood, see note above
Q5_K_M
275 MB
17.6334
+0.65%
1359
73.1
imatrix-guided
Q4_K_M
256 MB
17.8882
+2.10%
1555
77.0
imatrix-guided, recommended
KL-divergence vs F16
Mean PPL can mask token-level disagreement — two quants can tie on PPL while differing in the full predicted distribution. KLD/Δp are measured per-token; "Same top token" is the share of tokens where the quant agrees with F16 on the single most-likely next token.
Quant
Mean KLD
Median KLD
99th %ile KLD
RMS Δp
Same top token
Q8_0
0.00118
0.00082
0.00753
0.891%
97.94%
Q6_K
0.00472
0.00281
0.03898
1.706%
96.22%
Q5_K_M
0.01250
0.00709
0.10824
2.815%
93.96%
Q4_K_M
0.02842
0.01685
0.24259
4.131%
90.92%
Notably, Q5_K_M and Q6_K nearly tie on mean PPL (17.63 vs 17.64 above) but not on KLD — Q6_K is ~2.6x lower and has better top-token agreement, so it's the better pick when distributional fidelity matters more than the near-tie in PPL suggests.
Q8_0/Q6_K are static quantizations; Q5_K_M/Q4_K_M use an imatrix computed from 200 chunks of wikitext-2 wiki.train.raw. The same tensor-shape fallback noted above applies to all four.
Which Quant Should I Choose?
Q4_K_M — default pick. Smallest, fastest generation, +2.1% PPL vs F16. Highest KLD (0.028 mean); skip if you need tight distributional fidelity.
Q5_K_M — 275MB middle ground if Q4_K_M's KLD is too high, but Q6_K generally wins.
Q6_K — best top-token agreement short of Q8_0 (96.2%), though the tensor-shape fallback puts it near Q8_0's size. Worth the ~75MB over Q5_K_M when accuracy matters.
Q8_0 — safest, closest to FP16 (97.9% top-token, lowest KLD) if you have ~400MB to spare.
F16 — reference baseline, or if you'll re-quantize/fine-tune.
Search "Atom-350M GGUF", download your preferred quant, load it.
Ollama
ollama run hf.co/syaffers/Atom-350M-GGUF:Q8_0
text-generation-webui
Drop the .gguf file into your models/ folder and select the llama.cpp (GGUF) loader.
Prompt Format
ChatML:
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
