Kimi K3 is a ~2.9T parameter (104B active) mixture of experts model: 93 layers, 896 experts with 16 active per token, 7168 hidden size, 1M context, trained natively in MXFP4. The original only ships in MXFP4, so everything here comes from a direct MXFP4 to Q8_0 conversion, and the lower quants are requantized from that Q8_0.
Honesty first: this thing is enormous
Let's be upfront. This is a monster. The smallest quant we make, IQ1_S at ~1.56 bits, still weighs about 540 GB. That already blows past the 512 GB that even well specced machines top out at. To actually run any of these you need enough fast memory (RAM + VRAM combined) to hold the whole file. In practice that means a large RAM server or a serious multi GPU rig, not a laptop and not most workstations.
We know most people won't be able to run these locally, and that's okay. We made them for the few setups that can, for people who want to poke at the model, and as an archival base so the quants already exist the day the hardware or the tooling catches up. Rule of thumb: grab the biggest quant your memory can actually hold.
Does it even work?
Yes. After computing a quant, we run a full perplexity pass using llama-perplexity, and Q3_K_S came out at PPL 1.6066 +/- 0.02582. That is a healthy number. It means the quantized model is coherent and not broken (a broken quant gives perplexity in the tens or hundreds, or NaN).
Then, run a model with projector using --mmproj flag and include image with --image:
bash
1./build/bin/llama-mtmd-cli -m ./IQ1_S/Kimi-K3-IQ1_S-00001-of-00014.gguf \2 --mmproj ./mmproj-F16.gguf --image /tmp/test.jpg \3 -p "Describe this image in detail." --jinja
Plain mainline llama.cpp will reject these files with an unknown architecture error until Kimi K3 support lands upstream (tracking: ggml-org/llama.cpp#26185).
Vision works now!
Kimi K3 is a native vision language model, and as of this update the vision path works end to end in our fork: MoonViT-V2 support in clip/mtmd plus the converted vision tower, published right here as mmproj-F16.gguf and mmproj-BF16.gguf at the repo root. One small mmproj pairs with any text quant above, so nothing has to be re-downloaded. Verified with Q3_K_S: the model reads real images and describes them accurately.
Quants, quality
[!NOTE]
Every number is measured against Q8_0 as a base
Quant
Size
PPL(Q)
PPL(Q) / PPL(base)
Mean KL div
Top-1 match (%)
Δp RMS (%)
IQ1_S
590 GB
2.5244
1.7376
0.5392
78.706
35.446
IQ1_M
690 GB
2.1719
1.4949
0.3957
82.588
30.284
IQ2_XXS
769 GB
1.8794
1.2936
0.2594
87.231
23.883
Q3_K_S
1197 GB
1.8541
1.2762
0.2384
87.694
22.617
Q8_0
2955 GB
1.4528
1.00
0
100
0
PPL(q) - perplexity of the quant itself.
PPL / Q8_0 - the same number relative to
the reference, so 1.00 is indistinguishable and 1.63 means 63 % more perplexed;
KL div - how far the whole output distribution drifted, in nats.
Top-1 match - how often the quant picks the same next token as Q8_0; read this one if you read only one.
Δp RMS - how much its confidence moved, in percentage points.
These do not compare with numbers from other repositories: every publisher uses a different
reference model and a different corpus.
Speed
Measured with llama-bench on a single CPU box, no GPU: AMD EPYC 9554P (64 cores, 128 threads, 3.10 GHz), 1.5 TB RAM, 110 threads, model fully resident in RAM (page cache).
Quant
Size
pp512 (t/s)
tg128 (t/s)
IQ1_S
549 GiB
8.91
1.82
IQ1_M
643 GiB
IQ2_XXS
717 GiB
Q3_K_S
1115 GiB
10.91
1.70
Note the crossover. IQ1_S generates faster because token generation is memory-bandwidth bound and there is simply less weight to stream per token. It processes prompts slower because prompt processing is compute bound, and unpacking 1.5-bit codes costs more CPU per weight than 3-bit K-quants. Pick by workload: long prompts favor Q3_K_S, long generations favor IQ1_S.
About the imatrix
imatrix.dat was computed over 350 chunks of 512 tokens on a mixed corpus: bartowski's calibration_datav3, wikitext-2 train, source code, and Wikipedia.
That breadth matters here: with 896 experts and only 16 firing per token, a narrow corpus leaves most experts undersampled, and their weights fall back to default quantization. Expert coverage on this run landed at 99.4 to 99.8 percent.
Source is the Q3_K_S quant, not the full Q8_0: at 2.9 TB the Q8 does not fit in our 1.5 TB of RAM, while Q3_K_S at 1.2 TB does and runs cleanly on CPU. Computing an imatrix from a quantized model is standard practice, and its effect on the final 1 to 2 bit outputs is second order next to the quantization itself.
Attribution and license
This is a derivative of Kimi K3 by Moonshot AI, redistributed under the Kimi K3 License. The original copyright notice and license are kept in LICENSE.
