Views
No views yet
smol-IQ3_KS scored 77.3% on aider polyglot benchmark with 2x speed-up over similar sized mainline UD-IQ3_XXS! Details in discussion 14 here. Thanks Fernanda24!Q4_X quant runs on both on mainline llama.cpp and ik_llama.cpp. The other quants in this collection REQUIRE ik_llama.cpp fork to support the ik's latest SOTA quants and optimizations! Do not download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc!ik_llama.cpp can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.Q4_X patch!
Q4_X version scores perplexity equivalent to a full 1TB Q8_0 test quant using a one line patch to adjust q4_0 to better fit the original QAT target quantization. Discussions ongoing on llama.cpp PR#17064 and directly with moonshot on their huggingface discussions ai as it seems they only used 15 of 16 possible 4bit values possibly?1#!/usr/bin/env bash
2
3# Q4_0 (patched) routed experts approximating original QAT design
4# Q8_0 everything else
5
6custom="
7## Attention [0-60] (GPU)
8blk\..*\.attn_k_b\.weight=q8_0
9blk\..*\.attn_v_b\.weight=q8_0
10
11# Balance of attn tensors
12blk\..*\.attn_kv_a_mqa\.weight=q8_0
13blk\..*\.attn_q_a\.weight=q8_0
14blk\..*\.attn_q_b\.weight=q8_0
15blk\..*\.attn_output\.weight=q8_0
16
17## First Single Dense Layer [0] (GPU)
18blk\..*\.ffn_down\.weight=q8_0
19blk\..*\.ffn_(gate|up)\.weight=q8_0
20
21## Shared Expert [1-60] (GPU)
22blk\..*\.ffn_down_shexp\.weight=q8_0
23blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
24
25## Routed Experts [1-60] (CPU)
26blk\..*\.ffn_down_exps\.weight=q4_0
27blk\..*\.ffn_(gate|up)_exps\.weight=q4_0
28
29token_embd\.weight=q8_0
30output\.weight=q8_0
31"
32
33custom=$(
34 echo "$custom" | grep -v '^#' | \
35 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
36)
37
38numactl -N ${SOCKET} -m ${SOCKET} \
39./build/bin/llama-quantize \
40 --custom-q "$custom" \
41 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
42 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-Q8_0-Q4_0.gguf \
43 Q8_0 \
44 1281#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq4_kss
24blk\..*\.ffn_(gate|up)_exps\.weight=iq4_kss
25
26token_embd\.weight=iq6_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-smol-IQ4_KSS.gguf \
41 IQ4_KSS \
42 128474.772 GiB (3.973 BPW) and will be squash deleted to save on public quota soon. This new one uses q4_x patched and only applies imatrix to the iq3_k tensors but not to the q8_0 or q4_x. More details in discussion 4 here. It has almost the same perplexity so a good improvement.1#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=q4_0
24blk\..*\.ffn_(gate|up)_exps\.weight=iq3_k
25
26token_embd\.weight=iq6_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 --include-weights ffn_gate_exps \
40 --include-weights ffn_up_exps \
41 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
42 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-IQ3_K.gguf \
43 IQ3_K \
44 1281#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq3_ks
24blk\..*\.ffn_(gate|up)_exps\.weight=iq3_ks
25
26token_embd\.weight=iq4_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-smol-IQ3_KS.gguf \
41 IQ3_KS \
42 1281#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq3_ks
24blk\..*\.ffn_(gate|up)_exps\.weight=iq2_kl
25
26token_embd\.weight=iq4_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-IQ2_KL.gguf \
41 IQ2_KL \
42 1281#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq2_kl
24blk\..*\.ffn_(gate|up)_exps\.weight=iq2_kl
25
26token_embd\.weight=iq4_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-smol-IQ2_KL.gguf \
41 IQ2_KL \
42 1281#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq2_ks
24blk\..*\.ffn_(gate|up)_exps\.weight=iq2_ks
25
26token_embd\.weight=iq4_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-smol-IQ2_KS.gguf \
41 IQ2_KS \
42 128KT trellis quants generally are slower during TG given likely compute bottleneck if running on CPU, but if it is all you can fit then well...1#!/usr/bin/env bash
2
3custom="
4## Attention [0-60] (GPU)
5blk\..*\.attn_k_b\.weight=q8_0
6blk\..*\.attn_v_b\.weight=q8_0
7
8# Balance of attn tensors
9blk\..*\.attn_kv_a_mqa\.weight=q8_0
10blk\..*\.attn_q_a\.weight=q8_0
11blk\..*\.attn_q_b\.weight=q8_0
12blk\..*\.attn_output\.weight=q8_0
13
14## First Single Dense Layer [0] (GPU)
15blk\..*\.ffn_down\.weight=q8_0
16blk\..*\.ffn_(gate|up)\.weight=q8_0
17
18## Shared Expert [1-60] (GPU)
19blk\..*\.ffn_down_shexp\.weight=q8_0
20blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
21
22## Routed Experts [1-60] (CPU)
23blk\..*\.ffn_down_exps\.weight=iq1_kt
24blk\..*\.ffn_(gate|up)_exps\.weight=iq1_kt
25
26token_embd\.weight=iq4_k
27output\.weight=iq6_k
28"
29
30custom=$(
31 echo "$custom" | grep -v '^#' | \
32 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
33)
34
35numactl -N ${SOCKET} -m ${SOCKET} \
36./build/bin/llama-quantize \
37 --custom-q "$custom" \
38 --imatrix /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/imatrix-Kimi-K2-Thinking-Q8_0-Q4_0.dat \
39 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/-384x14B-BF16-00001-of-00046.gguf \
40 /mnt/data/models/ubergarm/Kimi-K2-Thinking-GGUF/Kimi-K2-Thinking-IQ1_KT.gguf \
41 IQ1_KT \
42 128--jinja --chat-template-file ./models/templates/Kimi-K2-Thinking.jinja.
You will also need to pass --special for it to output <think> and </think> tags correctly depending on endpoint and client used, thanks u/Melodic-Network4374 but note it will then also print out <|im_end|> so you can set your client to use that as a stop string.1# Example running Hybrid CPU+GPU(s) on ik_llama.cpp
2./build/bin/llama-server \
3 --model "$model"\
4 --alias ubergarm/Kimi-K2-Thinking-GGUF \
5 --ctx-size 32768 \
6 -ctk q8_0 \
7 -mla 3 \
8 -ngl 99 \
9 -ot "blk\.(1|2|3)\.ffn_.*=CUDA0" \
10 -ot "blk\.(4|5|6)\.ffn_.*=CUDA1" \
11 -ot exps=CPU \
12 --parallel 1 \
13 --threads 96 \
14 --threads-batch 128 \
15 --host 127.0.0.1 \
16 --port 8080 \
17 --no-mmap \
18 --jinja \
19 --chat-template-file updatedChatTemplate.jinja \
20 --special
21
22# Example running mainline llama.cpp
23# remove `-mla 3` from commands and you should be :gucci:--no-mmap to mmap() "troll rig" it paging weights read-only off of disk for a couple tok/sec maybe depending.--threads and --threads-batch as needed. For smaller CPUs I recommend setting them both the same equal to the number of physical cores. For an amd 9950x that would be -t 16 for example. Experiment on larger rigs especially with multiple socket NUMA considerations (avoid cross-NUMA memory access if possible).-amb 512 to fix the size of the MLA computation buffers. (only works on models with MLA style attention like Kimi-K2 and DeepSeek)