However, sometimes you want to see if a more classic quantization will be more suitable for your specific task, or simply want to fit (or use) a specific vram target. In this case, "one can't go wrong with bartowski builds" (c) reddit -- except for these are sadly missing any draft models, and a 20-30 %% gain in inference speed is not something to easily give up on.

Hence, I'll be adding here some "franken-merges" of original bartowski builds with 20 MTP layers added to it from unsloth/Qwen3.6-35B-A3B-MTP-GGUF model using a [conversion script] made by buzz.

All uploaded models were tested and add a ~20-30 % gain in inference speed compared to original ("ungrafted") models on AMD hardware.

(It is strongly recommended to use at most q8_0 quantizations for KV cache if your hardware supoorts it, especially if it has a chance to be more performant on it.)

llama.cpp invocation examples

This likely won't be too bad as a general suggestion -- but can be improved further by tailoring for a specific vram amount or device type; some examples are below:

llama-server  \
--model ${GGUF_FILENAME}  \
--flash-attn on  \
--n-gpu-layers-draft all  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

NB. For all of the examples below we assume a single user who mostly works on sequential and related tasks; your work pattern might be different, so check what works best for you.

llama.cpp invocation examples - find optimal mtp length

As for --spec-draft-n-max 4 setting above -- the exact number would obviously depend on your GPU model.

You might want to do a log scale search by trying

spec-draft-n-max := 1,2,4,8,... (unlikely more) while the inference speed grows or stays the same,
and then using binary search to narrow down the exact value that works best for you on your typical tasks;
for a quick control check, your server log will show something like:
- nb: showing results for the same model running two consequitive coding tasks

[37813] 812.35.434.815 I slot launch_slot_: id  0 | task 56272 | processing task, is_child = 0
[37813] 812.37.748.230 I slot print_timing: id  0 | task 56272 | prompt eval time =     310.31 ms /   177 tokens (    1.75 ms per token,   570.40 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |        eval time =    2002.97 ms /   400 tokens (    5.01 ms per token,   199.70 tokens per second)
[37813] 812.37.748.234 I slot print_timing: id  0 | task 56272 |       total time =    2313.28 ms /   577 tokens
[37813] 812.37.748.235 I slot print_timing: id  0 | task 56272 |    graphs reused =      52595
[37813] 812.37.748.238 I slot print_timing: id  0 | task 56272 | draft acceptance = 0.95482 (  317 accepted /   332 generated), mean len =  4.82
[37813] 812.37.749.091 I slot      release: id  0 | task 56272 | stop processing: n_tokens = 29120, truncated = 0

[37813] 812.38.114.222 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.988 (> 0.100 thold), f_keep = 1.000
[37813] 812.38.115.131 I slot launch_slot_: id  0 | task 56358 | processing task, is_child = 0
[37813] 812.38.767.224 I slot print_timing: id  0 | task 56358 | prompt eval time =     440.40 ms /   341 tokens (    1.29 ms per token,   774.29 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |        eval time =     211.57 ms /    20 tokens (   10.58 ms per token,    94.53 tokens per second)
[37813] 812.38.767.228 I slot print_timing: id  0 | task 56358 |       total time =     651.97 ms /   361 tokens
[37813] 812.38.767.229 I slot print_timing: id  0 | task 56358 |    graphs reused =      52603
[37813] 812.38.767.232 I slot print_timing: id  0 | task 56358 | draft acceptance = 0.33333 (   12 accepted /    36 generated), mean len =  2.33

As you can see, we could have also tried --spec-draft-n-max 5 and see if that would improve the benchmarks: basically as long as "mean len" frequently shows values which are greater than your spec-draft-n-max parameter -- you can still increment it by at least 1 and see if it improves your statistics (and waiting times).

Also, in some scenarios there could be some use for additional settings like --spec-draft-p-min 0.5.

I am no expert, but in my tests I had some luck with first determining the optimal --spec-draft-n-max value, and then seeing if raising up the acceptance min level (and/or the "split" values such as --spec-draft-p-split) will improve things a little more.

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf

this fits 24 GiB VRAM due to using quantized KV cache

NB: below, you would want to either disable reasoning (to make response times shorter for the price of sometimes less elaborate outcome), or enable reasoning, in which case --reasoning-preserve option might speed up things for you for certain work patterns.

(And so you would normally want to omit either --chat-template-kwargs or --reasoning-preserve entries below; however, if you just leave them as they are, it would still work.)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf  \
--no-mmproj  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning auto \
--chat-template-kwargs '{"enable_thinking":false}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL.bartowski.mtp.q8_k_xl.gguf

this shall also fit 24 GiB VRAM by using quantized KV cache and a slightly reduced context window size ;
for VRAM <= 16 GiB -- some further sacrifices are likely to be made, such as -cmoe, and possibly --n-gpu-layers auto
for mini-pc devices with integrated gpu memory, you would probably have enough ram, but you'd likely want to decrease --spec-draft-n-max -- somewhere down to 1 or 2, depending on your hardware (your mileage may vary, see above)

(Also see the note above on --reasoning-preserve option)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL.bartowski.mtp.gguf  \
--no-mmproj  \
--ctx-size 200000  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--chat-template-kwargs '{"enable_thinking":true}'  \
--reasoning-preserve  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S.bartowski.mtp.gguf

this shall still fit 24 GiB VRAM by using quantized KV cache and a substantially reduced context window size ;
for mini-pc devices with integrated gpu memory, you would probably have enough ram, but you'd likely want to decrease --spec-draft-n-max -- somewhere down to 1 or 2, depending on your hardware (your mileage may vary, see above)

(Also see the note above on --reasoning-preserve option)

llama-server  \
--model Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S.bartowski.mtp.gguf  \
--no-mmproj  \
--ctx-size 40960  \
--flash-attn on  \
--n-gpu-layers all  \
--n-gpu-layers-draft all  \
--kv-offload  \
--kv-unified  \
--cache-type-k q8_0  \
--cache-type-v q8_0  \
--temperature 1.0  \
--top-k 20  \
--top-p 0.95  \
--min-p 0  \
--reasoning auto \
--reasoning-preserve  \
--chat-template-kwargs '{"enable_thinking":true}'  \
--spec-type draft-mtp  \
--spec-draft-n-max 4  \
--cache-type-k-draft q8_0  \
--cache-type-v-draft q8_0

thread13/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF-MTP

About

llama.cpp invocation examples

llama.cpp invocation examples - find optimal mtp length

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.q8_k_xl.gguf

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL.bartowski.mtp.q8_k_xl.gguf

llama.cpp invocation examples - Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S.bartowski.mtp.gguf

thread13/
Kwaipilot_KAT-Coder-V2.5-Dev-GGUF-MTP