Views
No views yet
[!NOTE]
This repo contains Multi-Token Prediction (MTP) GGUF for LLaMA.cpp extracted from the base model (Qwen/Qwen3.6-27B).
It can be paired with a target model using the--spec-draft-modelflag.
See PR: https://github.com/ggml-org/llama.cpp/pull/22673If you’re looking for an MTP GGUF for transplanting/"grafting" onto your model, check out:
yes it can be loaded separately using--spec-draft-model. Theconvert_hf_to_gguf.pychanges have an option of--mtpwhich just outputs the MTP gguf.Using the "grafted" on MTP is more VRAM efficient though.Another thing is that-hfoption will try to look for the MTP gguf like it does formmprojin casespec-draft-type draft-mtpis mentioned.
python convert_hf_to_gguf.py ../Qwen3.6-27B --outtype bf16 --outfile ../Qwen3.6-27B-MTP/Qwen3.6-27B-MTP-bf16.gguf --mtp1INFO:hf-to-gguf:Loading model: Qwen3.6-27B
2INFO:hf-to-gguf:Model architecture: Qwen3_5ForConditionalGeneration
3INFO:hf-to-gguf:gguf: indexing model part 'model-00001-of-00015.safetensors'
4INFO:hf-to-gguf:gguf: indexing model part 'model-00008-of-00015.safetensors'
5INFO:hf-to-gguf:gguf: indexing model part 'model-00013-of-00015.safetensors'
6INFO:hf-to-gguf:gguf: indexing model part 'model-00015-of-00015.safetensors'
7INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
8
9INFO:hf-to-gguf:Exporting model...
10INFO:hf-to-gguf:token_embd.weight, torch.bfloat16 --> BF16, shape = {5120, 248320}
11INFO:hf-to-gguf:output.weight, torch.bfloat16 --> BF16, shape = {5120, 248320}
12INFO:hf-to-gguf:blk.64.nextn.eh_proj.weight, torch.bfloat16 --> BF16, shape = {10240, 5120}
13INFO:hf-to-gguf:blk.64.ffn_down.weight, torch.bfloat16 --> BF16, shape = {17408, 5120}
14INFO:hf-to-gguf:blk.64.ffn_gate.weight, torch.bfloat16 --> BF16, shape = {5120, 17408}
15INFO:hf-to-gguf:blk.64.ffn_up.weight, torch.bfloat16 --> BF16, shape = {5120, 17408}
16INFO:hf-to-gguf:blk.64.attn_k.weight, torch.bfloat16 --> BF16, shape = {5120, 1024}
17INFO:hf-to-gguf:blk.64.attn_q.weight, torch.bfloat16 --> BF16, shape = {5120, 12288}
18INFO:hf-to-gguf:blk.64.attn_v.weight, torch.bfloat16 --> BF16, shape = {5120, 1024}
19INFO:hf-to-gguf:output_norm.weight, torch.bfloat16 --> F32, shape = {5120}
20INFO:hf-to-gguf:blk.64.attn_norm.weight, torch.bfloat16 --> F32, shape = {5120}
21INFO:hf-to-gguf:blk.64.post_attention_norm.weight, torch.bfloat16 --> F32, shape = {5120}
22INFO:hf-to-gguf:blk.64.attn_k_norm.weight, torch.bfloat16 --> F32, shape = {256}
23INFO:hf-to-gguf:blk.64.attn_output.weight, torch.bfloat16 --> BF16, shape = {6144, 5120}
24INFO:hf-to-gguf:blk.64.attn_q_norm.weight, torch.bfloat16 --> F32, shape = {256}
25INFO:hf-to-gguf:blk.64.nextn.shared_head_norm.weight, torch.bfloat16 --> F32, shape = {5120}
26INFO:hf-to-gguf:blk.64.nextn.enorm.weight, torch.bfloat16 --> F32, shape = {5120}
27INFO:hf-to-gguf:blk.64.nextn.hnorm.weight, torch.bfloat16 --> F32, shape = {5120}
28
29INFO:gguf.gguf_writer:Writing the following files:
30INFO:gguf.gguf_writer:..\Qwen3.6-27B-MTP\Qwen3.6-27B-MTP-bf16.gguf: n_tensors = 18, total_size = 5.9G
31Writing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 5.94G/5.94G [00:46<00:00, 127Mbyte/s]
32INFO:hf-to-gguf:Model successfully exported to ..\Qwen3.6-27B-MTP\Qwen3.6-27B-MTP-bf16.gguf
1version = 1
2
3[*]
4flash-attn = on
5mlock = off
6mmap = on
7fit = on
8warmup = on
9batch-size = 256
10ubatch-size = 256
11cache-type-k = q4_0
12cache-type-v = q4_0
13kv-unified = true
14swa-full = true
15jinja = true
16direct-io = off
17cache-prompt = true
18cache-ram = 28672
19n-gpu-layers = 99
20reasoning = off
21reasoning-budget = 0
22min-p = 0
23presence-penalty = 1.5
24top-k = 40
25chat-template-kwargs = {"preserve_thinking": true}
26spec-default = true
27ctx-checkpoints = 64
28parallel = 1
29threads-http = 1
30ctx-size = 131072
31
32# --- MODELS ---
33[TeichAI/Qwen3.6-27B-Fable-5-Experimental-GGUF]
34alias = TeichAI/Qwen3.6-27B-Fable-5-Experimental-GGUF
35model = /root/.cache/llama.cpp/TeichAI/Qwen3.6-27B-Fable-5-Experimental-GGUF/Qwen3.6-27B-Fable-5-Distill.iq4_nl.gguf
36mmproj = /root/.cache/llama.cpp/mmproj/mmproj-Qwen3.6-27B-BF16.gguf
37spec-draft-model = /root/.cache/llama.cpp/mtp/Qwen3.6-27B-MTP-q4_0.gguf
38temperature = 0.7
39top-k = 20
40top-p = 0.8
41presence-penalty = 1.5
42repeat-penalty = 1.0
43seed = 42
44spec-type = draft-mtp,ngram-mod,ngram-map-k4v
45spec-draft-n-max = 3
46spec-draft-p-min = 0.50
47spec-draft-prio = 2
48spec-draft-prio-batch = 2
49spec-ngram-mod-n-match = 24
50spec-ngram-mod-n-min = 48
51spec-ngram-mod-n-max = 64
52spec-ngram-map-k4v-size-n = 8
53spec-ngram-map-k4v-size-m = 24
54spec-ngram-map-k4v-min-hits = 2