Base model: Qwen/Qwen3-Coder-Next
Perplexity (WikiText-2 test set, ctx=512, 584 chunks)
| Model | PPL | ± |
|---|
| Qwen3-Coder-Next-Q3mix (this) | 8.4837 | 0.06573 |
| Ubergarm smol IQ3_KS (reference) | 8.4649 | 0.06633 |
Model is compatible with mainline llama.cpp. However test was ran with
llama.cpp TurboQuant build](
https://github.com/TheTom/llama-cpp-turboquant)
Take note benchmark was not ran using mainline llama.cpp and ubergarm
model was benchmarked using ik_llama.cpp so test result is not exactly comparable.
This quant is inspired by ubergarm
/
Qwen3-Coder-Next-GGUF smol-IQ3_KS 30.728 GiB (3.313 BPW)
Qwen3-Coder-Next-Q3mix was quantized using:
#!/usr/bin/env bash
cat > /tmp/qwen3_tensors.txt << 'EOF'
attn_gate.weight=q6_k
attn_qkv.weight=q6_k
attn_output.weight=q6_k
attn_q.weight=q6_k
attn_k.weight=q6_k
attn_v.weight=q6_k
ssm_ba.weight=q6_k
ssm_out.weight=q6_k
ffn_down_shexp.weight=q6_k
ffn_gate_shexp.weight=q6_k
ffn_up_shexp.weight=q6_k
ffn_gate_inp.weight=q8_0
ffn_gate_inp_shexp.weight=q8_0
ffn_down_exps.weight=iq3_s
ffn_gate_exps.weight=iq3_s
ffn_up_exps.weight=iq3_s
token_embd.weight=iq4_nl
output.weight=q6_k
EOF
~/Documents/llama-cpp-turboquant/build/bin/llama-quantize
--tensor-type-file /tmp/qwen3_tensors.txt
--imatrix ~/imatrix-Qwen3-Coder-Next-BF16.dat
~/Qwen3-Coder-Next-f16.gguf
~/Qwen3-Coder-Next-Q3mix.gguf
IQ3_S
$(nproc)
So huge thanks for ubergarm for inspiration and expertise. This model is not intended to rival.
It's from personal need to run on mainline llama.cpp on 12gb vram and 32gb ram.
Best Practices from the base models card:
To achieve optimal performance, we recommend the following sampling parameters: temperature=1.0, top_p=0.95, top_k=40.
pipeline_tag: text-generation
tags:
