Qwimi-3.6-27B-Coder-MTP-GGUF

---

Contents

1. Source & conversion
2. Available quants
3. Which quant should I use?
4. Important: llama.cpp version requirement
5. How to run
6. Tool-calling with llama.cpp
7. Training summary
8. Known limitations
9. Benchmarks

---

1. Source & conversion

• Source: trjxter/Qwimi-3.6-27B-BF16 (merged full-precision model)
• Converted with: llama.cpp — convert_hf_to_gguf.py --outtype bf16, then llama-quantize per target quant
• Build: CPU-only (GGML_CUDA=OFF); quantization is a CPU-bound process and does not require GPU
• Conversion note: the merged repo's tokenizer_config.json originally carried a transformers-v5-internal tokenizer_class value that AutoTokenizer (used internally by the converter) could not resolve. This was corrected before conversion; if you reproduce this conversion yourself from an older snapshot of the BF16 repo, verify tokenizer_config.json reflects the base model's tokenizer class first.

2. Available quants

3. Which quant should I use?

• Best balance of quality/size: Q4_K_M — the standard recommended default for most use.
• Near-lossless, if you have the VRAM/RAM: Q8_0 (~30 GB) or Q6_K (~23 GB).
• Constrained VRAM: Q4_K_S or Q3_K_M/Q3_K_L — expect a noticeable but usually acceptable quality drop versus Q4_K_M.
• Extreme constraint: Q2_K_L — usable, but at 2-bit the model's coding accuracy and instruction-following will degrade more than the size savings suggest; only recommended if no larger quant fits your hardware.

4. Important: llama.cpp version requirement

5. How to run

6. Tool-calling with llama.cpp

7. Training summary

• Base: unsloth/Qwen3.6-27B, dense 27.8B, text-only SFT (vision frozen)
• Method: 4-bit QLoRA, r=64/alpha=64, language layers only
• Data: mixed single-run SFT — coding (82.5% of tokens) + tool-calling (4.8%) + agentic SWE-agent trajectories (12.7%), 22,359 total training rows, 72.5M training tokens
• 1 epoch, effective batch 16, 1,398 optimizer steps, A100 80GB
• Final eval loss: coding 0.4761, tool 0.0208, agentic 0.2892 — all three domains improved monotonically through the full run with no overfitting signal

8. Known limitations

• Quantization introduces its own quality loss on top of the base model's characteristics, increasing with lower bit-width — see §3 for guidance.
• This is a fresh/bleeding-edge architecture in llama.cpp (§4) — expect rougher edges in tooling support than for long-established architectures.

9. Benchmarks

1. A fully auto-scored 300-task custom benchmark covering coding, tool-calling, and agentic workflows.
2. A broader 200-task mixed benchmark containing standard coding tasks and personal Qwimi-oriented evaluations.

Important: the latency figures below measure average end-to-end wall time per task, not raw tokens per second. SFT can reduce task completion time by producing shorter, more direct, better-formatted responses and reaching the stop condition sooner. The results do not prove that the underlying Q6 inference kernels became faster.

---

9.1 Custom 300-task benchmark

Overall results

Category results

Pairwise outcomes

Category-level pairwise outcomes

---

9.2 Mixed 200-task benchmark

Overall auto-scored results

Pairwise auto-scored results

Category highlights

---

9.3 Combined interpretation

• Higher overall pass rate in both runs.
• Stronger performance on custom coding, tool-calling, HumanEval, and personal XML-tool tasks.
• A positive pairwise record:
  • 23–19 on the 300-task custom benchmark.
  • 7–2 on decisive paired tasks in the 200-task benchmark.
• Substantially lower task-completion time:
  • 49.4% lower on the custom 300-task suite.
  • 23.2% lower on the mixed 200-task suite.
• A remaining weakness in some areas:
  • -4.00 points on custom agentic accuracy.
  • -3.85 points on MBPP.

Qwimi is a modest but measurable coding/tool-use improvement over the base model, with unusually large task-level latency reductions. Its largest quality gains appear in coding and structured tool usage, while agentic and MBPP performance require further work.

---

9.4 Interpreting the speed results

• reaching the correct solution with fewer tokens;
• producing less unnecessary explanation or repetition;
• following the expected output format sooner;
• avoiding failed attempts, malformed tool calls, or self-correction loops;
• emitting the stop token earlier.

Qwimi completed the benchmark tasks faster.

---

9.5 Benchmark limitations

• Only the Q6 GGUF builds were compared; these results should not be assumed to apply identically to every quant listed in §2.
• Wall time depends on hardware, llama.cpp build, prompt length, output length, sampling settings, MTP support, and stop conditions.
• The 300-task suite is custom-authored and is not a replacement for established public benchmarks.
• Some categories in the 200-task run were excluded because their official scorer/export integration was incomplete; they were not counted as zero.
• Agentic tasks used keyword/structural auto-scoring and may not capture every aspect of repository-level software-engineering quality.
• More seeds, repeated runs, and token-level throughput measurements would improve confidence in the latency conclusions.