A 4-bit QLoRA SFT Multi-Token Prediction tune of Qwen3.6-27B fine-tuned for no-thinking agentic coding through a PI-style harness. Packaged as llama.cpp-compatible GGUF for local agent loops.
[!TIP]
For the strongest Pi-style coding-agent behavior, use the reasoning-trained release: bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF. See the technical writeup for the broader evaluation context. This no-thinking tune remains useful when you specifically want the lower-latency direct / instruct path.
⚡
MTP Decoding
Multi-Token Prediction drafts future tokens and accepts them when the main decode path agrees — cutting wall time on long reasoning, code generation, and tool-call setup.
🧩
No-Thinking by Design
Trained on Qwen3.6's non-thinking inference path. The model responds directly with tool calls, edits, and structured output — no <thinking> preamble eating wall time before the harness can act.
🧪
Local-First Throughput
Designed for llama.cpp-class runtimes on a single workstation, with MTP draft acceptance measured on real agent workloads.
🚀
Practical Speed
Built for workflows where waiting minutes per turn breaks the loop — the tune favors decisive output over scratch-pad expansion.
Context Window
128k tested
256k natively supported by the base · extensible up to 1M tokens via RoPE scaling.
MTP Draft Acceptance
~78%
Drafted future tokens accepted by the main decode path on agent workloads. 3 speculative steps · 4 draft tokens.
Internal reasoning traces were not exported into the SFT rows
Recommended quant
Q4_K_M as the default starting point
[!NOTE]
Qwen3.6-27B is a causal language model with a vision encoder. Image and video understanding are supported by pairing the language-model GGUF with the compatible Qwen3.6 mmproj-F16.gguf sidecar (see §4 Multi-modal inference). The MTP draft heads are kept at Q8_0 precision inside every quant via --tensor-type nextn=q8_0 at quantize time — speculative decoding works at any quant level, not just Q8_0 / bf16.
🧩 2. Why This Tune
Qwen3.6 supports both a thinking inference mode (which emits a <thinking>...</thinking> block before responding) and a non-thinking mode (where the model answers and acts directly). This release is fine-tuned specifically for the no-thinking path — the mode where agent loops actually live. In a PI-style harness running tool-call loops, every thinking token is wall time the harness can't dispatch the next action against, so the tune is shaped to make the no-thinking path quality where it counts: tool calls, repository edits, terminal commands, verifier feedback, and structured output.
This carries forward Qwen3.6's existing agentic-coding posture — frontend workflows, repository-level reasoning, and tool calling — but pulls the quality into the inference mode that local agent runtimes can budget for.
Terminal and shell task execution.
Repository inspection, patching, and test iteration.
Tool-call-shaped interactions and structured outputs.
DevOps runbooks, environment setup, and debugging loops.
Coding tasks where command use, file edits, and verifier feedback matter.
📦 3. Quantizations
Recommended starting point: Q4_K_M.
Quant
File size
VRAM (approx)
Suggested use
Q2_K
~11 GB
~13 GB
Smallest memory footprint; quality tradeoffs are expected.
Q3_K_S
~12 GB
~15 GB
Low-memory 3-bit option.
Q3_K_M
~14 GB
~16 GB
Balanced 3-bit option.
Q3_K_L
~15 GB
~17 GB
Higher-quality 3-bit option.
Q4_K_S
~16 GB
~18 GB
Smaller 4-bit option.
Q4_K_M
~17 GB
~19 GB
Default recommendation for most local use. Comfortable on a 24 GB GPU.
Q5_K_S
~19 GB
~21 GB
Higher-quality 5-bit option.
Q5_K_M
~20 GB
~22 GB
Strong quality/memory tradeoff; near the upper edge of a 24 GB GPU.
Q6_K
~22 GB
~25 GB
High-quality local inference if you have the memory.
Q8_0
~29 GB
~32 GB
Highest-precision quantized option.
bf16
~55 GB
~58 GB
BF16 GGUF reference, if present.
VRAM figures are rough estimates for GPU-offload inference (-ngl 99 -fa) at a moderate context (~32k) with quantized KV cache; they scale up with longer contexts.
[!IMPORTANT]
Every quant in this release ships with the MTP nextn prediction heads stored at Q8_0 precision, regardless of the overall quant target. That means speculative decoding works at any quant level — pick the smallest one that fits your VRAM and you still get the MTP throughput profile described in §6.
Some files may still be uploading. Check the Files tab for the exact artifacts currently available.
🚀 4. Quickstart
Run with llama.cpp (standard launch — works on any build):
bash
1# 128k context shown; base model natively supports 256k and is extensible to ~1M via RoPE scaling.2# Sampling values match Qwen3.6's recommended non-thinking-mode defaults — this is the inference3# path the tune was trained for, so prefer these.4llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M \5 --jinja -ngl 99 -fa -c 131072\6 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0\7 --presence-penalty 1.5
Run with upstream llama.cpp + MTP speculative decoding (ggml-org/llama.cpp, MTP support merged in PR #22673):
bash
1# The nextn prediction heads in this release activate via upstream's draft-mtp speculator.2# -np must be 1 with MTP (parallel slots are not yet supported alongside MTP).3llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M \4 --spec-type draft-mtp \5 --spec-draft-n-max 3\6 -np 1\7 --jinja -ngl 99 -fa -c 131072\8 --cache-type-k q8_0 --cache-type-v q8_0 \9 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0\10 --presence-penalty 1.5
Run with Ollama:
ollama run hf.co/bytkim/Qwen3.6-27B-MTP-pi-tune-GGUF:Q4_K_M
This release is compatible with the Qwen3.6 mmproj-F16.gguf sidecar for vision-language inference. A single mmproj file pairs with every quant in this release; the projector is architecturally tied to the base model's vision tower, not to the LM quant level, so download it once and reuse it.
The compatible mmproj can be downloaded from unsloth/Qwen3.6-27B-MTP-GGUF. The fine-tune in this release is language-only — the vision encoder weights have not been touched. Image / video understanding is therefore inherited unchanged from the upstream Qwen3.6-27B base model; this release does not claim to improve it, only to preserve it.
llama-server exposes an OpenAI-compatible /v1/chat/completions endpoint, so any client written against the OpenAI SDK can point at it directly — no client changes required:
python
1from openai import OpenAI
23client = OpenAI(4 base_url="http://localhost:8080/v1",5 api_key="not-needed",6)78resp = client.chat.completions.create(9 model="Qwen3.6-27B-MTP-pi-tune",10 messages=[11{"role":"system","content":"You are a precise coding agent."},12{"role":"user","content":"Write a Python function that merges overlapping intervals."},13],14)15print(resp.choices[0].message.content)
The same endpoint accepts tools=[...] for function calling and supports streaming via stream=True.
🧬 5. Training & Data Notes
Tuned on real agent traces, not synthetic generations or distilled chat — every training row is the trail of an assistant that actually executed the task end-to-end through a PI-style harness, exported as Qwen-compatible ChatML rows with tool schemas and runtime prompts preserved.
High-level task coverage spanned:
Terminal and shell-environment agent tasks.
Tool / function-calling interactions.
Multi-language code editing and repair tasks.
Repository issue resolution and test-driven patching.
Coding and API integration tasks.
Shell, package, migration, ops, and verifier-driven tasks.
🧭
Training philosophy
Trained with 4-bit QLoRA SFT. Training rows are exported with internal reasoning traces removed and assistant turns formatted in Qwen3.6's non-thinking style. The quality this release ships is the no-thinking path's quality — that's where the tune was trained, and that's where it should be run.
Specific dataset names and training-row counts are intentionally omitted from this initial card.
⚡ 6. MTP Throughput
MTP stands for Multi-Token Prediction. The model drafts likely future tokens and the runtime accepts them when they agree with the main decode path. On local agent work this matters because long reasoning, code generation, tool-call setup, and shell-oriented turns otherwise spend most of their wall time waiting on generation.
The numbers below describe the current local profile of the release. They are representative figures from internal runs against the PI harness — full benchmark publication is forthcoming and will replace these with task-success-rate tables in §7.
Raw decode profile
Prompt / Prefill
~615 tok/s
Reading and processing prompt / context tokens.
Decode / Generation
~40 tok/s
Raw generated-token speed reported by llama.cpp.
End-to-End Request
~71 tok/s
Full llama request throughput across prompt and decode.
MTP Draft Acceptance
~78%
Share of drafted future tokens accepted by the main decode path.
Agentic throughput
The agentic number is different from raw tokens/sec. It measures real task throughput across agent runs — including model generation, tool calls, shell commands, package installs, file I/O, and verifier-facing work.
Effective Output
~33 tok/s
Output tokens divided by full agent execution time.
Effective Total
~1.6k tok/s
Input + output tokens counted by the harness, end-to-end.
That makes it a more realistic agent-workflow number than plain decode speed — it includes time spent operating through the harness, not just time spent generating text.
📊 7. Coding Eval Benchmarks
🛰️
Releasing soon
Task-success benchmarks are tracked separately from throughput and will be published in a follow-up update to this card. Throughput answers how fast the local MTP stack moves; coding evals answer how often the agent actually solves the task — the two should not be inferred from one another.
The follow-up release will cover task-success rates across the high-level areas listed in §5: terminal/shell agent tasks, tool & function calling, multi-language code editing, repository issue resolution, and coding/API integration tasks.
[!IMPORTANT]
Throughput figures above are from the local MTP-enabled run. Task success rates should be reported only from completed eval runs, not inferred from speed.
🎯 8. Recommended Use Cases
Local coding-agent experiments.
Tool-heavy chat and function-calling experiments.
DevOps troubleshooting and runbook drafting.
Repository navigation, patch planning, and test iteration.
Long-context engineering workflows where local inference is preferred.
⚠️ 9. Limitations
This is a community release for research, evaluation, and workflow exploration.
Low-bit quantizations may reduce instruction following and tool-call reliability.
Coding-eval success rates are not finalized in this initial card.
This card does not claim safety alignment beyond the behavior inherited from the base model and the fine-tuning data.
📜 10. License
Released under the Apache 2.0 license, inherited from the upstream Qwen3.6-27B base model. You are free to use, modify, and redistribute the model and its derivatives subject to the terms of that license.
🙏 11. Acknowledgements
Thanks to the Qwen team for the Qwen3.6 base model and its MTP design, to the ggml-org / llama.cpp maintainers for native multi-token-prediction support in upstream, and to the broader open-source quantization tooling community whose work makes local-first inference of frontier models possible.
Built for local agent loops
Speed where it matters · context where it counts · MTP at every quant.
