GGUF quantizations of rico03/Qwen3.8-27B-Claude-Opus-Reasoning-Distilled — a LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, merged to 16-bit and quantized for local inference with llama.cpp / Ollama / LM Studio.
MTP (Multi-Token Prediction) head verified working for self-speculative decoding — see below, ~1.6× speedup.
🧠 What this is
Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers) plus a Multi-Token Prediction head. This fine-tune targets only the text reasoning path; the vision tower is present in the merge (unmodified base weights) but was not targeted by LoRA.
Base model:Qwen/Qwen3.8-27B (dense, 27B, hybrid linear+full attention, MTP head)
Method: LoRA (r=32, alpha=32) via Unsloth, merged to 16-bit, converted with llama.cpp (latest build — required for hybrid Gated DeltaNet operator support)
Training data: ~21.5K examples combining real Opus extended-thinking traces and reconstructed-reasoning traces over genuine Opus outputs (see Dataset Composition below)
Transparency note: 62% of the dataset has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. The lordx64 subset (38%) is the closest to authentic Opus thinking style. Full details on the base LoRA repo.
⚡ MTP / Speculative Decoding
The Multi-Token Prediction head from the base model (blk.64.nextn.*) survives the LoRA merge and GGUF conversion intact and is functional for self-speculative decoding — verified on Q8_0:
No separate draft model needed — the MTP head inside this same GGUF file acts as the draft. Output quality is unaffected (MTP proposes candidate tokens that the main model verifies; it doesn't change what gets generated, only how fast). Requires a recent llama.cpp build with --spec-type draft-mtp support.
📦 Available Quantizations
File
Size
Notes
qwen3.8-27b-opus-distill.f16.gguf
51 GB
Full precision, source for further quantization
qwen3.8-27b-opus-distill.Q8_0.gguf
28 GB
Near-lossless, recommended if VRAM allows
qwen3.8-27b-opus-distill.Q6_K.gguf
21 GB
Good quality/size trade-off
qwen3.8-27b-opus-distill.Q5_K_M.gguf
19 GB
qwen3.8-27b-opus-distill.Q4_K_M.gguf
16 GB
Practical for single consumer GPU (24GB card)
All quants verified to generate coherent output (Q8_0 sanity-tested with a technical prompt before upload). MTP speculative decoding available on all quants sharing the same architecture.
⚠️ Requires a recent llama.cpp build. This model uses Qwen3.8's hybrid Gated DeltaNet + Gated Attention architecture — older llama.cpp builds will not load it correctly. Build from source (main branch) if your package manager's version is stale.
🚀 Usage
llama.cpp
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20
With MTP speculative decoding (faster, same output distribution):
./llama-cli -m qwen3.8-27b-opus-distill.Q8_0.gguf -p "Explain the difference between TCP and UDP." -ngl 99 --temp 1.0 --top-p 0.95 --top-k 20 --spec-type draft-mtp
Ollama / LM Studio
Download the .gguf file matching your VRAM budget from the Files tab and point Ollama/LM Studio at it directly (Modelfile/import instructions per their respective docs).
Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template.
⚠️ Known Limitations
This checkpoint is a pipeline-validation run (150 steps, ~12.6% of one epoch), not a fully converged fine-tune. Expect it to show the target <think> format and some stylistic shift, but not robust generalization across task domains.
Dataset partially composed of reconstructed (not captured) reasoning traces — see Dataset Composition above
Text-only fine-tune; vision tower weights are present in the merge (unmodified base) but untested/unused in this GGUF export
Requires latest llama.cpp for hybrid attention + MTP support (see note above)
🙏 Acknowledgments
Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets, and the llama.cpp team for hybrid-architecture support.
