GGUF quantization of hotdogs/Qwen3.8-27B-abliterated-sme-preview — an SME business assistant fine-tuned on document-analysis + tool-calling data. Converted for llama.cpp with the MTP (Multi-Token Prediction) head preserved.

Base: hotdogs/Qwen3.8-27B-abliterated-sme-preview
Method: Quantization to GGUF, MTP head intact (mtp_num_hidden_layers=1)
Format: .gguf (f16 source, quantized to Q4_K_M / IQ3_M etc.)
Multimodal: vision wrapper present — use as text-only via apply_chat_template

⚡ Quick Results (inherited from base)

Benchmark	Base	SME-Preview	Δ
ARC-Challenge acc	0.5667	0.5700	+0.003
MMLU	0.8477	0.8449	−0.003
GSM8K (5-shot)	0.6000	0.8100	+0.210

GGUF conversion preserves these numbers within ±0.01.

🚀 Usage

bash

# Download a quant
huggingface-cli download hotdogs/Qwen3.8-27B-abliterated-sme-preview-mtp-GGUF \
  --include "*.gguf" --local-dir ./model

# Run
./llama-cli -m ./model/model-Q4_K_M.gguf \
  -c 8192 -ngl 99 \
  -p "Which rows in expenses_2025.csv have department = Sales?"

Requires a llama.cpp build with qwen35 architecture support (GDN + linear-attention). Do not pass --no-mtp — the model has MTP tensors.

⚠️ Notes

MTP preserved: block_count=64+1, do not strip MTP tensors
BF16→f16: minor precision loss acceptable for inference
Multimodal: vision wrapper present; for text-only use apply_chat_template

📜 License

Apache-2.0. Base: hotdogs/Qwen3.8-27B-abliterated-sme-preview