Qwen3.8-27B-Uncensored-Cyber — GGUF quantized with an agentic imatrix
GGUF quantizations of philbert440/Qwen3.8-27B-Uncensored-Cyber,
calibrated on an importance matrix built from real agentic-coding traffic rather than
generic English prose.
All model weights and behaviour are philbert440's work. This repository contributes
only the quantization and the calibration methodology — no fine-tuning, no merging, no
behavioural change of any kind was performed here.
Inherited from upstream, and stated plainly so nobody is surprised: this is a
de-refusal ("abliterated") model specialized for the cyber/offensive-security domain.
The vision tower and the MTP speculative-decoding head are preserved upstream, which is
what makes the multimodal path and the MTP graft work at all. Read
the base model's card
before using it — it documents the recipe and the evaluation.
Status (2026-08-20): the imatrix and both IQ4_XS artifacts are uploaded.
Sizes and SHA-256 values below are the Hugging Face LFS objects, checked against the
local production artifact for the MTP build.
Current FastLLM production artifact; includes the compatible MTP head
The two model files have identical IQ4_XS/imatrix text weights. The plus-mtp file adds
the MTP tensors from the matching upstream model; use it only with a runtime that
recognizes that graft. For a conventional GGUF loader, use the non-MTP file unless its
MTP support has been verified. The Q5_K_M comparison below was measured locally and is
not published in this repository.
Why a different calibration corpus
The usual llama.cpp calibration sets (wiki.train.raw, groups_merged.txt) are generic
prose. The tokens that actually break under quantization in agentic use are a different
population:
tool-call JSON and tool names (bash, read, web_search)
chat-template special tokens (<|im_start|>, <tool_call>)
long literals that must be reproduced verbatim — repository paths, package names,
commit hashes, device UUIDs
mixed Chinese/English (prose in Chinese, paths and code in English)
Those barely occur in generic prose, so a prose-derived imatrix treats their channels as
unimportant and the quantizer spends its error budget there first. The concrete failure
this targets: an agent writing /home/eze/Documents/PotouI for
/home/ezra/Documents/Proto-UI, then reading its own mangled output back and concluding
its context is inconsistent.
Calibration corpus
Extracted from real agent session logs (an agent CLI driving coding tasks against this
model) — not synthetic prompts:
size
1.00 MB
sessions sampled
42
dialogue turns
648
real tool calls
528 (real arguments, real paths)
<|im_start|> occurrences
1191
<tool_call> occurrences
547
chunks processed
584 @ -c 512
The corpus itself is not published. It is private working data — real repository
paths, real shell commands, real conversations. The resulting imatrix-agentic-v2.gguf is
published so the method is reproducible; rebuild your own corpus from your own traffic to
reproduce it end to end.
--parse-special is not optional here. The corpus is rendered with the model's own
chat-template markers. Without that flag llama.cpp tokenizes <|im_start|> as literal
text — <, |, im, _start, … — so the special tokens never enter the statistics at
all, which defeats the point of building this corpus. Measured effect on the same corpus
family: 0.342 → 0.299 tokens per byte (−12.7%), consistent with those markers collapsing
into single tokens.
Context stayed at 512, deliberately. The intuitive move is to match the 262 K context
this model is served at. Community testing says the opposite: for a fixed token budget a
small context usually beats 4096, because it yields more and more-diverse samples and a
better-conditioned estimate. 512 is also llama.cpp's long-standing default.
--output-tensor-type q8_0 --token-embedding-type q8_0. Output-layer quantization
error turns directly into picking the wrong token, which is precisely the verbatim-copy
failure above. Costs roughly 1.5 GB.
Honest caveats
Quantized from the Q8_0 release, not from BF16 (--allow-requantize). Q8_0 is
near-lossless and the error is dominated by the Q5/IQ4 step, but this is still a second
quantization.
Comparison scope is narrow. The measured numbers below use a held-out agentic
corpus drawn from different sessions than calibration (verified disjoint). They test
this deployment shape, not general writing, multilingual knowledge, or safety.
imatrix helps most below Q5_K_M. Community consensus puts the large gains in the
Q3/Q4 range. IQ4_XS is squarely in useful territory; Q5_K_M is near the upper edge
where the effect gets modest.
Calibration is domain-matched to agentic coding. For creative writing or general
chat, a generic imatrix may suit you better.
Context
Built while making this model usable for agentic work on a single V100 32 GB (SM70) —
FP16-only tensor cores, no INT8/FP8/FP4 TC, no cp.async. On that hardware decode is
memory-bandwidth bound, so IQ4_XS (~4.25 bpw) reads about 26% fewer bytes per token than
Q5_K_M (~5.33 bpw) and frees roughly 4 GB for KV cache — which is why it is the primary
target here rather than the largest quant that fits.
The production configuration used for the measurements below is FastLLM on one V100:
IQ4_XS-imatrix-fromq8-plus-mtp, turbo3 KV cache, batch 4, and a 262,144-token
limit. Multimodal use also requires the matching Qwen3.8 vision projector; it is a
separate artifact and is not embedded in either text-model GGUF here.
Lineage
Qwen3.8-27B — Qwen team
└─ philbert440/Qwen3.8-27B-Uncensored-Aggressive α=1.15, recipe-v2 refusal ablation
└─ philbert440/Qwen3.8-27B-Uncensored-Cyber residual-cyber peel, β=1.0, apply_from=4
└─ this repository quantization only
Quantized from philbert440's own Qwen3.8-27B-Uncensored-Cyber-Q8_0.gguf release.
Credits
philbert440 — the Cyber model, its Aggressive
base, and the Q8_0 GGUF these quantizations are derived from. Everything this repo is
useful for is downstream of that work.
ggml-org/llama.cpp — llama-imatrix and
llama-quantize. The importance-matrix approach itself comes from
PR #4861; the GGUF imatrix format
from PR #9400.
The llama.cpp community discussions on calibration data and context size
(#5006), which are why the
calibration context here stayed at 512 instead of being naively enlarged.
License
apache-2.0, inherited from the base model. These are quantizations of philbert440's
weights; the upstream terms govern.
