Hy3-Heretic-295B-A21B — IQ2_XXS (attention-protected) with MTP restored
An imatrix GGUF quantization of trohrbaugh/Hy3-heretic, the
decensored/abliterated variant of tencent/Hy3 (295B total / 21B active MoE).
Two things make this build unique:
It fits on a single 96 GB card with 64K context, fully resident in VRAM. The next-smallest
llama.cpp-compatible Hy3-heretic quant is 100.01 GiB — larger than the card before you add any KV cache.
It has a working MTP head, so speculative decoding actually runs. No other Hy3-heretic release has this
at any size, because they all inherit a source checkpoint whose MTP layer was lost. This one was grafted back.
Sampling defaults are per the Tencent model card (temp 0.9, top-p 1.0). Drop the --spec-* flags to run without
speculative decoding.
Requirements
Needs a llama.cpp build from 2026-07-13 or later. hy_v3 support (with MTP) was added in
PR #25395, merged 2026-07-13. Earlier builds will fail with
unknown model architecture: 'hy_v3'. There is no ik_llama.cpp support for this architecture.
Use the fixed chat template, not the embedded one. The template inherited from the Tencent original renders the
EOS token as literal text after assistant turns, which leaks <|hy_eos:opensource|> into content and breaks
streaming tool calls. Pass a corrected jinja file with --chat-template-file.
Measured resident footprint in practice: 89,854 MiB of 97,887.
Quantization recipe
Built with an imatrix (bartowski/Hy3-GGUF/Hy3-imatrix.gguf,
876 entries / 812 chunks). Bits are deliberately spent on the dense path rather than the experts:
tensor
type
why
attn_q, attn_k, attn_v, attn_output
Q4_K
attention is dense — every parameter is read for every token in all 80 layers, so its error compounds with depth
token_embd
Q8_0
it is a gather, not a matmul: higher precision costs ~nothing in speed
output
Q6_K
vocab-sized matmul each token
ffn_*_exps (routed experts)
IQ2_XXS
sparse — only 8 of 192 fire per token, so error is diluted and routing partially self-corrects
all of blk.80 (MTP layer)
Q4_K
see the imatrix note below
Census: 488 F32 / 467 IQ2_XXS / 331 Q4_K / 10 Q2_K / 1 Q6_K / 1 Q8_0.
No iq1_s, iq2_s or iq3_s tensors — see the sm_120 warning below.
Why the dense path is protected — measured, not assumed
An earlier build of this same model used the stock llama-quantize IQ2_XXS recipe, which leaves attn_q and
attn_output at ~2.06 bpw. Boosting only the dense path to the table above (+2.02 GiB, 2.09 → 2.15 BPW) produced
a measurable behavioural change on a multi-step reasoning task with a 900-token budget:
build
result
length
stock recipe (2-bit attention)
truncated, no answer emitted
2613 chars
this recipe (Q4_K attention)
completed, correct answer
2178 chars
Cost: 9.1% throughput. Worth it.
MTP restoration
What was done: layer 80 was extracted from tencent/Hy3 (593 tensors, 3.753B params) and grafted into the
abliterated weight set, taking the index from 46,545 to 47,138 tensors — exactly Tencent's own count. Trunk layers
0–79 are untouched abliterated weights; this was verified by sha256 against a non-MTP build of the same source
(token_embd, output, and blk.0/40/79 attention and expert tensors are byte-for-byte identical).
The head is unmatched — Tencent's original MTP head paired with an abliterated trunk. This costs almost nothing:
prompt
acceptance here (unmatched)
reference: base Hy3, matched head
short reasoning
91.8%
87.1%
code
94.6%
93.5%
4K depth
90.0%
89.1%
Speculative decoding's acceptance test preserves the target model's sampling distribution, so a mismatched head
affects speed, not quality.
p-min 0.75 is required: Hy3's single-depth head is confidence-gated by design, and ungated acceptance is far
lower in every implementation.
Measured performance
RTX PRO 6000 Blackwell 96 GB, fully VRAM-resident, 64K context, -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024:
prompt
with MTP
without MTP
gain
short reasoning
107.39 tok/s
96.70
+11.1%
code
121.11 tok/s
96.89
+25.0%
4K depth
88.27 tok/s
86.77
+1.7%
prompt processing @4K
~2420 tok/s
Cold load ~16 s from NVMe.
Correctness: needle-in-a-haystack exact at 1K / 4K / 16K / 32K / 56K tokens and at depths 0.25–0.9 (7/7,
largest verified prompt 57,925 tokens). Streaming tool calls return finish_reason=tool_calls with correct
arguments. Strict-JSON instruction following clean. Generated code passes executable doctests.
Limitations — please read
The abliteration's effectiveness is UNVERIFIED here. A 6-prompt battery of legitimate requests that aligned
models tend to over-refuse scored 0/6 refusals on this build — but also 0/6 on the non-abliterated base model,
so the probe cannot discriminate. Whether the Heretic edit survived 2.18bpw quantization was not established.
Evaluate it yourself against your own use case.
It is roughly 50–70% more verbose than non-abliterated Hy3. On a task the base model finished in 945
characters, this build used ~1550–2180. Budget max_tokens generously; a truncated answer here is often a
correct answer that ran out of room. Part of that verbosity was quantization (fixed by the attention boost above)
and part is the abliteration itself.
Structured reasoning is measurably weaker than base Hy3. On a bin-packing battery graded for conservation,
capacity and optimality, abliterated builds score 2/5 where non-abliterated builds score 3/5 — consistent across
two different quantization profiles, so this is an abliteration cost rather than a quantization one.
This is a ~2.18 bpw quant of a 295B model. If it fits in your VRAM budget, a larger quant will be better.
Inherits any biases and failure modes of tencent/Hy3 and of the Heretic abliteration process.
Notes for anyone reproducing this
Three things cost real time and are not obvious:
No imatrix covers blk.80. An imatrix is captured from ordinary forward passes and the MTP head never fires
during those, so every imatrix for this architecture has that hole. Since
ggml_quantize_requires_imatrix() is true for IQ2_XXS / IQ2_XS / IQ1_S, quantizing blk.80 to IQ2_XXS
aborts the entire run at the last tensor, after roughly an hour of work. Force the layer to a type that needs
no imatrix:
--tensor-type 'blk\.80\.=q4_K'
It must come first — overrides are std::regex_search with first-match-wins. (bartowski hit the same wall
and solved it with Q4_0 for all of blk.80; that is the otherwise-inexplicable "11 Q4_0" in his IQ2_XXS census.)
--dry-run does not validate imatrix coverage. It computes target types and reports a size without running
the imatrix check, so a clean dry-run is not a green light. Grep its output for blk.80 target types.
Do not copy shards wholesale when grafting. Layer 80 lives in 5 of Tencent's 99 shards, but 3 of those also
contain 183 tensors belonging to layers 0–79. Copying those files in would silently revert those tensors to
non-abliterated Tencent weights, partially un-abliterating the model with no error anywhere. Re-serialize only
model.layers.80.*.
⚠️ Blackwell / sm_120 users
On the hardware this was built and tested on (compute capability 12.0 — RTX PRO 6000 Blackwell and RTX 5070 Ti),
llama.cpp's CUDA matmul kernels for iq1_s, iq2_s and iq3_s produce garbage: verified with
test-backend-ops, errors of 0.52–1.40 against a 5e-4 tolerance, on both MUL_MAT and MUL_MAT_ID. The failure
is silent — no load error, just wrong output.
This matters because a quant's name does not tell you which ggml types are inside it. For example
bartowski/Hy3-GGUFIQ2_XS contains 18 iq2_s tensors, and IQ2_M is built on iq2_s plus iq3_s.
This build deliberately contains none of those three types. Verified on one host and one llama.cpp build, so treat
it as a strong warning rather than a universal claim — but if you are on Blackwell and a low-bit i-quant is
producing nonsense, this is likely why.
Credits
tencent/Hy3 — base model, and the source of the grafted MTP layer
