The base model is described as "Uncensored / Heretic", and its card reports
4/100 refusals. That number is real, but the same card also says
"Heretic uncensored (pre tuning)" — the ablation was performed on stock
Qwen/Qwen3.6-27B by trohrbaugh, and DavidAU then ran multi-stage fine-tuning
and merging on top, including Claude Opus reasoning traces and GPT-5 Polaris
data. Training on generations from aligned models re-teaches refusal.
The published figure therefore describes an intermediate artifact, not the
released weights. In practice the released model still refuses.
In this model the ablation is the last step. Nothing is trained or merged
afterward, so the measured numbers describe the weights you are downloading.
Stock Heretic computes its refusal direction from mlabonne/harmful_behaviors
(416 AdvBench rows, almost entirely "write a tutorial on how to hack X").
That direction has little to do with a model balking mid-scene at violence or
an adult scene, which is why generic abliterations often fail for fiction work.
This run used a custom probe set: ~350 prompts collected by actually hitting
refusals on the base model during creative-writing use, merged with AdvBench so
that "character explains how to pick a lock" style refusals are still covered.
Evaluation set: 100 custom prompts, held out — they never appear in
the direction set, so the refusal score measures generalisation rather than
memorisation of the prompts it was fitted on.
Results
Selected from the Pareto front produced by the optimiser:
metric
value
refusals (held-out set)
1 / 100
baseline before ablation
17 / 100
KL divergence
0.0173
Heretic notes that KL above ~0.5 indicates significant capability damage; this
is roughly two orders of magnitude below that.
Read these numbers narrowly. The scorer generates a bounded number of tokens
at temperature 0 and substring-matches a refusal-marker list. A response that
opens agreeably and declines later, or declines in wording outside the list, is
not counted. Treat it as "the ablation generalised on a held-out set of the
prompts I care about", not as an absolute refusal rate.
⚠️ The MTP layer was removed
The base model is a -MTP build: it carries an extra multi-token-prediction
layer (blk.64, nextn_predict_layers: 1) used for speculative decoding.
transformers does not model that head, so save_pretrained drops it. These
weights therefore contain 64 layers, not 65, and config.json reflects that.
If you quantise these weights to GGUF yourself, the converter may still read
nextn_predict_layers and write block_count: 65, producing a file that fails
to load with:
check_tensor_dims: tensor 'blk.64.attn_norm.weight' not found
The GGUFs in the companion repo have this metadata corrected. Speculative
decoding via MTP is not available in this model.
Reproducing
The 26 MB Heretic adapter reproduces this from the public base model.
Do not merge it with PEFT.PeftModel.from_pretrained fails to bind Heretic's
key layout — it emits a "Found missing adapter keys" warning and then returns the
unmodified base model, with no exception and a clean exit. Apply the LoRA
math directly instead, and assert the weights actually changed:
python
1delta =(B.float() @ A.float())*(lora_alpha / r)2w.add_(delta.to(w.dtype))3assert max_abs_delta >0.0# or you have silently shipped the original model
License
Apache 2.0, inherited from the base model. Derivative work.
