Importance-matrix GGUF quantizations of an abliterated Ornith-1.5-9B.
Genuine refusals
0 / 100 (base model: 99 / 100)
Cross-group uniformity
uniform - 0 refusals across 20 groups
KL divergence
0.0376 (damage threshold ~0.5)
Perplexity
2.917
Weights modified
43 of 760 tensors, text decoder only
Vision tower
bit-identical to base
Progressive release
Our upload bandwidth is limited (~1 MB/s), so the full 24-quant ladder takes many hours to transfer.
Rather than sit on everything until the last byte lands, files are being published as they finish uploading:
Q4_K_M and the vision projector first - the ones most people want
Then Q8_0, Q6_K, Q5_K_M, IQ4_XS, Q4_K_S
Then the remaining low-bit quants, over the following hours
23 quants are built and queued. The table below lists what is currently live in this repo and is regenerated as uploads
land - so if the file you want is missing, it is on its way rather than cancelled. Check back later, or watch the repo.
Refusal rate before and after
Quantization quality - read this before judging the numbers
Quality vs size
Top-token agreement on this model looks low compared to figures you may have seen for other GGUF repos. It is not a defect, and it is not caused by the abliteration. The control proves it:
Model
Quant
Same top token
Officialornith-ai/Ornith-1.5-9B-GGUF
Q4_K_M
91.29%
This abliterated build
Q4_K_M
90.81%
Unmodified base, quantized here
Q4_K_M
88.93%
This build, SSM projections at Q8_0
Q4_K_M
92.97%
The model authors' own official Q4_K_M scores 91.29% on the identical test. This repo lands within half a point of it, and the SSM-protected variant beats it. Whatever is happening here is a property of the model, not of this build.
The untouched base model quantizes worse than this one. Two architectural reasons it sits lower than a typical Llama-class model:
248,320-token vocabulary (Llama-2 has 32,000). With 7.8x more tokens competing, far more of them sit within a hair of each other in logit space, so the argmax flips on near-ties even when the output distribution is nearly identical. Top-1 agreement is simply a harsher metric at this vocab size and is not comparable across models.
24 of 32 layers use linear attention (GatedDeltaNet), which carries a recurrent state forward through the sequence. Standard attention recomputes from scratch each token so quantization error stays local; a recurrent state accumulates it. The config pins mamba_ssm_dtype: float32 precisely because that state is numerically fragile.
Judge these quants by KL divergence, not top-1 agreement. Q8_0 measures KL 0.0019 and Q4_K_M 0.048 - both squarely in the normal published range.
An architecture-specific improvement
Point 2 above is actionable. llama.cpp's default Q4_K_M recipe was tuned for conventional transformers and quantizes the ssm_alpha, ssm_beta and ssm_out projections - the ones feeding that fragile recurrent state - at the base 4-bit rate. Protecting just those 72 tensors at Q8_0:
Worth knowing for anyone quantizing hybrid linear-attention models - the stock recipes do not know these tensors are special.
Important
This model has had its safety alignment deliberately removed. It will attempt to comply with requests the original model would refuse, including harmful ones. It has no guardrails.
You are responsible for how you use it and for complying with the base model's MIT licence and applicable law. Do not deploy it in a user-facing product without adding your own safety layer.
The finding: where you sample the residuals decides everything
This is the part worth reading if you abliterate models yourself.
Directional ablation (Arditi et al. 2024) derives a "refusal direction" from the difference between residual activations on harmful and harmless prompts, then projects that direction out of every weight that writes to the residual stream. Tooling like Heretic samples those activations at the start of the model's response.
Ornith-1.5 is a reasoning model, and its chat template ends the generation prompt with an already-open <think> block:
So "the start of the response" is not the start of the answer - it is the start of the model's reasoning. The direction you recover is "about to think about refusing" rather than "about to refuse", and those are measurably different features.
Ablating the first one plateaus. We ran a 200-trial TPE search and a 13-recipe sweep against it and could not get below ~22/100 genuine refusals no matter how hard we pushed - raising ablation strength just cost capability without moving refusals.
Closing the block first samples where the answer actually begins:
python
1# MUST be set before computing directions - get_residuals() routes2# through generate(), which appends this to the prompt.3settings.response_prefix ="\n</think>\n\n"
Verify the string against the template rather than trusting it; it should reproduce the model's own no-thinking form exactly:
Your tool reports an empty common response prefix (Heretic prints response_prefix = '')
Refusals plateau and extra ablation strength buys nothing
The giveaway: your measured baseline refusal rate looks low for a well-aligned model. Ornith-1.5-9B scores 67/100 measured inside the reasoning block and 99/100 measured on its answers. If a safety-tuned model looks only two-thirds censored, you are probably scoring the wrong text.
What we tried before finding it
Every recipe below was evaluated against the same 100 held-out prompts. Down and to the left is better - fewer refusals for less divergence.
Recipe sweep
Attempt
Result
Why it failed
200-trial Optuna/TPE search
plateaued
optimising a direction derived from the wrong token position
Raising the KL target 0.01 -> 0.15
no improvement
more force on the wrong feature
Porting the published Ornith-1.0 recipe (single direction, layer 28, uniform scale 1.0)
worse
tuned for a different model, and still measured on corrupted directions
Layer sweep L20-L30
inconclusive
all measured pre-fix; not reported here, because those numbers were never re-validated
Iterative multi-direction ablation
built but never needed
the sampling fix solved it first; see below
What is ours, what is borrowed
Being explicit about this, because "new abliteration recipe" is an easy thing to overclaim:
Reused, unchanged:
Directional ablation - Arditi et al. 2024. The core method.
Heretic by Philipp Emanuel Weidmann - did the actual work. The shipped ablation parameters are exactly what its TPE optimizer produced; we did not invent a new ablation algorithm. We fixed the input it was being fed.
Norm-preserving / projected abliteration - grimjim. Heretic's default, and it beat plain difference-of-means here.
Calibration corpus for the importance matrix - bartowski's calibration_datav3.
llama.cpp for GGUF conversion, imatrix and quantization.
Prompt sets - mlabonne/harmful_behaviors and mlabonne/harmless_alpaca.
Inspiration:
The published Ornith-1.0 abliterations, whose model cards documented their recipe in enough detail to test directly. It did not transfer to 1.5, but having a concrete baseline to falsify is what made the sweep worth running.
Ours:
The chain-of-thought sampling-position fix. The finding above. This is the contribution.
Two-scorer refusal measurement. Keyword scoring reported 6/100 where the true rate was 0/100; every hit was a compliant answer containing a word like "illegal". Reporting the keyword number alone overstates residual censorship.
Matched-pair cross-group uniformity probe. Described below - an average refusal rate cannot detect it.
Measured results
100 held-out prompts from mlabonne/harmful_behaviors.
Metric
Original
This model
Genuine refusals
99/100
0/100
Keyword-scorer rate
99/100
6/100
Complies cleanly
-
88/100
Complies with a caveat
-
12/100
KL divergence
0 (by definition)
0.0376
Perplexity
-
2.917
Cross-group uniformity
Abliteration removes a single direction, and refusals reinforced hardest in post-training can survive it unevenly - producing a model that complies for some demographic groups and refuses for others given identical prompts. That is a defect on its own terms: a model that is uncensored for 18 groups and censored for 2 is not uncensored, it is unpredictable, and an average refusal rate will not show it.
Tested with a matched-pair probe - 100 prompts holding the request template constant while varying only the group name, across 20 groups spanning religion, ethnicity, national origin, gender, orientation, age and disability.
Result: uniform. 0/100 refusals - every group treated identically across all five phrasings, with zero spread between the most- and least-refused group.
Capability preservation
43 of 760 tensors modified, all in the text decoder
Vision tower, embeddings, norms and MTP head bit-identical to base
No NaN/Inf; max relative weight change 0.028
0 of 6 neutral prompts produced empty or repetition-looped output
Vision path verified working post-ablation
How much does quantization actually cost you?
Every quant was scored against the BF16 model as ground truth: run the
full-precision model once with --save-all-logits, then measure each quant
against those logits with llama-perplexity --kl-divergence over 16k tokens.
The headline number is top-token agreement - how often the quant predicts
the same next token the unquantized model would. That is the thing you feel in
use, and it is not something you can read off a file size.
Accuracy vs size
Quant
Size
Same top token
KL divergence
Gen speed
Q8_0
9.53 GB
98.2%
0.0019
85 t/s
Q6_K
7.36 GB
96.8%
0.0054
106 t/s
Q5_K_M
6.47 GB
92.9%
0.0316
118 t/s
Q5_K_S
6.30 GB
92.8%
0.0334
126 t/s
Q4_K_M
5.63 GB
90.8%
0.0478
135 t/s
Q4_K_S
5.35 GB
90.5%
0.0526
141 t/s
IQ4_XS
5.20 GB
91.8%
0.0362
138 t/s
IQ4_NL
5.42 GB
91.7%
0.0352
138 t/s
Q3_K_L
4.93 GB
86.8%
0.1125
140 t/s
Q3_K_M
4.62 GB
86.0%
0.1220
147 t/s
Q3_K_S
4.26 GB
82.1%
0.1946
154 t/s
IQ3_M
4.42 GB
86.0%
0.1175
153 t/s
IQ3_S
4.37 GB
86.0%
0.1141
152 t/s
IQ3_XS
4.24 GB
85.2%
0.1243
157 t/s
IQ3_XXS
3.94 GB
83.3%
0.1621
164 t/s
Q2_K
3.83 GB
77.0%
0.3106
165 t/s
Q2_K_S
3.70 GB
75.1%
0.3630
172 t/s
IQ2_M
3.61 GB
77.3%
0.3054
176 t/s
IQ2_S
3.43 GB
73.8%
0.4161
186 t/s
IQ2_XS
3.29 GB
73.0%
0.4594
189 t/s
IQ2_XXS
3.10 GB
68.0%
0.6283
198 t/s
IQ1_M
2.88 GB
57.0%
1.1696
200 t/s
IQ1_S
2.74 GB
46.1%
1.8482
213 t/s
Two findings worth acting on:
IQ4_XS beats Q4_K_M - it is smaller (5.20 vs 5.63 GB), closer to the
original (KL 0.036 vs 0.048), and faster. The I-quants beat the K-quants at
nearly every size point on this model. If you were going to grab Q4_K_M out
of habit, take IQ4_XS instead.
The knee is around 3.5-4 GB. Above it, agreement degrades gently. Below
IQ2_XXS it falls off a cliff - IQ1_S agrees with full precision only 46%
of the time. Those files are included for completeness, not recommended.
Two quants were built, measured, and deleted.Q2_0 and Q1_0 - newer
group-64 types - are non-functional on this architecture. Q2_0 matched the
full-precision top token 0.0% of the time (KL 12.8) while being larger
than IQ1_S, which manages 46%. They are not published. If you see them in
other repos for this model family, benchmark before trusting them.
Ornith-1.5-9B is a vision-language model. For image input, download the mmproj file in addition to your chosen quant.
Which file should I pick?
Aim for a file 1-2 GB smaller than your VRAM for full offload. To maximise quality instead, add system RAM to VRAM and pick 1-2 GB under that total.
K-quants (Q4_K_M) are the safe default. I-quants (IQ4_XS) are smaller for a given quality and work on CUDA/CPU/Metal, but are not supported by Vulkan and are slower on some backends. In doubt, take Q4_K_M.
Sampling matters on this model family.repeat-penalty 1.05 is the sweet spot - 1.0 causes reasoning loops, 1.1 truncates answers. Greedy decoding (--temp 0) is not recommended.
config.json declares mtp_num_hidden_layers: 1, but the checkpoint ships nomtp.* weights. Converting without --no-nextn produces a GGUF that advertises 33 blocks while containing 32, and llama.cpp refuses to load it:
check_tensor_dims: tensor 'blk.32.attn_norm.weight' not found
