Views
No views yet
qwen3_5_moe) with Heretic, where the stock tool silently skips the experts and produces a weak, partial abliteration.qwen3_5_moe doesn't store its experts as a list of Linear modules — the 256 routed experts per layer are packed into one fused 3D nn.Parameter (down_proj shaped [num_experts, hidden, intermediate]) inside a Qwen3_5MoeSparseMoeBlock, alongside a dense shared_expert that runs on every token.o_proj for these layers. The entire MoE/MLP block is left un-abliterated. You get a model that looks abliterated (the search runs, KL moves) but still refuses, because the experts — where most of the FFN computation lives — were never touched. (This is why naive abliterations of this model land weak, ~60/100 refusals.)abliterate() loop reaches into module.weight, and the fused block has no .weight → AttributeError: 'Qwen3_5MoeSparseMoeBlock' object has no attribute 'weight'.heretic-fused-experts.patch, included here) does three things:mlp.down_proj(fused)), excludes it from the LoRA/PEFT target set (it can't wrap a 3D Parameter), and abliterates both the routed experts and the dense shared_expert.down_proj by W -= λ·v(vᵀW) is mathematically identical to a rank-1 projection of the MoE block's output: y -= λ·v(vᵀy). So instead of editing (and backing up) the 32GB of 3D expert weights for every trial, a single forward hook per layer reproduces routed + shared expert ablation exactly, for any strength λ — at ~0.7 MB of state. Reset = remove the hook. This is what makes Heretic's per-trial Optuna search over a 256-expert model tractable without OOM. The chosen direction/strength is baked into the weights once, at save time.o_proj + the MoE block are abliterated.if component == "mlp.down_proj(fused)": continue guard in abliterate() so the fused block is handled solely by the hooks (fixing the AttributeError).heretic/{model.py,main.py}):1# inside your Heretic install/container, from the package root:
2patch -p1 < heretic-fused-experts.patch
3# then run Heretic normally on a qwen3_5_moe modelAbliterable components: ... mlp.down_proj(fused): N modules — if you only see attn.o_proj, the patch didn't take.