Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated-FP8
Vision-capable FP8 quantized fast abliterated distilled Qwen3.5-35B model made for Nvidia DGX Spark (~80GB VRAM is needed for full functionality)
Model Lineage
So first it was
Qwen/Qwen3.5-35B-A3B (BF16).
- Then Jackrong created a text-only, less chatty and better with tools version — Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled
- Then Huihui removed all refusals and put back the vision capabilities in huihui-ai/Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated
- Then I quantized it to FP8 using the conservative approach demonstrated by the Qwen team in Qwen/Qwen3.5-35B-A3B-FP8
Performance
Conservative approach to FP8 quantization caused minimum quality loss, while still bumping the speed from 31 t/s → 51 t/s on DGX Spark. With 262k context and some space for KV cache it uses 80GB VRAM (only).
Currently that's the best, fastest and abliterated model to be used on Nvidia DGX Spark, which also preserves all visual layers untouched.
I failed to find a case where this model will refuse to answer. It is especially funny to use with pictures ;). So far the best "tooling" skills — it really likes to Google stuff first even if it knows the answer.
I plan to test the quality of the model's output later and update this page.
Quantization Details
Quantized using the
FP8_DYNAMIC scheme from
llmcompressor (
>=0.10) with
compressed-tensors serialization.
Method
FP8_DYNAMIC is a data-free quantization scheme — no calibration dataset required. Weights are statically quantized to FP8 (per-channel, symmetric), while activations are dynamically quantized to FP8 (per-token, symmetric) at inference time.
Modules Excluded from Quantization
Matching the conservative strategy from
Qwen/Qwen3.5-35B-A3B-FP8:
| Module | Reason |
|---|
lm_head | Output head — precision-sensitive |
embed_tokens | Embedding layer |
linear_attn.conv1d, linear_attn.in_proj_a/b | Linear attention layers |
mlp.gate, mlp.shared_expert_gate | MoE router gates — routing precision matters |
model.visual.* | Entire visual encoder kept at BF16 |
mtp.* | Multi-token prediction layers |
Post-processing
The model was quantized via AutoModelForCausalLM (the only loader proven to work with llmcompressor for this architecture), then post-processed:
- Weight key renaming —
model.layers.X → model.language_model.layers.X to match the ConditionalGeneration format expected by vLLM
- Visual encoder restoration — BF16 vision encoder weights copied from the source model (since
AutoModelForCausalLM strips them)
- Config restructuring —
config.json rebuilt from the source model's nested structure with the quantization config injected
Resources
Disclaimer
It's an abliterated model. DO NOT use it if you think that all AIs need to be politically correct and boring.
