4-bit weight-only quantization (W4A16) of DavidAU's 40B "Chimera" — the 6-Core Fable Fusion Heretic Uncensored model. Quantized with Intel AutoRound recipe best, group_size 64, calibrated on high-quality reasoning traces.

Base model: DavidAU/Qwen3.6-40B-Fable-Fusion-6-Core-Deckard-Eleanor-Heretic-Uncensored

Quantization Recipe

Parameter	Value
Method	AutoRound 0.15.0 (`auto-round-best`)
Scheme	W4A16 (4-bit weights, 16-bit activations)
Group size	64
Symmetric	Yes
Iterations	1000
Calibration samples	5329 (from 37087 candidates)
Sequence length	2048
Batch size	1 (required for hybrid attention stability)
Format	`auto_round:auto_gptq`
Output size	~38 GB (down from ~80 GB BF16)
Quantization time	~4h 10min on 1× RTX 3090

Unquantized layers (kept BF16): lm_head, embed_tokens, model.visual (vision tower), all linear_attn projections (Gated DeltaNet), and MTP head via WOQ[RTN].

Hybrid Architecture Note

This model uses a 3:1 hybrid attention pattern — 72 linear-attention layers (Gated DeltaNet with Mamba SSM) and 24 full-attention layers. During quantization, --bs 1 is mandatory: the partial rotary embedding (partial_rotary_factor=0.25, head_dim=256) causes batch-dimension mismatches at higher batch sizes on full-attention layers.

Benchmarks (Custom, RTX 3090)

Custom generation benchmark, temperature=0.6, vLLM TP=2.

Test	Output tokens	tok/s	Finish
Speed baseline	800	37.45	stop
Reasoning + Math	2900	36.78	stop
Code generation	3241	37.62	stop
Creative writing (RU)	8192	35.27	length

Quality verdict: Reasoning task solved correctly (both the sheep logic puzzle and the two-train meeting problem). Code generation produced full production-ready Python with bitarray optimization, type hints, docstrings, error handling, and CLI benchmarking. Creative output sustained 8K+ tokens coherently.

Reference — base 27B Qwopus INT4 on same harness: 42–59 tok/s. The 40B quant is slower (more params) but produces longer, deeper reasoning chains.

Usage

vLLM (recommended, TP=2)

bash

vllm serve goldhub/Qwen3.6-40B-Fable-Fusion-6-Core-Deckard-Eleanor-Heretic-Uncensored-W4A16-AutoRound \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

SGLang (TP=1 PP=3)

This quantized Chimera successfully loads under SGLang with TP=1, PP=3:

bash

python -m sglang.launch_server \
  --model-path goldhub/Qwen3.6-40B-Fable-Fusion-6-Core-Deckard-Eleanor-Heretic-Uncensored-W4A16-AutoRound \
  --tp 1 --pp 3 \
  --trust-remote-code

MTP (Multi-Token Prediction)

The MTP head is preserved (mtp.layers.0, mtp.fc) and quantized via WOQ[RTN]. Under vLLM with MTP=3, speculative decoding is functional with variable acceptance (observed 13%–100% depending on token predictability), yielding ~25–45 tok/s effective throughput. For maximum acceptance stability, MTP=2 is recommended for long-form generation.

SGLang + MTP caveat

On SGLang with --pp 3 (pipeline parallel), MTP speculative decoding does NOT work due to overlap between pipeline stages. Use --tp 2 or vLLM if you need MTP. On vLLM with --tensor-parallel-size 2, MTP=3 works and is confirmed alive (acceptance 70-100%).

Chat Template

This repo ships a patched chat template (chat_template.jinja) from Qwen-Fixed-Chat-Templates v21.3 (froggeric/Qwen-Fixed-Chat-Templates).

The original template is preserved as chat_template.original.jinja for reference. The patched template fixes tokenizer/chat-formatting issues and improves instruction-following fidelity.

Acknowledgments

DavidAU — for the base 6-Core Chimera fine-tune
Intel AutoRound team — quantization framework
froggeric — fixed chat templates
Quantized by Viktor Zhuromskyy on consumer hardware (RTX 3090)

License

This quantized model inherits the base model's license. AutoRound is Apache-2.0.

🚀 Long Story Short: W4G64 AutoRound Quantization & MTP Confirmation

Successfully quantized the 40B Chimera to INT4 (W4G64) using AutoRound.

Weight: ~38 GB
VRAM: Fits comfortably on multi-GPU setups (e.g., 2x24GB or 3xRTX3090/4090) with room for large context.

⚡ MTP (Multi-Token Prediction) Status: ALIVE & KICKING

Initial concerns about MTP breaking during quantization or expansion are debunked. Telemetry confirms MTP=3 is active and providing significant generation speedups without degrading the "Heretic" creative quality or reasoning capabilities.

📊 W4G64 Benchmark Highlights (vLLM / SGLang)

Task	Performance / Quality	Notes
Speed	35 - 47 tok/s	Depends on context and MTP acceptance rate.
Coding	🟢 Flawless	Generated production-ready `Sieve of Eratosthenes` with `bitarray` optimization, type hinting, and `O(n log log n)` complexity analysis.
Math/Logic	🟢 Perfect	Step-by-step reasoning for complex word problems (trains, logic puzzles) without hallucinations.
Creative (RU)	🟢 Masterpiece	Deep, atmospheric cyberpunk/philosophical narratives ("РЕКУРСИВНЫЙ ВЫЗОВ", "ОГОНЬ ПАДАЮЩЕЙ ЗВЕЗДЫ"). Retains full uncensored/heretic alignment.
Long Context	🟢 Stable	Handles 8k+ tokens seamlessly.

🛠️ How to run (vLLM example)

bash

vllm serve goldhub/Qwen3.6-40B-Fable-Fusion-6-Core-Deckard-Eleanor-Heretic-Uncensored-W4A16-AutoRound \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --trust-remote-code