Qwen3.5-122B-A10B-abliterix UD-Q4_K_XL GGUF

Overview

At a glance

What is included

• *.gguf (2 files)
• Additional configuration, tokenizer, processor, or shard files (2 visible artifacts total)

Quick start

llama.cpp

Compatibility and responsible use

• Use a runtime that explicitly supports this format, architecture, and modality.
• Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
• Review the source model card and license before redistribution or deployment.
• Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
• Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

What is UD (User-Defined) Quantization?

How it differs from standard quantization

The UD methodology for this model

1. Architecture analysis: Qwen3.5-122B-A10B is a Mixture-of-Experts (MoE) model with 48 layers, 256 experts per layer (8 active), shared expert FFN, and SSM (state-space model) components. Each tensor was classified by its function:
   • Attention weights (attn_qkv, attn_gate, attn_q, attn_k, attn_v, attn_output): Quantized to Q8_0 — attention projections are highly sensitive to precision loss and benefit from near-lossless compression.
   • Expert FFN up/gate weights (ffn_up_exps, ffn_gate_exps): Quantized to Q4_K — these are the largest tensors in the model (256 experts x 3072x1024 each) and dominate the model size. Q4_K provides an excellent size/quality tradeoff for activation functions.
   • Expert FFN down weights (ffn_down_exps): Quantized to Q5_K — the down-projection is slightly more sensitive than up/gate, so Q5_K preserves more information here.
   • Shared expert FFN (ffn_down_shexp, ffn_up_shexp, ffn_gate_shexp, ffn_gate_inp_shexp): Quantized to Q8_0 — the shared expert processes every token, making it critical for quality.
   • Router/gating (ffn_gate_inp, ffn_gate_inp_shexp): Kept at F32 — the router decides which experts to activate. Even small precision errors here can cause misrouting, severely degrading output quality.
   • SSM components (ssm_a, ssm_conv1d, ssm_dt.bias, ssm_norm): Kept at F32 or Q8_0 — these are small tensors that play a structural role in the hybrid attention/SSM layers.
   • Norms (attn_norm, post_attention_norm, output_norm): Kept at F32 — normalization layers are tiny but critical for numerical stability.
   • Token embeddings (token_embd.weight): Quantized to Q8_0.
   • Output projection (output.weight): Quantized to Q8_0.
2. Importance matrix: A partial importance matrix (imatrix) was computed on coding-focused calibration data using llama-imatrix. This guides the k-quant block selection within each tensor, prioritizing precision on the most activation-sensitive dimensions. The imatrix was computed at n_ctx=512 across multiple calibration chunks before the quantization run.
3. Quantization: llama-quantize from llama.cpp was used with --tensor-type-file to apply the per-tensor type map, combined with --imatrix for importance-weighted block selection within each quantized tensor.

Resulting size

Usage with llama.cpp

Multi-GPU tensor split example (3x RTX 3090 + 2x RTX 3060)

Provided by

• Quantized by: groxaxo
• Base model: wangzhang/Qwen3.5-122B-A10B-abliterix
• Quantization engine: llama.cpp build 8736