You can deploy this model and start earning money today!

unidaikon/
Qwen3.5-35B-A3B-Q5_K_XXL-GGUF

Name: Qwen3.5-35B-A3B-Q5_K_XXL-GGUF
Availability: InStock

Views

No views yet

Overview

This repository provides a custom quantization of Qwen3.5-35B-A3B to Q5_K format, with a hybrid precision approach that keeps ssm and attention layers in high precision to preserve long-context performance. The resulting model size is approximately 23.8 GiB, optimized for systems with 32 GiB RAM + 8 GiB VRAM.

The Vision tower (mmproj) is the same file as the one in any other quantization repos.

Background

Currently, all gguf quantizations of Qwen3.5 compress ssm layers to low precision. For example, in Qwen3.5-35B-A3B-UD-Q5_K_XS:

blk.0.attn_qkv.weight 	[2,048, 8,192] 	Q5_K
...
blk.0.ffn_gate_inp_shexp.weight 	[2,048] 	F32
blk.0.ffn_gate_shexp.weight 	[2,048, 512] 	Q8_0
blk.0.ffn_up_exps.weight 	[2,048, 512, 256] 	Q5_K
...
blk.0.ssm_alpha.weight 	[2,048, 32] 	Q5_K
blk.0.ssm_beta.weight 	[2,048, 32] 	Q5_K
...
blk.0.ssm_out.weight 	[4,096, 2,048] 	Q5_K

This may cause issues in long-context scenarios:

ssm layers perform linear accumulation during generation, causing quantization errors to compound over time
ssm layers are small (2048×32 and 4096×2048), so quantization provides minimal performance gain
For certain tokens requiring minor knowledge updates, ssm_beta quantization may introduce noticeable degradation

This Quant: Keep ssm layers in BF16 precision.

Other Modificatoin

Higher precision to token embeddings (token_embd.weight)
- Qwen3.5 has much larger token list and so high precision can prevents token representation collapse
Higher attention matrix precision
- Full attention is critical in the SSM–FULL attention fusion architecture
- As only 25% layers have full attention, it's safe to have more bits without slow down inference

unidaikon/Qwen3.5-35B-A3B-Q5_K_XXL-GGUF

Overview

Background

Other Modificatoin

unidaikon/
Qwen3.5-35B-A3B-Q5_K_XXL-GGUF