2026-02-15: Requant to ensure quality - batch_size=1 (thread) and addition of Greek language and 34 multilingual dataset (request)
2026-02-14: Original quant, using LLMcompressor's new feature batch_size=32. batch_size may negatively impact calibration due to truncating or padding datasets, defeating the careful selection I made.
Overview
This strives to be the highest quality quant that can run on 192GiB VRAM
self-attention weights dequantized from the official version.
experts weights quantized using AWQ W4A16G32 scheme (4-bit weights, 16-bit activations, scaling factor per group of 32 weights)
High-quality large and diverse dataset with programming and devops focus
as well as domain-specific knowledge (math, sciences, medical, finance, business, humanities, philosophy, creative writing), general knowledge, pop culture and behavioral situations because we never code in a vacuum. And we want to make sure all experts are calibrated to the full range of their activations.
The model was tested with SGLang + 2x RTX Pro 6000, here is a script suitable for such configuration with the maximum 196,608 context length. This uses 92.5GiB of VRAM with the flashinfer backend.
--trust-remote-code is necessary until the transformers team merges github.com/huggingface/transformers/pull/42028
You have 2 reasoning parsers;
minimax, puts the reasoning content in a special field like DeepSeek models that is usually rendered in a specific manner in frontends.
minimax_append_think, puts the reasoning into <think>reasoning_content</think> and that is sent as normal text. Few frontends properly render that, I'm aware of Cherry Studio on Desktop and ChatterUI on Android.
The reason why minimax_append_think was introduced was Interleaved Thinking and having the model build upon it's previous thinking (usually frontends discard the thinking trace)
[!TIP]
💡In the sister model, I mentioned that with the recommended parameters the model tends to get stuck in repetition loops.
This does not seem to happen with SGLang hence "repetition_penalty: 1.10, frequency_penalty: 0.40" are not used.
There is no way to override such settings without editing generation_config.json anyway: https://github.com/sgl-project/sglang/issues/15487
The llmcompressor library was used with the following recipe:
yaml
1default_stage:2default_modifiers:3AWQModifier:4config_groups:5mlp_experts_projections:6# Include only MLP expert weights for 4-bit quantization7targets:["re:.*block_sparse_moe\\.experts\\.\\d+\\.(w1|w2|w3)$"]8weights:9num_bits:410type: int
11symmetric:true12group_size:3213strategy: group
14dynamic:false15# actorder: group16observer: memoryless_minmax
1718mappings:19-smooth_layer: re:.*post_attention_layernorm$20balance_layers:["re:.*w1$","re:.*w3$"]21-smooth_layer: re:.*w3$22balance_layers:["re:.*w2$"]23duo_scaling:true
The calibration set had 590 examples, 8192 sequence length, 60 programming languages, 12 spoken languages and is detailed at calibrate_software_engineer.yaml
Quantization theory and heuristics for manual tuning
In-depth overview of quantization theory and heuristics for manual tuning
Layers to quantize
Quantization should be focused on Linear layers (also called Dense or Fully-Connected layers i.e. MatMul+Bias)
In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1]
LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the
LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression.
Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized.
If there is enough bits, down projections should be prioritized.
According to [4]
Fig. 3: Maximum absolute value over layers for a LLaMA3-8B.
Each color represent a different projection and we clearly see that down_proj has the biggest
spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model
According to [5]
Figure 5(a) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting
that weight outliers are concentrated in the down-projection matrices Wdown
ℓ of the second layer and
the last two layers. Figures 5(b) and 5(c) provide detailed visualizations of these outliers in the last
two layers.
Mixture-of-Experts quantization (MoE)
Mixture-of-Experts require specific quantization techniques.
Mixed-precision quantization
Some layers have a higher impact on LLM performance.
According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers.
According to [3] on 2-bit quantization:
quantizing expert FFN layers do not seriously impact model quality
quantizing cross-attention has some impact
quantizing self-attention has a large impact
quantizing dense FFN has a very significant impact
Hence to preserve model quality we should choose not to quantize dense FFN layers and self-attention layers.
We notice that:
official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16:
According to [2], giving more bits to the first k blocks have a significantly higher impact on model quality than for the same last k blocks.
Expert quantization
When quantizing MoE, quantizing activations is tricky as only a subset of experts are activated per request. You have to make sure all experts are calibrated.
Visual showcase of why ensuring quantization of all MoE experts is important
Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)
Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia https://arxiv.org/pdf/2506.12044
Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)
Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen https://arxiv.org/pdf/2406.08155v1
Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)
Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla https://arxiv.org/pdf/2310.02410
Precision Where It Matters: A Novel Spike
Aware Mixed-Precision Quantization Strategy for
LLaMA-based Language Models (2025)
Lucas Maisonnave, Cyril Moineau, Olivier Bichler, and Fabrice Rastello https://arxiv.org/pdf/2504.21553
Systematic Outliers in Large Language Models (2025)
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang https://arxiv.org/pdf/2502.06415v2
