Qwen3 8B F2K - INT8 ConvRot
INT8 ConvRot quantization of Qwen3 8B tailored for FLUX.2 Klein 9B text conditioning in ComfyUI. The 16.38 GB BF16 checkpoint is reduced to approximately 10.21 GB.
Conversion
- Tool: silveroxides/convert_to_quant
- Format: INT8 row-wise with embedded ConvRot metadata
- ConvRot group size: 256
- Method: learned rounding (AdaRound) with low-memory streaming conversion
- Quantized: 224 matrices across 32 transformer blocks (~6.17B parameters, or 75.4% of matrix parameters)
The token embedding, LM head, and transformer blocks 0, 8, 17, and 26 remain BF16.
Why this is FLUX.2 Klein-specific
FLUX.2 Klein 9B conditions on Qwen3 hidden states 9, 18, and 27 rather than the model's final output. These states are produced immediately after transformer blocks 8, 17, and 26, so those three blocks remain BF16. Block 0 is also retained as the input boundary.
This selection protects the layers directly producing Klein's conditioning features while quantizing the remaining transformer blocks. It is therefore tailored to FLUX.2 Klein and is not intended as a general-purpose Qwen3 language-model quantization recipe.
Command
1ctq -i <input-model>.safetensors -o qwen_3_8b_f2k_int8_convrot.safetensors `
2 --int8 --scaling_mode row `
3 --convrot --convrot-group-size 256 `
4 --comfy_quant --save-quant-metadata `
5 --low-memory --device cuda `
6 --exclude-layers '(^model\.embed_tokens\.weight$|^lm_head\.weight$|^model\.layers\.(0|8|17|26)\.)' `
7 --verbose NORMAL
Quantization is lossy, so outputs are not bit-identical to the original BF16 checkpoint.
