Views
No views yet
Comfy-Org/vae-text-encorder-for-flux-klein-4b,
split_files/text_encoders/qwen_3_4b.safetensors), derived from the base language model
Qwen/Qwen3-4B.| File | Size | Format | Derived from |
|---|---|---|---|
qwen3_4b_int8_convrot.safetensors | 4.49 GiB | ConvRot INT8 (row-scaled) | Comfy-Org/vae-text-encorder-for-flux-klein-4b:split_files/text_encoders/qwen_3_4b.safetensors (BF16, 7.49 GiB) |
qwen3_4b_nvfp4_convrot_int8.safetensors | 3.51 GiB | NVFP4 (bulk MLP) + ConvRot INT8 (row-scaled, attention projections) | same source as above |
qwen3_4b_mixed_int4_int8_convrot.safetensors | 3.38 GiB | INT4 W4A4 ConvRot (bulk MLP) + ConvRot INT8 (row-scaled, attention projections) | same source as above |
embed_tokens and the first/last 2 of the model's 36 transformer blocks are kept at source BF16
precision in all three files.qwen3_4b_int8_convrot.safetensors: the remaining 224 attention/MLP weight tensors are
quantized to INT8 with ConvRot (group-wise Hadamard rotation, group size 256) and row-wise
scaling.qwen3_4b_nvfp4_convrot_int8.safetensors: the remaining 224 tensors are split by role — the
128 attention-projection tensors (self_attn.{q,k,v,o}_proj in the non-excluded blocks) are
INT8 with ConvRot (group size 256, row-scaled), and the 96 MLP tensors
(mlp.{gate,up,down}_proj) are NVFP4.qwen3_4b_mixed_int4_int8_convrot.safetensors: same role-based split as above, but the 96 MLP
tensors are INT4 W4A4 ConvRot (group size 256) instead of NVFP4, and the 128 attention-projection
tensors are promoted to INT8 ConvRot (row-scaled) to recover quality INT4 alone tends to lose on
those layers. Quality caveat: per this project's quantization procedure, INT4 ConvRot output
quality depends heavily on layer-selection method and rounding mode — this promotion criterion
(attention projections promoted, bulk MLP left at INT4) is the same role-based split used for the
NVFP4 variant above, not a result from a per-layer sensitivity sweep, and hasn't been separately
evaluated for quality beyond the functional check below.| File | Hardware |
|---|---|
qwen3_4b_int8_convrot.safetensors | Any modern GPU with usable INT8 tensor-core throughput — no Blackwell requirement. |
qwen3_4b_nvfp4_convrot_int8.safetensors | Requires a Blackwell GPU (SM ≥ 10.0) for NVFP4 inference. |
qwen3_4b_mixed_int4_int8_convrot.safetensors | No Blackwell dependency, but INT4 tensor-core throughput varies significantly by GPU generation — verify actual runtime behavior on target hardware. |
convert_to_quant (ctq):qwen3_4b_int8_convrot.safetensors):1ctq -i qwen_3_4b.safetensors -o qwen3_4b_int8_convrot.safetensors \
2 --int8 --scaling-mode row --dynamic-convrot --convrot-group-size 256 \
3 --layer-config qwen3_4b_layer_config_int8_convrot.json \
4 --comfy_quant --save-quant-metadataqwen3_4b_layer_config_int8_convrot.json,
included in this repo) excludes model.embed_tokens and model.layers.{0,1,34,35} from
quantization (kept at source BF16 precision); every other 2D weight tensor falls through to the
--int8 primary format.qwen3_4b_nvfp4_convrot_int8.safetensors):1ctq -i qwen_3_4b.safetensors -o qwen3_4b_nvfp4_convrot_int8.safetensors \
2 --nvfp4 --low-memory \
3 --custom-layers 'model\.layers\.(2|3|...|33)\.self_attn\.(q|k|v|o)_proj' \
4 --custom-type int8 --custom-scaling-mode row --custom-convrot --custom-convrot-group-size 256 \
5 --layer-config qwen3_4b_layer_config_nvfp4_skip_only.json \
6 --comfy_quant --save-quant-metadataqwen3_4b_layer_config_nvfp4_skip_only.json,
included in this repo) only holds skip: true entries for model.embed_tokens and
model.layers.{0,1,34,35}; --custom-layers/--custom-type route the attention-projection
layers in the remaining blocks to INT8 ConvRot; everything else falls through to the --nvfp4
primary format. (--custom-type/--fallback must be given alongside --nvfp4 on this ctq
build for --layer-config to be consulted at all — see this repo's quantization tooling notes
for why a bare --nvfp4 --layer-config ... invocation silently ignores the layer-config.)qwen3_4b_mixed_int4_int8_convrot.safetensors):1ctq -i qwen_3_4b.safetensors -o qwen3_4b_mixed_int4_int8_convrot.safetensors \
2 --int4 --dynamic-convrot --convrot-group-size 256 --low-memory \
3 --layer-config qwen3_4b_layer_config_mixed_int4_int8.json \
4 --comfy_quant --save-quant-metadataqwen3_4b_layer_config_mixed_int4_int8.json,
included in this repo) holds skip: true entries for model.embed_tokens and
model.layers.{0,1,34,35}, plus "format": "int8_tensorwise" (ConvRot, row-scaled) entries for
the attention-projection layers in the remaining blocks; every other 2D weight tensor falls
through to the --int4 primary format.--simple is omitted. Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
Copyright 2024 Alibaba Cloud
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Full terms: https://www.apache.org/licenses/LICENSE-2.0