The complete vision encoder and selected quantization-sensitive layers remain in BF16. Large projection matrices in the intermediate language-transformer blocks are quantized to row-wise INT8 with ConvRot.

Model details

Source model: DreamFast/Qwen3-VL-8B-Heretic-1.3.0
Base architecture: Qwen3-VL-8B
Intended runtime: ComfyUI
Quantization: Mixed BF16 / row-wise INT8 ConvRot
ConvRot group size: 256
Rounding method: SVD-based learned rounding (--simple was not used)
Original precision: BF16
Vision capability: Retained
License: Apache License 2.0

No layers were removed and no architectural changes were made. This is a lossy post-training quantization of the original ComfyUI safetensors checkpoint.

Precision layout

The following components remain in BF16:

Complete vision encoder (model.visual.*)
Token embedding (model.embed_tokens.weight)
Language-model head (lm_head.weight)
Final language-model normalization (model.norm.weight)
Input and post-attention normalization layers
Attention Q/K normalization layers
First language-transformer block (model.layers.0.*)
Last language-transformer block (model.layers.35.*)

The following projection matrices in language-transformer blocks 1–34 are stored as INT8 ConvRot:

self_attn.q_proj
self_attn.k_proj
self_attn.v_proj
self_attn.o_proj
mlp.gate_proj
mlp.up_proj
mlp.down_proj

This results in 238 INT8 projection matrices:

34 transformer blocks × 7 projection matrices = 238

Quantization metadata

Each quantized layer contains embedded ComfyUI quantization metadata equivalent to:

json

{
  "format": "int8_tensorwise",
  "orig_dtype": "torch.bfloat16",
  "convrot": true,
  "convrot_groupsize": 256,
  "per_row": true
}

Despite the internal int8_tensorwise format identifier, "per_row": true specifies row-wise scaling.

Conversion command

The model was converted with convert_to_quant:

bash

ctq \
  -i "qwen3-vl-8b-heretic-1.3.0.safetensors" \
  -o "qwen3-vl-8b-heretic-1.3.0-int8convrot.safetensors" \
  --int8 \
  --scaling_mode row \
  --convrot \
  --convrot-group-size 256 \
  --exclude-layers "(^lm_head\.weight$|^model\.embed_tokens\.weight$|^model\.norm\.weight$|^model\.layers\.(0|35)\.|_layernorm\.weight$|\.(q|k)_norm\.weight$|^model\.visual\.)" \
  --comfy_quant \
  --save-quant-metadata \
  --low-memory

--simple was intentionally omitted, enabling SVD-based learned rounding.

Verification

The resulting safetensors file contains:

text

BF16 tensors: 512
INT8 tensors: 238
F32 scale tensors: 238
U8 quantization-metadata tensors: 238

All 238 quantized projection matrices report:

text

convrot: true
convrot_groupsize: 256
per_row: true
orig_dtype: torch.bfloat16

Size and memory

Approximate weight storage:

Variant	Weight size
Original BF16	16.33 GiB
Mixed INT8 ConvRot	Approximately 10.2–10.4 GiB
Reduction	Approximately 6.0 GiB / 37%

Actual runtime VRAM usage also depends on activation memory, input resolution, visual-token count, context length, ComfyUI model offloading, and other models loaded in the workflow.

Usage with ComfyUI

Place the file in:

ComfyUI/models/text_encoders/

Load it with the same ComfyUI text-encoder workflow used for the original ComfyUI-format checkpoint.

A recent ComfyUI version with native INT8 ConvRot support is required. The file is intended for ComfyUI's native quantized safetensors runtime; it is not a GGUF model and should not be treated as a standard Transformers checkpoint.

Quality notice

Quantization is lossy, and outputs are not expected to be bit-identical to the BF16 source model.

The complete vision encoder and modality-merger path remain in BF16 to prioritize image-input fidelity. Image information is subsequently processed by the mixed-precision language transformer, so exact BF16 vision-language behavior is not guaranteed.

No formal benchmark comparing this quantization against the original BF16 checkpoint has been published for this repository.

Attribution

This repository contains quantized weights derived from:

Upstream model: DreamFast/Qwen3-VL-8B-Heretic-1.3.0
Upstream creator: DreamFast
Base model: Qwen/Qwen3-VL-8B-Instruct
Quantization tool: silveroxides/convert_to_quant

License

The upstream model is distributed under the Apache License 2.0, and this quantized derivative is redistributed under the same license.

See the included LICENSE file and the upstream model repository for the applicable terms, attribution, notices, and disclaimers.

craftingmod/Qwen3-VL-8B-Heretic-INT8

Qwen3-VL-8B-Heretic-1.3.0 — INT8 ConvRot (ComfyUI)