p1atdev/

CogView4-6B-quanto_int8

Quantization settings

• vae.: torch.bfloat16. No quantization.
• text_encoder.layers.:
  • Int8 with Optimum Quanto
  • Target layers:["q_proj", "k_proj", "v_proj", "o_proj", "mlp.down_proj", "mlp.gate_up_proj"]
• diffusion_model.:
  • Int8 with Optimum Quanto
  • Target layers: ["to_q", "to_k", "to_v", "to_out.0", "ff.net.0.proj", "ff.net.2"]

VRAM cosumption

• Text encoder (text_encoder.): about 11 GB
• Denoiser (diffusion_model.): about 10 GB

Samples

torch.bfloat16	Quanto Int8
VRAM 40GB (without offloading)	VRAM 28GB (without offloading)