Huihui-Qwen3.5-27B-W8A8-INT8 is an 8-bit checkpoint designed to reduce inference memory requirements, published by groxaxo. It is intended for open-source evaluation, reproducible experimentation, and compatible local or hosted inference workflows. The wording below is deliberately limited to what can be verified from this repository's metadata and artifacts.

At a glance

Field	Details
Format	INT8
Source / base	`huihui-ai/Huihui-Qwen3.5-27B-abliterated`
Intended task	`image-text-to-text`
License	`apache-2.0`

What is included

*.safetensors (1 file)
config.json
generation_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja
Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

Quick start

Runtime selection

Load the checkpoint with a runtime that supports its architecture and 8-bit weight format. Keep the repository's configuration and tokenizer/processor files together with the weight files.

Compatibility and responsible use

Use a runtime that explicitly supports this format, architecture, and modality.
Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
Review the source model card and license before redistribution or deployment.
Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.

Quantization or conversion changes numerical behavior, memory use, and throughput relative to the source checkpoint; validate quality on your own workload.

Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.

This is a W8A8 INT8 quantized version of huihui-ai/Huihui-Qwen3.5-27B-abliterated.

Quantization Details

Quantization Method: W8A8 INT8 (compressed-tensors format)
Base Model: Huihui-Qwen3.5-27B-abliterated (27B parameters)
Model Size: ~28GB (down from ~55GB FP16)

Usage with vLLM

bash

vllm serve /path/to/Huihui-Qwen3.5-27B-W8A8-INT8 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 8000 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --trust-remote-code \
  --enforce-eager

Note: --enforce-eager is required for this quantized model due to CUDA graph compatibility issues with compressed-tensors format.

Performance

Inference Speed: ~18-19 tokens/second (2x RTX 3090, TP=2)
Memory Usage: ~22GB VRAM per GPU

Limitations

Requires --enforce-eager flag in vLLM (no CUDA graphs)
Compatible with vLLM 0.16.1+
Uses compressed-tensors quantization format

Original Model

This model is quantized from huihui-ai/Huihui-Qwen3.5-27B-abliterated.

Please refer to the original model card for more details about the base model's capabilities and license.

groxaxo/Huihui-Qwen3.5-27B-W8A8-INT8

Huihui-Qwen3.5-27B-W8A8-INT8

Overview