Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated-AWQ
This repository provides an AWQ-quantized version of the original
huihui-ai/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated model.
The goal of this quantization is to reduce GPU memory usage and enable efficient inference deployment using frameworks such as vLLM.
Model Details
- Base model: huihui-ai/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated
- Quantization: AWQ
- Quantization tool: llm-compressor
- Precision: INT4 weight quantization
- Target inference engine: vLLM
Quantization Process
The model was quantized using llm-compressor with a calibration dataset and standard AWQ workflow.
Key steps:
- Prepare calibration dataset
- Run AWQ quantization using llm-compressor
- Export quantized weights compatible with vLLM
- Upload quantized model to HuggingFace
Intended Usage
This model is intended for:
- Efficient multimodal inference
- Deployment with vLLM
- Reduced VRAM consumption compared to FP16 models
Example use cases include:
- conversational agents
- multimodal assistants
- experimentation with large multimodal models on limited hardware
Acknowledgements
All credit for the original model goes to:
huihui-ai / Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated
This repository only provides an AWQ-quantized version for inference efficiency.
