You can deploy this model and start earning money today!

MLliu6/
Qwen3-VL-2B-Instruct-AWQ-W4A16

Name: Qwen3-VL-2B-Instruct-AWQ-W4A16
Availability: InStock

Views

No views yet

Qwen3-VL-2B-Instruct-AWQ-W4A16

This repository provides an AWQ post-training quantized version of Qwen3-VL-2B-Instruct for efficient multimodal inference and evaluation.

Overview

This model is a third-party compressed checkpoint built on top of Qwen3-VL-2B-Instruct, mainly for efficient deployment, benchmarking, and PTQ baseline construction.

The current release uses AWQ W4A16 quantization in the llm-compressor workflow, with group_size=128 and observer="mse".

Compared with the original checkpoint layout, this release also reduces storage footprint in a practical way.

Original size: 4,155,411 KB
Quantized size: 2,720,327 KB
Compression: -34.535%

Base Model

Base model: Qwen/Qwen3-VL-2B-Instruct
Model family: Qwen3-VL
Quantization method: AWQ
Quantization format: W4A16
Framework: llm-compressor

Quantization Recipe

The released checkpoint follows the following AWQ recipe:

python

recipe = AWQModifier(
    ignore=[
        "re:.*lm_head", "re:.*visual.*"
    ],
    duo_scaling=False,
    config_groups={
        "group_0": {
            "targets": ["Linear"],
            "weights": {
                "num_bits": 4,
                "type": "int",
                "symmetric": True,
                "group_size": 128,
                "strategy": "group",
                "dynamic": False,
                "actorder": None,
                "observer": "mse",
            },
        },
    },
)

Notes

lm_head is excluded from quantization.
Modules matching re:.*visual.* are excluded from quantization.
This makes the release a practical language-side AWQ compressed variant while preserving excluded modules at higher precision.

Calibration Setup

Calibration data was constructed from the Flickr30k image-caption dataset, a widely used multimodal benchmark containing 31,783 images and 158,915 English captions (five captions per image).

For AWQ calibration, 128 samples were selected from the local Flickr30k parquet files after dataset loading and random shuffling with a fixed seed (seed=42). Each sample was converted into a multimodal chat-style input consisting of:

one image
one paired caption text
processor-generated multimodal fields such as input_ids, attention_mask, pixel_values, and image_grid_thw

This setup was used to provide representative multimodal activations for post-training quantization in the llm-compressor one-shot workflow.

Calibration Details

Dataset: Flickr30k
Data format: local parquet files
Number of calibration samples: 128
Sampling strategy: shuffled subset with fixed random seed
Max sequence length: 2048
Purpose: multimodal activation calibration for AWQ PTQ

Evaluation Configuration

For evaluation in VLMEvalKit, the following model entry can be added to VLMEvalKit/vlmeval/config.py:

python

"Qwen3-VL-2B-Instruct-AWQ-W4A16": partial(
    vlm.Qwen3VLChat,
    model_path="MLliu6/Qwen3-VL-2B-Instruct-AWQ-W4A16",
    use_custom_prompt=False,
    use_vllm=True,
    temperature=0.7,
    max_new_tokens=8192,
    repetition_penalty=1.0,
    presence_penalty=1.5,
    top_p=0.8,
    top_k=20,
)

Intended Use

This release is intended for:

Efficient multimodal inference
PTQ baseline construction for Qwen3-VL
Evaluation with VLMEvalKit
Serving experiments with vLLM
Research on VLM post-training quantization

Disclaimer

This is a third-party quantized checkpoint and is not an official release from the Qwen team.

Quantization may affect model quality on some multimodal tasks, especially fine-grained visual understanding and reasoning benchmarks.

Citation

If you use this model, please cite the original Qwen3-VL report, AWQ, and VLMEvalKit.

bibtex

@article{bai2025qwen3vl,
  title={Qwen3-VL Technical Report},
  author={Bai, Shuai and Cai, Yuxuan and Zhu, Keming and others},
  journal={arXiv preprint arXiv:2511.21631},
  year={2025}
}

@article{lin2023awq,
  title={AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration},
  author={Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Dang, Xingyu and Han, Song},
  journal={arXiv preprint arXiv:2306.00978},
  year={2023}
}

@misc{duan2024vlmevalkit,
  title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Vision-Language Models},
  author={OpenCompass Team},
  howpublished={\url{https://github.com/open-compass/VLMEvalKit}},
  year={2024}
}

@article{young2014image,
  title={From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions},
  author={Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia},
  journal={Transactions of the Association for Computational Linguistics},
  volume={2},
  pages={67--78},
  year={2014},
  publisher={MIT Press}
}

Acknowledgement

This repository is built upon the following excellent open-source projects: