kaitchup/

Qwen3.5-0.8B-autoround-W4A16

This is Qwen/Qwen3.5-0.8B quantized with AutoRound to W4A16 (INT). The model is compatible with vLLM (tested: v0.16.1rc1). Tested with an H200. Currently under evaluation.

uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly
uv pip install git+https://github.com/huggingface/transformers.git

Thank you Verda for providing the needed compute. I used their H200s. Verda is a European, AI-focused cloud and GPU infrastructure provider with sovereignty, sustainability, data privacy, and performance at its core. Check them out if interested.