This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of Qwen/Qwen3.8-27B generated using Intel's AutoRound algorithm.

⚡ Quantization Details

The model was calibrated and quantized using production-grade settings to minimize accuracy degradation while significantly lowering VRAM requirements:

Quantization Algorithm: AutoRound
Bits / Precision: W4A16 (4-bit integer weights, 16-bit activation)
Group Size: 64 (provides higher reconstruction fidelity than standard 128)
Symmetric (sym): True
Calibration Samples (nsamples): 512
Tuning Iterations (iters): 600
Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) layers preserved in native bfloat16.

🚀 Usage & Quickstart

1. Inference via vLLM

For high-throughput production serving:

bash

vllm serve Vishva007/Qwen3.8-27B-W4A16-AutoRound \
    --quantization auto-round \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90

(Note: For the GPTQ format repo, you can set --quantization gptq if required by your backend).

📊 VRAM & Performance Benefits

Original Model (BF16): ~54 GB VRAM required (needs 2x A100/A6000 or 80GB VRAM)
Quantized Model (W4A16 Group 64): ~16–18 GB VRAM (can easily run on a single RTX 3090 / 4090 / A5000 24GB)
Speedup: Reduced memory bandwidth bottleneck leading to faster decoding token speeds.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`	Deploy to RunPod
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`	Deploy to RunPod
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`	Deploy to RunPod

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`	Deploy to RunPod
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`	Deploy to RunPod
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`	Deploy to RunPod

📚 Acknowledgments

Quantization performed using Intel AutoRound.
Base architecture provided by the Qwen Team.

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Qwen3.8-27B (W4A16 Quantized via AutoRound)