It achieves state-of-the-art accuracy retention by tuning weights for 1000 iterations with 512 calibration samples, significantly outperforming standard RTN (Round-to-Nearest) quantization.

Quantization Details

Method: AutoRound (Advanced Weight-Only Quantization)
Scheme: W4A16 (4-bit weights, 16-bit activations)
Symmetric: True
Group Size: 128
Tuning Config: 1000 iterations, 512 samples, batch size 8
Framework: Intel AutoRound

Quickstart

1. Install Dependencies

You need the auto-round library to run this model in its native format.

pip install auto-round transformers torch

2. Inference Code

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRoundConfig

model_id = "Vishva007/Qwen3-4B-Instruct-2507-W4A16-AutoRound"

# Load the model with AutoRound configuration
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "Explain quantum computing in one sentence."
messages = [{"role": "user", "content": prompt}]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs, skip_special_tokens=True))

Performance

Qwen3-4B-Instruct-2507 is the latest non-thinking instruct model from the Qwen team, featuring significant improvements in reasoning, coding, and instruction following.

This quantized version retains nearly 99% of the FP16 performance while reducing VRAM usage significantly, enabling deployment on consumer GPUs (e.g., RTX 3060/4060).

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`	Deploy to RunPod
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`	Deploy to RunPod
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`	Deploy to RunPod

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`	Deploy to RunPod
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`	Deploy to RunPod
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`	Deploy to RunPod

Citation

bibtex

@article{cheng2023optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}