kaitchup/

Qwen2.5-72B-Instruct-autoround-2bit-32g-4096-gptq

This is Qwen/Qwen2.5-72B-Instruct quantized with AutoRound in 2-bit (symmetric + gptq format) with a group size of 32 and calibration samples of 4096 tokens. The model has been created, tested, and evaluated by The Kaitchup. The model is compatible with vLLM and Transformers.

More details in this article: Accurate 2-bit Quantization: Run Massive LLMs on a Single Consumer GPU

Subscribe to The Kaitchup. This helps me a lot to continue quantizing and evaluating models for free.