This repository contains a compressed version of the Qwen2.5-32B-Instruct model, applying the Optimal Brain Restoration (OBR) framework as presented in the paper Optimal Brain Restoration for Joint Quantization and Sparsification of LLMs. OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, delivering significant speedup and memory reduction.

Code Repository: https://github.com/csguoh/OBR

Recent advances in Large Language Model (LLM) compression, such as quantization and pruning, have achieved notable success. However, as these techniques gradually approach their respective limits, relying on a single method for further compression has become increasingly challenging. In this work, we explore an alternative solution by combining quantization and sparsity. This joint approach, though promising, introduces new difficulties due to the inherently conflicting requirements on weight distributions: quantization favors compact ranges, while pruning benefits from high variance. To attack this problem, we propose Optimal Brain Restoration (OBR), a general and training-free framework that aligns pruning and quantization by error compensation between both. OBR minimizes performance degradation on downstream tasks by building on a second-order Hessian objective, which is then reformulated into a tractable problem through surrogate approximation and ultimately reaches a closed-form solution via group error compensation. Experiments show that OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, and delivers up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline.

This model is a 4-bit quantized, 50% unstructured sparse version of Qwen/Qwen2.5-7B-Instruct. It leverages the Optimal Brain Restoration (OBR) framework, a training-free method that aligns pruning and quantization by error compensation. OBR aims to minimize performance degradation on downstream tasks by using a second-order Hessian objective, reformulated into a tractable problem via surrogate approximation and group error compensation. This specific model instance uses the FlatQuant rotation scheme.

This model is intended for fast and memory-efficient text generation tasks where a standard Qwen2.5-7B-Instruct model would typically be used, but with significantly reduced computational overhead and memory footprint. It is particularly suitable for environments with limited memory or computational resources, or for deploying LLMs at scale.

As a compressed model, while it aims to retain performance, aggressive compression levels might lead to subtle degradation in certain niche tasks or highly sensitive applications. Users should evaluate its performance for their specific use cases. The base model's limitations (e.g., potential biases, factual inaccuracies) also apply.

The base model, Qwen/Qwen2.5-7B-Instruct, may carry inherent biases from its training data. Compression techniques like quantization and sparsification, even when carefully applied with OBR, may also introduce minor performance fluctuations compared to the full-precision, dense model.

Users should be aware of the trade-offs between model size/speed and potential minor performance shifts introduced by compression. Thorough evaluation on target tasks and datasets is recommended to ensure the model meets specific requirements.

This model is compatible with the Hugging Face transformers library. To get started, you can load the model using AutoModelForCausalLM and AutoTokenizer.

IMPORTANT: For compatibility with Qwen2.5 series models, ensure your transformers library version is transformers==4.45.0 or newer. You can install it via pip install transformers==4.45.0.

Here's a basic example for text generation:

For more detailed usage, including how to apply OBR to other base models (like Llama2, Mixtral) and integrate it into evaluation pipelines, please refer to the official GitHub repository's "Get Started" section.

The base model, Qwen/Qwen2.5-7B-Instruct, was trained on various datasets. The Optimal Brain Restoration (OBR) method is a post-training compression technique and does not involve additional training data for the compression process itself. However, it relies on a small calibration dataset (e.g., WikiText) to restore performance and optimize quantization/sparsity.

The OBR framework is training-free. The procedure involves applying pruning and quantization and then compensating for the induced errors using a second-order Hessian objective. Specific parameters for quantization (W4A4KV4) and sparsity (50%) are detailed in the paper and the GitHub repository.

Evaluation was conducted on standard benchmarks for LLMs, including WikiText perplexity and various zero-shot accuracy tasks. The paper provides detailed quantitative results and efficiency comparisons (runtime, FLOPs, TOPS).

Experiments show that OBR enables aggressive W4A4KV4 quantization with 50% sparsity on existing LLMs, delivering up to 4.72x speedup and 6.4x memory reduction compared to the FP16-dense baseline, while maintaining strong performance on downstream tasks.

If you find our work useful or helpful for your research, please feel free to cite our paper:

This work is based on previous works including QuaRot, SpinQuant, and FlatQuant. Users should follow the license of the corresponding backbone models.

For this specific model (HangGuo/QWen2.5-32B-FlatQuant-OBR-GPTQ-W4A4KV4S50), which is compressed using the FlatQuant method, the applicable license is MIT License. Please refer to the FlatQuant GitHub repository for full license details.

For any questions, feel free to contact cshguo@gmail.com.