Quantized the original Qwen3.5-4B model from BF16 to INT8 precision using bitsandbytes
Reduced memory footprint by 42.4% while maintaining model quality
Benchmarked on WikiText-2 (perplexity) and GSM8K (math reasoning)

Why I Did This

Memory Efficiency: Reduce GPU memory requirements to run on smaller GPUs
Faster Inference: INT8 operations can be faster on compatible hardware
Accessibility: Enable deployment on consumer-grade hardware with limited VRAM

Benchmark Results

Metric	Baseline (BF16)	INT8	Change
Memory	8.41 GB	4.84 GB	-42.4%
Perplexity (WikiText-2)	12.59	12.75	+1.30%
GSM8K Accuracy	86.00%	~86%	Minimal degradation

Key Findings

1.74x memory reduction with minimal quality loss
Only +1.30% perplexity increase on WikiText-2
Math reasoning capabilities preserved on GSM8K benchmark

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AxisQuant/Qwen3.5-4B-INT8",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")

messages = [{"role": "user", "content": "What is machine learning?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Technical Details

Quantization Method: bitsandbytes LLM.int8()
Base Model: Qwen/Qwen3.5-4B
Original Precision: BF16
Quantized Precision: INT8
Framework: Transformers + bitsandbytes

Limitations

Requires bitsandbytes library for optimal performance
Some operations may cast to FP16 during inference
Best suited for NVIDIA GPUs with INT8 tensor core support

Citation

If you use this model, please cite the original Qwen3.5 model:

bibtex

@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  publisher={Alibaba}
}

Author

Prashant Takale

Note: This model is for testing and experimental purposes only. There are still some improvements required in terms of inference speed and accuracy. I plan to explore better quantization methods like AWQ or GPTQ in future iterations for optimized performance.