Qwen2.5-7B-Instruct-INT8-Quanto – AI Model by Helllbos | AlphaNeural AI

You can deploy this model and start earning money today!

Helllbos/
Qwen2.5-7B-Instruct-INT8-Quanto

like

0

text-generation

model_hub_mixin

Qwen/Qwen2.5-7B-Instruct

Views

No views yet

This model has been pushed to the Hub using the PytorchModelHubMixin integration.

Base Model: Qwen/Qwen2.5-7B-Instruct
Quantization: INT8 (Optimum Quanto)
Library: Optimum Quanto
Framework: PyTorch
Documentation: https://huggingface.co/docs/optimum/quanto

Qwen2.5-7B-Instruct INT8 (Optimum Quanto)

This repository contains an INT8 quantized version of Qwen2.5-7B-Instruct created using Optimum Quanto.

The objective of this project is to reduce model size and memory usage while maintaining inference quality for research and local deployment.

Model Details

Base Model: Qwen/Qwen2.5-7B-Instruct
Quantization Method: Optimum Quanto
Weight Precision: INT8
Original Model Size: ~15.2 GB
Quantized Model Size: ~8.18 GB
Storage Reduction: ~46%

Usage

python

from optimum.quanto import QuantizedModelForCausalLM
from transformers import AutoTokenizer

model = QuantizedModelForCausalLM.from_pretrained(
    "YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)

tokenizer = AutoTokenizer.from_pretrained(
    "YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)

Benchmark

Benchmarking is currently in progress.

Planned evaluations include:

HellaSwag
ARC Challenge
Winogrande
GSM8K

Benchmark results will be added after evaluation.

Hardware

Quantization performed on:

Windows
Python 3.11
PyTorch
NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM)

Limitations

Quantization may slightly reduce model accuracy depending on the evaluation task.
Performance should be validated for production workloads.
GPU memory requirements depend on sequence length and runtime configuration.

License

This repository contains a quantized derivative of Qwen/Qwen2.5-7B-Instruct.

Please follow the original model's license and usage terms when using or redistributing this model.

This model has been pushed to the Hugging Face Hub using:

Framework: Optimum Quanto
Base Model: Qwen/Qwen2.5-7B-Instruct
Quantization: INT8
Library: huggingface_hub + Optimum Quanto
Documentation: https://huggingface.co/docs/optimum/quanto
Author
**Monishwaran K
GitHub: https://github.com/Monishwaran45
Hugging Face: https://huggingface.co/Helllbos ⭐ If you find this model useful, please consider giving it a Like ❤️ on Hugging Face.