You can deploy this model and start earning money today!

Roy2144/
Mistral-7B-Instruct-INT8

Name: Mistral-7B-Instruct-INT8
Availability: InStock

Views

No views yet

model_card = """

Mistral-7B-Instruct-INT8 (Quantized with llmcompressor)

This is a quantized version of mistralai/Mistral-7B-Instruct-v0.2, compressed using llmcompressor (a quantization toolkit developed by the vLLM team).

The goal of this quantization was to reduce memory and improve inference efficiency while preserving model accuracy. The result is a Hugging Face-compatible INT8 model with significantly lower memory usage and good performance on commodity GPUs.

Quantization Details or recipe

The quantization was done using the llmcompressor library. The following modifiers and configuration were used:

python

from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.transformers import oneshot

recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(scheme="W8A8", targets="Linear", ignore=["lm_head"]),
]

oneshot(
    model="mistralai/Mistral-7B-Instruct-v0.2",
....
....
)

How to use the model

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Roy2144/Mistral-7B-Instruct-INT8")
tokenizer = AutoTokenizer.from_pretrained("Roy2144/Mistral-7B-Instruct-INT8")

inputs = tokenizer("Why is quantization useful for LLMs?", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=64)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Benchmark Results

Benchmarked on a single NVIDIA A100 40GB GPU using:
- Transformers backend (FP16 and INT8 variants). Note: The current implementation of transformers does not include optimized kernels for direct INT8 computation. So transformers (INT8) is optional here.
- vLLM engine with INT8 inference acceleration for improved performance.

Metric	Transformers (INT8)	Transformers (FP16)	vLLM (INT8)
Avg Latency	150,045 ms ❌	2,278 ms ✅	1,100 ms ⚡
Throughput	0.51 tokens/sec ❌	33.72 tokens/sec ✅	76.47 tokens/sec ⚡
Peak VRAM	20.5 GB ✅	29.0 GB ❌	20.5 GB ✅

Benchmarking Process Overview

To evaluate the impact of INT8 quantization and compare different inference backends, we conducted controlled benchmarks using the following setup:

Model: mistralai/Mistral-7B-Instruct-v0.2 (base) and its quantized variant
Device: Single NVIDIA A100 GPU (40GB)
Backends Evaluated:
- Transformers (PyTorch) for both FP16 and INT8
- vLLM runtime for INT8 acceleration
Prompt Set: 5 diverse user-like queries related to model usage, quantization, and vLLM
Inference Config: max_new_tokens=64, batch_size=1

We measured three key metrics:

Latency (end-to-end response time)
Throughput (tokens/sec)
Peak VRAM Usage

The benchmarking script ran each prompt individually across the backends, capturing time and memory performance using PyTorch and CUDA APIs.

Scope & Future Work

This project focused on compressing and benchmarking the Mistral-7B-Instruct model using both:

Hugging Face Transformers (FP16 and INT8 inference)
vLLM runtime with OpenAI-compatible HTTP interface

Future improvements and scope of work include:

Load testing with GuideLLM for more robust, concurrent benchmark evaluation
End-to-end latency profiling with multi-turn prompts and larger batch sizes
Accuracy comparison between original and quantized outputs (task-level metrics)
Exploring hybrid quantization + sparsity for further efficiency

GuideLLM is a promising benchmarking tool that can simulate realistic OpenAI-style load and measure key inference metrics with concurrency and scale.

Credits

Base model: mistralai/Mistral-7B-Instruct-v0.2
Quantized with: llmcompressor