You can deploy this model and start earning money today!

Roy2144/
Meta-LLama

Name: Meta-LLama
Availability: InStock

Views

No views yet

Quantized Meta-LLaMA-3.1-8B-Instruct vLLM Inference Project

This project benchmarks the inference performance of:

vLLM server with Quantized (W8A8) Meta-LLaMA-3.1-8B-Instruct model using LLM Compressor
Hugging Face Transformers running full-precision Meta-LLaMA-3.1-8B-Instruct in float16.

We evaluate:

Quantized Meta-LLaMA-3.1-8B-Instruct model using LLM Compressor, applying SmoothQuant + GPTQ techniques with W8A8 quantization scheme.
Uploaded the quantized model to Hugging Face Hub: Roy2144/Meta-LLama.
Deployed the quantized model using vLLM OpenAI-compatible API server.
Ran Benchmarks:
- vLLM server performance (Quantized)
- Hugging Face Transformers local inference (float16)
Measured:
- Latency per request
- Throughput (tokens/sec)
- Response quality comparison
Analyzed differences between quantized and float16 models.

Metric	vLLM + Quantized Meta-LLaMA W8A8	HF Transformers + Meta-LLaMA float16
Avg Latency	0.62 seconds	2.66 seconds
Avg Throughput	163.99 tokens/sec	30.46 tokens/sec

vLLM + Quantization massively boosts inference speed (~4–5x faster than float16).
Quantized Meta-LLaMA-3.1-8B-Instruct maintains excellent response quality despite INT8 compression.
HF Transformers serve as a baseline for unoptimized float16 inference.

Prompt: "Explain quantization in simple words for a beginner."

Both vLLM Quantized and HF float16 models generated detailed, coherent responses.
Quantized model responses were slightly more concise.