This repository contains a 4.0bpw EXL3 quantization of the Meta-Llama-3-8B-Instruct model, optimized specifically for high-speed inference on NVIDIA RTX GPUs.

Why EXL3?

EXL3 (part of the ExLlamaV2 ecosystem) offers the best performance-to-vram ratio for NVIDIA users. On an RTX 5070 (12GB), this model provides lightning-fast token generation while leaving enough VRAM headroom for large context windows or background tasks.

Model Highlights

Format: EXL3 (ExLlamaV2)
Precision: 4.0 bits per weight (bpw)
Architecture: Llama 3
VRAM Friendly: Fits comfortably in 8GB and 12GB cards.

VRAM Usage Estimates

Context Length	VRAM Usage (Approx)
4,096 tokens	~5.3 GB
8,192 tokens	~6.2 GB
16,384 tokens	~7.8 GB

How to use

This model is designed for use with ExLlamaV2 loaders.

Local Inference (TabbyAPI / Oobabooga)

Point your loader to this repository folder.
Select the ExLlamaV2 loader.
Ensure you have the latest CUDA drivers (12.x recommended).

Integration (Python)

python

from exllamav2 import ExLlamaV2, ExLlamaV2Config, ExLlamaV2Tokenizer

config = ExLlamaV2Config()
config.model_dir = "path/to/Llama-3-8B-Instruct-EXL3-4.0bpw"
config.prepare()

model = ExLlamaV2(config)
model.load()

FerrisIntelligence/Llama-3-8B-Instruct-EXL3-4.0bpw

Llama-3-8B-Instruct-EXL3-4.0bpw