from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16)
tok = AutoTokenizer.from_pretrained('AnishRacherla/aya-expanse-8b-compressed-final-int4', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('AnishRacherla/aya-expanse-8b-compressed-final-int4', quantization_config=bnb, device_map='auto', trust_remote_code=True)

Benchmark vs Baseline (FLORES-200, 100 En→Zh)

Metric	Baseline int4	This model	Δ
VRAM	5.31 GB	4.89 GB	-0.42
Tokens/sec	9.2	9.8	+0.6
COMET	0.5095	0.8553	+0.3458