Step	Method	Detail
1	Layer Pruning	32 → 28 layers, 16 GB → 14 GB
2	LoRA Fine-tuning	QLoRA on WMT19 En-Zh (36k samples)
3	Weight Merging	`merge_and_unload()` — no PEFT at runtime
4	int4 Quantization	NF4 double-quant via bitsandbytes

How to Load

python

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained('AnishRacherla/aya-expanse-8b-compressed-final', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('AnishRacherla/aya-expanse-8b-compressed-final', quantization_config=bnb, device_map='auto', trust_remote_code=True)

Benchmark (FLORES-200, 100 samples, En→Zh)

Metric	fp16 (merged)	int4 — this model	Δ
VRAM	13.34 GB	4.89 GB	-8.46 GB
Load Time	101.5s	47.2s	-54.3s
Tokens/sec	14.9	10.7	-4.2
COMET Score	0.8642	0.8553	-0.0089