This repository provides a custom IQ-Hybrid quantization engineered to maximize response coherence and perplexity while fitting comfortably within 8 GB VRAM consumer GPUs.

By using an importance-matrix (imatrix) driven greedy allocation pipeline with class-specific quantization ladders, this build achieves better perplexity than the official Q6_K quantization while saving over 1.47 GB of disk and VRAM footprint.

📊 Perplexity & VRAM Benchmark

Evaluated using llama-perplexity on standard wikitext-2-raw (512 context chunks):

File	Size	BPW	PPL (Wikitext-2)	8GB VRAM Usability
Ornith-1.5-9B-SHQ-6.5GB	6.55 GB	~6.1	8.9089	Full GPU Offload (up to 16k context)
Ornith-1.5-9B-SHQ-5.4GB	5.38 GB	~5.0	9.2262	Full GPU Offload (Long context 32k+)
Official Q6_K	6.85 GB	6.56	9.4833	Tight (Risk of OOM on 8GB VRAM)
Official Q5_K_M	6.40 GB	5.50	~9.70+	Fits in 8GB

⚡ Key Highlights

Outperforms Official Q6_K with 21% Less Memory: Achieves superior language modeling accuracy (lower PPL) than the official 6.85 GB build while reducing file size down to 5.38 GB.
Engineered for 8GB GPUs: Leaves ~1.8–2.0 GB of VRAM headroom on 8GB hardware (RTX 3060/4060, Intel Arc A750, RX 6600/7600), allowing full layer offloading and extended context lengths (32k+ tokens) without CPU/RAM spillover.
Non-Linear Vocabulary Retention: The large token-embedding and output matrices are preserved using Gaussian-mapped IQ4_NL, capturing semantic fidelity far better than traditional linear block quantizers.
Protected Attention & SSM States: Critical attention projections, Mamba/SSM recurrent parameters, and layer norms are shielded in high-precision formats (Q6_K, Q8_0, and F32), channeling aggressive compression only into less sensitive feed-forward layers.

This project was developed to push the limits of modern architectures on consumer 8GB hardware. If you found this optimization useful and would like to support future quantizations and hardware upgrades, you can support me here:

ko-fi

JASouth/Ornith-1.5-9B-IQ-Hybrid

Update: If I get enought tips I will release the framework, my experiments and my conclusions about quantizations. If you want information for free, start a discussion

🦅 Ornith-1.5-9B IQ-Hybrid Quantization (8GB VRAM Optimized)

📊 Perplexity & VRAM Benchmark

⚡ Key Highlights

JASouth/
Ornith-1.5-9B-IQ-Hybrid