You can deploy this model and start earning money today!

JASouth/
Ornith-1.5-9B-MTP-Elastic-IQ-HybridQuant

Name: Ornith-1.5-9B-MTP-Elastic-IQ-HybridQuant
Availability: InStock

transformers

gguf

text-generation

ornith-ai/Ornith-1.5-9B

Views

No views yet

Update: If I get enought tips I will release the framework, my experiments and my conclusions about quantizations. If you want information for free, start a discussion

Experimental Elastic (`_E`) GGUF Quantizations

Disclaimer: These quantization builds are provided strictly for research and experimental benchmarking purposes. The official Q6_K build is included in the evaluation tables solely as an external baseline for comparative analysis.

Overview

This repository provides a custom IQ-Hybrid quantization engineered to maximize response coherence and perplexity while fitting comfortably within 8 GB VRAM consumer GPUs.

This repository provides custom Elastic quantization variants (Q5_E, Q4_E, and Q3_E), generated using importance matrices (imatrix) calibrated against domain corpora sourced from AtomicChat/calib-corpora.

These builds investigate tensor-level allocation dynamics, vocabulary-to-body noise trade-offs, and quality retention across varying memory footprints.

Updete: Versions uploaded with MTP in their name have optimized (Q8_0) MTP.

Perplexity Benchmarks

Evaluations were conducted using llama-perplexity on the standard benchmark Wikitext-2 (wikitext-2-raw/wiki.test.raw) at context size $c = 4096$.

Wikitext-2 Test Results

Quantization / Model Variant	File Size	Perplexity (PPL)	Error Margin (±\ σ )	Status / Role
Original BF16	~18.0 GB	8.9492	± 0.06728	Uncompressed Base Reference
Official Q6_K	7.03 GB	8.7090	± 0.06451	Upstream Baseline (Comparison)
Custom Q5_E	6.91 GB	9.1529	± 0.06959	Experimental Elastic Build
Custom Q4_E	~5.99 GB	8.6587	± 0.06313	Experimental Elastic Build
Custom Q3_E	~5.07 GB	8.1402	± 0.05477	Experimental Elastic Build
Custom_MTP Q5_E	6.91 GB	9.0391	± 0.06819	Experimental Elastic Build-MTP optimized version
Custom_MTP Q4_E	~5.99 GB	8.6977	± 0.06356	Experimental Elastic Build-MTP optimized version
Custom_MTP Q3_E	~5.07 GB	8.3722	± 0.05690	Experimental Elastic Build-MTP optimized version

Evaluation Notes & Methodology

Baseline Reference (Q6_K): Included strictly as an external control point to observe how standard uniform K-quantization compares with variable-allocated Elastic schemes under identical evaluation parameters.
Calibration Corpora: Importance matrices were computed using calibration datasets sourced from AtomicChat/calib-corpora. (calib_longctx.txt, eval_code_full.txt, eval_agentic.txt, eval_neutral.txt)
Evaluation Command:
bash
1llama-perplexity \ 2 -m <model-path>.gguf \ 3 -f wikitext-2-raw/wiki.test.raw \ 4 -c 4096 \ 5 -b 4096 \ 6 --ubatch-size 512 \ 7 --flash-attn on \ 8 --threads 6

This project was developed to push the limits of modern architectures on consumer 8GB hardware. If you found this optimization useful and would like to support future quantizations and hardware upgrades, you can support me here:

ko-fi

JASouth/Ornith-1.5-9B-MTP-Elastic-IQ-HybridQuant