DrontGPTQ is a simple quantizer for GPT-like models.
Supports:
2 bit ✅
3 bit ✅
4 bit ✅
5 bit ✅
6 bit ✅
7 bit ❌
8 bit ✅
12 bit ✅
16 bit ✅
32 bit ✅
Consists of just two files.
DrontQ.py – a very simple quantizer. Converts HF F32 to .dq format (Dront quant).
DQuse.py – a simple way to run .dq models.
To use this, scroll to the end of the files; you will find the main variables there, which you should adjust to your needs.
Quantizes local models—specifically GPT-like ones.
Most Dront gpt-like models will also be released in DQ quantization! Especially the new VdrontV
