Nexesenex/

princeton-nlp_Sheared-LLaMA-2.7B_iMat_Custom_Quant_Stategies-GGUF

Notes only for now, rework needs to be done.

Q2_K_S

Master 900.24 MiB (2.80 BPW) PPL = 9.2374 +/- 0.05597

PR : 0.97 GB (2.86 BPW) 0.90 GiB (2.86 BPW) PPL over 655 chunks for n_ctx=512 = 8.8572 +/- 0.05438

Q2_K

Master 979.89 MiB (3.04 BPW) PPL = 8.4255 +/- 0.05158

PR : 1.02 GB (3.01 BPW) 0.95 GiB (3.01 BPW) PPL over 655 chunks for n_ctx=512 = 8.4388 +/- 0.05174

Q2_K_L

PR : 1.07 GB (3.18 BPW) 1.00 GiB (3.18 BPW) PPL over 655 chunks for n_ctx=512 = 8.1990 +/- 0.04955

Q3_K_M

Master 1.24 GiB (3.94 BPW) PPL = 7.5814 +/- 0.04537

PR : 1.33 GB (3.95 BPW) 1.24 GiB (3.95 BPW) PPL over 655 chunks for n_ctx=512 = 7.5797 +/- 0.04537

Q3_K_L

Master 1.35 GiB (4.29 BPW) PPL = 7.5425 +/- 0.04506

PR : 1.45 GB (4.29 BPW) 1.35 GiB (4.29 BPW) PPL over 655 chunks for n_ctx=512 = 7.5425 +/- 0.04506

IQ1_XS

PR : 0.63 GB (1.86 BPW) 0.59 GiB (1.86 BPW) PPL over 655 chunks for n_ctx=512 = 20.1826 +/- 0.13290

PR2 : 0.64 GB (1.91 BPW) 0.60 GiB (1.91 BPW) PPL over 655 chunks for n_ctx=512 = 17.2270 +/- 0.11191

IQ1_S

Master 600.66 MiB (1.87 BPW) PPL = 19.5631 +/- 0.12335

PR : 0.65 GB (1.94 BPW) 0.61 GiB (1.94 BPW) PPL over 655 chunks for n_ctx=512 = 16.8894 +/- 0.10891

PR2 : 0.67 GB (1.97 BPW) 0.62 GiB (1.97 BPW) PPL over 655 chunks for n_ctx=512 = 15.0819 +/- 0.09635

IQ1_M

Master 646.43 MiB (2.01 BPW) PPL = 14.6658 +/- 0.09171

PR : 0.70 GB (2.06 BPW) 0.65 GiB (2.06 BPW) PPL over 655 chunks for n_ctx=512 = 13.7678 +/- 0.08796

PR2 : 0.70 GB (2.07 BPW) 0.65 GiB (2.07 BPW) PPL over 655 chunks for n_ctx=512 = 13.0640 +/- 0.08332

IQ1_XL

PR : 0.73 GB (2.16 BPW) 0.68 GiB (2.16 BPW) PPL over 655 chunks for n_ctx=512 = 12.4043 +/- 0.07821

PR2 : 0.73 GB (2.16 BPW) 0.68 GiB (2.16 BPW) PPL over 655 chunks for n_ctx=512 = 11.5807 +/- 0.07290

IQ2_XXS

Master 722.70 MiB (2.24 BPW) PPL = 10.7338 +/- 0.06621

PR : 0.79 GB (2.34 BPW) 0.74 GiB (2.34 BPW) PPL over 655 chunks for n_ctx=512 = 9.9385 +/- 0.06080

PR2 : 782.00 MB (2.32 BPW) 745.78 MiB (2.32 BPW) PPL over 655 chunks for n_ctx=512 = 9.9519 +/- 0.06097

IQ2_XS

Master 789.96 MiB (2.45 BPW) PPL = 9.6386 +/- 0.05831

PR : 0.86 GB (2.54 BPW) 0.80 GiB (2.54 BPW) PPL over 655 chunks for n_ctx=512 = 9.1694 +/- 0.05600

PR 2 : 0.84 GB (2.50 BPW) 0.79 GiB (2.50 BPW) PPL over 655 chunks for n_ctx=512 = 9.1616 +/- 0.05577

IQ2_S

Master 853.19 MiB (2.65 BPW) PPL = 8.9655 +/- 0.05484

PR : 0.90 GB (2.66 BPW) 0.84 GiB (2.66 BPW) PPL over 655 chunks for n_ctx=512 = 8.9321 +/- 0.05441

PR 2 : 0.90 GB (2.65 BPW) 0.83 GiB (2.65 BPW) PPL over 655 chunks for n_ctx=512 = 8.8710 +/- 0.05385

PR 3 : 0.88 GB (2.60 BPW) 0.82 GiB (2.60 BPW) PPL over 655 chunks for n_ctx=512 = 8.8873 +/- 0.05410

IQ2_M

Master 914.20 MiB (2.84 BPW) PPL = 8.4373 +/- 0.05110

PR : 0.95 GB (2.82 BPW) 0.89 GiB (2.82 BPW) PPL over 655 chunks for n_ctx=512 = 8.4881 +/- 0.05125

PR 2 : 0.96 GB (2.84 BPW) 0.89 GiB (2.84 BPW) PPL over 655 chunks for n_ctx=512 = 8.3724 +/- 0.05052

PR 3 : 0.96 GB (2.83 BPW) 0.89 GiB (2.83 BPW) PPL over 655 chunks for n_ctx=512 = 8.3566 +/- 0.05040

IQ2_XL

PR : 1.00 GB (2.97 BPW) 0.93 GiB (2.97 BPW) PPL over 655 chunks for n_ctx=512 = 8.2687 +/- 0.04968

PR 2 : 1.01 GB (3.00 BPW) 0.94 GiB (3.00 BPW) PPL over 655 chunks for n_ctx=512 = 8.1774 +/- 0.04925

IQ3_XXS

Master 998.70 MiB (3.10 BPW) PPL = 8.0228 +/- 0.04817

PR : 1.09 GB (3.22 BPW) 1.01 GiB (3.22 BPW) PPL over 655 chunks for n_ctx=512 = 7.8982 +/- 0.04752

IQ3_XS

Master 1.06 GiB (3.36 BPW) 7.7736 +/- 0.04626

PR : 1.19 GB (3.53 BPW) 1.11 GiB (3.53 BPW) PPL over 655 chunks for n_ctx=512 = 7.7329 +/- 0.04586

PR 2 : 1.16 GB (3.44 BPW) 1.08 GiB (3.44 BPW) PPL over 655 chunks for n_ctx=512 = 7.7327 +/- 0.04591

IQ3_S

Master 1.11 GiB (3.53 BPW) PPL = 7.6667 +/- 0.04548

PR : 1.23 GB (3.64 BPW) 1.14 GiB (3.64 BPW) PPL over 655 chunks for n_ctx=512 = 7.6479 +/- 0.04551

IQ3_M

Master 1.17 GiB (3.73 BPW) PPL = 7.6557 +/- 0.04521

PR : 1.30 GB (3.84 BPW) 1.21 GiB (3.84 BPW) PPL over 655 chunks for n_ctx=512 = 7.5941 +/- 0.04558

PR 2 : 1.28 GB (3.80 BPW) 1.19 GiB (3.80 BPW) PPL over 655 chunks for n_ctx=512 = 7.5864 +/- 0.04569

IQ3_XL

PR : 1.32 GB (3.89 BPW) 1.22 GiB (3.89 BPW) PPL over 655 chunks for n_ctx=512 = 7.5774 +/- 0.04555

PR 2 : 1.33 GB (3.95 BPW) 1.24 GiB (3.95 BPW) PPL over 655 chunks for n_ctx=512 = 7.5446 +/- 0.04516

IQ3_XXL

PR : 1.35 GB (3.99 BPW) 1.26 GiB (3.99 BPW) PPL over 655 chunks for n_ctx=512 = 7.5468 +/- 0.04529

PR 2 : 1.37 GB (4.07 BPW) 1.28 GiB (4.07 BPW) PPL over 655 chunks for n_ctx=512 = 7.5119 +/- 0.04497

IQ3_XXXL

PR : 1.40 GB (4.13 BPW) 1.30 GiB (4.13 BPW) PPL over 655 chunks for n_ctx=512 = 7.4913 +/- 0.04487

IQ4_XS

Master 1.36 GiB (4.32 BPW) PPL = 7.4449 +/- 0.04439

PR : 1.46 GB (4.32 BPW) 1.36 GiB (4.32 BPW) Final estimate: PPL over 655 chunks for n_ctx=512 = 7.4449 +/- 0.04439

FP16

Master 5.03 GiB (16.00 BPW) PPL = 7.3511 +/- 0.04375