GGUF quantized version of OpenMath2-Llama3.1-8B
project original
source (finetuned model)
Q_2_K (not nice)
Q_3_K_S (acceptable)
Q_3_K_M is acceptable (good for running with CPU)
Q_3_K_L (acceptable)
Q_4_K_S (okay)
Q_4_K_M is recommanded (balance)
Q_5_K_S (good)
Q_5_K_M (good in general)
Q_6_K is good also; if you want a better result; take this one instead of Q_5_K_M
Q_8_0 which is very good; need a reasonable size of RAM otherwise you might expect a long wait
f16 is similar to the original hf model; opt this one or hf also fine; make sure you have a good machine
*the latest update includes Q_4_0, Q_4_1 (belong to Q4 family) and Q_5_0, Q_5_1 (Q5 family)
how to run it
use any connector for interacting with gguf; i.e.,
gguf-connectorthe chart and figure above are from finetuned model (nvidia side); those are used for comparing between the finetuned model and the base model; and the base model is from meta
