Llamacpp Quantizations of Meta-Llama-3.1-8B
Using
llama.cpp release
b3472 for quantization.
Download a file (not the whole branch) from below:
Benchmark Results
Results have been computed using:
| Benchmark | Quant type | Metric |
|---|
| WinoGrande (0-shot) | Q8_0 | 74.1121 +/- 1.2311 |
| WinoGrande (0-shot) | Q4_K_M | 73.1650 +/- 1.2453 |
| WinoGrande (0-shot) | Q3_K_M | 72.7703 +/- 1.2511 |
| WinoGrande (0-shot) | Q3_K_S | 72.3757 +/- 1.2567 |
| WinoGrande (0-shot) | Q2_K | 68.4294 +/- 1.3063 |
| HellaSwag (0-shot) | Q8_0 | 79.41645091 |
| HellaSwag (0-shot) | Q4_K_M | 79.05795658 |
| HellaSwag (0-shot) | Q3_K_M | 79.41645091 |
| HellaSwag (0-shot) | Q3_K_S | 76.93686517 |
| HellaSwag (0-shot) | Q2_K | 72.16689902 |
| MMLU (0-shot) | Q8_0 | 39.4703 +/- 1.2427 |
| MMLU (0-shot) | Q4_K_M | 39.5349 +/- 1.2431 |
| MMLU (0-shot) | Q3_K_M | 38.8889 +/- 1.2394 |
| MMLU (0-shot) | Q3_K_S | 37.2739 +/- 1.2294 |
| MMLU (0-shot) | Q2_K | 35.4651 +/- 1.2163 |
Downloading using huggingface-cli
First, make sure you have hugginface-cli installed:
pip install -U "huggingface_hub[cli]"Then, you can target the specific file you want:
huggingface-cli download fedric95/Meta-Llama-3.1-8B-GGUF --include "Meta-Llama-3.1-8B-Q4_K_M.gguf" --local-dir ./If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:
huggingface-cli download fedric95/Meta-Llama-3.1-8B-GGUF --include "Meta-Llama-3.1-8B-Q8_0.gguf/*" --local-dir Meta-Llama-3.1-8B-Q8_0You can either specify a new local-dir (Meta-Llama-3.1-8B-Q8_0) or download them all in place (./)
Reproducibility
