You can deploy this model and start earning money today!

localmodels/
LLaMA-7B-ggml

Name: LLaMA-7B-ggml
Availability: InStock

Views

No views yet

LLaMA 7B ggml

From Meta: https://ai.meta.com/blog/large-language-model-llama-meta-ai

Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`

Quantized using an older version of llama.cpp and compatible with llama.cpp from May 19, commit 2d5db48.

k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`

Quantization methods compatible with latest llama.cpp from June 6, commit 2d43387.

Provided files

Name	Quant method	Bits	Size	Max RAM required	Use case
llama-7b.ggmlv3.q2_K.bin	q2_K	2	2.80 GB	5.30 GB	New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors.
llama-7b.ggmlv3.q3_K_L.bin	q3_K_L	3	3.55 GB	6.05 GB	New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
llama-7b.ggmlv3.q3_K_M.bin	q3_K_M	3	3.23 GB	5.73 GB	New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
llama-7b.ggmlv3.q3_K_S.bin	q3_K_S	3	2.90 GB	5.40 GB	New k-quant method. Uses GGML_TYPE_Q3_K for all tensors
llama-7b.ggmlv3.q4_0.bin	q4_0	4	3.79 GB	6.29 GB	Original quant method, 4-bit.
llama-7b.ggmlv3.q4_1.bin	q4_1	4	4.21 GB	6.71 GB	Original quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
llama-7b.ggmlv3.q4_K_M.bin	q4_K_M	4	4.05 GB	6.55 GB	New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K
llama-7b.ggmlv3.q4_K_S.bin	q4_K_S	4	3.79 GB	6.29 GB	New k-quant method. Uses GGML_TYPE_Q4_K for all tensors
llama-7b.ggmlv3.q5_0.bin	q5_0	5	4.63 GB	7.13 GB	Original quant method, 5-bit. Higher accuracy, higher resource usage and slower inference.
llama-7b.ggmlv3.q5_1.bin	q5_1	5	5.06 GB	7.56 GB	Original quant method, 5-bit. Even higher accuracy, resource usage and slower inference.
llama-7b.ggmlv3.q5_K_M.bin	q5_K_M	5	4.77 GB	7.27 GB	New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K
llama-7b.ggmlv3.q5_K_S.bin	q5_K_S	5	4.63 GB	7.13 GB	New k-quant method. Uses GGML_TYPE_Q5_K for all tensors
llama-7b.ggmlv3.q6_K.bin	q6_K	6	5.53 GB	8.03 GB	New k-quant method. Uses GGML_TYPE_Q8_K for all tensors - 6-bit quantization
llama-7b.ggmlv3.q8_0.bin	q8_0	8	7.16 GB	9.66 GB	Original quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users.

localmodels/LLaMA-7B-ggml

LLaMA 7B ggml

Original llama.cpp quant methods: q4_0, q4_1, q5_0, q5_1, q8_0

k-quant methods: q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K

Provided files

localmodels/
LLaMA-7B-ggml

Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`

k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`