Views
No views yet
llama-quantize).| File | Quantization | ~Size |
|---|---|---|
model-Q4_K_M.gguf | Q4_K_M | ~637 MB |
model-Q5_K_M.gguf | Q5_K_M | ~746 MB |
model-Q8_0.gguf | Q8_0 | ~1.1 GB |
1llama-cli -m model-Q4_K_M.gguf -p "Hello" -n 128
2# or
3llama-server -m model-Q4_K_M.gguf --host 0.0.0.0 --port 8080TinyLlama/TinyLlama-1.1B-Chat-v1.0llama.cpp/convert_hf_to_gguf.py (F16), then llama-quantizeysingh-aiml/tinyllama-quantization-gguf