# Download the model
wget https://huggingface.co/canes33/Llama-3.1-8B-Q8R16-GGUF/resolve/main/model-Q8R16.gguf

# Run inference
./llama-cli -m model-Q8R16.gguf -p "Your prompt here"

Using Python (llama-cpp-python)

python

from llama_cpp import Llama

llm = Llama(model_path="model-Q8R16.gguf")
output = llm("Your prompt here", max_tokens=100)
print(output)

Using other tools

This GGUF model is compatible with:

llama.cpp
text-generation-webui
KoboldCpp
GPT4All
LM Studio
LoLLMS Web UI
ctransformers

Model File Info

Filename: model-Q8R16.gguf
Quantization: Q8R16