Sagicc/granite-8b-code-instruct-Q5_K_M-GGUF

Name: granite-8b-code-instruct-Q5_K_M-GGUF
Availability: InStock

This model was converted to GGUF format from ibm-granite/granite-8b-code-instruct using llama.cpp after addded support for small Granite Code models in b3026 'llama.cpp release'. Refer to the original model card for more details on the model.

For now only works with llama.cpp

Use with llama.cpp

Install llama.cpp through brew.

brew install ggerganov/ggerganov/llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Sagicc/granite-8b-code-instruct-Q5_K_M-GGUF --model granite-8b-code-instruct.Q5_K_M.gguf -p "You are an AI assistant"

Server:

llama-server --hf-repo Sagicc/granite-8b-code-instruct-Q5_K_M-GGUF --model granite-8b-code-instruct.Q5_K_M.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

git clone https://github.com/ggerganov/llama.cpp &&             cd llama.cpp &&             make &&             ./main -m granite-8b-code-instruct.Q5_K_M.gguf -n 128