You can deploy this model and start earning money today!

eformat/
granite-3.0-8b-instruct-Q4_K_M-GGUF

Name: granite-3.0-8b-instruct-Q4_K_M-GGUF
Availability: InStock

Views

No views yet

eformat/granite-3.0-8b-instruct-Q4_K_M-GGUF

Not all tools (vllm, llama.cpp) seem to support the new model config params it seems (25/10/2024).

json

# config.json
"model_type": "granite"
"architectures": [
  "GraniteForCausalLM"
]

This gguf conversion done using old ones

json

# config.json
"model_type": "llama"
"architectures": [
  "LlamaForCausalLM"
]

This gguf loads OK - tested using:

bash

# llama.cpp
./llama-server --verbose --gpu-layers 99999 --parallel 2 --ctx-size 4096 -m ~/instructlab/models/granite-3.0-8b-instruct-Q4_K_M.gguf

bash

# vllm
vllm serve ~/instructlab/models/granite-3.0-8b-instruct-Q4_K_M.gguf