Views
No views yet
| File | Quantization | Size | Parameter Precision | Quality | Recommend for |
|---|---|---|---|---|---|
*-Q4_K_M.gguf | Q4_K_M | ~5.0 GB | 4-bit | medium quality | PC with 8 GB VRAM recommend, 8 GB RAM works but is very slow |
*-Q8_0.gguf | Q8_0 | ~8.5 GB | 8-bit | high quality | recommend for >=10 GB VRAM like a 3080 |
*-f16.gguf | F16 | ~16 GB | 16-bit | Full | Rarely needed, the 8 bit is nearly as good with half the size |
1# With llama.cpp
2./main -m DeepDataDemon-8B-Q4_K_M.gguf -p "Your prompt here"
3
4# With Ollama
5ollama create deepdatademon -f ModelfileFROM ./DeepDataDemon-8B-Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>"""
PARAMETER stop <|start_header_id|>
PARAMETER stop <|end_header_id|>
PARAMETER stop <|eot_id|>