Meta-Llama-3-70B-Instruct-abliterated-v3.5-IMat-GGUF

• Files
  • IMatrix
  • Common Quants
  • All Quants
• Downloading using huggingface-cli
• Inference
  • Simple chat template
  • Chat template with system prompt
  • Llama.cpp
• FAQ
  • Why is the IMatrix not applied everywhere?
  • How do I merge a split GGUF?

Files

IMatrix

Common Quants

All Quants

Downloading using huggingface-cli

huggingface-cli download legraphista/Meta-Llama-3-70B-Instruct-abliterated-v3.5-IMat-GGUF --include "Meta-Llama-3-70B-Instruct-abliterated-v3.5.Q8_0/*" --local-dir ./
# see FAQ for merging GGUF's

Inference

Simple chat template

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

{user_prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{assistant_response}<|eot_id|><|start_header_id|>user<|end_header_id|>

{next_user_prompt}<|eot_id|>

Chat template with system prompt

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{user_prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{assistant_response}<|eot_id|><|start_header_id|>user<|end_header_id|>

{next_user_prompt}<|eot_id|>

Llama.cpp

FAQ

Why is the IMatrix not applied everywhere?

How do I merge a split GGUF?

1. Make sure you have gguf-split available
   • To get hold of gguf-split, navigate to https://github.com/ggerganov/llama.cpp/releases
   • Download the appropriate zip for your system from the latest release
   • Unzip the archive and you should be able to find gguf-split
2. Locate your GGUF chunks folder (ex: Meta-Llama-3-70B-Instruct-abliterated-v3.5.Q8_0)
3. Run gguf-split --merge Meta-Llama-3-70B-Instruct-abliterated-v3.5.Q8_0/Meta-Llama-3-70B-Instruct-abliterated-v3.5.Q8_0-00001-of-XXXXX.gguf Meta-Llama-3-70B-Instruct-abliterated-v3.5.Q8_0.gguf
   • Make sure to point gguf-split to the first chunk of the split.