We have quantized the Meta-Llama3-8b-Instruct model into three variants:
Q5_KM
Q4_KM
IQ4_XS
These quantized models offer improved efficiency while maintaining performance.
Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub.
To learn more about our company and services, check out our website at SandLogic.
Architecture: Optimized transformer with Grouped-Query Attention (GQA)
Parameters: 8 billion
Context Length: 8k tokens
Training Data: New mix of publicly available online data (15T+ tokens)
Knowledge Cutoff: March, 2023
Model Capabilities
Llama 3 is designed for multiple use cases, including:
Responding to questions in natural language
Writing code
Brainstorming ideas
Content creation
Summarization
The model understands context and responds in a human-like manner, making it useful for various applications.
Use Cases
Chatbots: Enhance customer service automation
Content Creation: Generate articles, reports, blogs, and stories
Email Communication: Draft emails and maintain consistent brand tone
Data Analysis Reports: Summarize findings and create business performance reports
Code Generation: Produce code snippets, identify bugs, and provide programming recommendations
Model Variants
We offer three quantized versions of the Meta-Llama3-8b-Instruct model:
Q5_KM: 5-bit quantization using the KM method
Q4_KM: 4-bit quantization using the KM method
IQ4_XS: 4-bit quantization using the IQ4_XS method
These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.
Usage
pip install llama-cpp-python
Please refer to the llama-cpp-python documentation to install with GPU support.
Basic Text Completion
Here's an example demonstrating how to use the high-level API for basic text completion:
bash
1from llama_cpp import Llama
23llm = Llama(4model_path="./models/7B/llama-model.gguf",
5verbose=False,
6# n_gpu_layers=-1, # Uncomment to use GPU acceleration7# n_ctx=2048, # Uncomment to increase the context window8)910output = llm(11"Q: Name the planets in the solar system? A: ", # Prompt12max_tokens=32, # Generate up to 32 tokens13stop=["Q:", "\n"], # Stop generating just before a new question14echo=False # Don't echo the prompt in the output15)1617print(output["choices"][0]["text"])
Download
You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.
By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.
We thank Meta for developing and releasing the original Llama 3 model.
Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.
