This is a quantized version of the Meta Llama 3.2 1B Instruct model, optimized for efficient inference using the GGUF format with Q4_K_M quantization. The model has been quantized to 4-bit precision using the K-quant method (Medium quality), providing a good balance between model size, inference speed, and quality.

This model has been trained to provide helpful, accurate, and contextually appropriate responses regarding the RadiPro's services. Since RadiPro AI agency is a rather small company with limited number of services the chatbot's main purpose is to demonstrate to clients what potential implementation on their platform might look.

Model Details

Base Model: meta-llama/Llama-3.2-1B-Instruct
Quantization Method: Q4_K_M (4-bit K-quant, Medium quality)
Format: GGUF (GPT-Generated Unified Format)
Model Size: ~700MB (quantized from ~2GB)
Architecture: Transformer-based language model
Context Length: 128K tokens
Languages: Primarily English, with multilingual capabilities

Quantization Details

Quantization Type: Q4_K_M
Bits: 4-bit
Method: K-quant (K-means quantization)
Quality: Medium (balanced quality/size trade-off)
Compression Ratio: ~3x reduction in model size

How to Use

Using llama.cpp

bash

# Download llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make

# Run inference
./main -m radipro-chatbot-Llama-3.2-1B-Instruct.Q4_K_M.gguf \
       -p "Your prompt here" \
       -n 128

Using Python with llama-cpp-python

python

from llama_cpp import Llama

# Load the model
llm = Llama(
    model_path="radipro-chatbot-Llama-3.2-1B-Instruct.Q4_K_M.gguf",
    n_ctx=2048,  # Context window
    n_threads=4,  # Number of CPU threads
)

# Generate text
response = llm(
    "What is artificial intelligence?",
    max_tokens=128,
    temperature=0.7,
    top_p=0.9,
)

print(response['choices'][0]['text'])

Using Ollama

bash

# Create a Modelfile
cat > Modelfile << EOF
FROM ./radipro-chatbot-Llama-3.2-1B-Instruct.Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

# Create and run
ollama create radipro-chatbot -f Modelfile
ollama run radipro-chatbot

Performance

Inference Speed: Significantly faster than FP16/BF16 models
Memory Usage: ~1-2GB RAM (depending on context size)
Quality: Maintains ~95-98% of original model quality
Best For: CPU inference, edge devices, resource-constrained environments

System Requirements

Minimum RAM: 2GB
Recommended RAM: 4GB+
CPU: Any modern CPU (x86_64, ARM64)
GPU: Optional (supports GPU acceleration via llama.cpp with CUDA/Metal)

Limitations

Reduced Precision: 4-bit quantization may result in slight quality degradation compared to the full-precision model
Context Window: While the base model supports 128K tokens, practical context limits depend on available memory
Language: Primarily optimized for English, though multilingual capabilities exist
Bias: This model has been specifically trained to serve as a RadiPro chatbot. It won't answer general questions and it will always try to steer conversation towards the agency and their services.

Ethical Considerations

This model is intended for research and educational purposes. Users should be aware of potential biases and limitations. Please review the Llama 3.2 Community License for usage terms.

Citation

If you use this model, please cite the original Llama 3.2 model:

bibtex

@misc{llama32,
  title={Llama 3.2},
  author={Meta AI},
  year={2024},
  howpublished={\url{https://llama.meta.com/llama3/}}
}

License

This model is released under the Llama 3.2 Community License. Please review the license terms before use.

Acknowledgments

Base Model: Meta AI for the Llama 3.2 1B Instruct model
Quantization: GGUF format and quantization tools
Community: The open-source AI community for tools and support

Model Card Contact

For questions, issues, or contributions related to this quantized model, please open an issue in the repository.

Note: This is a quantized model card. For the full-precision model, please refer to the original Llama 3.2 1B Instruct model.

raditotev/radipro-chatbot-Llama-3.2-1B-Instruct-GGUF

RadiPro Chatbot - Llama 3.2 1B Instruct (GGUF Q4_K_M)

Model Description