Views
No views yet
1# Download the model
2huggingface-cli download QuantLLM/Llama-3.2-3B-5bit-gguf Llama-3.2-3B-5bit-gguf.Q5_K_M.gguf --local-dir .
3
4# Run with llama.cpp
5./llama-cli -m Llama-3.2-3B-5bit-gguf.Q5_K_M.gguf -p "Hello, how are you?" -n 1281# Create a Modelfile
2echo 'FROM ./Llama-3.2-3B-5bit-gguf.Q5_K_M.gguf' > Modelfile
3
4# Create the model
5ollama create llama-3.2-3b-5bit-gguf -f Modelfile
6
7# Run
8ollama run llama-3.2-3b-5bit-gguf.gguf file from this repository1from llama_cpp import Llama
2
3llm = Llama.from_pretrained(
4 repo_id="QuantLLM/Llama-3.2-3B-5bit-gguf",
5 filename="Llama-3.2-3B-5bit-gguf.Q5_K_M.gguf",
6)
7
8output = llm(
9 "Write a story about a robot:",
10 max_tokens=256,
11 echo=True
12)
13print(output["choices"][0]["text"])| Property | Value |
|---|---|
| Base Model | meta-llama/Llama-3.2-3B |
| Format | GGUF |
| Quantization | Q5_K_M |
| License | apache-2.0 |
| Created | 2025-12-20 |
| Quantization | Bits | Use Case |
|---|---|---|
| Q2_K | 2-bit | Minimum size, experimental |
| Q3_K_M | 3-bit | Very constrained environments |
| Q4_K_M | 4-bit | Recommended for most users |
| Q5_K_M | 5-bit | Higher quality, more memory |
| Q6_K | 6-bit | Near-original quality |
| Q8_0 | 8-bit | Best quality, largest size |
1from quantllm import turbo
2
3# Load and quantize any model
4model = turbo("meta-llama/Llama-3.2-3B")
5
6# Export to any format
7model.export("gguf", quantization="Q5_K_M")