This is the GGUF quantized version of AVALON-2B, the first sub-3B self-reflective language model. Optimized for on-device inference with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.
Available Files
Filename
Quantization
Size
Quality
Use Case
avalon-2b-q4km.gguf
Q4_K_M
1.5 GB
Best balance
Recommended
Performance Benchmarks
Tested on Apple Silicon with llama.cpp:
Device
Chip
RAM
Speed (tok/s)
Memory
MacBook Air
Apple M3
16 GB
40.2
2.1 GB
MacBook Pro
Apple M3 Pro
18 GB
52.4
2.1 GB
Mac Studio
Apple M2 Ultra
64 GB
78.6
2.0 GB
iPhone 15 Pro
Apple A17 Pro
8 GB
12.4
1.8 GB
Quality Retention
Metric
Full Model
Q4_K_M
Retention
MMLU
62.04%
61.42%
99.0%
Self-RAG Accuracy
82.5%
80.5%
97.6%
Perplexity
Baseline
+1.6%
98.4%
Usage
Ollama (Easiest)
bash
1# Create Modelfile2cat> Modelfile <<'EOF'
3FROM ./avalon-2b-q4km.gguf
45TEMPLATE """{{ if .System }}<|im_start|>system
6{{ .System }}<|im_end|>
7{{ end }}{{ if .Prompt }}<|im_start|>user
8{{ .Prompt }}<|im_end|>
9{{ end }}<|im_start|>assistant
10{{ .Response }}<|im_end|>
11"""
1213PARAMETER stop "<|im_end|>"
14PARAMETER temperature 0.7
1516SYSTEM """You are AVALON, a self-reflective AI assistant. Generate [Retrieval] for time-sensitive questions and [No Retrieval] for factual knowledge. End responses with [Utility:X]."""
17EOF1819# Create and run20ollama create avalon-2b -f Modelfile
21ollama run avalon-2b "What is quantum computing?"
llama.cpp
bash
1# Download2wget https://huggingface.co/nuroai/Avalon-2B-GGUF/resolve/main/avalon-2b-q4km.gguf
34# Run inference5./llama-cli -m avalon-2b-q4km.gguf \6 -p "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"\7 -n 128 --temp 0.7
User: What is the capital of Japan?
Assistant: [No Retrieval]Tokyo is the capital of Japan.[Utility:5]
User: What's the current Bitcoin price?
Assistant: [Retrieval]I need current market data to answer this...[Utility:4]
