SmolLM2-135M-Instruct-Reasoning-GGUF
GGUF conversions of:
Avtrkrb/SmolLM2-135M-Instruct-bnb-4bit-reasoning
Available Quantizations
| Quantization | Recommended Use |
|---|
| Q4_0 | Same as Q4_K_M |
| Q4_K_M | Best balance of quality and size |
| Q5_K_M | Better quality |
| Q6_K | Better quaity smaller than Q8_0 |
| Q8_0 | Highest quality quantized version |
Chat Template
This model uses the standard SmolLM2 chat template.
<|im_start|>system
You are a helpful AI assistant named SmolLM, trained by Hugging Face
<|im_end|>
<|im_start|>user
Hello
<|im_end|>
<|im_start|>assistant
Ollama Example
FROM ./SmolLM2-135M-Instruct-bnb-4bit-reasoning-Q4_K_M.gguf
TEMPLATE """
{{ .Prompt }}
"""
llama.cpp Example
./llama-cli
-m SmolLM2-135M-Instruct-bnb-4bit-reasoning-Q4_K_M.gguf
Notes
When using GGUF versions, ensure the same chat template used during training is applied during inference. Incorrect templates can significantly reduce model quality.
Source Model
Avtrkrb/SmolLM2-135M-Instruct-bnb-4bit-reasoning
