Property	Value
Base model	JetBrains/Mellum2-12B-A2.5B-Thinking
Architecture	Mellum (MoE)
Total parameters	12B
Active parameters	2.5B
Experts	64 (8 per token)
Context length	128K (sliding window)
Quantization	Q4_K_M
File size	~7.5 GB

Usage

bash

llama-server \
  -m Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf \
  -c 32000 \
  -ngl 99 \
  --jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --port 18081

For thinking mode, use reasoning_budget to cap reasoning tokens:

bash

curl http://localhost:18081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Mellum2-12B-A2.5B-Thinking-Q4_K_M","messages":[{"role":"user","content":"Explain quicksort"}],"max_tokens":500,"reasoning_budget":512}'

Low VRAM (6GB GPU)

For GPUs with limited VRAM, offload MoE experts to CPU:

bash

llama-server \
  -m Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf \
  -c 32000 \
  -ngl 99 \
  -ncmoe 20 \
  -t 8 \
  --jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0

Tested on GTX 1060 6GB: ~18 t/s with ncmoe=20, ~22 t/s with ncmoe=10.

Acknowledgements

JetBrains for the original Mellum2 model
llama.cpp for the inference engine

developerjeremylive/Mellum2-12B-A2.5B-Thinking-GGUF-etheroi

Mellum2-12B-A2.5B-Thinking GGUF (Q4_K_M)

Model Details

Usage

Low VRAM (6GB GPU)

Acknowledgements

developerjeremylive/
Mellum2-12B-A2.5B-Thinking-GGUF-etheroi