This repository contains GGUF quantizations of JetBrains/Mellum2-12B-A2.5B-Instruct for efficient local inference with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

Model details

Base model: JetBrains/Mellum2-12B-A2.5B-Instruct
Format: GGUF
Architecture: Mellum2
Task type: Instruction-following assistant
Context length: 131,072 tokens
License: Apache 2.0

Available quantizations

File	Quantization	Size	Notes
`Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf`	Q4_K_M	8.07 GB	Lower-memory deployment with good quality-to-size tradeoff
`Mellum2-12B-A2.5B-Instruct-Q5_K_M.gguf`	Q5_K_M	9.21 GB	Higher-quality 5-bit deployment with a modest size increase

Intended use

This model is intended for:

local chat and assistant workflows
coding assistance
tool-use experiments
CPU-friendly or memory-constrained inference
users who want a balance between quality and memory usage

Notes

These quantizations were produced with llama.cpp. During conversion, some tensors may require fallback quantization depending on the model architecture, which is expected and does not prevent successful inference.

Usage

llama.cpp

./llama-server -m Mellum2-12B-A2.5B-Instruct-Q5_K_M.gguf --jinja --port 8000

Ollama

Create a Modelfile pointing to the GGUF file:

FROM ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf

Then run:

bash

ollama create mellum2-q4 -f Modelfile
ollama run mellum2-q4

License

Released under the same license as the base model: Apache 2.0.

Acknowledgments

Base model: JetBrains/Mellum2-12B-A2.5B-Instruct
Quantization performed with llama.cpp

NANI-Nithin/Mellum2-12B-A2.5B-Instruct-GGUF

Mellum2-12B-A2.5B-Instruct-GGUF