Mellum2 Thinking is a reasoning-augmented Mixture-of-Experts assistant model. It has 64 experts with 8 active per token, a 131,072-token context window, and emits reasoning in <think>...</think> blocks before the final answer.

Conversion details

Source: JetBrains/Mellum2-12B-A2.5B-Thinking
Format: MLX safetensors
Quantization: affine, 6 bits, group size 64
License: Apache-2.0
EOS token: <|im_end|> (token ID 28)

The upstream config.json and generation_config.json identify token ID 0 as the EOS token, while the tokenizer identifies <|im_end|> (ID 28) as EOS. This conversion uses token ID 28 so MLX generation stops at the end of the assistant turn.

Usage

bash

pip install -U mlx-lm

mlx_lm.chat \
  --model mlx-community/Mellum2-12B-A2.5B-Thinking-6bit \
  --max-tokens 8192 \
  --temp 0.6 \
  --top-p 0.95

Benchmarks

Benchmarked with oMLX 0.5.7 and MLX-LM 0.31.3 on an Apple M5 Max MacBook Pro with 128 GB unified memory, using the built-in Python-code workload and 128 generated tokens. Results vary with hardware, runtime versions, context length, and generation settings.

Prompt tokens	TTFT	Prefill	Generation	End-to-end	Peak memory
4,096	717.8 ms	5,706.1 tok/s	145.9 tok/s	1.6 s	9.90 GB
16,384	3,436.8 ms	4,767.2 tok/s	132.8 tok/s	4.4 s	10.2 GB
32,768	9,389.4 ms	3,489.9 tok/s	114.7 tok/s	10.5 s	10.5 GB
65,536	23,584.3 ms	2,778.8 tok/s	87.4 tok/s	25.1 s	11.2 GB
131,072	69,353.6 ms	1,889.9 tok/s	54.9 tok/s	71.7 s	12.7 GB

At a 4,096-token prompt, batch generation achieved 196.4, 187.8, and 194.2 aggregate generation tok/s at 2, 4, and 8 concurrent requests, respectively, compared with 145.9 tok/s for one request.

Model provenance

For the original model card, training details, benchmark results, and usage guidance, see the upstream JetBrains checkpoint.