Kimi K2.5 optimized to run comfortably on a Mac Studio M3 512G.
Other MLX options require 450G+, which
is tight even with 500G of usable memory. This quant fits into ~380G with room to spare, giving you the flexibility
to use longer contexts, run other models in parallel, and open up 3 browser tabs without OOM'ing.
If you're looking to use Kimi K2.5 as the core of a "Claude Code in a box" setup, you've come to the right place.
Update: Uploaded a v2 that improves perplexity while keeping the same size.
Update: Created an even smaller 2.5 bit version that
uses less memory while maintaining the same perplexity as v1!
Usage
sh
1# Start server at http://localhost:8080/v1/chat/completions
2uvx --from mlx-lm --with tiktoken \
3 mlx_lm.server \
4 --host 127.0.0.1 --port 8080 \
5 --trust-remote-code \
6 --model spicyneuron/Kimi-K2.5-MLX-2.8bit
78# Kimi K2.5 requires tiktoken + remote code for the tokenizer
Methodology
Quantized with a mlx-lm fork, drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs.
MLX quantization options differ than llama.cpp, but the principles are the same:
Sensitive layers like MoE routing, attention, and output embeddings get higher precision (BF16, 8, 4)
More tolerant layers like MoE experts get lower precision (2, 3)
This one is comparable to Unsloth's UD-Q2_K_XL
in size, but loads and runs noticeably faster thanks to MLX. Compared to the 3 bit MLX, it's faster, uses 80G less
memory, yet has lower perplexity.
