oh yeah! 100% faster for math/code stuff.
.
Macbook M4 Max: high power (10 TPS on low-power, GPU draws only 5 watts...less than your brain)
system prompt: "You are Fuse01. You answer very direct brief and concise"
.
Try this model in Visual Studio Code with the Roo Code extension. Starting in Architect Mode and letting it auto switch to Code Mode.... it actually spits decent code for small projects with multiple files.
Getting close to last year's Claude Sonnet for small projects. It actually stays reasonably stable even with Roo Code's huge 10k system prompt. The model still shits the bed for big projects but better after adding roo-code-memory-bank.
So far (Feb 20, 2025) this is the only model & quant that runs fast on Mac, spits decent code on projects AND works with Speculative Decoding.
The Model
bobig/FuseO1-DeepSeekR1-Qwen2.5-Coder-32B-Preview-Q8 was
converted to MLX format from
FuseAI/FuseO1-DeepSeekR1-Qwen2.5-Coder-32B-Preview
using mlx-lm version
0.21.4. (FYI: the mlx-lm version should be the same in Base model and Draft model)
1from mlx_lm import load, generate
2
3model, tokenizer = load("bobig/FuseO1-DeepSeekR1-Qwen2.5-Coder-32B-Preview-Q8")
4
5prompt = "hello"
6
7if tokenizer.chat_template is not None:
8 messages = [{"role": "user", "content": prompt}]
9 prompt = tokenizer.apply_chat_template(
10 messages, add_generation_prompt=True
11 )
12
13response = generate(model, tokenizer, prompt=prompt, verbose=True)
Maybe check out this new Q4 lossless from NexaAI and tell the MLX community how to improve mlx-lm to get 8-bit quality at 4-bit speed!
