You can deploy this model and start earning money today!

darthcrawl/
EVA-Qwen2.5-32B-v0.2-mlx-6bit

Name: EVA-Qwen2.5-32B-v0.2-mlx-6bit
Availability: InStock

not-for-all-audiences

text-generation

conversational

EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2

quantized

apache-2.0

6-bit

Views

No views yet

EVA-Qwen2.5-32B-v0.2-mlx-6bit

MLX 6-bit quant of EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2. Disk: ~25 GB. Active memory: ~27 GB plus KV cache.

Quantization

bash

mlx_lm.convert --hf-path EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2 \
  --mlx-path ./EVA-Qwen2.5-32B-v0.2-mlx-6bit -q --q-bits 6

REQUIRED: EOS patch before serving

Same as the 4-bit. Source ships incomplete eos_token_id. Patch needed for ChatML turn boundaries:

bash

python3 -c "
import json
p = './EVA-Qwen2.5-32B-v0.2-mlx-6bit/generation_config.json'
g = json.load(open(p))
g['eos_token_id'] = [151643, 151645]
json.dump(g, open(p, 'w'), indent=2)
"

This quant has it applied.

Recommended sampling

Same as 4-bit:

yaml

temperature: 1.0
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Known mlx_lm.server bugs

Same as the 4-bit (XTC tokenizer bug, seed ignored, prompt-cache size cap recommended).

License

Apache-2.0 from base.

darthcrawl/EVA-Qwen2.5-32B-v0.2-mlx-6bit