8-bit is statistically indistinguishable from BF16 in most evaluations. Use this if you want maximum quality and have the memory; otherwise 4-bit is fine for most use cases.

Quantization

bash

mlx_lm.convert --hf-path TheDrummer/Anubis-70B-v1.2 \
  --mlx-path ./anubis-70b-v1.2-mlx-8bit -q --q-bits 8

Recommended sampling (RP)

yaml

temperature: 0.95
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Speculative decoding

Same draft as 4/6-bit (mlx-community/Llama-3.2-3B-Instruct-uncensored-6bit). --num-draft-tokens 7.

Known mlx_lm.server bugs

seed parameter silently ignored in 0.31.3.
Prompt-cache KV grows unbounded; set --prompt-cache-size 4 or lower at this size to avoid Metal GPU timeouts.

License

Inherits llama3.1 from base.

darthcrawl/anubis-70b-v1.2-mlx-8bit

Anubis-70B-v1.2-mlx-8bit

Quantization

Recommended sampling (RP)

Speculative decoding

Known mlx_lm.server bugs

License

darthcrawl/
anubis-70b-v1.2-mlx-8bit