You can deploy this model and start earning money today!

darthcrawl/
anubis-70b-v1.2-mlx-4bit

Name: anubis-70b-v1.2-mlx-4bit
Availability: InStock

Views

No views yet

Anubis-70B-v1.2-mlx-4bit

MLX 4-bit quant of TheDrummer/Anubis-70B-v1.2. Llama-3.1-70B base, RP finetune. Disk: ~37 GB. Active memory while serving: ~40 GB plus KV cache.

bash

mlx_lm.convert --hf-path TheDrummer/Anubis-70B-v1.2 \
  --mlx-path ./anubis-70b-v1.2-mlx-4bit -q --q-bits 4

mlx-lm 0.31.x. No further patches needed for this quant.

yaml

temperature: 0.95
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Top-p 1.0, top-k 0, presence/frequency penalties 0. Drop temp to 0.85 if outputs go florid; raise to 1.05 if stiff.

Pairs cleanly with mlx-community/Llama-3.2-3B-Instruct-uncensored-6bit as a draft model. --num-draft-tokens 7 is a good default; 8 if your prompts are predictable, 5-6 for highly varied input.

seed parameter silently ignored in mlx-lm 0.31.3 chat-completions. Different seed values produce byte-identical output. Workaround if you care about reproducibility: vary the prompt itself per request.
Prompt-cache KV grows unbounded at large --prompt-cache-size. On a 70B 4-bit + Mac unified memory, set --prompt-cache-size 8 or lower to avoid Metal GPU timeout aborts. The --prompt-cache-bytes flag does not enforce its limit reliably.

Inherits llama3.1 from base. Not-for-all-audiences per source finetune.