Variant	Folder	Quantization	Size	Best fit
MLX 8-bit	`mlx-8bit/`	affine, group size 64	36.85 GB	Highest-quality local MLX run.
MLX 6-bit	`mlx-6bit/`	affine, group size 64	28.19 GB	Balanced memory and quality.
MLX 4-bit	`mlx-4bit/`	affine, group size 64	19.53 GB	Smallest footprint and easiest local run.

Local benchmark notes

Initial local testing was performed on a Mac Studio with an M4 Max and 64 GB unified memory using oMLX.

4-bit

Screenshot: benchmarks/grug-35b-a3b-4bit-omlx.png

Test	TTFT	Prompt TPS	Generation TPS	End-to-end	Throughput	Peak memory
pp1024 / tg128	566.6 ms	1807.2 tok/s	103.9 tok/s	1.805 s	638.2 tok/s	19.35 GB
pp4096 / tg128	2258.3 ms	1813.7 tok/s	100.2 tok/s	3.545 s	1191.5 tok/s	20.08 GB
pp8192 / tg128	4690.0 ms	1746.7 tok/s	97.1 tok/s	6.019 s	1382.2 tok/s	20.50 GB
pp16384 / tg128	12050.5 ms	1359.6 tok/s	90.3 tok/s	13.478 s	1225.1 tok/s	20.38 GB

Continuous batching, pp1024 / tg128:

Batch	Generation TPS	Speedup	Prompt TPS	Prompt TPS/request	TTFT	End-to-end
1x	103.9 tok/s	1.00x	1807.2 tok/s	1807.2 tok/s	566.6 ms	1.805 s
2x	116.1 tok/s	1.12x	1468.4 tok/s	734.2 tok/s	1394.6 ms	3.600 s
4x	180.0 tok/s	1.73x	1549.7 tok/s	387.4 tok/s	2530.3 ms	5.488 s

6-bit

Screenshot: benchmarks/grug-35b-a3b-6bit-omlx.png

Test	TTFT	Prompt TPS	Generation TPS	End-to-end	Throughput	Peak memory
pp1024 / tg128	645.4 ms	1586.6 tok/s	87.1 tok/s	2.119 s	543.5 tok/s	45.58 GB
pp4096 / tg128	3175.8 ms	1289.8 tok/s	82.9 tok/s	4.732 s	892.6 tok/s	45.16 GB
pp8192 / tg128	6701.5 ms	1222.4 tok/s	80.9 tok/s	8.294 s	1003.1 tok/s	45.30 GB
pp16384 / tg128	22365.0 ms	732.6 tok/s	76.7 tok/s	24.045 s	686.7 tok/s	45.68 GB

The 6-bit batching run shown in benchmarks/grug-35b-a3b-6bit-prefill-cap.png hit oMLX's prefill safety cap at the tested settings: predicted peak exceeded 45.5 GB, which was 90% of the effective 50.5 GB ceiling.

8-bit

The 8-bit variant could not be thoroughly tested under the default local memory cap. In the initial oMLX test environment, oMLX projected 54.62 GB total memory use against a 51.84 GB effective ceiling. The model files themselves are 36.85 GB; the higher runtime estimate includes the current oMLX process footprint, MLX runtime/allocator overhead, buffers, and KV/cache planning.

The 8-bit variant should be retested after raising the Apple GPU wired-memory cap and restarting the local serving process, for example:

sudo sysctl iogpu.wired_limit_mb=59392

Throughput numbers are not published yet. This section will be updated after a successful full benchmark run.

Usage

Download only the variant you want:

python

from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-35B-A3B-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)

Run with mlx-lm:

bash

python -m mlx_lm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Reply with a short explanation of what this model is." \
  --max-tokens 256

Local compatibility note

The source config uses model_type: qwen3_5_moe_text. At conversion time, mlx-lm 0.31.3 did not include a native loader for that exact model type, so the conversion used a local compatibility shim mapping the model to the existing Qwen3-Next style MLX implementation and adapting the published weight names.

Consumers may need an mlx-lm build that supports qwen3_5_moe_text, or an equivalent local compatibility shim, until upstream support is available.

Provenance and attribution

Source model: kai-os/Grug-35B-A3B
Relationship: MLX quantized derivatives of the source model
Source revision used locally: f6349f9c7beba10ad44ce0210b0a0f6fba414a05
Conversion tool: mlx-lm 0.31.3
Quantization mode: affine, group size 64

Please refer to the source model card for upstream training details, intended use, limitations, acknowledgements, and license context.

Limitations

Quantization can change output quality, numerical behavior, and edge-case performance. These files are intended for local MLX inference on Apple Silicon. Use the source model repo for the original Transformers/safetensors weights.