This is the v0 baseline in an iterative quantization series. It uses mlx_lm.convert's default Gemma-4 predicate (everything 4-bit, router.proj 8-bit). Subsequent versions (v1+) explore mixed-precision and AWQ/DWQ recipes for better quality.

See the variant index below for newer revisions.

Quantization Recipe

Component	Bits	Group size	Notes
`router.proj` (per-layer)	8	64	mlx-lm built-in Gemma-4 override
Everything else	4	64	including dense MLP, experts, attention
Mode	`affine`	—	mlx-lm default

Conversion command:

bash

mlx_lm.convert \
  --hf-path coder3101/gemma-4-26B-A4B-it-heretic \
  --mlx-path output/gemma-4-26B-A4B-it-heretic-mlx-4bit \
  -q --q-bits 4 --q-group-size 64

Benchmarks (Apple M4 Pro 48GB, mlx-lm 0.31.2)

Quality

Metric	Value	Dataset
Perplexity	156.93 ± 2.77	`allenai/tulu-3-sft-mixture` (256 samples × 512 tok = 131k tokens)
Eval time	226 s	batch size 8
Eval peak memory	21.6 GB	—

Generation Speed

Metric	Value
Prefill	769 tok/s (avg over 5 trials)
Generation	75.1 tok/s (avg over 5 trials)
Inference peak memory	14.7 GB

Test config: prompt_tokens=512, generation_tokens=128, batch_size=1.

Disk Footprint

Format	Size
Original (bf16)	~52 GB
This MLX 4-bit	13 GB

Quality Note ⚠

This v0 baseline shows PPL ~157 vs mlx-community/... PPL ~109 (30% worse). Root cause analysis identified that mlx-community uses an asymmetric MoE recipe: 8-bit on the always-on dense MLP path + 4-bit on sparse experts. Tracking versions on HF lets us reproduce and improve from this baseline iteratively.

Usage

python

from mlx_lm import load, generate

model, tokenizer = load("BRlin/gemma-4-26B-A4B-it-heretic-mlx-4bit")
prompt = "Hello, how are you?"
response = generate(model, tokenizer, prompt=prompt, max_tokens=100, verbose=True)

CLI:

bash

mlx_lm.generate --model BRlin/gemma-4-26B-A4B-it-heretic-mlx-4bit \
  --prompt "Explain quantization in one paragraph." --max-tokens 200

Variant Index

Version	Repo	Recipe	PPL	Gen TPS	Disk	Status
v0 (this)	`BRlin/gemma-4-26B-A4B-it-heretic-mlx-4bit`	Standard 4-bit	156.93	75.1	13 GB	baseline
v1	`gemma-4-26B-A4B-it-heretic-mlx-mixed-4-8`	8-bit dense MLP + router, 4-bit experts	119.87 (−23.6%)	67.6	14 GB	recommended default
v2	`BRlin/gemma-4-26B-A4B-it-heretic-mlx-awq-mixed-4-8`	v1 + AWQ calibration	TBD	TBD	TBD	planned
v3	`BRlin/gemma-4-26B-A4B-it-heretic-mlx-dwq-mixed-4-8`	v1/v2 + DWQ distillation	TBD	TBD	TBD	planned

v1 is recommended for most use cases — large quality gain, modest speed cost.

Hardware & Software

Hardware: Apple M4 Pro, 48 GB unified memory
Software: macOS 15, mlx 0.31.1, mlx-lm 0.31.2, Python 3.12.9

Known Risks

Metal kernel bug (ml-explore/mlx#3393): Gemma-4 26B-A4B (128 experts top-8) produces garbage on base M4 (10 GPU cores). This v0 was converted on M4 Pro (20 cores) and produces coherent output, but untested on lower-end M4.
PLE / ScaledLinear sensitivity: Gemma-4 has Per-Layer Embeddings and ScaledLinear layers that are quantization-sensitive. Future v1+ recipes may need to keep these in higher precision.

License

Inherits from base model: Gemma Terms of Use.

Acknowledgements

coder3101 — original Heretic-aligned weights
mlx-community — reference MLX conversions
ml-explore/mlx, ml-explore/mlx-lm — framework