4-bit (group size 64) MLX conversion of garagelm/hybrid-gpt-232m for Apple Silicon. A 232M-parameter research model with hybrid local+global attention (sliding window w=64 on 12 of 16 layers, full causal attention on layers 3/7/11/15), pretrained on 1.0B tokens entirely on one Apple M4 Pro.

MLX is the deployment-relevant runtime for this project: it is what the efficiency claims were measured under, and it outperforms PyTorch+MPS for on-device inference.

What 4-bit costs. Measured on identical held-out windows (16 x 1024 tokens of the pretraining validation split), not estimated:

	val loss	on disk
float16	2.9689	464 MB
4-bit	2.9869	130 MB
delta	+0.0180 nats	3.55x smaller

Measured performance

M4 Pro (48GB), 4-bit (group size 64), 128 decode tokens, peak memory 1.10 GB. KV cache is read from the live cache arrays, not inferred.

prompt	TTFT	prefill tok/s	decode tok/s	KV cache
64	7 ms	9,158	549	1.56 MB
256	21 ms	12,139	544	2.35 MB
512	41 ms	12,540	539	3.40 MB
896	73 ms	12,231	528	4.97 MB

The bounded KV cache is the point of the architecture: 12 of 16 layers keep only their last 64 keys, so cache growth is dominated by the 4 global layers. At full 1024 context that is ~30% of an all-global equivalent.

Usage

model_type: hybrid_gpt is not in mlx-lm's registry yet, so pass the bundled module explicitly. The module in this repo is written against mlx-lm's model interface, so generate, streaming, prompt caching and the server all work.

python

import sys
from pathlib import Path
from huggingface_hub import snapshot_download
from mlx_lm.utils import load_model, load_tokenizer
from mlx_lm import generate

path = Path(snapshot_download("garagelm/hybrid-gpt-232m-mlx-4bit"))
sys.path.insert(0, str(path))
import hybrid_gpt

model, _ = load_model(path, get_model_classes=lambda config: (hybrid_gpt.Model, hybrid_gpt.ModelArgs))
tok = load_tokenizer(path)
print(generate(model, tok, prompt='The water cycle begins when', max_tokens=100))

Correctness

Two gates run before this repo is written, both in the research repo's release/mlx/export.py:

Module parity — the hybrid_gpt.py shipped here matches the reference MLX port on a 128-token prefill plus 8 cached decode steps (max|logit diff| 1.5e-05, top-1 agreement 1.000). The gate runs on the fp32 weights, so it is a property of the module rather than of this precision. That port is itself logit-parity-gated against the PyTorch implementation, so the module here is transitively pinned to PyTorch.
Precision parity — the exported weights are compared against fp32 on identical inputs; the numbers are recorded in export_stats.json.

Limitations

A 232M-parameter research artifact, not an assistant. Fluent text, unreliable facts, 1024-token context, English only. No instruction tuning or alignment of any kind — see the chat variant for that.

Evaluations, training recipe and the full architecture writeup are on the upstream card: garagelm/hybrid-gpt-232m.