MLX MX FP8 (micro-scaled 8-bit float) quantization of Nanbeige/Nanbeige4.2-3B, converted with mlx-lm.

Nanbeige4.2-3B is a compact agentic model built on Nanbeige4.2-3B-Base. Its Looped Transformer architecture reuses the same transformer layers across multiple passes to increase effective capacity without adding parameters. With only 3B non-embedding parameters, it outperforms larger models (Qwen3.5-9B, Gemma4-12B) on general-agent, code-agent, and reasoning benchmarks (per the base model's card: SWE-Bench Verified 63.6, GPQA-Diamond 87.4, HMMT-Feb-2026 82.8).

Nanbeige4.2-3B uses a Looped Transformer architecture (model_type: "nanbeige"): the same 22 decoder layers run twice (num_loops: 2), with a full RMSNorm applied between and after each pass — a way to add effective depth without adding parameters (3B non-embedding params, ~4B total). This architecture had no mlx-lm support at all prior to this release. A minimal, from-scratch MLX port (mlx_lm/models/nanbeige.py) was written for this conversion, implementing the loop mechanism with per-loop KV cache isolation (each loop's attention only attends to its own loop's keys/values) and covering the subset of the (very large) upstream NanbeigeConfig that this checkpoint actually uses — the full reference implementation also defines n-gram embeddings, hyper-connections, and depth-attention, none of which are active here (num_loops=2 is the only non-default architectural flag Nanbeige4.2-3B sets).

This is not yet in an official mlx-lm release. A PR adding this support is open against ml-explore/mlx-lm. Until it merges, install mlx-lm from the PR branch to load this model:

The chat template supports enable_thinking (default on — set False for non-thinking mode) and preserve_thinking (whether prior turns' reasoning is kept in multi-turn contexts; the base model card recommends False for general chat/QA and True for multi-turn tool-use/agentic workflows). Per the base model's recommendations: temperature 1.0 for agentic/tool-use tasks, 0.6 for reasoning/chat tasks; eos_token_id is 166101.

CLI:

Verified via mlx_lm.generate through the model's chat template, on an Apple M4 Pro:

Prompt: "Explain the difference between TCP and UDP in two sentences."
Output (mxfp4): "...We need to provide an explanation that is concise, exactly two
sentences... TCP (Transmission Control Protocol) is a connection-oriented, reliable,
error-correcting protocol... UDP (User Datagram Protocol) is a connectionless,
unreliable, but lightweight protocol..."

Prompt: "Write a haiku about the ocean at night."
Output (mxfp8): "...haiku structure: three lines with syllable counts: 5, 7, 5...
Elements to include: night setting, ocean imagery (waves, stars reflected,
moonlight, darkness)..."

All four quantized variants preserve the base model's thinking-native behavior (private reasoning before the final answer) intact through quantization.

Measured with mlx_lm.generate on 5 GPQA-Diamond prompts (the same benchmark the base model reports its 87.4 score on), Apple M4 Pro, max_tokens=400:

4-bit and 8-bit each cluster tightly regardless of MX-float vs affine-int encoding — pick based on quality tolerance (smaller/faster 4-bit vs higher-fidelity 8-bit) rather than the encoding scheme.

A note on accuracy: we attempted a full 25-sample GPQA-Diamond accuracy run across all variants, but this is a thinking-native model that reasons at length on hard graduate-level questions — most responses didn't reach a final answer within a 2048-token budget (across all variants), making the resulting accuracy numbers a measurement of response-length variance rather than model quality. We don't have a reliable accuracy comparison to publish here as a result; treat the decode-speed numbers above as the trustworthy signal from this exercise.

A note on correctness validation: this quantization required first porting Nanbeige's Looped Transformer architecture to mlx-lm (see "New architecture support" below). That port was validated against the official modeling_nanbeige.py reference implementation (run via transformers, float32) on held-out prompts: computed logits from both implementations for the same inputs and compared next-token predictions and full logit distributions. After aligning RoPE frequency initialization between the two implementations, results matched closely — logit cosine similarity ≥0.9999 and identical next-token predictions on every test prompt.

optiq-5bpw was not produced for this release — it requires the mlx-optiq package, which wasn't available in the conversion environment.