LFM2.5-230M-MLX-attn8-base6
A
custom mixed-precision MLX quant of Liquid AI's
LFM2.5-230M:
6-bit base weights + 8-bit
attention. On an M4 MacBook Air it runs
~17% faster than stock 8-bit at identical
quality — the speed of a 6-bit model with the correctness of 8-bit.
A faster sibling,
attn8_base5 (5-bit base + 8-bit attention,
~35% faster than
8-bit, same quality), lives on the
attn8_base5 branch.
Why mixed precision
Quantizing LFM2.5 uniformly trades quality for speed, and the damage is concentrated in
specific weights. Per-task probing on extraction / classification / factual prompts showed:
- Attention (q/k/v/out, ~5% of params) must stay 8-bit. This is what controls
"extract directly" vs "write a regex script." Drop it to 6-bit and the model starts
dumping Python regex instead of answering an "extract the emails" instruction.
- Embeddings (the tied lm_head / logit matrix, ~29% of params) need ≥5-bit. At 4-bit
they corrupt both extraction and factual recall.
- Everything else (the ~48% MLP, conv blocks, embeddings) tolerates 6-bit fine.
So this model keeps attention at 8-bit and everything else at 6-bit (group size 64,
affine). It's effectively "the known-good 6-bit model, patched at exactly the one
capability 6-bit broke" — the smallest change that recovers full 8-bit quality while
beating 8-bit speed. (config.json records the per-layer bits: 24 layers @ 8-bit, 59 @ 6-bit.)
Benchmarks (M4 MacBook Air 16 GB, mlx-lm 0.31.3, greedy, 256 tok)
Speed = median of 3 timed runs. Quality = a fixed set of 6 checkable
extraction/classification/factual tasks; an arithmetic task all variants fail is excluded
— this isn't a math model.
| variant | gen tok/s | peak GB | quality |
|---|
| LiquidAI MLX-4bit | ~492 | 0.168 | 4/6 — fumbles email extraction & "capital of Japan" |
| LiquidAI MLX-6bit | ~390 | 0.208 | 5/6 — fixes facts, still regex-dumps on extraction |
| this model (attn8-base6) | ~381 | 0.213 | 6/6 — clean, matches 8-bit |
attn8_base5 branch | ~438 | 0.187 | 6/6 — clean |
| LiquidAI MLX-8bit | ~325 | 0.263 | 6/6 — clean |
| LiquidAI MLX-bf16 | ~188 | 0.477 | 6/6 — clean |
Before → after: stock 8-bit @ ~325 tok/s → this model @ ~381 tok/s (+17%) at the
same 6/6 quality. Numbers are machine-specific; reproduce with the recipe below.
Usage
1from mlx_lm import load, generate
2from mlx_lm.sample_utils import make_sampler
3
4# default branch = attn8_base6 (6-bit base + 8-bit attention)
5model, tok = load("lilcheaty/LFM2.5-230M-MLX-attn8-base6")
6
7# faster sibling (5-bit base + 8-bit attention):
8# model, tok = load("lilcheaty/LFM2.5-230M-MLX-attn8-base6", revision="attn8_base5")
9
10prompt = tok.apply_chat_template(
11 [{"role": "user", "content": "Extract all emails from: 'Reach ana@acme.io or sales@acme.io'. Comma-separated."}],
12 add_generation_prompt=True,
13)
14print(generate(model, tok, prompt, max_tokens=128, sampler=make_sampler(temp=0.0)))
CLI:
1python -m mlx_lm generate --model lilcheaty/LFM2.5-230M-MLX-attn8-base6 \
2 --prompt "Summarize the benefits of on-device LLMs." --max-tokens 128
How it was built
1from mlx_lm.convert import convert
2
3def predicate(path, module):
4 # attention at 8-bit, everything else at 6-bit; group size 64
5 bits = 8 if "self_attn" in path else 6
6 return {"group_size": 64, "bits": bits, "mode": "affine"}
7
8convert(
9 hf_path="LiquidAI/LFM2.5-230M-MLX-bf16",
10 mlx_path="attn8_base6",
11 quantize=True, q_group_size=64, q_bits=6,
12 quant_predicate=predicate,
13)
Intended use & limitations
LFM2.5-230M is a small extraction / classification / lightweight-agentic model — great
for data extraction, structured output, and on-device tasks. It is not for heavy
reasoning, math, or long-form creative writing (it cannot reliably do multi-digit
arithmetic, by design). This quant preserves the base model's capabilities and limits.
License & attribution
Derivative of
LiquidAI/LFM2.5-230M
(© Liquid AI), redistributed under the
LFM Open License v1.0 (lfm1.0). All credit for
the model goes to Liquid AI; this repo only re-quantizes the weights for MLX. Please review
the base model's license for your use case.
