Calibrated 6-bit MLX quantization of poolside/Laguna-S-2.1
(118B total, 8B activated per token), produced with oMLX oQ at
level 6 enhanced — 6.27 bits/weight effective, 92 GB on disk. Data-driven mixed precision:
bits are allocated per tensor from an imatrix-calibrated sensitivity map, not a fixed rule.
For Apple Silicon.
92 GB on disk, down from 235 GB BF16
48 layers, 47 of them MoE with 256 routed experts + 1 shared, top-10 (L0 is a dense MLP);
interleaved attention (12 global with YaRN to 1M context, 36 sliding-window 512)
Peak memory in my tests: 86.8 GB at 1k context, 90.0 GB at 64k — you'll want a 128 GB Mac
Converted and tested on a Macbook Pro M5 Max 128GB 40 GPU
Requirements
mlx-lm doesn't support the laguna architecture yet — there's an open PR:
mlx-lm#1223. Until it lands, use mlx-vlm
(0.6.3+), which implements laguna as a text-only model:
oMLX serves it directly from 0.5.3 on — it vendors that PR and patches it into mlx-lm at
import, so no model setting is needed. On earlier builds, discovery decides between the mlx-lm and
mlx-vlm loaders by looking for a vision sub-config, and laguna has none — so it lands on mlx-lm and
fails with Model type laguna not supported. Set model_type_override: "vlm" in the model's
settings, then refresh discovery (omlx restart): the load failure is cached per entry until the
next discovery pass, so setting the override alone won't clear it.
Quantization
oQ6e allocates bits per tensor from an importance-matrix calibration pass over calibration data.
The 6-bit base lands on the experts; 248 of the 386 dense-spine tensors — attention, embeddings,
lm_head, routers — were boosted to 8 bits. Output is standard MLX affine quantization — no
custom kernels or runtime required.
Like the oQ5e (and unlike the three smaller variants), this build was calibrated with omlx's
newer adaptive imatrix collection: up to 1024 samples, extended until every routed expert is
covered, instead of a fixed 128 samples.
How it was quantized
oQ at level 6 enhanced — imatrix-calibrated, group size 128. I had to patch omlx to route laguna
through the mlx-vlm loader. At 235 GB the model doesn't fit in 128 GB of RAM, so calibration ran
against a uniform 4-bit proxy on disk rather than the FP weights, which shifts the bit allocation
slightly.
Conversion check
Smoke-tested after conversion with mlx_vlm.generate: coherent — solved 17 * 24 = 408, no
repetition loop. 42.4 tok/s generation on a short prompt, peak 92.6 GB.
Performance
Measured with oMLX's benchmark harness on a Macbook Pro M5 Max 128GB 40 GPU, single request,
128 generated tokens:
prompt
gen tok/s
prefill tok/s
TTFT ms
peak GB
1k
53.0
940.3
1090
86.83
4k
53.1
996.0
4114
86.98
8k
50.1
876.9
9343
87.17
16k
47.4
799.2
20502
87.54
32k
43.1
731.3
44807
88.33
64k
32.9
575.5
113871
89.95
Continuous batching at 1k prompt / 128 generated:
batch
tg tok/s
speedup
TTFT ms
E2E s
1
53.0
1.00x
1090
3.51
2
71.0
1.34x
2210
5.81
4
90.1
1.70x
4275
10.06
8
114.0
2.15x
5667
16.20
Benchmarks & Variants
mmlu_pro, mathqa and winogrande, n=300 seeded samples each, thinking off, identical questions across
every variant. The bf16 row is the hosted API, measured the same way. Standard error at this n is
around 2.5 points, so oQ4e through oQ6e aren't separated by this run.
Accuracy vs bits per weight, three benchmarks, n=300
Treat this as a rough sighting, not a verdict. Three benchmarks at n=300 cover a narrow slice of what
the model does — no long-context work, no agentic loops, no real code — and at this sample size most
of the ladder above 3.6 bpw sits inside the error bars. I ran them to size the drop between levels,
not to rank the variants against each other. Test the one you're considering on your own workload
before trusting any of it.
Usage
bash
1# mlx-vlm — plain mlx-lm doesn't support the laguna architecture2uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-S-2.1-oQ6e \3 --prompt "Explain Bayes' theorem in two sentences." --max-tokens 30045# oMLX — discovers the model from the HF cache; set model_type_override: "vlm" first6omlx serve
License
OpenMDW-1.1, inherited from
the base model. Refer to the original model card for architecture, benchmarks, and intended use.
