8-bit MLX quantization of MERaLiON/MERaLiON-3-10B, the instruction-tuned Singaporean multilingual speech-language model from A*STAR / I²R — in a loadable single-tree layout with a working MLX inference harness.

2026-07-07 republish: this repo previously shipped a split component layout (encoder.safetensors / adaptor.safetensors / decoder-*.safetensors) that no MLX runner could load directly. It now ships the standard model.safetensors.index.json tree produced by pipelines/mlx_direct_quantize, verified end to end with audio transcription through the MLX harness below.

At a glance


Source	`MERaLiON/MERaLiON-3-10B` @ `3d5c2f772641b1cfeba35743df3db32a07db8c48` (bf16, ~19 GB)
Format	MLX (Apple Silicon native), single safetensors tree
Quantization	8-bit affine, group-size 64 (decoder 2-D tensors only)
Disk size	10.5 GB (11,221,638,720 bytes)
Decoder	Gemma-2-9B-it, 295 tensors quantized (169 decoder 1-D preserved bf16)
Encoder	Whisper-large-v3 derived, 487 tensors preserved bf16
Adaptor + bridges	speech MLP adaptor, `ln_speech`, `speech_encoder_layer_weighted_sum` — 7 tensors preserved bf16

Unlike the 3B-ASR sibling, MERaLiON-3-10B mixes its speech encoder's per-layer outputs with learned softmax weights (speech_config.use_weighted_layer_sum=true); the 1-D weight vector rides through unquantized with the rest of the speech tower.

Verification (2026-07-07, M-series Mac, 128 GB)

Composite-load self-test: PASS (decoder quantized, speech tower bf16, finite logits, decode bitwise-matches the mlx_lm gemma2 forward).
Audio transcription on the upstream MERaLiON sample clips, greedy HF-parity sampler:
- code-switched Singlish: near-verbatim (there's like two quarters ma then 他会教).
- code-switched Tamil: fluent Tamil with minor word errors, comparable to the 3B-ASR-8bit reference behaviour.
- The Cantonese/Hokkien samples are known-degenerate for the whole MERaLiON-3 family (upstream torch references also collapse on them) and are informational only.

Usage

Runs on the MERaLiON-3 MLX harness in majek/pipelines/meralion3_mlx (Whisper encoder + weighted layer sum + adaptor in MLX, Gemma-2 decoder via mlx_lm.models.gemma2):

bash

# one-shot transcription
python -m pipelines.meralion3_mlx.generate \
    --path majentik/MERaLiON-3-10B-MLX-8bit \
    --wav your_clip.wav

# local HTTP endpoint (POST wav bytes or multipart "file")
python -m pipelines.meralion3_mlx_serve \
    --model majentik/MERaLiON-3-10B-MLX-8bit --port 8321

Stock mlx-lm alone cannot load this model (composite audio architecture, model_type=meralion3); the harness routes the speech tower and decoder from the same checkpoint tree. The prompt template is the upstream one: Instruction: <TextHere> \nFollow the text instruction based on the following audio: <SpeechHere> inside Gemma-2 chat turns.

Quantization recipe

Only the Gemma-2-9B decoder's 2-D tensors are quantized (where >95 % of parameters live). The Whisper encoder, speech adaptor, layer-sum weights, norms, embeddings and all 1-D tensors are preserved bf16 — they are small and disproportionately sensitive to quantization noise.

Reproduce:

bash

python -m pipelines.mlx_direct_quantize --model meralion3-10b \
    --base-dir /tmp/mlx-direct-release/meralion3-10b/base \
    --out-dir /tmp/mlx-direct-release/meralion3-10b/8bit \
    --bits 8 --mode affine --group-size 64

Intended use

Speech understanding and transcription for Singapore-context English, Mandarin, Malay, Tamil (plus Indonesian, Thai, Vietnamese), including code-switched speech and Singlish — locally on Apple Silicon.

Licence

Derivative of MERaLiON-3-10B under the MERaLiON-3-Public-Licence, which permits derivatives with notices. This repository redistributes quantized weights with the required attribution; upstream model, training data and evaluation details are documented on the base model card.