Inkling is a 975B-total / 41B-active sparse-MoE, natively multimodal model
(text + image/video + audio → text). This is the full multimodal conversion:
all three towers (text backbone, HMLP vision, dMel audio) are ported; the
multi-token-prediction head is dropped (inference-irrelevant).
MLX supports FP4 modes and Thinking Machines ships an
Inkling-NVFP4 checkpoint — so for
the record, we benchmarked round-trip reconstruction error (‖W − Ŵ‖ / ‖W‖ vs bf16) on real
Inkling expert weights:
Scheme
bits/weight
reconstruction error
affine int4 (group 64)
4.50
~9.1%
nvfp4 (group 16)
4.50
~10.2%
mxfp4 (group 32)
4.25
~12.3%
Affine int4 is the most faithful: it is asymmetric (per-group scale and zero-point, 16
uniform levels), which centers on Inkling's near-Gaussian expert weights better than
symmetric FP4's fixed non-uniform levels. FP4's real payoff is heavy-tailed activations and
native Blackwell FP4 tensor cores — neither helps weight fidelity on Apple Silicon, where MLX
would dequantize FP4 anyway. So these builds use affine int4.
⚠️ Loading requires the bundled inkling_mlx loader
The inkling_mm_model architecture is not in stock mlx-lm / mlx-vlm, so this
repo bundles a minimal, numerically-validated MLX implementation under inkling_mlx/.
pip install mlx mlx-lm transformers
python
1from inkling_mlx.load import load
2from inkling_mlx.generate import greedy_generate
3from transformers import AutoTokenizer
45model, config = load("/path/to/this/repo")6tok = AutoTokenizer.from_pretrained("/path/to/this/repo", trust_remote_code=True)7ids = tok("The capital of France is")["input_ids"]8print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
Needs an Apple-Silicon Mac with enough unified memory to hold the weights (≈ the
size above).
Status & caveats
Text generation works end-to-end via an incremental KV + short-convolution cache.
Multimodal is supported end-to-end: the vision/audio towers and their
preprocessing (InklingProcessor — image patchify/normalize, audio log-mel→dMel,
validated ~1e-7 vs the reference) are included. Pass images/audio via the processor.
Quantized: attention / MLP / expert projections, token embed+unembed, and the
vision/audio matmuls. Kept in higher precision: the MoE router, RMSNorms, the four
short-convolutions per layer, and the relative-position bias.
Conversion is streaming (tensor-by-tensor; the ~1.9 TB bf16 model never fully loads
into RAM) and was validated with fp32 numerical parity against transformers PR #47347.
License: Apache-2.0 (inherits the base model).
