[!WARNING]
Research preview — not extensively tested. This model has not been extensively
evaluated for coherence or alignment/safety. It can and will produce text that is
incoherent, factually wrong, nonsensical, or otherwise unsafe. It is not suitable for
any production or user-facing use. It is shared for research and curiosity only. Do not rely
on its outputs. Unofficial and not affiliated with or endorsed by Thinking Machines.
A heavily-pruned, LoRA-healed, 4-bit MLX derivative of
thinkingmachines/Inkling-Small,
built to run on a 16 GB Apple-silicon Mac.
What it is
Inkling-Small is a large multimodal Mixture-of-Experts model (256 routed experts per MoE layer).
This preview keeps only 8 of those 256 experts (a ~97% cut of the routed parameters), quantizes
the result to 4-bit in MLX, and then LoRA-heals the wreckage on a small general-English
corpus so it produces coherent text again. It is text-only (the vision/audio towers are dropped).
The result is a small (~8 GB) model that generates fluent, structured English — but with tiny
capacity and no reliable world knowledge. It will confidently state false things (e.g. that the
capital of France is "1.5 million") and cannot do arithmetic. Think of it as a demonstration that
an aggressively-pruned MoE can be revived into coherence on consumer hardware — not as a useful
assistant.
How it was made
Prune — stream only 8 experts/layer from the original checkpoint (top-6-of-8 routing kept).
Quantize — 4-bit affine (group size 64) in MLX.
Heal — LoRA (rank 16) on attention + expert projections, trained on ~6M tokens of
FineWeb-Edu. Best checkpoint
at cross-entropy ≈ 2.23 (perplexity ≈ 9). The adapter is shipped separately from the base
because fusing it into the 4-bit weights measurably degrades quality.
Usage
Apple-silicon Mac required (MLX).
bash
1pip install mlx mlx-lm transformers numpy
2python run.py --prompt "The three primary colors are"
Greedy decoding loops badly on a model this small; run.py applies a repetition penalty (default
1.3), which you should keep. Layout: 4-bit base weights + adapter/adapters.safetensors (LoRA) +
bundled inkling_mlx/ runtime + run.py.
Example output (--prompt "The three primary colors are"):
The three primary colors are the red, white and black. The other two include yellow.
These four shades of color have been used in a variety of applications:
- Red: Used to indicate danger or urgency
- White: ...
Fluent and structured — and also wrong (the primary colors are not "red, white and black"). That is
representative: coherent in form, unreliable in fact.
Limitations
Not safety- or alignment-tested. No red-teaming, no refusal training, no evaluation. May emit
false, biased, offensive, or otherwise harmful content.
Not factual. ~97% of the routed experts were deleted; world knowledge is largely gone.
Narrow. Healed on a small general-web sample; tends toward generic prose.
Bundled inkling_mlx/ runtime: from
mlx-community/Inkling-Small-mlx
(Apache-2.0), with a one-line modification in moe.py (stop_gradient on expert indices) so the
model can be LoRA-trained; identity at inference.
Released under Apache-2.0, inheriting the base model's license. This is a modified derivative
work; it is unofficial and not affiliated with, sponsored by, or endorsed by Thinking Machines.
