Sarvam-105B MLX Q6
This is an MLX 6-bit affine quantization of sarvamai/sarvam-105b, converted
from the public BF16 checkpoint abhinand/sarvam-105b-bf16.
The model is a base/pretrain text-generation model using the sarvam_mla
architecture: DeepSeek-V3-style MLA attention plus sigmoid MoE routing. It is
not an instruction-tuned chat model.
Why Q6
We tested both q4 and q6 locally on Apple Silicon. The q4 conversion loads and
runs, but it is not usable: it collapses to whitespace with probability 1.0
across sanity prompts. The q6 conversion is coherent and passed real-weight
validation, so q6 is the published practical checkpoint.
Local validation summary:
| Quant | Size | Result |
|---|
| q4, group size 64 | 56 GB | Broken: collapsed whitespace forward |
| q6, group size 64 | 80 GB | Coherent top-token and generation checks |
Example q6 top-token probes:
| Prompt | Top prediction |
|---|
The capital of France is | Paris |
def add(a, b):\n return | a |
Validation
Measured locally on an Apple Silicon M5 Max host with 128 GB unified memory.
- Real BF16 source checkpoint loaded through the MLX
sarvam_mla port during
conversion.
- q6 generation smoke passed.
- Warm short-context decode measured about 46 tok/s, with about 86 GB peak
memory.
- Long-context absorbed-MLA checks ran at 2k, 8k, and 16k context.
- Prefix-cache and prompt-lookup decoding checks passed on suitable workloads.
The q4 collapse was investigated against RoPE, config mapping, and the
kv_b_proj sanitize path; those matched the reference implementation. The
failure is attributed to naive 4-bit quantization of this large
DeepSeek-style MLA-MoE model's activation outliers.
Loading
This artifact includes the custom configuration/modeling files referenced by
auto_map. Use MLX tooling with remote/local custom code support as needed.
1from mlx_lm import load, generate
2
3model, tokenizer = load("pierrelamy/sarvam-105b-mlx-q6")
4response = generate(model, tokenizer, prompt="The capital of France is", max_tokens=32)
5print(response)
Provenance
- Base model:
sarvamai/sarvam-105b
- BF16 conversion source:
abhinand/sarvam-105b-bf16
- Format: MLX safetensors
- Quantization: affine 6-bit, group size 64
- Tensor payload reported by
model.safetensors.index.json: 86,202,965,504 bytes
- Parameters reported by
model.safetensors.index.json: 106,031,767,424
Caveats
- This is a base model, not an instruct model.
- q4 is intentionally not published because it collapsed in local validation.
- The checkpoint is large: expect roughly 80 GB on disk and about 86 GB peak
memory in the measured short-context generation setup.
