BaseQuant_XL is a data-agnostic static mixed-precision recipe — no calibration data, no iMatrix, no activation-weighted sampling. It allocates precision by layer importance using a simple heuristic: routing-critical layers stay at full precision, every-token layers get 8-bit (near-lossless), and sparse MoE experts get 6-bit (natural redundancy from 128 experts, only 6 active per token). This contrasts with data-dependent methods (iMatrix, AWQ, GPTQ, oQ, oQ4e) that can skew representation toward well-represented domains.

Quantization Recipe (BaseQuant_XL 6/8)

Precision	Layers	Rationale
bf16	`lm_head`, shared expert	Routing/output-critical — errors here are qualitatively different
8-bit	embeddings, attention (q/k/v/o_proj), Mamba2 (in_proj, out_proj, conv1d)	Every-token layers — 8-bit is near-lossless
6-bit	routed experts (128 experts × 23 MoE layers + MTP MoE)	Bulk parameters — natural redundancy tolerates lower precision

The MoE router gate (gate.weight, e_score_correction_bias) is kept in full precision as bare arrays (not quantizable layers).

6.834 bits per weight, ~25 GB (6 shards)
Group size: 64

Model Architecture

Nemotron-H — hybrid Mamba2 + Transformer + MoE architecture:

52 layers: 23 Mamba2 (SSM), 6 Attention, 23 MoE
1 NextN (MTP) prediction layer: [attention, moe] — Multi-Token Prediction for speculative decoding
30B total parameters, ~3B active per token (6 of 128 experts active)
1 shared expert with shared_expert_overlap: true
mlp_hidden_act: relu2, routed_scaling_factor: 2.5
tie_word_embeddings: false
Context length: 262,144 tokens
ChatML-style template with <think> reasoning and tool-call support

What Makes Lightning Different

Mamba-2 hybrid architecture — interleaves SSM layers with MoE/Attention for higher throughput and lower memory than pure Transformer
Multi-Token Prediction (MTP) — trained with MTP layers that predict multiple future tokens, enabling self-speculative decoding
3B active / 30B total — efficient for single-device deployment

Inference Parameters

temperature: 1.0
top_p: 0.95
top_k: 40
min_p: 0.01
repeat_penalty: 1.05
reasoning_parser: nemotron_v3
tool_call_parser: qwen3_coder

Thinking mode is controlled via the chat template kwarg enable_thinking (default: true).

Usage

python

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain quantum entanglement."}],
    add_generation_prompt=True,
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Also compatible with LM Studio and oMLX — point it at the model directory.