Views
No views yet
Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs
nemotron_h hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from.| Property | Value |
|---|---|
| Base | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (~30B total, ~3B active) |
| Method | OptiQ mixed-precision, sensitivity-driven (uniform-4-bit reference) |
| Precision | per-layer 4/8-bit: 36 layers at 4-bit, 127 at 8-bit |
| Experts | 46 fused expert tensors: 36 at 4-bit, 10 at 8-bit |
| On disk | 22.8 GB |
| Speculative decoding | MTP head preserved in optiq/mtp.safetensors |
llama.cpp's naming for mixed quants, the "4bit" label denotes the family, not the weighted average.| Metric | Score |
|---|---|
| MMLU (5-shot, 969 samples) | 78.9% |
| GSM8K (1000 samples) | 85.5% |
| IFEval (full set, strict) | 68.4% |
| BFCL-V3 simple (200 calls) | 83.0% |
| HumanEval (164 problems, pass@1) | 87.8% |
| HashHop (long-context retrieval) | 40.0% |
| Capability Score (mean of 6) | 73.94 |
pip install "mlx-optiq>=0.4.20"1import optiq # registers the nemotron_h MoE + MTP sidecar
2from mlx_lm import load, generate
3
4model, tok = load("mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit")
5msgs = [{"role": "user", "content": "Explain why the sky is blue."}]
6prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
7print(generate(model, tok, prompt=prompt, max_tokens=600))<think> block before its answer, so give it room and read the text after </think>.optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-expertsOptiqEngine(path, stream_experts="auto"), and from optiq eval --stream-experts for benchmarking.