NVIDIA-Nemotron-3.5-Lightning-30B-A3B-mlx-oQ4e
An MLX quantization of
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16,
produced with oMLX's quantizer (hence the
oQ4e suffix).
Quantization
The base setting is 4-bit, group size 64, affine mode. This is a mixed-precision
quant, not a uniform one: 117 modules carry per-module overrides, all at group size 64
and affine mode.
| Precision | Modules |
|---|
| 4-bit (base) | all 128 routed experts across the 23 MoE layers |
| 5-bit | 28 modules — mamba in_proj/out_proj, some attention projections |
| 6-bit | 32 modules — mamba in_proj/out_proj, attention q/k/v/o_proj |
| 8-bit | 57 modules — all shared_experts.{up,down}_proj, some attention projections, backbone.embeddings, lm_head |
Quantization was imatrix-guided. The calibration pass used oMLX's
oqe_code_multilingual set (128 samples at sequence length 512) and reached full
expert coverage — all 5888 routed experts were activated during calibration, with no
zero-count experts.
Resulting size on disk: ~17 GiB.
A uniform 8-bit conversion of the same upstream model, from the same calibration run,
is available at
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-mlx-oQ8e.
Architecture notes
The upstream model is a hybrid: Mamba-2 state-space layers, MoE layers, and attention
layers interleaved — 52 layers total (23 mamba, 23 MoE, 6 attention), with 128 routed
experts, 6 active per token, plus 1 shared expert. model_type is nemotron_h;
architectures is ["NemotronHForCausalLM"].
Upstream describes MTP (multi-token prediction) layers, and its deployment guidance
targets vLLM on CUDA. This conversion contains no MTP weights — config.json
reports num_nextn_predict_layers: 0 and the safetensors index holds no MTP tensors.
MTP and speculative decoding have not been exercised here at all.
Testing
This quant has been tested in oMLX only. No claims are made about behavior under
any other runtime, and no benchmarks or quality evaluations have been run against it.
Attribution
All model weights and design credit belong to NVIDIA. This repository redistributes a
quantized conversion of their release. See the upstream model card for the model's
capabilities, intended use, training details, and license terms:
https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
