Views
No views yet
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, an open 30B mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. Converted for use with llama.cpp using imatrix-calibrated quantization.| Quantization | File | Size | Use case |
|---|---|---|---|
| IQ4_XS | NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf | 18.7 GB | Fits entirely on a 24 GB GPU (e.g. 16 GB + 8 GB dual) with room for a 256K KV cache |
| Property | Value |
|---|---|
| Base model | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 |
| Architecture | NemotronHForCausalLM (nemotron_h_moe) — hybrid Mamba-2 + MoE + Attention with Multi-Token Prediction (MTP) |
| Parameters | ~30B total / ~3B active per token |
| Experts | 128 routed + 1 shared, 6 routed active per token |
| Layers | 52 (interleaved; ~6 attention, rest Mamba-2/MoE) |
| Hidden size | 2,688 |
| Context length | 262,144 (256K native, up to ~1M via rope scaling) |
| Vocab size | 131,072 |
| Reasoning | Yes (thinking mode, <think> blocks) |
| Tool calling | Native (<tool_call> XML format) |
| Modalities | Text |
| Languages | en, es, fr, de, it, ja |
impacte/NVIDIA-Nemotron-3.5-Lightning-GGUF/
├── NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf # IQ4_XS imatrix GGUF (18.7 GB)
└── .gitattributes # LFS tracking1# IQ4_XS (24 GB GPU, full 256K context)
2llama-server \
3 -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
4 --ctx-size 262144 \
5 --port 8080--cache-type-k q4_0 --cache-type-v q4_0) to keep the KV cache footprint minimal:1llama-server \
2 -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
3 --ctx-size 262144 \
4 --cache-type-k q4_0 \
5 --cache-type-v q4_0 \
6 --port 80801curl http://localhost:8080/v1/chat/completions \
2 -H "Content-Type: application/json" \
3 -d '{
4 "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS",
5 "messages": [
6 {"role": "user", "content": "Explain what a Tauri v2 app is."}
7 ]
8 }'1# Pull the pre-built Ollama tag (256K context, q4_0 KV cache)
2ollama run oamazonasgabriel/nemotron-3.5-lightning:iq4-xs-256k-24gbGPU1from llama_cpp import Llama
2
3llm = Llama(
4 model_path="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf",
5 n_ctx=262144,
6 n_gpu_layers=-1, # offload all layers to GPU
7)NVIDIA-Nemotron-3.5-Lightning-30B-A3B is an open 30B-parameter mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. It uses a hybrid Mamba-2 + MoE + attention architecture with only ~6 attention layers out of 52, so the KV cache stays tiny even at very long contexts. The model supports native tool calling and thinking mode, and is multilingual (en, es, fr, de, it, ja).nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — released under the OpenMDW-1.1 license. Review before commercial use.convert_hf_to_gguf.py (bf16) and llama-quantize (IQ4_XS with imatrix calibration).Note: This is a GGUF conversion of a model under the OpenMDW-1.1 license. Ensure your use complies with that license.