GGUF builds of EryriLabs/Lightning-Herald-30B-A3B, my Hermes-Agent tune of NVIDIA Nemotron 3.5 Lightning (30B total, 3B active). The full model card, benchmarks and training details live on the main repo. This one is just the files.

Files

Quant	Size	Notes
IQ4_XS	19 GB	The exact file used in the before/after benchmark. Best fit for one 24GB card with a big context.
Q4_K_M	25 GB	Standard 4-bit. Needs two GPUs or some CPU offload on 24GB.
Q5_K_M	27 GB
Q6_K	35 GB
Q8_0	35 GB	Close to lossless.

All made with llama.cpp b10413 (convert_hf_to_gguf.py then llama-quantize) from the released BF16 weights. You need a llama.cpp build with Nemotron-H support, so b10413 or newer.

Run

llama-server -m Lightning-Herald-30B-A3B-IQ4_XS.gguf --jinja -ngl 999 -c 32768 --temp 1.0 --top-p 0.95

--jinja matters: it makes llama.cpp render the chat template with your tools and parse tool calls into structured tool_calls. The sampling above is NVIDIA's recommendation for the base model and what the benchmark used.

Point Hermes Agent at http://127.0.0.1:8080/v1. A config preset is on the main repo.

Licence

OpenMDW-1.1, inherited from the base. Training data was Apache 2.0.

Dwain Barnes, EryriLabs

EryriLabs/Lightning-Herald-30B-A3B-GGUF

Lightning-Herald-30B-A3B GGUF

Files

Run

Licence

EryriLabs/
Lightning-Herald-30B-A3B-GGUF