You can deploy this model and start earning money today!

thecodehaider/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF

Name: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF
Availability: InStock

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

quantized

endpoints_compatible

conversational

Views

No views yet

⚡ Quantized with QuantizeLab

This model was quantized to GGUF format using QuantizeLab, the fastest SaaS to quantize models under 33B parameters. Join hundreds of developers downloading our optimized quants!

Fast Conversion: Direct upload to Hugging Face.
Optimized Sizes: Support for all major GGUF bit-rates.
Hardware Free: No local GPU required for quantization.

👉 Try QuantizeLab Now

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF (Q4_K_M)

Q4_K_M GGUF quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, produced with llama.cpp by Quantizelab.dev.


File	`model-Q4_K_M.gguf`
Quantization	`Q4_K_M`
Size on disk	24.52 GB
Base model	nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Run it (full GPU offload)

GGUF defaults to CPU. To get GPU speed you must offload every layer — a single layer left on the CPU takes generation from ~25 tok/s to ~3 tok/s. -ngl 999 simply means "offload all of them".

bash

# llama.cpp
llama-cli -hf thecodehaider/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF:model-Q4_K_M.gguf -ngl 999 -c 4096 -p "Hello"

# local file
llama-cli -m model-Q4_K_M.gguf -ngl 999 -c 4096 -cnv

# OpenAI-compatible server
llama-server -m model-Q4_K_M.gguf -ngl 999 -c 4096 --port 8080

bash

# Ollama
ollama run hf.co/thecodehaider/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF:Q4_K_M

python

# llama-cpp-python
from llama_cpp import Llama
llm = Llama(model_path="model-Q4_K_M.gguf", n_gpu_layers=-1, n_ctx=4096)
print(llm("Hello", max_tokens=128)["choices"][0]["text"])

Will it fit your GPU?

Weights plus ~1.2 GB of KV-cache and compute overhead at a 4k context.

GPU	VRAM	Fits fully offloaded?	Headroom for context
NVIDIA T4 / RTX 4060	16 GB	No	offload partially (`-ngl` lower) or use CPU
RTX 3090 / 4090 / A10	24 GB	No	offload partially (`-ngl` lower) or use CPU
A100 40GB	40 GB	Yes	~14.3 GB (4k+ context)

If a row says No, lower -ngl until it fits, or run on CPU (GGUF works either way — it is just slower).

Notes

Q4_K_M is the recommended balance of size and quality; Q8_0 and above will not fully offload to a 16 GB card for models past ~8B.
Reduce -c (context) first when you hit out-of-memory: the KV cache grows linearly with context length.