This is a calibrated W4A16 quantization of NVIDIA Nemotron 3.5 Lightning 30B-A3B, intended for high-throughput NVIDIA GPU serving with vLLM. It has been validated on a single 24 GB RTX 3090 at a 640K production context window.
Source checkpoint: NVIDIA's BF16 release at revision 63a200063804e06fdb41d6717e43bc92f67859d2
Governing license: OpenMDW 1.1; the unmodified upstream license is included as LICENSE.
This is a third-party, post-training quantization. useful-quants did not train, fine-tune, align, or otherwise develop Nemotron. The conversion does not change NVIDIA's intended-use, safety, or license terms; see the upstream model card for those terms, language coverage, architecture, reasoning behavior, training/data disclosure, and NVIDIA's source-model benchmark suite.
Context terminology matters: NVIDIA specifies an architectural model context of up to 1M tokens. The 640K recommendation and 744K maximum here are deployment validations for this exact W4A16 artifact on one 24 GB RTX 3090 profile; they are not model-wide limits or guarantees for other hardware/software configurations.
Quick start
Install a current vLLM release compatible with your CUDA/PyTorch installation, then serve the model:
On the validated vLLM 0.27.1 stack, automatic dtype selection resolves to BF16 for this checkpoint. --dtype bfloat16 is optional but is used in the reproducible 3090 profile to make that choice explicit. --max-num-seqs 32 is required on this 24 GB hybrid-Mamba deployment: vLLM's generic 256-slot default exceeds the available Mamba cache blocks. This compressed-tensors artifact was validated with vLLM; direct Transformers inference is not documented here.
NVIDIA's chat template supports enable_thinking=True for reasoning mode and False for direct-answer mode. The temperature=1.0 and top_p=0.95 example values are NVIDIA's upstream sampling defaults, not settings derived by this quantization.
RTX 3090 24 GB — validated long-context profile
The reproducible production profile is examples/start_vllm_3090_640k.sh. It uses vLLM 0.27.1, PyTorch 2.13, Triton 3.7.1, compiled CUDA graphs, BF16/FP32 cache behavior, priority scheduling, chunked prefill, 32 sequences, 8,192 batched tokens, and a fixed 4,466,765,988-byte cache reservation:
bash examples/start_vllm_3090_640k.sh
Observed with that profile: peak VRAM was 24,076 MiB (~98.0%); a 655,344-token near-limit single request plus decode passed; and 32 concurrent resident requests totaling 670,688 prompt tokens passed. The deterministic capacity stress had no OOM, preemption, or recomputation. Explicit cache reservation makes the memory envelope reproducible across restarts while retaining CUDA graphs; VLLM_DISABLE_COMPILE_CACHE=1 is intentionally not used.
3090 boundary
The validated boundary sweep on the preceding 3090 stack found 736K and 744K successful. At 744K, near-limit decode and 32-way residency both passed, but there was less than one cache page of nominal slack. At 748K the engine could not initialize: it required 4.44 GiB of cache against 4.43 GiB available. Thus 744K is a demonstrated boundary, not an operational recommendation; 640K is deliberately retained as the production setting.
With --max-num-batched-tokens 16384, estimated cache capacity fell to 563,824 tokens and the server could not initialize at 640K. 8192 is therefore the validated throughput/context compromise on this 24 GB profile.
For long-running mixed workloads, vLLM can interleave short batch work and long agentic contexts with priority scheduling. A high-priority request cannot, however, displace running work solely to obtain a sequence slot when all 32 slots are occupied. Reserve 1–2 ingress slots for high-priority/agent traffic: for example, admit bulk work to at most 30–31 slots.
Measured throughput
Current-stack vLLM 0.27.1 measurements on the RTX 3090 profile above, using
steady-state decode after warm-up. The full 11-point curve is more useful than
a three-point headline:
Active requests
Aggregate decode throughput
1
214.7 tok/s
8
817.4 tok/s
16
1,099.9 tok/s
32
1,462.8 tok/s
Matched vLLM / llama.cpp saturation comparison
RTX 3090 matched W4A16 vLLM and IQ4_XS llama.cpp throughput
This reader-facing comparison uses the same Nemotron 3.5 Lightning 30B-A3B
checkpoint lineage in both formats: this W4A16 artifact in vLLM and the
verified IQ4_XS GGUF in llama.cpp. It is a separate common short-context
workload (--max-model-len 4096 for vLLM): cache-isolated, unique prompts,
greedy decoding, and 256 fixed visible output tokens per request. It is not a
replacement for the 640K capacity validation above.
At the normal 370 W board limit, W4A16 reaches 1,819 aggregate visible tok/s
at B16—87% of its B32 result of 2,089 tok/s—making B16 the practical knee
for this matched workload. B32 remains the absolute-throughput setting. The
dashed 275 W lines are an optional efficiency overlay, not the default
performance configuration: they preserve low-concurrency throughput but reduce
W4A16 B16/B32 saturation throughput. The IQ4_XS power-limit difference is
small by comparison.
These are hardware-, prompt-, sampling-, and runtime-specific measurements,
not universal model-performance claims. The exact matched methodology, raw
results, and both power-limit CSVs are in
benchmarks/rtx3090-matched-vllm-llamacpp.md.
The separately measured 640K production curve, including TPOT/TTFT and raw
five-repeat data, remains in
benchmarks/rtx3090-throughput-curve.md.
Prefix caching and batch work
This validated 640K profile batches independent requests continuously. It does
not enable prefix caching by default: in vLLM 0.27.1, hybrid Mamba models
leave it opt-in because the required Mamba cache all mode is experimental.
Prefix caching only helps requests that share an identical token prefix; it
does not improve a batch of unrelated prompts. Add --enable-prefix-caching
only for a shared-prefix workload and soak-test the exact context/cache profile
you intend to use. A CUDA:0 smoke of this exact 640K profile entered the
experimental Mamba warm-up path and was not completed, so it is intentionally
not in the validated long-context command. A smaller-context lab profile may
use the flag, but it is a separate experimental deployment choice rather than
a property of the published 640K validation.
Quality evaluation
We evaluated 2,000 multiple-choice items (500 each) using vLLM likelihood scoring. This is quantization evidence for this artifact, not a replacement for NVIDIA's full source-model benchmark suite.
Task
W4A16
ARC-Challenge
62.0%
HellaSwag
67.8%
PIQA
87.0%
Winogrande
72.8%
Equal-weight aggregate
72.4%
These are standalone W4A16 measurements. They are not used for a cross-format
quality claim, because such a claim requires the same source checkpoint and
identical evaluation methodology. Full methodology:
benchmarks/quality-scorecard.md.
IFEval — Nemotron 3.5 Lightning 30B-A3B
On the original 541-prompt Google IFEval set, evaluated by Google's reference
strict/loose checker with a shared NVIDIA-formatted direct-answer prompt and
greedy decoding:
Format
Prompt strict
Instruction strict
Prompt loose
Instruction loose
This W4A16 artifact
73.94%
81.29%
77.45%
83.69%
bartowski GGUF IQ4_XS
74.49%
82.01%
78.00%
84.53%
Both rows derive from the NVIDIA Nemotron 3.5 Lightning 30B-A3B BF16
checkpoint lineage, use the same 541 Google IFEval prompts, one shared
NVIDIA-formatted direct-answer prompt manifest, identical greedy generation
settings, and Google's reference evaluator. The measured difference on this
run is small (403/541 versus 400/541 prompt-strict); it should not be treated
as a general quantization ranking. See benchmarks/ifeval.md
for checkpoint, GGUF metadata, runtime hashes, prompt construction, and raw
result locations.
No directly comparable BF16 perplexity was measured. A source-BF16 comparison was not practical on the available 24 GB hardware, and no incompatible PPL result is presented as a degradation estimate. For this instruction/reasoning checkpoint, the paired 2,000-item scorecard is the directly measured quantization-quality evidence.
