Extracted from NVIDIA's published
Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark
checkpoint. This repository contains the drafter and the plugin shim that attaches it;
it is not a standalone model and will not load as one.
The shim is required. Without it the parent model still starts, serves, and reports
healthy — it simply runs slower, with no error anywhere. If you are benchmarking the
parent model and cannot reproduce its published throughput, check that the shim is
loaded first.
Not an NVIDIA release
This is an independent derivative. It is not an official NVIDIA
product, it is not endorsed by, affiliated with, sponsored by, or supported by NVIDIA
Corporation, and NVIDIA has not reviewed, validated, or approved it.
"NVIDIA", "Nemotron", and "DGX Spark" are trademarks of NVIDIA Corporation, used here
solely to identify the origin of the source model. No trademark rights are claimed and
no affiliation is implied. For NVIDIA's official drafter, see
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark.
At a Glance
Type
Speculative draft model
For
Brazley-Nemotron-3.5-Lightning-Omni
Size
1.3 GB
Precision
W4A16_NVFP4. Note that config.json declares "dtype": "bfloat16" at top level; the operative algorithm is in quantization_config
Resolves in vLLM as
Qwen3DSparkModel — natively supported in vLLM 0.27.1
Serving depth
num_speculative_tokens: 3
Also contains
nano_omni_eagle3.py + nano_omni_eagle3-0.1.dist-info/ — the required shim
Release Date
08/20/2026
Model Overview
Description:
A speculative draft model — a small model that proposes several tokens ahead, which the full model verifies in a single step. Accepted proposals are effectively free; rejected ones cost the verification.
Its dimensions close exactly against the parent's transplanted text tower: hidden 2688, vocab 131072, 32 heads / 2 KV, head_dim 128 on both sides, and eagle_aux_hidden_state_layer_ids [2, 6, 20, 30, 42, 52] against the parent's 52 layers. It was trained by NVIDIA against stock Nemotron 3.5 Lightning and runs here unmodified against a merged model.
It serves through vLLM's DSpark path, which resolves it automatically from its architectures field.
vLLM's DSpark path requires the target model to satisfy the SupportsEagle3 protocol. NemotronHForCausalLM declares it. The merged parent, however, resolves to NemotronH_Nano_VL_V2 — the Omni multimodal wrapper — which does not, because stock Nano Omni has no speculative-capable text tower. The parent has one only by virtue of the Lightning transplant.
SupportsEagle3 is a @runtime_checkable Protocol and supports_eagle3() is a plain isinstance(), duck-typed on five members. Everything downstream already handles a multimodal wrapper: the protocol's set_aux_hidden_state_layers walks get_language_model(), so does dspark/utils.py::load_dspark_model, and NemotronH_Nano_VL_V2.forward returns the text tower's output verbatim, so an aux-hidden-state tuple passes straight through.
The shim therefore ships as a standard vllm.general_plugins entry point, bind-mounted into the stock image. No forked engine, no rebuilt image, nothing to maintain.
Two read-only bind mounts alongside the parent model's launch. Both the .py and the .dist-info/ are required — the .dist-info is what makes vLLM discover the entry point.
If you edit the shim, clear the bytecode cache. A read-only bind mount lets a .py and its .pyc disagree indefinitely, and neither one tells you which was executed.
Verifying it loaded
A draft head can build with zero loaded weights and skip silently, presenting as "speculation is on" at roughly 0% acceptance. Loading weights took 8.88 seconds reports presence, not correctness.
The shim compares every directly-comparable draft tensor against the same tensor read from the checkpoint file. On a good load:
[nano-omni-eagle3] DRAFT WEIGHTS VERIFIED: 118 checkpoint tensors -> 92 draft
parameters (702.4M elements); 39/39 directly-comparable tensors match the
checkpoint EXACTLY, 0 mismatches. Acceptance numbers from this run are trustworthy.
It raises rather than serving on a mismatch, and raises rather than certifying when nothing was comparable:
python
1ifnot compared:2raise AssertionError("DRAFT LOAD UNVERIFIABLE:... NOT ONE was directly
3 comparable. Refusing to certify the drafter loaded -- an unverifiable
4 gate must not report success.")56if mismatched:7raise AssertionError("DRAFT WEIGHTS DID NOT LOAD:... A drafter with unloaded
8 weights accepts ~0%and looks identical to a working one from outside.9 Refusing to serve.")
Tensor comparison rather than a parameter count, because 118 checkpoint tensors become 92 parameters when quantized scales fuse — a count mismatch is expected and proves nothing, while an exact match on an unquantized 131072×2688 tensor cannot happen by chance.
Behavioural corroboration: a correctly loaded drafter shows ~66% position-0 acceptance with a clean positional decay.
Evaluation
Measured against the parent model with speculation off, identical conditions (max_tokens 2048, temperature 1.0, top_p 0.95, enable_thinking false, first request discarded, every configuration launched fresh):
Depth
c=1
vs off
c=8
vs off
Accepted / step
off
80.76 tok/s
—
245.93 tok/s
—
—
n=2
81.01
+0.3%
283.36
+15.2%
1.124
n=3 — shipped
91.46
+13.2%
276.50
+12.4%
1.337
n=4
81.52
+0.9%
266.18
+8.2%
1.482
n=7
81.70
+1.2%
237.39
−3.5%
1.655
n=3 is a peak rather than a default: n=2 and n=4 both fall off it at concurrency 1, and n=7 — the drafter's native block size — is worse at concurrency 8. Accepted-tokens-per-step keeps rising with depth while served throughput falls, which is why the shipped depth is chosen on tok/s.
Per-position acceptance against the merged parent at n=3: 66.1 / 40.5 / 24.7 (greedy, c=1), within 0.2 points of the same drafter's figures against stock Lightning.
These depth-sweep rows were measured under an earlier shim revision whose gate verified one tensor rather than all comparable tensors; the acceptance figures on the parent model's card were taken under the current gate.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This repository contains no Omni-derived material, so the NVIDIA Open Model Agreement
governing part of the parent model does not apply here and is deliberately not shipped
with it — including it would misstate what this repository contains.
If you redistribute this drafter or a derivative of it, you must retain a copy of the
OpenMDW-1.1 agreement and all copyright notices and notices of origin. The license
terminates your grant if you initiate patent or copyright litigation concerning these
materials.
No warranty. OpenMDW-1.1 provides these materials "AS IS", without warranty of any
kind. This derivative is provided on the same basis.
This repository is dual-licensed by component
The two files are covered by different grants. Which one applies depends on which
file you are using, not on the repository as a whole.
The shim is original work by Nik Brazley and contains no NVIDIA code — it declares an
existing vLLM protocol against an existing vLLM class and verifies loaded tensors against
the checkpoint file. It is released under the Apache License, Version 2.0, which
matches the license of vLLM itself, the project whose protocol it implements. The full
unmodified license text ships as LICENSE.shim-Apache-2.0.
Copyright 2026 Nik Brazley. Licensed under the Apache License, Version 2.0.
The file carries an SPDX header so the grant travels with it. That matters here more than
usual: the normal way to use this shim is to drop it into a vLLM plugins directory, which
separates it from every license file in this repository.
