First published NVFP4 build of this model · 2026-07-14
This is a mixed-precision quantization of tencent/Hy3 (295B-A21B MoE). It was built for one goal: a faster decode rate without sacrificing quality. Decode speed on a large model is set by memory bandwidth — how many bytes of weights each token must read — and NVFP4 was chosen exactly to cut those bytes. The official FP8 release reads ~19.6 GB per token; this build reads ~14.4 GB. The measured speed follows that arithmetic. It loads on stock vLLM, and the multi-token-prediction (MTP) draft head is preserved bit-for-bit, so speculative decoding works.
Highlights
+12% single-stream decode over the official FP8 release — 108.5 vs 96.9 tok/s plain, 117.7 vs 105.7 with MTP¹, measured on the same box and engine. This build's plain decode beats the official FP8 running with its speculation on.
Quality held — +2.35% wikitext-2 perplexity vs the FP8 release, zero mid-word corruption in 101,400 tokens under Tencent's recommended sampling. Calibrated GPTQ, not data-free rounding.
The MTP draft head ships whole — all 593 tensors byte-identical BF16; the official releases quantize theirs. Speculative decoding runs at the checkpoint's full trained acceptance.
Stock vLLM 0.24 — no fork, no pinned commit; 262,144-token context on fp8 KV; NVFP4 is tensor-core-native on Blackwell.
First published NVFP4 build of this model — 178 GB on disk, down from 557 GB BF16, and smaller than the official 4-bit GGUF even while carrying the full draft head.
Ships the fix for Chinese characters appearing in English output — every quant of this model has this problem to varying extents because it comes from the base model; even the official FP8 leaks (receipts under decision 5). The fix is easy to use and works with any quant.
¹ MTP throughput requires a two-line vLLM patch, submitted upstream as PR #48767 — see decision 4. All speed numbers measured on 4× RTX PRO 6000 Blackwell (SM120), single stream.
The decisions
Precision map: routed experts NVFP4 at ~96% of weights, every-token layers FP8 at ~2.6%, draft head and logit-adjacent tensors BF16 at ~1.6%
1 · Routed experts: 4-bit NVFP4, solved with GPTQ against calibration data
45,504 matrices — 79 MoE layers × 192 experts × gate/up/down — at FP4-E2M1, group size 16, FP8-E4M3 scales.
Why 4-bit here. The experts hold ~96% of the weights, and the 8 of 192 that fire still hold ~60% of the parameters each token touches. Four-bit here does two things at once: the artifact fits in 178 GB instead of 557, and the per-token read drops by ~5 GB versus the FP8 release — which is the entire speed win. Weight bytes are the decode bottleneck; the experts are where the bytes are.
Why calibrated. Data-free rounding is not safe on this checkpoint. An earlier data-free (RTN) NVFP4 build of ours corrupted rare tokens under Tencent's recommended sampling (temperature 0.9, top_p 1.0): about one event per thousand characters, single words written half in English and half in Mandarin. That build was deleted; the rate is a production observation. GPTQ solves each matrix against real activations on the same 4-bit grid, and the defect class vanished: zero mid-word corruption in 101,400 generated tokens.
2 · Every-token layers: FP8
560 matrices — attention (q/k/v/o), shared experts, and the dense layer-0 MLP — at FP8, per-channel weight scales, dynamic per-token activation scales.
Why. These layers run on every token, so they set both the speed floor and the quality floor. A predecessor build with every linear layer at 4-bit cost +3.41% perplexity. Holding these layers at FP8 recovered quality to +2.35%, and the build still reads 5 GB/token less than the official FP8. Tencent keeps exactly these layers at 8-bit in both of their own quantized releases. Their sensitivity data and ours agree.
3 · The MTP draft head: untouched BF16
All 593 tensors of the draft layer, byte-identical to source. Embeddings, the output head, router gates, norms, and expert biases also stay BF16 — small, logit-adjacent tensors that are not worth any risk.
Why. A draft head earns its keep through acceptance rate, and under speculative decoding acceptance is speed: every accepted draft token is a token the target did not have to decode alone. The head stays at full 16-bit on the belief that any quantization there spends acceptance — and spending acceptance is spending speed — to save a few GB of disk that never justify it. The official 4-bit GGUF quantizes its draft head; this build keeps it whole.
4 · Serve speculation at depth 1
Why. The head is a single layer trained for depth 1 — num_nextn_predict_layers: 1 in the base config. Position-0 acceptance is ~57%, and it is set by that training, not by precision: this build's full-BF16 head measures the same acceptance as the official FP8's partly-quantized one (56.9% vs 57.7%). Past position 0, acceptance collapses. Measured on this build, single-stream: depth 1 = 117.7 tok/s, depth 2 = 102.1, depth 3 = 109.6 — conditional acceptance falls to ~41% at position 1 and ~37% at position 2, and every depth past 1 loses to depth 1 (depth 2 lands below even plain decode). Depth 1 pays +8.5% over plain. Deeper pays less.
One engine fact. vLLM needs a two-line patch for MTP to be profitable on this model family. HYV3MTP in vllm/model_executor/models/hy_v3_mtp.py lacks the @support_torch_compile decorator that DeepSeekMTP carries, so the draft layer runs eager against a CUDA-graphed target, and one eager draft cycle costs about one full target step. Add the import and the decorator, matching DeepSeekMTP. Upstream: issue vllm-project/vllm#48766 and PR #48767 — the PR's two-line diff is the exact local patch to apply to your install until it merges. Plain decode needs no patch.
5 · Stop Chinese characters from appearing in the output — and ship the fix
Why a fix is needed at all. The base checkpoint leaks Chinese synonyms into pure-English prose, at every precision. At semantic slots it places the Chinese word at rank 2–3 of the next-token distribution with double-digit probability.
Measured at a real trigger slot (mid-sentence English narrative, where the next word is "denial" or "denying"), forced-token logprobs, both arms served identically:
Arm
logprob(否认)
≈ probability
rank
Official FP8
−0.98
~37%
2
This build
−1.66
~19%
3
A second trigger repeats the pattern (FP8 −4.30, this build −6.09). The official FP8 leaks harder than this build at both measured slots. In long-form English generation this surfaces about twice per 100K tokens — rare, never zero.
Why the weights cannot fix it. The leak survives Tencent's own FP8. It is in the checkpoint's training, not in anyone's rounding. And a rank-2 token at 37% cannot be filtered by any top_p, top_k, or min_p setting.
Why this exact mechanism. The sampler can do what the weights cannot: ban every token containing a Chinese, Japanese, or Korean (CJK) script character, passed as allowed_token_ids. Control tokens are exempt — their names carry fullwidth-bar punctuation from the CJK block, and banning them removes end-of-sequence, leaving the model unable to stop; the ban governs text, grammar tokens stay emittable. vLLM enforces allowed_token_ids on both the plain and the speculative-decoding sampling paths. A custom logits processor silently fails to bind to MTP-drafted tokens; bad_words re-tokenizes ~48K strings per request. The ~1 MB payload costs milliseconds locally. Assembling a CJK character from raw byte tokens would take several consecutive tokens the model gives negligible probability in English, so banning whole tokens closes the leak.
The kit, in cjk-ban/: generate-allowed-ids.py (model-agnostic — scans any tokenizer, bans Han, kana, Hangul, and CJK punctuation, keeps control tokens emittable) and hy3-allowed-token-ids.json (pregenerated for this model: 47,295 banned, 73,523 allowed, of 120,818). The fix is not specific to this build: it works with any quant of this model — or the unquantized release — on anything that honors allowed_token_ids.
python
1import json
2from openai import OpenAI
34allowed = json.load(open("cjk-ban/hy3-allowed-token-ids.json"))["allowed_token_ids"]5client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")6out = client.chat.completions.create(7 model="hy3-nvfp4",8 messages=[{"role":"user","content":"Write a scene where he denies it."}],9 temperature=0.9,10 extra_body={"allowed_token_ids": allowed},11)
Verified live under MTP at the strongest measured trigger, production sampling: unbanned, 4 of 8 samples leak 否认. Banned, 0 of 8. Greedy decode picks " denial". Generation terminates normally under the ban — finish_reason stop, verified live on this build. If you want English output with zero Chinese characters — from this build or from any serving of this checkpoint — use this.
Measured results
Single-stream decode: this build 117.7 tok/s with MTP depth 1 and 108.5 plain, official FP8 105.7 and 96.9
All numbers measured on 4× RTX PRO 6000 Blackwell: vLLM 0.24.0, TP4, kv-cache fp8, single-stream decode, against the official tencent/Hy3-FP8 served identically on the same machine. Single-stream is deliberate: this build is for a personal workstation, and a workstation serves one person at a time. Aggregate multi-stream throughput is a data-center metric.
Config
Official FP8
This build
Plain decode (tok/s)
96.9
108.5 (+12.0%)
MTP depth 1 (tok/s)¹
105.7
117.7 (+11.4%)
MTP depth 2 (tok/s)
87.2
102.1
MTP depth 3 (tok/s)
—
109.6
Perplexity, wikitext-2 (66,055 tokens)
4.8624
4.9768 (+2.35%)
Stray Chinese-character events at temp 0.9 / top_p 1.0
0 / 50,598 tokens
2 / 101,400 tokens²
MTP position-0 acceptance
57.7%
56.9%
² Both events are the same whole-word substitution documented under decision 5 — a base-model behavior the official FP8 exhibits more strongly at the measured trigger slots. The rates are statistically inseparable, and the ban kit makes the rate exactly zero. Zero mid-word corruption in either arm.
On the depth rows: the base model is trained for an MTP depth of exactly one — num_nextn_predict_layers: 1 in Tencent's config, one draft module trained to predict one step ahead. Higher depths re-run that module on its own output, which its training never covered, so acceptance collapses past position 0 and every depth above 1 measures slower than depth 1. That is a property of the checkpoint, not of this quantization. Run depth 1.
This build is designed and tested on 4× RTX PRO 6000 Blackwell (96 GB each). Two DGX Spark units hold the weights in 256 GB of combined unified memory, and NVFP4 is native to their Blackwell silicon too. Two RTX PRO 6000 cards (192 GB) might load it with a much smaller context window. Neither two-device setup is tested here.
~37 GiB of weights per GPU at TP4; fp8 KV at ~162 KiB/token gives 4–5 concurrent full-256K streams at 0.90 utilization on 4× 96 GB cards.
First boot includes torch.compile — raise the readiness timeout (VLLM_ENGINE_READY_TIMEOUT_S=1800) so the engine is not killed mid-compile.
Reasoning parser: version-dependent. This checkpoint family emits namespaced thinking tags (<think:opensource> … </think:opensource>) as single tokens, and the chat template opens the section inside the prompt, so completions usually carry only the closer. On vLLM 0.24.x, --reasoning-parser hy_v3 expects the bare pre-release <think> pair and returns empty content — serve those versions with no reasoning parser and split client-side: everything up to the last </think:…> closer is reasoning, the rest is the answer. From 0.25.0, vllm-project/vllm#47192 resolves the namespaced pair natively via the token_suffix field this artifact's tokenizer_config ships.
SGLang cannot serve this artifact (no W4A16-FP4 scheme, no mixed-precision checkpoint support as of 0.5.x). This is a vLLM-lane build.
Versus the official 4-bit release (AngelSlim GGUF)
Tencent's AngelSlim team published Q4_K_M GGUFs of this model on 2026-07-14. Their recipe independently confirms this precision map: every-token layers at 8-bit, routed experts at 4-bit class. The differences:
AngelSlim Q4_K_M
This build
MTP draft head
quantized (eh_proj q8_0)
BF16, byte-identical
Expert 4-bit method
data-free K-quant (no imatrix published)
GPTQ, Hessian-calibrated per expert
Weight format
ggml K-blocks
NVFP4 (E2M1 + FP8 scales), tensor-core native on SM120
The official 4-bit release holds some expert tensors above 4-bit — the down projections at 5-bit, and the last layers' at 6-bit. That is the release team's own sensitivity signal. A follow-up build is planned that studies those tensors and holds them at 8-bit instead.
Quantization details
Full build recipe and integrity gates
llmcompressor 0.12, one oneshot pass, sequential pipeline, two modifiers on disjoint targets: GPTQ→NVFP4 on routed experts, FP8-dynamic on attention/shared/dense-0.
NVFP4 (nvfp4-pack-quantized in compressed-tensors): E2M1 values, group 16 on the input dimension, per-group FP8-E4M3 scales, F32 global scales shared exactly across fused groups (q/k/v; expert gate/up) — 15,168 fused pairs verified byte-equal.
Integrity gates: every source tensor accounted for (45,504 NVFP4 triplets, 560 FP8 pairs, 1,074 BF16 pass-through, zero missing or extra); 17.9×10⁹ scale elements, zero NaN, zero zeros; all 593 MTP-layer tensors byte-identical to source.
config.json carries one serving fix: the FP8 target list names the shared-expert projections under the flattened module path vLLM resolves at load time.
License and provenance
Apache-2.0. Tencent's LICENSE ships verbatim in this repository. The weights here are a modified version of tencent/Hy3: quantized as described above; they are not the original weights and are not affiliated with or endorsed by Tencent. "Tencent" and "Hy3" are used descriptively to identify the source model.
Quantization, forensics, and documentation by Auroter, with AI assistance (Claude); all measurements taken and verified on the hardware named above.
