NVFP4 quantization of tencent/Hy3 — Tencent's Hy3 (final) 295B Mixture-of-Experts model (21B active, 192 routed + 1 shared expert, 8 experts/token, 256K context, unified fast/deep-reasoning) — for serving on vLLM.
This is a weight-only NVFP4 (NVFP4-A16) checkpoint. The routed-expert FFN tensors — which are ~98% of the model — are quantized to NVFP4 (NVIDIA's 4-bit float, E2M1 with FP8-E4M3 per-16-block scales and an FP32 per-tensor global scale). Every outlier-sensitive path is deliberately kept in BF16: all self-attention (Q/K/V/O), the shared expert, the MoE routers/gates, the dense-MLP layer, the MTP head, token embeddings, lm_head, and all norms. Activations stay BF16 (no activation quantization). Because the experts dominate the memory footprint, quantizing only them recovers almost all of the size reduction while protecting quality where it matters most.
Produced with NVIDIA ModelOpt 0.45.0 via a memory-frugal shard-streaming pass (no calibration data — weight-only NVFP4 derives its scales from the weights themselves), so the packing is exactly what vLLM's NVFP4 loader expects.
About LibertAI
LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.
If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.
Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200, GB10/DGX Spark) NVFP4 weights run on native FP4 tensor-core kernels, and the 4-bit expert weights cut both VRAM and the memory bandwidth that dominates MoE decode. This gives format parity with the NVFP4 path used by vLLM / SGLang / TensorRT-LLM.
Requires a vLLM build with HYV3ForCausalLM support (vLLM ≥ 0.23) and a CUDA-13 image on GB10. A full 2× GB10 deployment recipe (multinode fabric, memory tuning, the exact kernel gotchas) is in deploy/. Verified serving on a 2× NVIDIA GB10 (Grace Blackwell, sm_121) cluster at tensor-parallel = 2 — coherent generation, correct arithmetic, hy_v3 tool calls, and 256K needle retrieval.
Benchmarks — 2× NVIDIA GB10 (DGX Spark)
Served on two GB10 boxes (Grace Blackwell, sm_121, ~120 GB unified each) at tensor-parallel 2 — marlin FP4 MoE, fp8 KV cache, prefix caching, --enforce-eager --max-model-len 262144 --gpu-memory-utilization 0.90.
Hy3-NVFP4 benchmarks on 2× GB10 / DGX Spark
KV cache: 286,640 tokens — a full 256K-token request fits (weights ~84.5 GB/node).
Throughput scales with concurrency (128-tok completions, greedy):
Concurrency
1
4
8
16
32
Aggregate tok/s
13.3
41.7
65.2
99.8
165.8
Per-request tok/s
13.3
10.5
8.2
6.3
5.2
Prefix caching: a repeated ~6.3K-word prompt is served in 0.32 s vs 3.95 s cold — a 92% cut.
Long context: ~120K-token prefill ≈ 138 s.
Note on speed. On GB10 (sm_121) CUDA graphs livelock during capture with the marlin MoE kernel, so --enforce-eager is required and single-stream tops out around 13 tok/s — aggregate throughput is where the two-box setup delivers. A native FP4 MoE path (vLLM's b12x / CuTe-DSL) will lift the single-stream ceiling once its sm_121 build ships. On B200 (sm_100) or RTX 50-series (sm_120) the native NVFP4 kernels run without this constraint.
Provenance & method
Base: tencent/Hy3 (BF16, 99 shards, 598 GB).
Quantizer: shard-streaming ModelOpt NVFP4QTensor weight-only pass; expert FFN → NVFP4, everything else copied through in BF16.
Verification: per-expert round-trip cosine ≈ 0.995 vs the BF16 source.
