Hy-MT2-30B-A3B NVFP4 Experts GGUF
An expert-only NVFP4 derivative of tencent/Hy-MT2-30B-A3B for native
English-Persian and Persian-English translation on NVIDIA Blackwell GPUs.
This is not a blanket 4-bit conversion. The 128 routed experts in layers 1-47
are NVFP4; attention, embeddings, routers, shared experts, layer 0, and the
language-model head remain BF16. The routed experts account for 94.4198% of the
30,064,719,872 source parameters.
Provenance
- Source:
tencent/Hy-MT2-30B-A3B
- Source revision:
d3ead4dba61c09aac60a261a96ad1df3e705febb
- Quantizer: NVIDIA ModelOpt 0.45.0 at
ec87a82927d003986d44fb7f4fa8b3d10c31b095
- Runtime/converter: llama.cpp at
ebb546b7e961bd46fd9ed0387ffd14ca86b6fe1b
- Calibration: 240 English/Persian Wikipedia samples, 65,487 tokens, collected
on 2026-08-11; benchmark sources and references were excluded
- Artifact size: 19,353,433,056 bytes
- SHA-256:
3f5edeb940b56ad851cce84cc55de7812d97dc80aeb4a6d96aa20f044bec891f
ModelOpt fake-quant validation on eight held-out calibration rows produced
0.979527 prefill-logit cosine similarity and 0.973255 first-decode-logit cosine
similarity, with 8/8 top-1 agreement in both cases.
Validation
The native GGUF was tested on an RTX PRO 6000 Blackwell Server Edition with a
64-row balanced development set (32 translations in each direction). References
are synthetic development evidence, not an independent human test set.
| Runtime | Format | Output tok/s | en-fa chrF++ | fa-en chrF++ |
|---|
| vLLM | BF16 source | 859.40 | 56.86 | 66.00 |
| llama.cpp | NVFP4 GGUF | 413.92 | 56.84 | 65.49 |
Both runs used batch/concurrency 64 and a maximum of 128 generated tokens. The
NVFP4 run produced no empty translations and no exact source copies. Its output
agreement with BF16 was 88.01 chrF++ for English-to-Persian and 93.66 chrF++ for
Persian-to-English.
RTX 5080 Laptop (Stallion)
The same GGUF and llama.cpp revision were also validated on a 16 GB RTX 5080
Laptop GPU at an 85 W power limit. With 38 layers offloaded to CUDA and the
remaining weights in system RAM, a single 8K-context slot used 15,144 MiB VRAM
and generated at 53.9 tok/s server-side (51.9 tok/s end-to-end).
An eight-slot throughput run used 8K total context (1,024 tokens per slot), 64
requests, and a 128-token output cap:
| Concurrency | Output tok/s | Total tok/s | Requests/s |
|---|
| 8 | 85.63 | 231.87 | 1.69 |
That run produced no empty outputs or exact source copies. Development chrF++
was 56.17 for English-to-Persian and 65.80 for Persian-to-English.
llama.cpp
Use a CUDA build that includes SM120 and the NVFP4 MMQ kernel. The exact tested
llama.cpp revision is listed above.
1cmake -S . -B build-sm120 \
2 -DGGML_CUDA=ON \
3 -DCMAKE_CUDA_ARCHITECTURES=120
4cmake --build build-sm120 --config Release -j --target llama-cli llama-server
5
6build-sm120/bin/llama-cli \
7 -m Hy-MT2-30B-A3B-NVFP4-Experts.gguf \
8 -ngl 99 -c 4096 -fa on \
9 -n 256 --temp 0
The amount of GPU offload is hardware- and workload-dependent. Leave headroom
for the KV cache and other processes instead of assuming the complete 19.35 GB
file fits in VRAM.
The tested 16 GB / 8K Stallion configuration used -ngl 38 -c 8192 -np 1 for
single-request serving. For the aggregate batch-8 result it used
-ngl 38 -c 8192 -np 8; llama.cpp therefore assigned 1,024 context tokens to
each parallel slot.
Important limitation
The intermediate ModelOpt safetensors pass fake-quant logit checks, but their
current native and emulated vLLM deployment paths generated invalid text during
this build. Those files are quarantined and are not the supported artifact. The
GGUF listed by the exact checksum above is the validated runtime artifact.
The development set was used to validate conversion fidelity. It must not be
reported as a fresh held-out measure of general translation quality.
