NVIDIA-Nemotron-3.5-Lightning-30B-A3B — ROCmFP4-STRIX_LEAN
TL;DR
NVIDIA Nemotron 3.5 Lightning 30B-A3B (hybrid
Mamba-2 + MoE + Attention) quantized to
ROCmFP4-STRIX_LEAN (software FP4, GGUF type 106) for
AMD Strix Halo (gfx1151) on the
ROCmFPX fork family — we serve and benchmark these files on
our lab runtime (full source:
pugant/strix-halo-llamacpp-lab, upstream by
charlie12345/ROCmFPX). Runs at
84.3 tok/s generation on a Radeon 8060S iGPU — the fastest of any model tested on this hardware. 15.7 GiB footprint.
⚠️ This GGUF is for the ROCmFPX fork of llama.cpp on AMD RDNA 3.5 only. It will not load in stock llama.cpp.
⚠️ Critical warnings — read before use
- Type 106 is invalid for stock llama.cpp.
Q4_0_ROCMFP4_STRIX_LEAN (type 106) is a custom type from the charlie12345/ROCmFPX fork. Loading this file with a stock build will fail.
- Requires a ROCmFPX fork build. Recommended: our lab runtime — full source + replication guide at
pugant/strix-halo-llamacpp-lab/rocmfpx (the exact build these benchmarks run on: per-request drafter routing, DFlash2 drafter support, reasoning budget, spec-boundary cache salvage). Upstream charlie12345/ROCmFPX loads these files too (any build with the custom GGML types, e.g. the matching kyuz0 Docker toolbox for gfx1151).
- FP4 is software-dequantized on RDNA 3.5. gfx1151 has no native FP4 hardware; the kernel dequantizes FP4→FP16 in software. The speedup vs. Q4_K_M comes from the smaller memory footprint and the fork's optimized ROCm dequant path, not from hardware FP4.
- Unified memory required. On Strix Halo the VRAM partition is tiny (512 MB). Run with
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 and HSA_OVERRIDE_GFX_VERSION=11.5.1.
-fit off is mandatory for this architecture. llama-server with default -fit on deadlocks during "fitting params to device memory" on the 128-expert MoE. Always pass -fit off.
Benchmarks
Hardware: AMD RYZEN AI MAX+ 395, Radeon 8060S (gfx1151, 128 GB unified memory), ROCm 7.2.4.
Method:
llama-bench -ngl 999 -fa on -p 512 -n 128. All benchmarks for this model were run in
ROCm containers (HIP backend). Since then we also benchmarked ROCmFPX quants on the Vulkan (RADV) build of the fork — see the comparison table in
Qwen3.6-35B-A3B-MTP-Q6_0_ROCMFPX; those numbers are on a different backend and not directly comparable.
| Format | tg128 (tok/s) | pp512 (tok/s) | Size |
|---|
| ROCmFP4-STRIX_LEAN | 84.28 | 1051 | 15.72 GiB |
| Q4_K_M (same base) | 63.64 | 813 | 22.82 GiB |
ROCmFP4-STRIX_LEAN is +32% faster at generation and 31% smaller than Q4_K_M.
Note: Q4_K_M falls back on 134/401 tensors (the Mamba/SSM tensors are not compatible with Q4_K_M block sizes); ROCmFP4 quantizes them natively (1/401 fallback), which is a large part of why it wins here.
Compared to other Strix Halo ROCmFP4 models: this is the fastest yet (+19% vs grug-35b-v2 at 70.92, +26% vs Ornith-1.0-35B at 66.68).
Quantization details
- Type:
Q4_0_ROCMFP4_STRIX_LEAN (type 106) — fork-specific.
- Preset: STRIX_LEAN (uses
q4_0_rocmfp4_fast dequant).
- Bitrate: 4.28 BPW.
- imatrix: bartowski calibration (185 entries covering the quantizable tensors).
- Tensors: 401 (237 f32 — norms/SSM-state/bias; 164 ROCmFP4 — weight matrices).
- SHA256:
91316f85d52d378ff963c42f293660f2ca094f8bce0b08ea481ff620bc5056a7
Architecture: nemotron_h_moe
Nemotron-H is a hybrid SSM-Transformer. Each of the 52 layers is one of:
- Mamba-2 (SSM): linear-time recurrent state-space layer.
ssm.conv_kernel=4, ssm.state_size=128, 64 heads. Handles long-range context with fixed KV cost.
- MoE: 128 routed experts (6 active per token) + 1 shared expert.
expert_feed_forward_length=1856, expert_shared_feed_forward_length=3712.
- Multi-head attention: 32 heads, 2 KV heads (GQA),
head_dim=128, RoPE θ=10000. Inserted periodically for global mixing.
Stats: 31.58 B total parameters, ~3.5 B active per token, 131072 vocab, 1M context length.
Note: the ROCmFPX fork's converter does not emit the MTP/nextn tensors, so this build is a plain (non-speculative) model. The base model's MTP layer is not used here.
imatrix
The importance matrix from bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF (185 entries, 822 chunks). Paths in the GGUF metadata have been neutralized to basenames during sanitization.
Usage
1docker run --rm -d --name nemotron \
2 --device /dev/kfd --device /dev/dri --group-add video --group-add render \
3 -e HSA_OVERRIDE_GFX_VERSION=11.5.1 -e GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
4 -p 1234:1234 -v /path/to/models:/models \
5 <rocmfpx-fork-image> llama-server \
6 -m /models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-ROCmFP4-STRIX_LEAN.gguf \
7 -ngl 999 -c 32768 -fa on --jinja --host 0.0.0.0 --port 1234 -fit off
The model has a chat template with a thinking/reasoning mode (the <think> block surfaces in reasoning_content on the OpenAI-compatible endpoint).
Attribution
- Base model: NVIDIA,
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (OpenMDW-1.1).
- Converter & quantization: charlie12345/ROCmFPX fork (MIT).
- imatrix: bartowski.
- Toolbox: kyuz0 llama.cpp-amd-gpu.
- Quantized by: pugant.
License
OpenMDW License Agreement, version 1.1 (OpenMDW-1.1) — inherited from the upstream NVIDIA model. See LICENSE.
Acknowledgements
Thanks to NVIDIA for releasing Nemotron under a permissive license, to the ROCmFPX fork authors for the FP4 quantization work, and to bartowski for the calibration imatrix.
Limitations
- FP4 dequantization is software-only on gfx1151; no native FP4 hardware is used.
- MTP/speculative decoding is not enabled in this build.
- Validated on Strix Halo (gfx1151) only; behavior on other RDNA versions is untested.
Disclaimer
Not affiliated with or endorsed by NVIDIA, AMD, ggml-org, bartowski, kyuz0, or the ROCmFPX maintainers. Provided "as is", without warranty.
Software
- Runtime (recommended): our lab build of the ROCmFPX fork — full source at
pugant/strix-halo-llamacpp-lab/rocmfpx
(per-request drafter routing, DFlash2 drafter, reasoning budget, spec-boundary cache salvage;
the exact build used for the benchmarks above). Upstream:
charlie12345/ROCmFPX — the fork that defines the
ROCmFPX presets/types.
- Lab repo (patches, benchmarks, full replication guide):
pugant/strix-halo-llamacpp-lab
Everything here is experimental and provided as-is, at your own risk.
