This is a GGUF conversion of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 using llama.cpp's convert_hf_to_gguf.py using --outtype auto, and without MTP layers.

Thanks to no MTP layers included, you can run this on 32 GB VRAM with maximum context, at q8_0 KV cache quantization.

Verified working correctly with latest llama.cpp on 2x RTX 5060 Ti 16 GB and the following command (add other things you need):

llama-server -b 1024 -ub 256 \
  -ctk q8_0 -ctv q8_0 \
  --kv-offload -ngl 999 --load-mode none -fa on \
  --ctx-size 1048576 --parallel 1

No expert-layer offloading to RAM required. Solid performance. Here's an example run, where I fed it the PDF of my book Lo and Behold, X! — Denizens of the Ivory Fortress

0.09.472.316 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.09.472.434 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.12.554.658 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  16389, progress = 0.30, t =   3.08 s / 5317.30 tokens per second
0.12.750.235 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  17413, progress = 0.32, t =   3.28 s / 5312.43 tokens per second
0.12.945.471 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  18437, progress = 0.34, t =   3.47 s / 5308.64 tokens per second
0.13.145.359 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  19461, progress = 0.36, t =   3.67 s / 5298.53 tokens per second
0.13.344.785 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  20485, progress = 0.38, t =   3.87 s / 5290.09 tokens per second
0.13.546.788 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  21509, progress = 0.40, t =   4.07 s / 5279.14 tokens per second
0.13.749.650 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  22533, progress = 0.42, t =   4.28 s / 5268.17 tokens per second
0.13.953.626 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  23557, progress = 0.44, t =   4.48 s / 5256.88 tokens per second
0.14.159.157 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  24581, progress = 0.45, t =   4.69 s / 5244.84 tokens per second
0.14.366.205 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  25605, progress = 0.47, t =   4.89 s / 5232.18 tokens per second
0.14.574.330 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  26629, progress = 0.49, t =   5.10 s / 5219.45 tokens per second
0.14.783.860 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  27653, progress = 0.51, t =   5.31 s / 5206.34 tokens per second
0.14.995.848 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  28677, progress = 0.53, t =   5.52 s / 5191.92 tokens per second
0.15.206.569 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  29701, progress = 0.55, t =   5.73 s / 5179.70 tokens per second
0.15.419.823 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  30725, progress = 0.57, t =   5.95 s / 5166.15 tokens per second
0.15.633.912 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  31749, progress = 0.59, t =   6.16 s / 5152.84 tokens per second
0.15.849.117 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  32773, progress = 0.61, t =   6.38 s / 5139.52 tokens per second
0.16.065.595 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  33797, progress = 0.63, t =   6.59 s / 5126.08 tokens per second
0.16.285.412 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  34821, progress = 0.64, t =   6.81 s / 5111.00 tokens per second
0.16.507.478 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  35845, progress = 0.66, t =   7.04 s / 5095.22 tokens per second
0.16.730.801 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  36869, progress = 0.68, t =   7.26 s / 5079.53 tokens per second
0.16.955.928 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  37893, progress = 0.70, t =   7.48 s / 5063.56 tokens per second
0.17.182.549 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  38917, progress = 0.72, t =   7.71 s / 5047.54 tokens per second
0.17.410.245 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  39941, progress = 0.74, t =   7.94 s / 5031.75 tokens per second
0.17.640.124 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  40965, progress = 0.76, t =   8.17 s / 5015.51 tokens per second
0.17.870.636 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  41989, progress = 0.78, t =   8.40 s / 4999.77 tokens per second
0.18.102.121 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  43013, progress = 0.80, t =   8.63 s / 4984.32 tokens per second
0.18.336.681 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  44037, progress = 0.82, t =   8.86 s / 4967.94 tokens per second
0.18.572.206 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  45061, progress = 0.83, t =   9.10 s / 4951.89 tokens per second
0.18.807.463 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  46085, progress = 0.85, t =   9.34 s / 4936.79 tokens per second
0.19.042.969 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  47109, progress = 0.87, t =   9.57 s / 4922.30 tokens per second
0.19.282.733 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  48133, progress = 0.89, t =   9.81 s / 4906.38 tokens per second
0.19.524.114 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  49157, progress = 0.91, t =  10.05 s / 4890.43 tokens per second
0.19.766.907 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  50181, progress = 0.93, t =  10.29 s / 4874.57 tokens per second
0.20.011.015 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  51205, progress = 0.95, t =  10.54 s / 4858.82 tokens per second
0.20.256.793 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  52229, progress = 0.97, t =  10.78 s / 4843.04 tokens per second
0.20.504.567 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  53253, progress = 0.99, t =  11.03 s / 4827.09 tokens per second
0.20.627.932 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  53765, progress = 1.00, t =  11.16 s / 4819.60 tokens per second
0.20.740.520 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  54021, progress = 1.00, t =  11.27 s / 4794.17 tokens per second
0.23.879.480 I slot print_timing: id  0 | task 0 | n_decoded =    242, tg =  80.58 t/s, tg_3s =  80.58 t/s
0.26.884.770 I slot print_timing: id  0 | task 0 | n_decoded =    483, tg =  80.39 t/s, tg_3s =  80.19 t/s
0.29.893.026 I slot print_timing: id  0 | task 0 | n_decoded =    724, tg =  80.29 t/s, tg_3s =  80.11 t/s
0.32.904.506 I slot print_timing: id  0 | task 0 | n_decoded =    965, tg =  80.23 t/s, tg_3s =  80.03 t/s
0.35.905.707 I slot print_timing: id  0 | task 0 | n_decoded =   1205, tg =  80.18 t/s, tg_3s =  79.97 t/s
0.38.910.402 I slot print_timing: id  0 | task 0 | n_decoded =   1445, tg =  80.13 t/s, tg_3s =  79.88 t/s
0.41.919.359 I slot print_timing: id  0 | task 0 | n_decoded =   1685, tg =  80.07 t/s, tg_3s =  79.76 t/s
0.44.931.096 I slot print_timing: id  0 | task 0 | n_decoded =   1925, tg =  80.03 t/s, tg_3s =  79.69 t/s
0.47.833.210 I slot print_timing: id  0 | task 0 | prompt eval time =   11403.79 ms / 54025 tokens (    0.21 ms per token,  4737.46 tokens per second)
0.47.833.212 I slot print_timing: id  0 | task 0 |        eval time =   26956.97 ms /  2156 tokens (   12.50 ms per token,    79.98 tokens per second)
0.47.833.213 I slot print_timing: id  0 | task 0 |       total time =   38360.76 ms / 56181 tokens
0.47.833.217 I slot print_timing: id  0 | task 0 |    graphs reused =       2147
0.47.835.579 I slot      release: id  0 | task 0 | stop processing: n_tokens = 56180, truncated = 0

SAIFIINDUSTRIES/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-noMTP-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B NVFP4 noMTP GGUF

SAIFIINDUSTRIES/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-noMTP-GGUF