This is a MoE model — 35B total parameters, ~3B activated per token (8 of 256 experts). The expert FFN tensors — both routed experts (*_exps) and shared experts (*_shexp), 240 tensors total — are quantized to NVFP4 (NVIDIA's 4-bit float with E4M3 block scales), repacked from the calibrated r0b0tlab/nex-n2-mini-nvfp4 checkpoint (NVIDIA ModelOpt v0.44). Because the experts dominate the model's memory footprint, NVFP4-quantizing them gives most of the size reduction; the remaining tensors (attention, linear-attention blocks, embeddings) use a conventional GGUF quant.
About LibertAI
LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.
If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.
Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the expert matmul — the dominant compute cost during MoE inference. On older GPUs the path falls back to dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.
Files
File
Size
Experts
Other tensors
When to pick
Nex-N2-mini-NVFP4-Q4_K_M.gguf
19.8 GB
NVFP4
Q4_K_M (imatrix)
Recommended — smallest and fastest for serving on Blackwell
Nex-N2-mini-NVFP4-Q8_0.gguf
20.7 GB
NVFP4
Q8_0
Higher quality non-expert tensors
Nex-N2-mini-NVFP4-BF16.gguf
22.9 GB
NVFP4
BF16
Max quality (preserves source precision for non-expert tensors)
mmproj-Nex-N2-mini-F16.gguf
903 MB
—
F16 vision tower
Required for image input — reusable with any Nex-N2-mini GGUF
Performance
Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build 85f99dca8.
Variant comparison (single-stream, llama-bench 512 in / 64 out)
Variant
Size
PP512 (tok/s)
TG64 (tok/s)
NVFP4-Q4_K_M
18.41 GiB
9514
259
NVFP4-Q8_0
19.30 GiB
10096
234
NVFP4-BF16
21.31 GiB
9678
193
Batched serving vs stock Q4_K_M (honest comparison)
Prompt processing is ~10% faster on NVFP4 at every batch size (~9900–10200 vs ~8600–9300 tok/s). Single-stream decode still slightly favors stock Q4_K_M's MMQ kernel; from 4 concurrent streams up — the realistic serving regime — the NVFP4 variant wins on total throughput while using 1.5 GiB less VRAM.
(This is the first MoE release where our NVFP4 expert path beats stock K-quants in batched serving — earlier this year the MMQ kernel still won; upstream NVFP4 MoE optimization has since closed the gap.)
Long context on a single GPU
The hybrid linear-attention architecture keeps the KV cache small (only 10 of 40 layers carry full-attention KV, 2 KV heads), so the full 256k context plus vision fits a single RTX 5090 with room to spare:
Nex-N2 uses "Agentic Thinking" with adaptive reasoning depth — the chat template enables <think> blocks by default.
These GGUFs embed a fixed chat template. The upstream nex-agi template prefills the assistant turn with '<think>' (no trailing newline) while rendering past assistant reasoning as '<think>\n…'. That inconsistency breaks llama.cpp's reasoning extraction: the parser never recognizes the forced-open think block, so the full chain-of-thought (and a stray </think>) leaks into content instead of reasoning_content — on every llama.cpp build, regardless of --reasoning-format. Other community GGUFs of this model embed the upstream template and inherit the bug. Our embedded template adds the missing newline, so reasoning_content / content separation and tool-call parsing work out of the box with stock llama-server --jinja.
Template fix: broken vs fixed API responses
About the architecture
Nex-N2-mini is built on the Qwen3.5-MoE architecture (qwen35moe in GGUF): a hybrid linear-attention MoE with 40 layers (3 of every 4 layers use linear attention, every 4th is full attention), 256 experts (8 active per token) plus a shared expert, totalling 35B parameters with ~3B active. The upstream config declares a 1-layer MTP head, but the published checkpoints do not include MTP weights, so no MTP/speculative variant can be produced from public weights. The ModelOpt source keeps attention projections, linear-attention blocks, embeddings, and lm_head at BF16 — routed + shared expert FFNs (40 layers × 6 tensors) are NVFP4. The variants above differ only in how those non-expert tensors are stored.
