Nex-N2-Pro — OneComp AutoBit sub-4-bit (avg 3.2 bits/weight)
A
sub-4-bit, mixed-precision (2/3/4/8-bit) quantization of
nex-agi/Nex-N2-Pro (a ~397B-parameter
MoE model, ~17B active, derived from Qwen3.5-397B-A17B).
The weights average 3.2 bits/weight (~3.35 bpw effective incl. group overhead),
shrinking the model from ~740 GB (BF16) to ~161 GB so the full model fits and
serves on a single workstation with 2×96 GB GPUs.
⚠️
This checkpoint uses a custom mixed-bit MoE format and does NOT load with
stock vLLM / Transformers / llama.cpp. Serving requires the mixed-bit MoE plugin
(
mmzz164/OneCompression,
MIT) — see
How to serve. If you want a plug-and-play quant, use one of the standard
GGUF/AWQ builds instead.
TL;DR
| |
|---|
| Base model | nex-agi/Nex-N2-Pro (≈397B MoE, A17B; built on Qwen3.5-397B-A17B) |
| Method | OneComp AutoBit (ILP mixed 2/3/4/8-bit) + QEP + MSE clipping + act_order |
| Avg bits | 3.2 bpw (≈3.35 effective) |
| Size | ~161 GB (41 safetensors shards) |
| Quality (WikiText, EN) | BF16 3.773 → 4.177 PPL (+10.7%) |
| Throughput | ~18.8 tok/s on 2× RTX PRO 6000 (Blackwell) |
| Min hardware | ~192 GB total VRAM (e.g. 2×96 GB) |
Why mixed-precision sub-4-bit?
Uniform 4-bit formats (e.g. NVFP4 ≈ 4.5 bpw effective) put this 397B model at
~220–240 GB — too large for 192 GB of VRAM. Going sub-4-bit is the only way to
fit it. OneComp AutoBit spends bits where they matter (per-module ILP allocation:
8-bit for sensitive projections, 2-bit for robust/rarely-routed experts) and uses
GPTQ-family error propagation (QEP) plus MSE clipping and act_order to
hold quality at this bit budget.
Quantization details
- Bit allocation — OneComp AutoBit solves a per-module mixed-integer allocation
over {2,3,4,8}-bit GPTQ quantizers (group size 128) to hit the 3.2-bpw target,
weighting by activation-aware error. Guards: an RTN fallback for numerically
degenerate layers and a 4-bit floor for non-expert (attention) projections.
- Rounding — GPTQ with Quantization Error Propagation (QEP) (Fujitsu
Research), MSE range-clipping, and act_order (static-groups variant, so
the packed layout stays kernel-compatible).
- Post-quant surgery — members of a vLLM fused group are made bit-uniform
(low member promoted to 8-bit RTN), and the shared-expert gate/up projections are
kept in BF16 (the vLLM loader fuses and skips them otherwise).
- Kept in BF16 — vision tower, embeddings, LM head, router gates, norms.
Evaluation
WikiText perplexity, measured against the BF16 model under the same harness:
| WikiText PPL (EN) | Δ vs BF16 |
|---|
| BF16 (base) | 3.773 | — |
| This model (3.2 bpw) | 4.177 | +10.7% |
Note: perplexity is a proxy. Downstream task behavior was spot-checked (arithmetic,
code, EN/JA translation, "thinking" mode) and looked consistent with the base model,
but no formal benchmark suite was run on this quantized checkpoint.
Performance
On 2× RTX PRO 6000 (Blackwell, 96 GB each) with a custom grouped fused-MoE
Triton kernel + CUDA graphs: ~18.8 tok/s decode (greedy outputs are
bit-identical to the non-graph reference path). Throughput was measured with a tight
VRAM config (short max_model_len); longer context trades against the VRAM headroom.
Hardware requirements
- ~192 GB total VRAM (validated on 2×96 GB Blackwell). The ~161 GB of weights
leave only a small KV-cache/activation budget on 192 GB, so context length and
concurrency are limited. Cards with more VRAM relax this.
- Kernel was built/tested on CUDA 13.0 (Blackwell); other GPUs may need kernel
retuning.
How to serve
This is a custom mixed-bit MoE checkpoint. Stock loaders cannot read it. To serve:
- Install OneComp (MIT) — the
quantization/serving base.
- Add the mixed-bit grouped fused-MoE serving plugin (the part that actually loads
and runs per-expert 2/3/4/8-bit MoE — not in upstream) from
mmzz164/OneCompression:
mixed_moe.py (the mixed_gptq FusedMoEMethodBase), plus grouped_moe.py and
fused_dq_gemm.py (the Triton kernels). See
MIXED_BIT_MOE_SERVING.md.
- Serve with vLLM using pipeline-parallel across your two GPUs and the
mixed_gptq
quantization, e.g. (illustrative):
1vllm serve <path-to-this-model> \
2 --quantization mixed_gptq \
3 --pipeline-parallel-size 2 \
4 --trust-remote-code
gpu_memory_utilization, max_model_len,
CUDA-graph capture sizes) depend on your VRAM; see MIXED_BIT_MOE_SERVING.md.
If you only need the model's capabilities and not this specific compression, prefer
a standard quantization of the base model.
Limitations
- Not a standard format — requires the OneComp
mixed_gptq plugin; will not load
in vanilla vLLM / Transformers / llama.cpp / Ollama.
- High VRAM floor (~192 GB) — out of reach for most single machines.
- Vision path not validated — the base is a vision-language model; the vision
tower is kept in BF16 but the image→text path was not evaluated after quantization.
- Lossy — +10.7% WikiText PPL vs BF16. Within a "quality-preserving" band for
general use, but not lossless.
License & attribution
Licensed under Apache 2.0, inherited through the lineage:
Qwen3.5-397B-A17B (Apache 2.0) → nex-agi/Nex-N2-Pro (Apache 2.0) → this
quantization (Apache 2.0).
Per Apache 2.0, this is a derivative work with the following significant change:
post-training weight quantization to ~3.2 bits/weight using OneComp AutoBit + QEP +
MSE clipping + act_order. Please retain attribution to Qwen and Nex-AGI, and
propagate any upstream NOTICE files.
This model's capabilities, training, and behavior are entirely those of the base
model nex-agi/Nex-N2-Pro; only the weight precision was changed.
Acknowledgements
- nex-agi for Nex-N2-Pro.
- Alibaba Qwen for the Qwen3.5 foundation.
- Fujitsu Research for OneComp / OneCompression
(MIT), including the AutoBit allocator and QEP.
- Mixed-bit MoE serving kernels:
mmzz164/OneCompression (MIT).
Citation
If you use this checkpoint, please cite the base model and OneComp:
1@misc{nex-n2-pro-onecomp-3p2bit,
2 title = {Nex-N2-Pro — OneComp AutoBit sub-4-bit (3.2 bpw) quantization},
3 note = {Derivative of nex-agi/Nex-N2-Pro; quantized with Fujitsu OneComp (AutoBit + QEP)},
4 year = {2026}
5}
