True W4A4 NVFP4 (ModelOpt 0.45) of XYZAILab/XYZ-Aquila-mini @ 4d3b0c45663b8b40f9e6e57b64ff815c09c49904, served natively on NVIDIA GB10 / SM121 with vLLM 0.25.0.

Native model context: max_position_embeddings = 262144 (~256k).
Served window depends on host KV (max_model_len); NIAH uses a max-context protocol (25/50/90% of served max_model_len).

Runtime

bash

docker pull ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

docker run --gpus all --ipc=host --network host \
  -e PORT=18082 \
  -e MODEL_ID=/models/model \
  -e SERVED_MODEL_NAME=xyz-aquila-nvfp4 \
  -e QUANTIZATION=modelopt_mixed \
  -e KV_CACHE_DTYPE=fp8 \
  -e ATTENTION_BACKEND=flashinfer \
  -e MOE_BACKEND=flashinfer_b12x \
  -e LINEAR_BACKEND=flashinfer_cutlass \
  -e LANGUAGE_MODEL_ONLY=1 \
  -e GPU_MEMORY_UTILIZATION=0.85 \
  -e MAX_MODEL_LEN=32768 \
  -e MAX_NUM_SEQS=16 \
  -v $PWD:/models/model:ro \
  ghcr.io/r0b0tlab/xyz-aquila-mini-nvfp4-vllm:v0.25.0-sm121-702f4814

Long-context tip: raise MAX_MODEL_LEN toward 262144 with MAX_NUM_SEQS=1 and chunked prefill (MAX_NUM_BATCHED_TOKENS can stay 16384). Host-admit before claiming full native ctx.

Native markers: FlashInferCutlassNvFp4LinearKernel, FLASHINFER_B12X MoE, FlashInfer attn arch=sm121, FP8 KV. No Marlin/emulation.

Quant recipe

Eligible MLP/experts/lm_head: NVFP4 W4A4
Attention: FP8
KV cache at serve: FP8
Excluded: vision, routers, norms, embeddings, GDN aux, MTP (MTP off in derived config only)
Calibration: 512 windowed rows
Export: ~21 GiB, 3 shards; hf_quant 291 layers (NVFP4 161 / FP8 130)

Quality results

Suite	Score	Notes
BFCL v4 `multi_turn_base` (200)	64.00% campaign · 68.00% core-subset re-run	bfcl-eval==2025.12.17 · 0 infra errors
BFCL AST-600 micro	33.33% prior · 33.50% core-subset	multiple / parallel / parallel_multiple
AST multiple	35.0% / 35.5%	200 cases
AST parallel	46.5% / 48.5%	200 cases
AST parallel_multiple	18.5% / 16.5%	200 cases
API canaries	5/5 PASS	identity, zh math, JSON, tool, needle
NIAH MAX_CONTEXT_TEST (served M=native 262144)	3/3 PASS	depths 65,520 / 131,040 / 235,872 (25/50/90% of M−64)
NIAH (legacy served M=32768)	3/3 PASS	~8k/16k/29k — not full native capacity

Do not use diluted BFCL “Overall Acc” CSV columns when only a subset of categories was run.

Throughput (GB10, vllm bench serve)

1024 in / 256 out, ignore_eos, temp 0, 3 reps drop-first:

Conc.	Output tok/s	Total tok/s	mean TTFT ms	mean TPOT ms
1	64.6	325.4	694	12.8
2	82.4	415.3	2010	16.3
4	164.5	829.1	641	21.9
8	230.5	1162.1	1009	30.9
16	305.7	1540.9	1854	45.3

Conversion gates

Gate	Status
B1 BF16 CPU real-residency baseline (~70.2 GiB)	PASS
C W4A4 PTQ	PASS
D packed NVFP4 export audit	PASS
E SM121 native serve	PASS

Runtime repo: https://github.com/r0b0tlab/xyz-aquila-mini-nvfp4-sm121-vllm
Bench client: https://github.com/r0b0tlab/r0b0bench (core / core-subset profiles)
Evidence HTML: see runtime repo docs/evidence-report.html

r0b0tlab/
XYZ-Aquila-mini-NVFP4

XYZ-Aquila-mini NVFP4 (SM121)

Runtime

Quant recipe

Quality results

Throughput (GB10, vllm bench serve)

Conversion gates

Related

License

r0b0tlab/XYZ-Aquila-mini-NVFP4

XYZ-Aquila-mini NVFP4 (SM121)

Runtime

Quant recipe

Quality results

Throughput (GB10, vllm bench serve)

Conversion gates

Related

License

r0b0tlab/
XYZ-Aquila-mini-NVFP4