One file that puts every routed expert of Qwen3.6-35B-A3B on a single NVIDIA DGX Spark, at the full 262,144-token native context. This is the offline-built 2-bit plane store for vLLM-Moet: all 10,240 routed experts (40 MoE layers × 256) of nvidia/Qwen3.6-35B-A3B-NVFP4, requantized to sign-symmetric 2-bit planes and serialized per-expert-slot into one O_DIRECT-servable file. The serving stack reads experts straight from this file into a GPU cache pool, so the expert tensors are never materialized in RAM. Serving and every number on this card ran on one GB10 box.


file	`qwen36-planes.moet2pf` — 8.44 GiB (9,060,192,256 bytes)
format	`MOET2PF1` v1: 4096-aligned header/table/body, per-slot sha256 (contract: `tools/plane_file.py`)
built from	`nvidia/Qwen3.6-35B-A3B-NVFP4` @ `491c2f1ea524c639598bf8fa787a93fed5a6fbce` (pinned in the header)
contents	routed-expert FFN weights only — 10,240 slots (40 × 256), 0.844 MiB/slot; the per-layer shared expert is not in this file
license	Apache-2.0 (follows the upstream checkpoint; this file is a requantization of its expert weights)

This file does not replace the checkpoint — it pairs with it. At boot vLLM still reads the upstream NVFP4 checkpoint for attention, shared experts, gates and tokenizer; this file serves only the routed experts. You need both.

The model underneath

35B total / 3B activated. 40 layers, all MoE (no dense prefix), 256 routed experts per layer with top-8 routing plus one shared expert, moe_intermediate_size 512, hidden size 2048, vocab 248,320. Attention is hybrid on a full_attention_interval of 4 — 10 full-attention layers (16 heads, 2 KV heads, head_dim 256) and 30 linear-attention (gated-delta-net-style) layers. max_position_embeddings is 262,144. Upstream the checkpoint is mixed precision: FP8 attention-side tensors, W4A16_NVFP4 group-16 expert tensors, FP8 KV cache. This repo's 2-bit planes are a further requantization applied on top of the NVFP4 experts.

Naming footnote: the checkpoint's transformers architecture is Qwen3_5MoeForConditionalGeneration (model_type: qwen3_5_moe), so logs say "Qwen3.5-MoE" while the model is Qwen3.6. That is a library naming lag, not a different checkpoint — expect it in the boot banner and in docs/gb10.md.

Measured

One GB10 / DGX Spark (compute capability 12.1, aarch64 Grace host, 119.7 GiB unified memory shared host+device), driver 580.95.05 / CUDA 13.0, official vLLM v0.24.0 plus this repo's overlay patch, hand-written SM120 SASS cubins (built for compute_cap 12.0, loaded and run on sm_121). Plane file served O_DIRECT into a 10 GiB GPU cache pool: 12136 slots x 0.84 MiB against 10,240 distinct experts, so the whole expert set is resident with headroom. Boot re-verifies the file: OK: 10240 slots verified in 3.7s (O_DIRECT).

At --max-model-len 262144 --max-num-seqs 16, the engine reports a GPU KV cache of 2,698,418 tokens — 10.29x maximum concurrency for 262,144-token requests.

Aggregate decode: median 248 tok/s (typical range 208–285, batch 16; measured over 3,661 Running: 16 reqs engine samples, full spread 139–336).
Prefill: 1,356–1,725 tok/s (from the needle runs below).
Single-stream decode: ~46–50 tok/s.

262K needle-in-a-haystack — 4/4

Ran immediately after boot on a --max-num-seqs 1 boot of the same recipe (so the prefill rates are uncontended); prefill tok/s is prompt_tokens / latency.

target	prompt tokens	depth	latency	result	prefill tok/s
~32K	30,921	0.50	19 s	PASS	1,627
~128K	124,183	0.50	72 s	PASS	1,725
~250K	242,707	0.25	179 s	PASS	1,356
~250K	242,707	0.75	179 s	PASS	1,356

MATH-500 — 377/500 = 75.4% under a two-pass 16K→32K budget protocol

HuggingFaceH4/MATH-500, all 500 problems, thinking mode, temp 0.6 / top_p 0.95, per-request seed 0, 16-way concurrency, answer = last \boxed{} after </think>. The headline number is not a single pass: pass 1 ran every item at max_tokens=16384 and scored 329/500 = 65.8% with 192 items stopping on finish_reason=length; pass 2 re-ran exactly those 192 items at max_tokens=32768 and the results were merged, giving 377/500 = 75.4%. So 75.4% is accuracy with a 32K thinking budget where 16K was not enough, and 65.8% is the straight one-pass-at-16K figure.

level	1	2	3	4	5	all
correct	39/43	76/90	86/105	97/128	79/134	377/500
accuracy	90.7%	84.4%	81.9%	75.8%	59.0%	75.4%

Cost of the two passes on the one box: 9,483,273 completion tokens (~9.5M) over 652.2 min of wall time (299.0 + 353.2 min, ~10.9 h), served by a single continuous server process. Zero request errors across all 692 generation calls.

Meeting Marathon — one hard coding problem, three inference configs

Weighted interval scheduling with a cooldown and a cap K; the generated solution is graded by 200 randomized tests plus 7 edge cases against a brute-force-validated reference DP. The perf column is the wall-clock runtime of the model's own generated code on an n=100k, K=100 case with a 60 s budget — it is not a decode-speed metric.

run	finish_reason	completion tokens	latency	grading	perf
no-think, temp 0	stop	4,332	86 s	PASS 207/207	14.4 s
think, temp 0	length (40,960 cap)	40,960	887 s	FAIL (no code block)	n/a
think, temp 0.6	length (40,960 cap)	40,960	2,545 s	PASS 207/207	1.5 s

The think-temp-0.6 run overlapped the concurrent 16-way MATH-500 sweep on the same server, so its latency is not an isolated single-stream measurement; its grading and perf number are unaffected (they measure the generated code, not the model's speed).

Use

The serving stack must exist first: clone vLLM-Moet and install official vLLM v0.24.0 with the repo's overlay patch applied — container path: the repo README's Dockerfile.sm120-v024; native GB10 sequence: docs/gb10.md. The recipe below assumes the clone lives at /workspace/vllm-Moet. If it lives elsewhere, run vllm serve from the checkout root or put <clone>/tools on PYTHONPATH — the patched store locates tools/plane_file.py via the import path, the current directory, or the literal /workspace/vllm-Moet path, and refuses to boot otherwise.

bash

# 1. plane file + upstream checkpoint
hf download 9prodhi/Qwen3.6-35B-A3B-moet2pf qwen36-planes.moet2pf --local-dir /workspace/models
hf download nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --revision 491c2f1ea524c639598bf8fa787a93fed5a6fbce

# 2. arm the file tier and serve (the env of the observed 262K boot)
export VLLM_MOE_W2=1
export VLLM_MOE_W2_DELTA=0
export VLLM_MOE_W2_BASE_FILE=/workspace/models/qwen36-planes.moet2pf
export VLLM_MOE_W2_BASE_FILE_FIRST_LAYER=0
export VLLM_MOE_W2_BASE_CACHE_GB=10
export VLLM_MOE_W2_BASE_MISS_TOL=1000000
export VLLM_MOE_W2_CUBIT_DIR=/workspace/vllm-Moet/kernels/cubins-sm120
# the observed boots additionally carried (via the harness venv):
export PYTHONPATH=/workspace/vllm-Moet/tools:$PYTHONPATH
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas   # CUDA-13 Triton/ptxas mismatch guard
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1       # set only AFTER step 1's downloads

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --revision 491c2f1ea524c639598bf8fa787a93fed5a6fbce \
  --host 127.0.0.1 --port 8010 \
  --gpu-memory-utilization 0.60 \
  --served-model-name qwen36-moet \
  --max-model-len 262144 --max-num-seqs 16

(--revision on the serve line is a post-hoc addition to the recipe, not part of the observed command: the observed boots ran offline against the pinned snapshot, so they were revision-pinned by construction. Online, without it, a future upstream push to the NVFP4 repo would silently serve weights that do not match the revision pinned in this file's header.)

VLLM_MOE_W2_BASE_FILE_FIRST_LAYER=0 is required: Qwen3.6-MoE is all-MoE and has no first_k_dense_replace in its config, so the store cannot derive the first MoE layer and refuses to boot without the explicit override.

The boots behind this card were launched through the repo harness tools/gb10/g1p_serve.sh <logfile> --served-model-name qwen36-moet --max-model-len 262144 --max-num-seqs 16, which hardcodes --host 127.0.0.1 --gpu-memory-utilization 0.60 --max-model-len 4096 --max-num-seqs 1 and appends the caller's flags after them. vLLM logs WARNING ... Found duplicate keys --max-model-len and takes the last value (262144) — harmless, but expect it.

Two quirks worth knowing before you trust the first answer:

Fire one throwaway request after boot. In tolerant miss mode (MISS_TOL=1000000) the very first request on a fresh boot returns garbage (it echoed the prompt) while the file-backed base cache is first populated; every request after that is coherent. Every eval script used for this card opens with a discarded warm-up.
No reasoning parser — thinking arrives inside content. The engine runs with reasoning_parser='', so there is no separate reasoning_content field: the <think>...</think> block is part of message.content and the answer follows the literal string </think>. Split on it (content.split("</think>")[-1]) and give max_tokens real headroom (≥256 at an absolute minimum) or the reply is cut off mid-reasoning before an answer ever appears.

bash

# warm-up (discard this response)
curl -s http://127.0.0.1:8010/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen36-moet","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' >/dev/null

# real request
curl -s http://127.0.0.1:8010/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen36-moet",
       "messages":[{"role":"user","content":"Compute 1+2+...+100. Put the final answer in \\boxed{}."}],
       "temperature":0.6,"top_p":0.95,"seed":0,"max_tokens":16384}' \
  | python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"].split("</think>")[-1])'

The same GB10 ground rules gate the boot itself: on a unified-memory box, check MemAvailable before launching — the boot transient draws from the same pool the host runs on, and a raw boot into an already-loaded box can freeze it (preflight pattern: docs/gb10.md §c; the boots behind this card went through the repo harness, not a bare vllm serve).

Stop the server with a SIGTERM (or the harness stopfile, for the *_boot.sh orchestrators in tools/gb10/). Per this repo's GB10 ground rules, never SIGKILL a server on a unified-memory box — it strands unified memory until the host reboots.

Run with your coding agent

Paste the block below into Claude Code (or any coding agent) on the target box. It is self-contained: it encodes every quirk on this card — the pinned revisions, the plane-file discovery rule, the boot guard, the warm-up discard, and the </think> parsing — so the agent lands on the working path first try and fails fast on the wrong hardware.

text

# Serve Qwen3.6-35B-A3B (2-bit expert planes) via vLLM-Moet — agent instructions

You are setting up a local OpenAI-compatible server for Qwen3.6-35B-A3B, using a
2-bit expert plane file that is read straight from disk (experts never materialize
in RAM). Follow these steps exactly; do not improvise versions or flags.

## 0. Preflight — fail fast, in order
- GPU: NVIDIA Blackwell SM120-class (DGX Spark / GB10, compute capability 12.x),
  driver for CUDA 13. If `nvidia-smi` shows anything else, STOP and report
  "unsupported hardware" — the kernels are hand-written SM120 SASS cubins.
- Disk: >= 40 GB free (8.44 GiB plane file + ~23.5 GB checkpoint + caches).
- If this is a unified-memory box (DGX Spark): check `MemAvailable` in
  /proc/meminfo before any boot, and NEVER `kill -9` a vLLM process here — it
  strands unified memory until reboot. Stop servers with SIGTERM only.

## 1. Serving stack
git clone https://github.com/9prodhi/vLLM-Moet /workspace/vllm-Moet
Install official vLLM v0.24.0 (exactly) and apply the repo's overlay patch
patch/vllm-moet-v0.24.0.patch — follow the repo README (container path:
Dockerfile.sm120-v024) or docs/gb10.md (native GB10 sequence) verbatim.
Do NOT upgrade vllm; the patch targets the v0.24.0 tag only.
Keep the clone at /workspace/vllm-Moet, or run the serve from the checkout root:
the patched store finds tools/plane_file.py only via import path, CWD, or that
literal path, and refuses to boot otherwise.

## 2. Artifacts (both required — the plane file pairs with the checkpoint)
hf download 9prodhi/Qwen3.6-35B-A3B-moet2pf qwen36-planes.moet2pf --local-dir /workspace/models
hf download nvidia/Qwen3.6-35B-A3B-NVFP4 --revision 491c2f1ea524c639598bf8fa787a93fed5a6fbce
Integrity (optional but cheap):
  sha256sum /workspace/models/qwen36-planes.moet2pf
  # must be 57793c44cfbd2e4232e7d1284ae3e008ba310eefb2b575ef5dcc75b63e17d327

## 3. Serve
export VLLM_MOE_W2=1
export VLLM_MOE_W2_DELTA=0
export VLLM_MOE_W2_BASE_FILE=/workspace/models/qwen36-planes.moet2pf
export VLLM_MOE_W2_BASE_FILE_FIRST_LAYER=0    # required: all-MoE model, no auto-derive
export VLLM_MOE_W2_BASE_CACHE_GB=10
export VLLM_MOE_W2_BASE_MISS_TOL=1000000
export VLLM_MOE_W2_CUBIT_DIR=/workspace/vllm-Moet/kernels/cubins-sm120
export PYTHONPATH=/workspace/vllm-Moet/tools:$PYTHONPATH
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1   # only AFTER step 2's downloads

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --revision 491c2f1ea524c639598bf8fa787a93fed5a6fbce \
  --host 127.0.0.1 --port 8010 \
  --gpu-memory-utilization 0.60 \
  --served-model-name qwen36-moet \
  --max-model-len 262144 --max-num-seqs 16
# Smaller boxes: reduce --max-model-len (e.g. 32768) before touching anything else.
# Healthy boot log includes: "OK: 10240 slots verified" — the file self-verifies.

## 4. First requests — two quirks, both mandatory
1) The FIRST request after boot returns garbage while the file-backed cache
   populates. Fire one throwaway and discard it:
   curl -s http://127.0.0.1:8010/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"qwen36-moet","messages":[{"role":"user","content":"hi"}],"max_tokens":8}' >/dev/null
2) There is NO reasoning parser: thinking arrives inside message.content as
   <think>...</think>. Take content.split("</think>")[-1] as the answer, give
   max_tokens real headroom (thousands, not hundreds), and use temperature 0.6 /
   top_p 0.95 for thinking mode (temp 0 thinking can loop forever).

## 5. Success criteria — report these
- Boot log shows "10240 slots verified"; server READY.
- Warm-up sent and discarded.
- A real request (e.g. "Compute 1+2+...+100, final answer in \boxed{}") returns
  coherent text after </think> containing \boxed{5050}.
Reference numbers (one GB10): ~46-50 tok/s single-stream decode, ~248 tok/s
aggregate at batch 16, full 262,144-token context. Model card:
https://huggingface.co/9prodhi/Qwen3.6-35B-A3B-moet2pf

Verify

Every slot carries a sha256 and the serving stack re-verifies the whole file at each boot (10240 slots verified in 3.7s, O_DIRECT). Offline:

python3 tools/build_plane_file.py verify --dst /workspace/models/qwen36-planes.moet2pf

The pinned checkpoint identity is in the file itself, not just in this card — plane_file.read_header() returns checkpoint_repo_id: 'nvidia/Qwen3.6-35B-A3B-NVFP4' and revision: '491c2f1ea524c639598bf8fa787a93fed5a6fbce', so you can confirm what these planes were built from without trusting the README.

Slot-table sha256: 15f85d45e91a94b8804b9122b2ad3efdfc970f4d49bcc06265a5099b96b2af94

Whole-file checksum: sha256:57793c44cfbd2e4232e7d1284ae3e008ba310eefb2b575ef5dcc75b63e17d327

Reproduce instead of downloading

The plane file is offline output of tools/build_plane_file.py, whose slot bytes are byte-identical to what the in-memory loader would stage. The recorded qwen36 build: 40 layers × 256 experts = 10,240 slots, 0.844 MiB/slot, --device cuda, 1.6 min at 107.9 experts/s, with a post-build golden check of 200 random experts byte-exact against the store-staged loader bytes.

The fully worked, documented build recipe in this repo is the DeepSeek-V4-Flash one in docs/gb10.md §b — use it as the template. The exact qwen36 build command line was not recorded anywhere, so rather than present a fabricated transcript: the invocation almost certainly took the shape below, inferred from the builder's own auto-derivation of repo-id/revision from an HF-cache snapshot path, not from a logged command.

bash

# INFERRED shape, not a recorded command — see docs/gb10.md §b for the verified DS4 example
python3 tools/build_plane_file.py build \
  --src ~/.cache/huggingface/hub/models--nvidia--Qwen3.6-35B-A3B-NVFP4/snapshots/491c2f1ea524c639598bf8fa787a93fed5a6fbce \
  --dst /workspace/models/qwen36-planes.moet2pf \
  --device cuda

To reproduce the evals: boot exactly as in Use, send the warm-up, then drive /v1/chat/completions with model: qwen36-moet. MATH-500 used the prompt suffix "Please reason step by step, and put your final answer within \boxed{}.", temp 0.6 / top_p 0.95 / seed 0, 16 concurrent workers, max_tokens 16,384 for pass 1 and 32,768 for the pass-2 rerun of the truncated items, and graded the last \boxed{} after </think> by normalized string match. The driver scripts were throwaway session harnesses and are not shipped in the repo; the protocol above is the whole contract.

Honest limitations

The headline 75.4% is budget-extended, not single-pass. One pass at 16K tokens scores 65.8%. Quote whichever matches your own token budget.
The hardest problems still run out of room. Even at 32,768 tokens, 114 of the 500 MATH-500 items still stopped on finish_reason=length, and 90 produced no extractable \boxed{} answer at all. Those all count as wrong.
Grading was pure normalized string matching. The harness has a sympy-equivalence fallback, but sympy.parsing.latex.parse_latex needs antlr4-python3-runtime, which is not installed in this environment — confirmed live: it raises ImportError: LaTeX parsing requires the antlr4 Python package, and the fallback swallows the exception and returns False. So every equivalence check silently failed and mathematically-correct answers in a different format were marked wrong. This biases the reported accuracy down, by an amount nobody has measured.
Thinking at temperature 0 can loop. The think/temp-0 Meeting Marathon run degenerated into a verbatim repeating reasoning block and burned its whole 40,960-token cap without ever emitting an answer. That is an inference-config artifact, not a plane-file defect — use temp 0.6 / top_p 0.95 for thinking mode.
The first request after boot is garbage under tolerant miss mode. See quirk 1 above. If you skip the warm-up you will see it.
No apples-to-apples comparison against the unquantized or NVFP4 checkpoints. For provenance only: NVIDIA's own card publishes a BF16-vs-NVFP4 eval table measured on GB300, which characterizes their NVFP4 checkpoint, not this 2-bit build — no equivalent suite has been run here, so there is no measured quality delta to quote for the 2-bit planes.
Sample sizes are small outside MATH-500. Needle is 4 prompts; Meeting Marathon is one problem at n=1 per configuration. The needle prefill rates come from a --max-num-seqs 1 boot and the decode figures from the batch-16 boot; they are not from the same run.

Attribution and license

Three layers of provenance, all of which apply:

Alibaba — base weights, Qwen/Qwen3.6-35B-A3B, Apache-2.0 (LICENSE).
NVIDIA — NVFP4 quantization via Model Optimizer (ModelOpt v0.44.0), released as nvidia/Qwen3.6-35B-A3B-NVFP4 on 05/28/2026. Their card states, verbatim: "This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case; see link to Non-NVIDIA (Qwen3.6-35B-A3B) Model Card from Alibaba."
This repo — vLLM-Moet requantizes that checkpoint's routed experts to sign-symmetric 2-bit planes and ships them as the single file in this repository.

License: Apache-2.0, inherited from the upstream checkpoint. This file contains requantized expert weights derived from it and carries the same terms.

9prodhi/Qwen3.6-35B-A3B-moet2pf

Qwen3.6-35B-A3B — 2-bit expert plane file (.moet2pf) for vLLM-Moet

The model underneath

Measured

262K needle-in-a-haystack — 4/4

MATH-500 — 377/500 = 75.4% under a two-pass 16K→32K budget protocol

Meeting Marathon — one hard coding problem, three inference configs

Use

Run with your coding agent

Verify

Reproduce instead of downloading

Honest limitations

Attribution and license

9prodhi/
Qwen3.6-35B-A3B-moet2pf

Qwen3.6-35B-A3B — 2-bit expert plane file (`.moet2pf`) for vLLM-Moet