Qwen3.6-35B-Hybrid-3.25bpw
This is a saliency-directed hybrid-precision candidate ablation for
Qwen3.6-35B-A3B. It mirrors the allocation policy of the GLM hybrid release:
the 64 highest-scoring routed experts in each of Qwen's 40 MoE layers remain
in NVIDIA ModelOpt NVFP4, while the other 192 experts per layer are stored as
NF3g32.
The repository uses the canonical release name:
Qwen3.6-35B-Hybrid-3.25bpw.
Important runtime and size notes
This is not directly loadable by stock vLLM. runtime/ contains the
tested Docker compatibility loader: it verifies hybrid.index.json,
materializes the NF3g32 tail into a disposable ModelOpt-NVFP4 cache, and then
starts vLLM normally with CUDA graphs enabled. It is not a native fused
mixed-NVFP4/NF3 vLLM kernel.
3.25bpw is the nominal routed-expert code allocation (25% four-bit and
75% three-bit codes). It is not a physical global bits-per-weight claim.
NF3g32 stores an FP16 scale per 32 values, so its physical tail storage is
3.5 bits/scalar. The safetensor payload in this release is 19.0 GiB (about
20 GB on disk), including unchanged non-expert tensors and scales. It is not
a 12 GB checkpoint.
Provenance and construction
- Parent checkpoint:
nvidia/Qwen3.6-35B-A3B-NVFP4.
- Parent ModelOpt index SHA-256:
d67403a4e9793c0ba8a136baf14b3b76ec7b32c822267978084895e07ebd8a3e.
- Expert ranking:
mateowilliam/qwen3.6-35b-a3b-reap-observations at
c0fbef14f8bedccc42048cfdf3edce0e58e35d70.
- Allocation manifest SHA-256:
873cba8b4d2cede25090e9b4774cbf5704b7296d8d994cec54f80b4e0ee1bed2.
- Package index SHA-256:
662f1cfa438c052938369f7e94521b054bcb648600f16be905f46679b130fbb1.
The public saliency artifact records the Qwen3.6 model family and the matching
40-layer × 256-expert geometry, but not an immutable source config revision.
Accordingly, this remains an ablation candidate rather than a provenance-closed
production release.
Validation performed
- A deterministic evenly-spaced sample of 240 of 23,040 NF3 tail projections
compared 251,658,240 decoded values to their parent ModelOpt-NVFP4 values:
MAE 0.001525, RMSE 0.001997, relative RMSE 0.18865.
- Four fixed first-token probes against the preserved parent server selected
the same greedy token in 4/4 cases. The mean absolute selected-token
logprob difference was 0.0852. The observed top-logprob-table symmetric
divergence was 0.0687; this is explicitly not a full-vocabulary KL result.
- The supplied Docker service completed a real chat request and an automatic
Qwen XML tool-call parsing request at a 200,000-token context setting, with
CUDA graph capture enabled.
These are conversion and serving checks only. Full-vocabulary logits/KL,
representative generation quality, a native mixed-expert runtime, and the
post-release two-Spark Terminal-Bench 2.1 / GPQA Diamond gate remain open.
Docker use
Build the included runtime image from this repository root:
1docker build -t qwen36-nf3-hybrid:local -f runtime/Dockerfile runtime
2
3HYBRID_MODEL_DIR="$PWD" \
4HYBRID_CACHE_DIR=/path/to/persistent-materialization-cache \
5docker compose -f runtime/compose.yml up -d
The service uses port 8012 and exposes the model ID
Qwen3.6-35B-Hybrid-3.25bpw. Its default configuration is a 200,000-token
window, FP8 KV cache, Qwen reasoning parsing, Qwen XML tool parsing, automatic
tool choice, prefix caching, chunked prefill, async scheduling, and CUDA
graphs. It does not request eager execution or disable CUDA graphs.
The Docker base image is the DGX Spark vLLM runtime used to validate this
release. Replace it with an equivalent vLLM environment only after verifying
ModelOpt-NVFP4 support, Qwen hybrid-attention support, and the compatibility
materializer.
Files
base-*.safetensors: unchanged parent tensors and the selected NVFP4 expert
tensors.nf3-tail-*.safetensors: custom NF3g32 codes, scales, and input scales for
the remaining expert projections.hybrid.index.json: package schema, tensor maps, and tail records.runtime/: Docker compatibility loader and its source conversion utilities.
The custom NF3g32 storage format is documented in runtime/scripts/nf3g32.py.
It is a release-local format, not an interoperable safetensors quantization
standard.
