Qwen3.5-122B-A10B-FP8-CacheReady
FP8-quantized Qwen3.5-122B with routing canonicalization for deterministic MoE expert selection.
This is the FP8 variant of
dystrio/Qwen3.5-122B-A10B-CacheReady, built directly from
Qwen/Qwen3.5-122B-A10B-FP8 (Qwen's official FP8 checkpoint).
Only router (gate) weight matrices were modified. In Qwen's FP8 checkpoint, gate weights are already stored in bf16 precision (excluded from quantization by Qwen). All expert weights, attention weights, embeddings, and quantization scales are byte-for-byte identical to Qwen/Qwen3.5-122B-A10B-FP8. Approximately ~45% of experts fell into equivalence groups across the model.
What CacheReady does
MoE routers choose top-k experts per token. When multiple experts produce near-identical outputs (equivalence groups), the router can flip between them due to tiny numerical differences from quantization, batch shape, or execution order. CacheReady canonicalizes router weights so equivalent experts receive identical routing scores, making expert selection deterministic by construction.
Benchmark results
All benchmarks: 2x NVIDIA H100 NVL 96GB, vLLM 0.18.0, enforce_eager=True, tensor_parallel_size=2.
Routing determinism
| Model | Texts | Determinism (FP8) |
|---|
| Original FP8 | 20 | 100% |
| FP8-CacheReady | 20 | 100% |
Quantized routing stability also verified at 100% agreement for both models.
Prefix caching throughput (FP8, 2x H100 NVL, TP=2)
| Model | Workload | Without Cache | With Cache | Cache Effect |
|---|
| Original FP8 | Shared prefix | 812 tok/s | 819 tok/s | 1.01x |
| Original FP8 | Unique prefix | 840 tok/s | 832 tok/s | 0.99x |
| FP8-CacheReady | Shared prefix | 834 tok/s | 806 tok/s | 0.97x |
| FP8-CacheReady | Unique prefix | 820 tok/s | 804 tok/s | 0.98x |
On this hardware configuration (2x H100 NVL, TP=2), the original FP8 model does not exhibit the prefix caching degradation observed in the bf16 variant. Both models show roughly neutral cache behavior.
Comparison with bf16 CacheReady (4x H100 PCIe, TP=4)
The
bf16 CacheReady variant showed a dramatic difference on 4x H100 PCIe with TP=4:
| Model | Shared-prefix cache effect |
|---|
| Original bf16 (TP=4) | 0.65x (caching hurts — 35% slower) |
| bf16 CacheReady (TP=4) | 1.31x (caching helps — 31% faster) |
The routing instability that causes prefix caching failures depends on the serving configuration. Higher tensor parallelism (TP=4), PCIe interconnect (vs NVLink), and different batch patterns can trigger expert selection flips that invalidate cached KV states. At TP=2 on NVLink, the problem doesn't manifest — but it may on larger deployments.
When does CacheReady matter?
The patch has zero quality cost and zero performance cost. It is most likely to help when:
- Higher TP counts (TP=4, TP=8) — more expert partition boundaries, more routing divergence opportunities
- PCIe interconnect — less deterministic execution order than NVLink
- Mixed-precision serving — nvfp4 quantization amplifies routing noise more than fp8
- Multi-tenant / high-concurrency — varying batch shapes across requests with shared prefixes
If your serving config already produces stable routing (as our 2x H100 NVL TP=2 bench shows), CacheReady is a free safety net. If your config triggers instability (as the bf16 TP=4 PCIe bench shows), CacheReady fixes it.
Quality preservation
No measurable quality change. The same routing canonicalization patch from the
bf16 CacheReady variant applies directly because Qwen stores gate weights in bf16 precision even in FP8 checkpoints. The patch only modifies router weight rows for experts that produce near-identical outputs.
Usage
Drop-in replacement for Qwen/Qwen3.5-122B-A10B-FP8. No code changes needed.
With vLLM
1python -m vllm.entrypoints.openai.api_server \
2 --model dystrio/Qwen3.5-122B-A10B-FP8-CacheReady \
3 --tensor-parallel-size 2 \
4 --enable-prefix-caching \
5 --trust-remote-code
With transformers
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained(
4 "dystrio/Qwen3.5-122B-A10B-FP8-CacheReady",
5 torch_dtype="auto",
6 device_map="auto",
7 trust_remote_code=True,
8)
9tokenizer = AutoTokenizer.from_pretrained("dystrio/Qwen3.5-122B-A10B-FP8-CacheReady")
Compatibility
- vLLM >= 0.17 (native FP8 support)
- SGLang
- Any framework that loads FP8 safetensors checkpoints
Related Models
| Variant | Precision | Size | Shared-prefix improvement | Link |
|---|
| CacheReady (bf16) | bf16 | ~240 GB | 0.65x → 1.31x (TP=4 PCIe) | dystrio/Qwen3.5-122B-A10B-CacheReady |
| CacheReady (FP8) | FP8 | ~120 GB | neutral at TP=2 NVLink | this model |
Citation
1@misc{dystrio_cacheready_fp8_2026,
2 title={Routing Canonicalization for Deterministic MoE Inference (FP8)},
3 author={Dystrio},
4 year={2026},
5 url={https://huggingface.co/dystrio/Qwen3.5-122B-A10B-FP8-CacheReady}
6}
