Qwen3-MoE Dummy
Dummy Qwen3-MoE model for testing. Not trained — weights are randomly initialized.
Produces garbage output but uses the real Qwen3-MoE architecture (qwen3_moe),
so vLLM loads and serves it correctly with expert parallelism enabled.
Purpose
Validate MoE deployment plumbing without downloading a real model (~10 seconds
to load vs hours for a real MoE). Use cases:
- Multinode DP+EP (data parallel + expert parallel) deployment validation
- LeaderWorkerSet topology checks
- InferencePool and router-scheduler pod verification
- CI/CD smoke tests for MoE support
Architecture
Adapted from Qwen/Qwen3-235B-A22B with dimensions scaled down to ~20M parameters:
| Parameter | Value |
|---|
| Architecture | Qwen3MoeForCausalLM |
| Total parameters | 19.9M |
| Hidden size | 64 |
| Intermediate size | 128 |
| MoE intermediate size | 128 |
| Experts | 8 (top-2 routing) |
| Attention heads | 2 |
| KV heads | 1 |
| Head dim | 32 |
| Hidden layers | 2 |
| Tensor type | BF16 |
| Format | Safetensors |
License
Apache-2.0. Weights are randomly initialized — no trained data, no IP.
Tokenizer from Qwen/Qwen3-235B-A22B (Apache-2.0).
How this model was created
Generated with transformers.AutoModelForCausalLM.from_config() with random
initialization. No training data or pretrained weights were used.
