Qwen3-3B-A0.9B
This repository contains the current best local checkpoint exported from the Qwen3 MoE architecture exploration in this workspace.
Files
- Model weights in Hugging Face format
- Architecture config:
qwen3_3p1b_a0p85b_moe_30biso_4l.json
- Recovery finetune config:
recover_dialogue_qwen3_3p1b_30biso_recovery_cn_v1.yaml
- Smoke evaluation snapshot:
candidate_v1_smoke_suite.json
Architecture Summary
- Base family: Qwen3 MoE
- Hidden size:
2048
- Layers:
4
- Attention heads:
32
- KV heads:
4
- Experts:
128
- Active experts per token:
8
- MoE intermediate size:
768
- Dense intermediate size:
6144
- Dtype:
bfloat16
Current Best Local Status
This upload corresponds to the checkpoint currently documented as the best working local baseline in:
README.mddocs/stage1/qwen3_moe_4layer_recovery.md
Its practical status is:
- It can handle simple QA and part of short Chinese dialogue.
- It is not yet a fully repaired dialogue model.
- Later recovery branches did not consistently outperform this baseline.
Tokenizer
The tokenizer used during local experiments is the official Qwen tokenizer from the Qwen3-4B-Instruct release.
Tokenizer files are not re-exported in this checkpoint bundle because local training followed the same save style as the official weight export workflow.
