Views
No views yet
transformers for the native HF implementation.| Hyperparameter | Value | Original MiMo |
|---|---|---|
num_hidden_layers | 5 | 48 |
layer_types | [full, sliding×4] | matches pattern |
mlp_layer_types | [dense, sparse×4] | matches pattern (layer 0 dense, rest MoE) |
hidden_size | 2048 | 4096 (ratio 2.0) |
intermediate_size | 8192 | 16384 (ratio 2.0) |
moe_intermediate_size | 1024 | 2048 (ratio 2.0) |
num_attention_heads / num_key_value_heads | 16 / 1 | 64 / 4 (ratio 4.0) |
head_dim / v_head_dim | 192 / 128 | 192 / 128 |
n_routed_experts / num_experts_per_tok | 64 / 2 | 256 / 8 (ratio 4.0) |
| parameters | 2.34B | 300B |