The model has 30.2M total parameters but activates only 8.9M per token, and was initialised by sparse-upcycling DynamicMind-Mini. It keeps the same custom 8k-token byte-level BPE tokenizer with digit-aware tokenization, so inference cost per token is unchanged from the dense model while total capacity is 3.4x larger.
Model Details
Field
Value
Total parameters
30,150,912
Active parameters per token
8,917,248
Architecture
Sparse MoE Llama-style decoder
Layers
9
Hidden size
256
Routed experts per layer
14
Shared experts per layer
1
Experts activated per token
2 routed + 1 shared
Expert intermediate size
256
Attention heads
8
KV heads
2
Vocabulary size
8,192
Context length
1,024
Embeddings
Tied input/output embeddings
Weight format
safetensors
Mixture of Experts
Each block's dense MLP is replaced by one always-on shared expert plus 14 fine-grained routed experts, of which the router selects the top 2 per token. Shared + top-2 at intermediate size 256 reproduces the dense model's exact active parameter count.
Load balancing uses an auxiliary-loss-free bias (DeepSeek-V3 style): a per-expert bias steers selection toward idle experts while the combining weights come from the unbiased softmax, so balancing costs no gradient interference.
Routing is per token, per layer — a single sequence touches many different experts, and the 9 routers are independent of each other.
Tokenizer
DynamicMind-MoE uses the same digit-aware 8k tokenizer as DynamicMind-Mini.
Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.
Digit IDs:
Token
ID
1
9
2
10
3
11
4
12
5
13
6
14
7
15
8
16
9
17
0
18
Training
Field
Value
Initialisation
Sparse upcycling from DynamicMind-Mini
Tokens seen
10,008,133,632
Optimizer steps
8,484
Sequence length
1,024
Tokens per optimizer step
1,179,648
Peak learning rate
1e-4
Min learning rate
1e-5
LR schedule
Cosine decay after 200 warmup steps
Optimizer
AdamW (betas 0.9, 0.95)
Weight decay
0.1
Gradient clipping
1.0
Seed
1337
Hardware
1x RTX 3060 12GB, 39.5h
Data mixture: FineWeb-Edu sample-10BT 50.0%, Cosmopedia-v2 27.8%, FineMath-4plus 22.2%.
Embeddings, attention and norms transferred 1:1 from the dense model. The dense MLP (intermediate size 768) splits exactly into three 256-wide slices — because down_proj sums over the intermediate axis — and each expert was seeded from one slice plus small noise to break router symmetry.
Benchmarks
Self-reported results from the official BananaMind Base Bench 1.1 script, all measured with the same runner, dtype (bfloat16) and GPU.
Model
Total params
Active params
Overall Elo
BananaMind-2-Medium
55.9M
55.9M
1,037
GPT-2
124M
124M
990
BananaMind-2-Nano
12.1M
12.1M
915
DynamicMind-MoE
30.2M
8.9M
912
Pythia-14m-deduped
14M
14M
909
BananaMind-2-MoE
26.1M
—
903
DynamicMind-Mini
8.9M
8.9M
868
Detailed DynamicMind-MoE result
Category
Passed
Elo
Overall
137 / 350
912
Language completion
38 / 50
1,115
Logical reasoning
17 / 50
970
World knowledge
25 / 50
951
Commonsense
23 / 50
923
Context tracking
15 / 50
843
Quantitative
11 / 50
822
Code completion
8 / 50
793
Against the dense DynamicMind-Mini the MoE gains +44 Elo overall at identical inference cost, improving in six of seven categories. Quantitative is the exception (822 vs 837), despite FineMath being 22% of the corpus.
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. Cross-tokenizer comparisons (e.g. against GPT-2's 50k vocabulary) carry a residual tokenizer effect that mean-per-token log-probability does not fully remove.
Usage
This model uses custom architecture code, so load it with trust_remote_code=True.
Install dependencies:
pip install -U transformers safetensors torch
Run inference:
python
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
34model_id ="DedeProGames/DynamicMind-MoE"56tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)78model = AutoModelForCausalLM.from_pretrained(9 model_id,10 trust_remote_code=True,11 torch_dtype=torch.bfloat16 if torch.cuda.is_available()and torch.cuda.is_bf16_supported()else torch.float16,12).cuda().eval()1314prompt ="The meaning of life is "15input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)1617with torch.no_grad():18 output = model.generate(19 input_ids=input_ids,20 max_new_tokens=64,21 do_sample=False,22 repetition_penalty=1.1,23 pad_token_id=tokenizer.eos_token_id,24 eos_token_id=tokenizer.eos_token_id,25)2627print(tokenizer.decode(output[0], skip_special_tokens=True))
Limitations
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 8.9M active parameters it reproduces register and structure well (encyclopedic text reads encyclopedic, code keeps valid indentation) but is frequently wrong on facts and arithmetic. Keep a finite generation limit and do not use it for high-stakes decisions.
