You can deploy this model and start earning money today!

aldigobbler/
smollmv2-360Mx4E-MoE-v0.1-unaligned-gates

Name: smollmv2-360Mx4E-MoE-v0.1-unaligned-gates
Availability: InStock

pytorch

Views

No views yet

Highly experimental proper MoE

!! THE GATE IS NOT ALIGNED FULLY !!

Based off of smollmv2. (Llama) MoE-ified then further trained on a general dataset.

info:

MoE layers: [8, 12, 16, 20, 24, 28] 
Top-k: 2 (activates 50.0% of experts per token) 
Hidden size: 960 
Total parameters: 494,554,560 
Trainable parameters: 494,554,560 
Auxiliary loss weight: 0.01

training loss: Total Loss = 6.4659, LM Loss = 5.9851, Aux Loss = 48.0835

val loss: Total Loss: 0.8298, LM Loss: 0.7697, Aux Loss: 6.0092

aldigobbler/smollmv2-360Mx4E-MoE-v0.1-unaligned-gates

Highly experimental proper MoE

!! THE GATE IS NOT ALIGNED FULLY !!

info:

aldigobbler/
smollmv2-360Mx4E-MoE-v0.1-unaligned-gates