MoE model based on Ling V2 MoE arch, with 32 experts, 2 activated, no shared experts, no dense layers, trained on about 100M tokens of FineWeb-2 pol-Latn split, using tokenizer taken from EuroLLM 1.7B
Just a test to validate pipeline before training bigger 4B 256 expert model on 100B tokens.
