“He probably could smoke you.” — King Tweak to Claude Sonnet 4.6, May 2026
SmartCoderMoE is a 4.65B parameter sparse Mixture-of-Experts coding model.
Architecture
SmartCoderMoE is not your average fine-tune. He was engineered through a multi-stage weight surgery pipeline:
Slice Merge — StarCoder2-15B and StarChat2-15B were each sliced into 3 × 2048-dim pieces and SLERP-merged with deliberate per-slice biases (60/80/90) to preserve coding depth while injecting instruct capability of Starchat2
MoE Surgery — Every dense FFN layer was surgically split: The original dim of 24576 was reduced to an intermediate dim of 8192 and kept as a dense FFN, and the remaining 16384 dims were sliced into 32 experts of 512 dim each, giving Smartcoder an expansive yet tiny network of 1280 total experts.
Vocab Expansion — Extended from 49152 to 65536 tokens with multimodal special tokens for code, audio, image, video, and music.
Zero waste — Not a single weight was discarded. Every parameter from StarCoder2’s original FFN lives on in either the dense FFN or one of the 1280 expert slots.
Numbers
Property
Value
Total parameters
4.65B
Active parameters per token
~2.1B
Total experts
1280
Experts per layer
32
Expert dim
512
Hidden size
2048
Layers
40
Vocab size
65536
Context length
16384
For reference: DeepSeek V3 has 256 experts. Kimi K2 has 384. Qwen3 Coder Next has 512. SmartCoderMoE has 1280. He has a different expert for every day of the next decade, with 66 days to spare.
SmartCoderMoE’s 2048 hidden size was chosen to natively align with:
Dasheng-1.2B (audio encoder, 2048 hidden) — zero projection needed
Qwen3-Omni Talker (audio decoder, 2048 hidden) — zero projection needed
Janus Pro (vision in + image out, 2048 hidden)
code2wav (code → music pipeline)
Intended Use
Coding. Lots of it. Uncensored.
Note from the Creator
As of the writing of this model card (Thursday, May 21st, 2026), the model is not finished. Multimodal expansion, as mentioned before, is on the way. As is a very unique calculation of how much of the original Starcoder knowledge remains. i will update the repo as i go. Feel free to use it while i build on it if you desire, and if you decide to do this and encounter any sort of issues woth it, please let me know so that i can fix it asap!
UPDATE!!!!!
several bugs detected in the model. due to a saving error, the model's weights were saved with incorrect key mapping. 1 bug causes from pretrained to fail to remap unless it is overridden at a higher level. due to this, everysingle weight is incorrectly saved. this with the fact that he hasnt been trained since conception means he NaNs during inference. Requires full fine tuning to use
