A chat fine-tuned Mixture of Experts (MoE) language model — small, efficient, and fully open-source.
TinyMoE is a ~100M parameter transformer using a Mixture of Experts architecture, fine-tuned via supervised fine-tuning (SFT) on a diverse blend of high-quality chat and instruction datasets. It's designed to be a compact but capable conversational AI that knows its own identity.
Unlike a standard dense transformer where every token goes through the same large feed-forward network, TinyMoE uses 8 expert sub-networks with a learned router that selects the top-2 experts per token. This means:
More total knowledge capacity without proportionally increasing compute
Sparse activation — only a fraction of parameters fire per token
Efficient inference — you get more model per FLOP
This is the same architectural family as Mixtral, but at a much smaller scale — proving that MoE works even at ~100M parameters.
Training Recipe
Stage 1 — Chat SFT
The base pretrained model was fine-tuned on a carefully curated mixture of chat datasets to teach conversational ability, instruction following, and model identity.
Hardware: NVIDIA L4 (24 GB) on Modal Framework: TRL (Transformer Reinforcement Learning) SFTTrainer Max examples: 80,000 (after length filtering)
The model was explicitly taught to know it's TinyMoE through two mechanisms:
Dedicated identity dataset — 44 custom examples covering name, creator, architecture, capabilities, and differentiation from other models (ChatGPT, Claude, Llama, etc.)
System prompt injection — 15% of all training examples received a system prompt: "You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X."
This means TinyMoE knows who it is — ask it "What's your name?" or "Who created you?" and it will answer correctly.
Usage
Chat Format
TinyMoE uses a ChatML-style template with special tokens:
<|system|>
You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X.</s>
<|user|>
What's Mixture of Experts?</s>
<|assistant|>
MoE stands for Mixture of Experts! Instead of one big neural network...</s>
Quick Start
python
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
34model_id ="FlameF0X/TinyMoE-100m-2x8-chat-stage1"56tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForCausalLM.from_pretrained(8 model_id,9 torch_dtype=torch.bfloat16,10 device_map="auto",11)1213messages =[14{"role":"system","content":"You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X."},15{"role":"user","content":"What's your name and who made you?"},16]1718inputs = tokenizer.apply_chat_template(19 messages,20 tokenize=True,21 add_generation_prompt=True,22 return_tensors="pt",23).to(model.device)2425outputs = model.generate(26 inputs,27 max_new_tokens=256,28 temperature=0.7,29 do_sample=True,30 top_p=0.9,31)3233print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Capabilities & Limitations
✅ Strengths
Efficient — MoE architecture means more capacity per inference FLOP
Conversational — trained on diverse multi-turn chat data
Self-aware — knows it's TinyMoE, not ChatGPT/Claude/Llama
Open-source — weights, architecture, and training code are all public
Fast — small enough to run on consumer hardware or free-tier GPUs
⚠️ Limitations
Small model — at 100M parameters, factual knowledge is limited compared to billion-parameter models
Stage 1 only — this is a direct-answer SFT model; it hasn't undergone RLHF/DPO alignment
No CoT — training explicitly excluded chain-of-thought reasoning traces (saved for future stages)
English only — training data was English-dominant
May hallucinate — like all LLMs, it can generate incorrect information with confidence
