A from-scratch 1.5B-parameter hybrid Mixture-of-Experts language model, pre-trained end-to-end on a single AMD MI300X (ROCm) using an MLX training stack. This is a base model — pre-training only, no instruction/chat fine-tuning yet.
Author: Geramy L. Loveless · Trained Aug 2026 · Framework: MLX (ROCm backend)
LemonSeed's efficiency comes from the hybrid: most layers are linear-attention (Gated-DeltaNet) with an O(1) recurrent state, and only every 4th layer is softmax attention with a growing KV cache. So decode is cheap and mostly constant-memory.
Per-layer state carried between steps:
GDN layers → a fixed-size recurrent state S (a [v_heads, head_dim, head_dim] matrix). Does not grow with sequence length.
Attention layers (3, 7, 11, 15, 19) → a standard KV cache (grows with context; sliding-window 256).
Prefill — process a prompt of length T (one pass)
h = embed(tokens) # [T, D]
for each layer L:
x = rmsnorm(h)
if L is attention:
q,k,v = proj(x) # GQA: 16 q-heads, 2 kv-heads
y = causal_softmax_attn(q,k,v) # RoPE, sliding window 256
cache[L].kv = (k, v) # save for decode
else: # Gated DeltaNet (linear attention)
# chunked delta-rule scan over the T tokens:
# S_t = S_{t-1} (I - beta_t k_t k_tᵀ) + beta_t v_t k_tᵀ (delta rule)
# y_t = q_t · S_t
y, S = gdn_scan(x) # O(T); keep final S
cache[L].state = S # save for decode
h = h + out_proj(y) # residual
x = rmsnorm(h)
keep = mod_gate(x) # Mixture-of-Depths: ~top-15% of tokens
r = moe(x[keep]) # routed: top-2 of 8 experts (+ shared always-on)
h = h + shared(x) ; h[keep] += r # scatter routed output back
logits = tie_embed(rmsnorm(h))[-1] # distribution for the FIRST new token
Decode — generate token t+1 from the saved state (one token in, one out)
h = embed(new_token) # [1, D]
for each layer L:
x = rmsnorm(h)
if L is attention:
q,k,v = proj(x)
cache[L].kv.append(k, v) # KV grows by 1 (window-capped at 256)
y = attend(q, cache[L].kv) # O(context) — but only ~5 layers
else: # GDN — O(1), constant memory
S = cache[L].state
S = S*(I - beta·kkᵀ) + beta·v·kᵀ # in-place delta-rule update
y = q · S # read
cache[L].state = S
h = h + out_proj(y)
x = rmsnorm(h)
if mod_gate(x): # is this token routed this layer?
h = h + shared(x) + moe_top2(x)
else:
h = h + shared(x)
logits = tie_embed(rmsnorm(h)) # sample -> next token, repeat
Why it's fast: 15 of 20 layers are GDN → O(1) per decode step, no KV growth. Only the 5 attention layers pay O(context). And MoE runs just top-2 of 8 experts per token, so ~0.46 B active params/token despite 1.5 B nominal. Net: a small KV footprint and near-constant-time decode.
Training
From scratch (random init) — no distillation, no continued-pretrain from another model.
Objective: next-token cross-entropy + load-balance aux loss + router z-loss.
Curriculum: domain-ordered (per-block loss swings 2.5–3.7 are the curriculum, not instability).
Honest capacity note ⚠️
By weight, only 68 of 160 routed experts actually trained (the rest sit near zero-init), so the effective size is ~0.90 B, not the full 1.51 B. The router dispatched tokens across all experts (balanced), but the tight 0.15 score band ran the model effectively top-1 for most of training, starving non-top experts of gradient — a classic winner-take-all collapse, worst in the early layers, full in the deep layers.
This is lost parameter budget, not lost quality — the model works well on what it trained, and MoE only runs the top-2 per token, so dead experts cost memory but no compute. It's documented transparently here rather than hidden. The next model uses a wider score band from step 0 and a per-layer expert schedule to fix it.
Using the untrained experts — expansion slots for post-training (RL / tools / agents)
Here's the useful part: those 92 untrained experts (~0.6 B of near-zero capacity) are empty expansion slots. You can deliberately fill them during post-training with new skills — tool-calling, agentic planning, chain-of-thought reasoning — without growing the model and with catastrophic-forgetting protection built in (freeze the trained base, train only the fresh experts). The inference cost stays the same (still top-2 per token).
Why this works in post-training when mid-pretrain revival did not: a converged base has no unmet demand, so any expert you re-seed is redundant and gets pruned within ~1k steps (we tried; it collapses). But a new task the model can't yet domanufactures demand — the task gradient has nowhere good to go in the existing experts, so the fresh experts find a real niche and stick. You're not reviving generic capacity; you're colonizing empty slots with a skill that needs them.
Which experts are dead (target map, from the final checkpoint)
tier
layers
dead experts (indices)
best for
fully dead (0/8 live)
0, 1, 9, 10, 13
all 8
hardest — no in-layer sibling to clone; use the shared expert as the seed
sparse (1–2 live)
2, 5, 6, 7, 8, 11
most of 0–7
best targets — has a live sibling to clone and room; a re-seeded expert finds a niche
mid (3–6 live)
3, 4
{0,1,2,4,6}, {1,2,4,5,7}
good targets — plenty of dead slots + diversity
full (7–8 live)
12, 14–19
only L12→{7}, L14→{0}
leave alone (deep layers are already fully used)
Note the dead capacity sits in the early/mid layers; the deep layers (14–19) are full. So mid-level skills (structural patterns, composition, formatting like tool-call syntax) map cleanly onto the available slots; the deep layers will consume the new mid-layer features downstream.
How to target them (recipe)
Re-seed the dead experts to a usable scale — clone a live sibling in the same layer (or the shared expert for fully-dead layers) + ~10% noise, so they start with useful features (random init just gets pruned). Also clone the router row so they get competitive routing weight.
bash
1# in the lemonseed repo — targeted, incremental, or whole-model:2python scripts/revive_experts.py IN.safetensors OUT.safetensors \3 --clone --noise=0.1 --layers=2,5,6,7,8,11 --max-per-layer=2
Widen the score band so non-top experts actually receive combine-weight + gradient (the tight 0.15 band is what starved them):
export LEMONSEED_MOE_SCORE_BAND=0.5 # true top-2
Freeze the base, train only the fresh experts — this forces the new skill into the empty slots and prevents forgetting (the trained backbone + live experts can't drift). Unfreeze everything for a short final anneal if you want.
Drive it with task demand — SFT on tool-call / agentic / reasoning traces, then RL (GRPO/PPO). The strong, specific reward signal is what makes the re-seeded experts stick and specialize.
Monitor with scripts/expert_health.py each ~1k steps — a colonized expert's w_up std should climb (staying above 1e-4) instead of decaying back. If it decays, the layer had no demand for it; move to a different layer.
Start with the sparse layers (2, 5, 6, 7, 8, 11) — they have both a live sibling to clone from and open slots, so they take most reliably. The fully-dead layers (0, 1, 9, 10, 13) are the stretch goal (seed from the shared expert) and need the strongest task signal.
Tooling for all of this — revive_experts.py (re-seed/clone), expert_health.py (measure), and the LEMONSEED_MOE_SCORE_BAND knob — ships in the lemonseed-docs repo.
Capabilities & intended use
Base model for research / continued fine-tuning. It is not instruction-tuned — it completes text, it doesn't follow chat instructions out of the box.
Arithmetic: solid with chain-of-thought / scratchpad prompting (the Qwen3.6 digit-split tokenizer helps); one-shot answers are noisier.
Good starting point for: SFT, RLHF/GRPO, long-context extension, and MoE research.
How to run
LemonSeed uses a custom architecture (hybrid GDN + MoE + MoD), so it is not loadable via transformers. Files here: model.safetensors, config.json, tokenizer.json.
▶ Primary engine — LSE (Lemon Seed Engine)
The reference runtime is LSE — Lemon Seed Engine: a modular C++ training and inference engine built for this architecture, running on the HRX native ROCm runtime. Ops record into a lazy DAG, fuse, JIT to AMDGPU code objects via amd_comgr, and dispatch through the HRX ABI (not HIP). It's the performant, first-class way to load and serve LemonSeed.
bash
1git clone https://github.com/Geramy/LSE &&cd LSE
2cmake -S . -B build -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo && cmake --build build
3# point it at model.safetensors + config.json + tokenizer.json from this repo
Reference path (Python / MLX)
For a simple/reference implementation, the MLX loader:
python
1from lemonseed.model import load_model
2from lemonseed.tokenizer import load_tokenizer
3from lemonseed.generate import generate_text
45model = load_model("model.safetensors")# + config.json alongside6tok = load_tokenizer("qwen3.6")# or point at the bundled tokenizer.json7print(generate_text(model, tok,8"Question: What is 8 × 6?\nLet's think step by step.\nAnswer:",9 max_new_tokens=64, greedy=True))
Training data attribution: the corpus includes openly-licensed educational material, notably MIT OpenCourseWare (CC BY-NC-SA). Because that source is NonCommercial, this model is intended for research / non-commercial use. If you plan commercial use, review the data licensing first. Credit to MIT and the open-education community — solo foundation-model work runs on open resources.
Limitations
Small (~0.9 B effective) base model — expect hallucination, limited world knowledge, no safety tuning.
Not instruction-tuned; no chat template applied.
~half the routed experts are untrained (see capacity note) — a fine-tune (esp. one that colonizes the dead experts) can recover capacity.
Built entirely on MLX (ROCm backend) — hat tip to the MLX maintainers. Trained through a machine migration and a full-disk crash, finished clean at val loss 1.78.
