BananaMind-2-SLMoE is an experimental sequence-level mixture-of-experts language model. The checkpoint theoretically contains 25.45M total parameters, but one fixed set of 13 experts is selected for an entire message, leaving 7.90M parameters active per message.

This is not a production model. It is a research checkpoint built to test whether sequence-level MoE routing works at this scale at all. A V2 is planned with routing, expert-balance, and implementation fixes.

Model summary

Property	Value
Effective size	E8M
Active parameters per message	7,902,208
Theoretical total parameters	25,449,472
Layers	8
Hidden size	256
Attention heads / KV heads	8 / 2
Experts	64
Active experts per message	13
Expert intermediate size	56
Router prefix	First 32 valid tokens
Context length	4,096
Vocabulary	8,192
Expert MLP	SwiGLU
Position encoding	RoPE, theta 100,000

The router reads a causal prefix and selects one top-13 route. That same route is then reused for every token in the message and generated response. This is different from token-level MoE models, which may select a different route for each token.

Why sequence-level MoE?

This model is a small-scale test of how this idea could work; it is research, not a production-ready model. The longer-term idea is that sequence-level routing could make very large sparse models usable on smaller machines. For example, a hypothetical 744B-total, E30B model could keep inactive experts on disk and load only its selected 30B active set into RAM for a message. With sufficient quantization, and provided the shared weights and KV cache also fit, that could potentially bring such a model within reach of a consumer PC.

The same basic offloading idea works with normal token-level MoE. The problem is that its selected experts can change at every token, so experts not already in RAM may need to be loaded from disk repeatedly during generation. Disk I/O would make that extremely slow. Sequence-level MoE chooses one expert set for the message, loads it from disk once, and reuses it for the entire response.

Expert utilization

The final checkpoint does not show full expert collapse, but expert usage is not fully balanced.

A 48-prompt routing probe found:

38 of 64 experts selected at least once
23.43 effective experts across the probe
47 unique top-13 routes across 48 prompts
normalized routing entropy of 0.758
five experts present in every tested route

The checkpoint therefore has meaningful route diversity, while still showing a persistent group of dominant experts. It should not be described as having perfect expert specialization.

Benchmarks

Model	Parameters	ARC Easy	HellaSwag	PIQA	ARC Challenge	ArithMark 3	ArithMark 2
BananaMind-2-SLMoE E8M	7.90M active / 25.45M total	33.92	27.16	55.44	23.12	33.60	26.32
BananaMind-2-Nano	9.97M	36.20	27.50	55.98	23.38	33.70	27.68
BananaMind-2-MoE	25.1M total	34.64	27.45	56.37	21.16	33.80	28.44

ARC Easy, HellaSwag, PIQA, ARC Challenge, and ArithMark 3 use normalized continuation accuracy. ArithMark 2 uses raw continuation accuracy.

We're not claiming SLMoE beats token-level MoE here. This run used ~2× the training tokens and ~4× the active compute (top-13 vs top-1) compared to BananaMind-2-MoE, so the comparison isn't controlled. The point of this checkpoint is to show that sequence-level routing trains stably at this scale — not that it's the better architecture.

BananaMind Base Bench 1.1

Metric	Result
Overall Elo	887
Accuracy	36.29% (127/350)
Weighted accuracy	33.90%
Language completion	64.00%
Commonsense	32.00%
World knowledge	48.00%
Context tracking	34.00%
Quantitative	24.00%
Logical reasoning	36.00%
Code completion	16.00%

These are research results, not guarantees of downstream quality. Scores can vary with evaluator version, precision, and batching configuration.

Usage

This repository uses custom Transformers architecture code, so trust_remote_code=True is required.

Standard RAM mode

RAM mode loads all 64 experts as regular model parameters. Computation remains sparse: only the selected 13 experts are evaluated for each message.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/BananaMind-2-SLMoE"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
    expert_storage="ram",
).to(device)

inputs = tokenizer("The color of the sky is", return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        only_use_active_experts=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Disk-backed active-expert mode

Disk mode does not register the complete expert bank as in-memory model parameters. The router runs first, only the selected expert slices are read from model.safetensors, and those slices are cached for the response.

python

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
    expert_storage="disk",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        only_use_active_experts=True,
    )

In disk mode, the full 64-expert bank stays in SafeTensors storage. The active expert slices temporarily occupy system memory and the target device while the request runs. The operating system may retain recently accessed file pages in its disk cache. Disk mode is inference-only and is most efficient with use_cache=True.

For a batch containing multiple messages, each message receives its own top-13 route. The total number of distinct experts materialized across the batch can therefore exceed 13.

Training

The checkpoint was pretrained for approximately 60B tokens with AdamW using a curriculum containing FineWeb-HQ, FineWeb-Edu, DCLM, Cosmopedia v2, FineMath, and NPSet2 data.

Limitations

This is an experimental base model, not an instruction-tuned assistant.
It is not intended for production or high-stakes use.
Expert usage remains concentrated even though full collapse was not observed.
A single route is fixed for the full response and cannot adapt token by token.
Disk-backed inference trades memory residency for per-request I/O latency.
Generated text may be incorrect, repetitive, biased, or unsafe.

License

Apache-2.0.