BananaMind 2 Micro is a 2,933,193-parameter base causal language model
trained from scratch on 74,998,349,824 tokens. It is a pretrained completion
model, not an instruction-tuned chat model.
Highlights
2.93M parameters with tied input and output embeddings
4,096-token context window
2,048-token custom tokenizer
GQA, pre-RoPE QK normalization, SwiGLU, and RMSNorm
XSA refresh path with detached attention readout, causal depthwise gating,
and token-embedding reinjection
Trained with stock PyTorch Muon for matrix parameters and AdamW for
embeddings and one-dimensional parameters
Benchmarks
All results below use the final checkpoint. BananaMind Base Bench 1.1 and
ArithMark use raw continuation likelihood rather than generated answers.
Benchmark
Metric
Score
BananaMind Base Bench 1.1
Overall Elo
874
BananaMind Base Bench 1.1
Raw accuracy
34.57% (121/350)
BananaMind Base Bench 1.1
Weighted accuracy
32.39%
ArithMark 2.0
Accuracy
24.92% (623/2,500)
ArithMark 3.0
acc_norm
34.00% (340/1,000)
ArithMark 3.0
Raw accuracy
33.70% (337/1,000)
Base Bench category Elo
Category
Elo
Language completion
898
Commonsense
860
World knowledge
899
Context tracking
815
Quantitative
837
Logical reasoning
911
Code completion
927
Small-model comparison
ARC, PIQA, HellaSwag, and ArithMark 3 values are acc_norm. ArithMark 2
uses raw accuracy. CMA-1M-Mini was not run on either ArithMark version, so
those cells are intentionally blank.
Base Bench values for GPT-S2-5M and CMA-1M-Mini come from the BananaMind
Base Bench leaderboard. GPT-S-5M was not listed there when this card was
prepared, so its complete official local Base Bench 1.1 result is shown.
Parameter efficiency
The comparison chart uses raw BananaMind Base Bench 1.1 accuracy and removes
the four-choice random baseline before normalizing by model size:
The result is accuracy percentage points above random per 100K parameters.
For example, 30% raw accuracy becomes 5 points above random before the size
normalization. Higher is better.
This is a narrow parameter-efficiency measurement, not a complete model
quality score. The chart can be regenerated with:
python make_parameter_efficiency_chart.py
Architecture
Field
Value
Parameters
2,933,193
Layers
9
Hidden size
128
Intermediate size
512
Query / KV heads
4 / 2
Head dimension
32
Context length
4,096
Vocabulary
2,048, tied
RoPE theta
100,000
Attention
GQA with pre-RoPE QK norm
MLP
SwiGLU
Normalization
RMSNorm
Refresh kernel
Strictly causal depthwise convolution, kernel 9
The refresh path reads the detached attention output as a signal, combines a
learned projection with a strictly causal depthwise convolution, and gates a
projection of the original token embedding. Its learned residual scalar was
initialized to zero. During cached generation, convolution history is carried
alongside each layer's key/value cache.
Training
Field
Value
Tokens seen
74,998,349,824
Optimizer for matrices
Stock torch.optim.Muon
Peak Muon learning rate
0.05
Optimizer for embeddings / 1D parameters
AdamW
Peak AdamW learning rate
0.004
Precision
bfloat16 autocast
Hardware
8 x NVIDIA RTX PRO 6000 Blackwell Server Edition
Training time
11,973 seconds (3h 19m 33s)
Curriculum
Token range
FineWeb-Edu
DCLM
Cosmopedia v2
FinePhrase
FineMath
NPSet2
0.00B-15.00B
55%
30%
6%
4%
4%
1%
15.00B-30.00B
48%
26%
9%
8%
7%
2%
30.00B-48.75B
42%
20%
12%
12%
11%
3%
48.75B-63.75B
36%
16%
14%
15%
15%
4%
63.75B-75.00B
30%
12%
15%
18%
21%
4%
The exact source revisions and observed source-token totals are preserved in
checkpoint_metadata.json. The tokenizer corpus and revisions are preserved
in tokenizer_training_manifest.json.
Usage
This repository contains custom Transformers architecture code. Review
configuration_bananamind2micro.py and modeling_bananamind2micro.py before
enabling trust_remote_code.
