You can deploy this model and start earning money today!

Views

No views yet

BananaMind-2-Mini

Name: BananaMind-2-Mini
Availability: InStock

BananaMind-2-Mini is a small decoder-only causal language model trained from scratch by BananaMind on a 30B-token curriculum. It is our first model in the BananaMind 2 Series!

The model has 25,178,752 parameters, a 4,096 token context window, and a custom 8k-token digit-aware byte-level BPE tokenizer.

Model Details

Field	Value
Parameters	25,178,752
Architecture	BananaMind2Mini decoder-only Transformer
Layers	14
Hidden size	384
Intermediate size	1,024
Attention heads	6
KV heads	2
Head dim	64
Attention style	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	100,000
Normalization	RMSNorm
RMSNorm epsilon	1e-6
Vocab size	8,192
Context length	4,096
Embeddings	Tied input/output embeddings
Weight format	safetensors
HF architecture	`BananaMind2MiniForCausalLM`
HF model type	`bananamind2_mini`
Final checkpoint	`runs/bananamind2-mini/final.pt`
Final training step	55,485
Tokens seen	29,999,726,592

Credits to AxiomicLabs and GPT X2 125M for the architecture inspiration.

Tokenizer

BananaMind-2-Mini uses a custom 8k byte-level BPE tokenizer trained from FineWeb-Edu text with digit-aware pre-tokenization.

Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.

Digit IDs:

Token	ID
`0`	19
`1`	20
`2`	21
`3`	22
`4`	23
`5`	24
`6`	25
`7`	26
`8`	27
`9`	28

Examples:

text

18  -> [20, 27]
227 -> [21, 21, 26]

Special token IDs:

Token	ID
`<pad>`	0
`<bos>`	1
`<eos>`	2
`<unk>`	3

Training Data

BananaMind-2-Mini was trained on a 30B-token mix of web, educational, synthetic textbook, and math data.

Dataset	Target Tokens	Share
FineWeb-Edu	16.5B	55%
DCLM	9.0B	30%
Cosmopedia-v2	3.0B	10%
FineMath-4+	1.5B	5%
Total	30.0B	100%

The run used a progressive curriculum rather than sampling the final aggregate mix from the first token.

Phase	Token Range	Mix
Web-heavy start	0B to 7.2B	60% FineWeb-Edu, 38% DCLM, 1% Cosmopedia-v2, 1% FineMath-4+
Curriculum ramp	7.2B to 8.0B	Ramps toward more synthetic and math data
Main mix	8.0B to 18.0B	55% FineWeb-Edu, 32% DCLM, 8% Cosmopedia-v2, 5% FineMath-4+
Aggregate taper	18.0B to 23.2B	Tapers toward the final aggregate target
Final mix	23.2B to 30.0B	50.957% FineWeb-Edu, 20.766% DCLM, 20.043% Cosmopedia-v2, 8.234% FineMath-4+

Training Setup

Field	Value
Sequence length	4,096
Micro batch	12
Gradient accumulation	11
Effective batch	132 sequences
Tokens per optimizer step	540,672
Planned optimizer steps	55,486
Actual final step	55,485
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	2.3e-3
Warmup steps	1,750
LR schedule	Warmup-stable-decay with cosine decay
Decay ratio	0.15
Weight decay	0.1, then 0.01 after 12B tokens
Gradient clipping	1.0
Z-loss coefficient	1e-4 until 12B tokens, then off
Compile	PyTorch compile enabled
Seed	1337

Evaluation

Self-reported benchmark scores using lm_eval. Scores may vary a bit depending on harness version, runtime settings, dtype, and evaluation environment.

All task scores below use acc_norm,none. Average is the mean over ARC Easy, PIQA, ARC Challenge, and HellaSwag.

Model	Average	ARC Easy	PIQA	ARC Challenge	HellaSwag
BananaMind-2-Mini	38.72	39.86	59.63	25.68	29.72
Zupra-1.6-50M-Instruct-Ultra-exp	39.47	43.14	59.47	25.60	29.68
BananaMind-1.5-Base	39.46	42.47	60.61	23.98	30.77
MiniBananaMind-v4-9M	35.07	34.97	55.39	23.04	26.87
Pythia-31M	34.79	34.01	56.47	21.42	27.28

Higher is better on the y-axis, and more parameters are farther right on the x-axis. The top-left region represents the most score-efficient models.

Repository Files

File	Description
`config.json`	Transformers config for `bananamind2_mini`
`model.safetensors`	Final exported model weights
`tokenizer.json`	Custom 8k digit-aware tokenizer
`tokenizer_config.json`	Tokenizer metadata
`generation_config.json`	Default generation config
`configuration_bananamind2mini.py`	Custom Transformers config class
`modeling_bananamind2mini.py`	Custom Transformers model class
`checkpoint_metadata.json`	Source checkpoint, step, and token metadata

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run inference:

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "BananaMind/BananaMind-2-Mini"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Suggested Generation Settings

For stable continuations:

do_sample=False
repetition_penalty=1.1
max_new_tokens=64 to 160

For more varied text:

do_sample=True
temperature=0.6 to 0.8
top_p=0.9
top_k=50
repetition_penalty=1.1
max_new_tokens=64 to 192

Intended Use

BananaMind-2-Mini is intended for lightweight language-model research, local experimentation, text continuation, tokenizer experiments, and small-model training comparisons.

Because this is a base model, prompts should be written as continuation prompts rather than chat messages.

License

Apache 2.0

🍌