The model has 49,559,552 parameters, a 3,072-token context window, and a custom 12,288-token digit-aware byte-level BPE tokenizer. It supports system prompts, multi-turn conversations, grouped-query attention, QK normalization, RoPE, tied embeddings, and KV-cached generation.
Model Details
Field
Value
Parameters
49,559,552
Base model
BananaMind/BananaMind-2-Medium
Architecture
BananaMind2Medium decoder-only Transformer
Layers
12
Hidden size
512
Intermediate size
1,920
Attention heads
8
KV heads
2
Attention style
Grouped-query attention with QK norm
MLP
SwiGLU
Position embeddings
RoPE
Vocabulary size
12,288
Context length
3,072
Embeddings
Tied input/output embeddings
Generation cache
KV cache supported
Weight format
safetensors
Training type
Full-parameter supervised fine-tuning
Instruction Tuning
Field
Value
Dataset
HuggingFaceTB/smol-smoltalk
Dataset split
train
Dataset rows
460,341
Epochs
1
Final optimizer step
4,304
Packed tokens processed
423,037,024
Supervised assistant tokens
325,659,108
Sequence length
3,072
Micro batch
8
Gradient accumulation
4
Effective batch
32 sequences
Peak learning rate
2e-4
Warmup
100 steps
LR schedule
Constant after warmup
Optimizer
AdamW
Betas
0.9, 0.95
Weight decay
0.1
Gradient clipping
1.0
Seed
1337
System and user messages were provided as context but masked from the loss. Only assistant content and the assistant-ending EOS token contributed to the training objective. All model parameters were trainable; no adapters or LoRA modules were used.
Chat Template
The tokenizer includes a Jinja chat template for system, user, and assistant messages. Its rendered structure is:
The role markers are plain text encoded with the existing tokenizer. No additional vocabulary entries were introduced during fine-tuning.
Usage
This model uses custom architecture code, so load it with trust_remote_code=True.
pip install -U transformers safetensors torch
python
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
34model_id ="BananaMind/BananaMind-2-Medium-Chat"56tokenizer = AutoTokenizer.from_pretrained(7 model_id,8 trust_remote_code=True,9)1011device ="cuda"if torch.cuda.is_available()else"cpu"12dtype =(13 torch.bfloat16
14if torch.cuda.is_available()and torch.cuda.is_bf16_supported()15else torch.float32
16)1718model = AutoModelForCausalLM.from_pretrained(19 model_id,20 trust_remote_code=True,21 torch_dtype=dtype,22).to(device).eval()2324messages =[25{26"role":"system",27"content":"You are a concise and helpful assistant.",28},29{30"role":"user",31"content":"Write a Python function that squares a number.",32},33]3435inputs = tokenizer.apply_chat_template(36 messages,37 add_generation_prompt=True,38 return_tensors="pt",39 return_dict=True,40)41inputs ={name: tensor.to(device)for name, tensor in inputs.items()}4243with torch.inference_mode():44 output = model.generate(45**inputs,46 max_new_tokens=256,47 do_sample=False,48 repetition_penalty=1.1,49 pad_token_id=tokenizer.eos_token_id,50 eos_token_id=tokenizer.eos_token_id,51 use_cache=True,52)5354new_tokens = output[0, inputs["input_ids"].shape[1]:]55print(tokenizer.decode(new_tokens, skip_special_tokens=True))
For multi-turn chat, append each generated assistant response to messages, add the next user message, and apply the chat template again.
Suggested Generation Settings
For stable responses:
do_sample=False
repetition_penalty=1.1
max_new_tokens=128 to 256
For more varied responses:
do_sample=True
temperature=0.6 to 0.8
top_p=0.9
repetition_penalty=1.1
max_new_tokens=128 to 256
Always keep a finite generation limit. Small models can enter repetitive continuations on difficult or contradictory prompts.
Benchmarks
Self-reported scores using lm_eval, the official ArithMark 2.0 script, and the internal BananaMind Instruction Bench. Scores may vary slightly by evaluation setup.
Model
Average
HellaSwag
ARC Easy
ARC Challenge
PIQA
ArithMark 2.0
BananaMind-2-Medium-Chat
39.05
31.71
43.43
24.40
60.66
29.92
Veyra2-Apricot-50M-Base
38.81
31.28
42.47
23.29
62.13
28.96
Supra 50M Instruct
38.38
29.09
44.40
27.30
59.47
29.12
Supra 1.5 50M Instruct
38.37
29.26
43.94
26.11
59.41
29.80
ARC Easy, ARC Challenge, PIQA, and HellaSwag use acc_norm,none.
BananaMind Instruction Bench (Internal)
Model
Overall (100)
General (40)
Multi-turn (25)
System Prompts (20)
Context Recall (10)
Code (5)
BananaMind-2-Medium-Chat
38.0
22.5
56.0
25.0
60.0
80.0
Supra 1.5 50M Instruct*
21.0
27.5
20.0
5.0
30.0
20.0
Supra 1.0 50M Instruct*
15.0
27.5
0.0
10.0
20.0
0.0
* BananaMind Instruction Bench contains 100 examples: 40 General, 25 Multi-turn, 20 System Prompts, 10 Context Recall, and 5 Code. Supra 1.5 50M Instruct received conversation context through its ### Input field. Supra 1.0 50M Instruct used a copied checkpoint with 3.0x linear RoPE scaling, extending its context from 1,024 to 3,072 tokens without additional long-context training so it could fit the benchmark context.
BananaMind 2 Medium Chat benchmarks
Repository Files
File
Description
config.json
Transformers configuration
model.safetensors
Fine-tuned model weights
tokenizer.json
Custom 12,288-token tokenizer
tokenizer_config.json
Tokenizer metadata
chat_template.jinja
System/user/assistant chat template
generation_config.json
Generation configuration
configuration_bananamind2medium.py
Custom Transformers config class
modeling_bananamind2medium.py
Custom Transformers model class
sft_metadata.json
Fine-tuning provenance and token counts
banner.png
Model-card banner
benchmarks.png
Standard and internal benchmark comparison chart
Intended Use
This model is intended for small-model research, local chat experiments, instruction-tuning studies, educational demonstrations, and comparisons of compact language models.
