This model card provides a detailed overview of the BantamForCausalLM model, a transformer-based architecture designed for adaptive, efficient language modeling through hybrid dense and sparse computation.
Model Details
Model Description
The Bantam model is a 20-layer causal language model combining dense Transformer blocks with Mixture-of-Experts (MoE) layers. It features layer-wise dynamic attention, progressive context scaling, and grouped multi-query attention, designed for efficient large-scale language modeling with balanced compute utilization.
Model type: Causal Language Model (Transformer-based)
Language(s): Multilingual (55 languages, see dataset summary below)
Model Sources
Paper: Pending publication
Demo: Coming soon
Uses
Direct Use
The Bantam model can be used directly for text generation, completion, summarization, and instruction following. It supports context windows up to 2048 tokens and operates efficiently on GPUs using bfloat16 precision.
Downstream Use
Bantam can be fine-tuned for downstream NLP tasks, such as translation, dialogue modeling, educational content generation, or creative writing. Its multilingual and mixed-domain dataset allows flexible adaptation.
Out-of-Scope Use
The model is not intended for high-stakes or safety-critical domains, such as legal, medical, or financial decision-making. It should not be used for generating misinformation or biased outputs without human oversight.
Bias, Risks, and Limitations
Bantam inherits biases from its multilingual datasets, which include content from the internet, curated knowledge bases, and open-source text corpora. It may underperform on underrepresented languages or dialects.
Additionally, as a relatively small model (≈285M parameters), hallucinations and factual inaccuracies are expected — especially when reasoning beyond the scope of the training data.
Recommendations
Users should implement output filtering, content moderation, and continuous evaluation on domain-specific benchmarks to identify and mitigate bias or performance issues.
Model Capabilities
Bantam demonstrates strong multilingual competence across 55 languages and is capable of generating informative, coherent, and contextually aware text in each of them.
The model was designed to leverage many small attention heads in early layers to capture linguistic and grammatical structures, transitioning to larger, more abstract reasoning in later layers. This design improves logical coherence and narrative flow across diverse languages despite the model’s compact size.
How to Get Started with the Model
Before loading the model, install the Bantam CLI:
pip install bantam-cli
if you want to inference it directly via the bantam-cli you can run:
bantam-cli chat --model Theoistic/Bantam-285m
or initialize the model in Python:
python
1import bantam # lazy imports2import bantam.tokenization_bantam # registers BantamFastTokenizer with AutoTokenizer3import bantam.modeling_bantam # registers config/model with AutoConfig/AutoModel456from transformers import AutoModelForCausalLM, AutoTokenizer
789tokenizer = AutoTokenizer.from_pretrained("Theoistic/Bantam-285m")10model = AutoModelForCausalLM.from_pretrained("Theoistic/Bantam-285m")111213prompt ="Once upon a time,"14inputs = tokenizer(prompt, return_tensors="pt")151617# Remove unsupported keys (like token_type_ids) before generation18if"token_type_ids"in inputs:19del inputs["token_type_ids"]202122outputs = model.generate(**inputs, max_length=100)23print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
The model was trained on the Bantam Dataset, a multilingual and multi-domain collection of JSONL files designed to support general-purpose language modeling. It includes content from knowledge bases, refined educational text, tiny fictional stories, and curated data for linguistic diversity.
Others: Arabic, Hebrew, Amharic, Turkish, Finnish, Korean, Japanese, Swahili, Vietnamese, Thai, Greek, Persian, and more.
Note: The dataset itself is not publicly released; this summary represents the linguistic and structural diversity of the data used for training.
Training Procedure
Preprocessing
A large portion of the training data was deduped, normalized and categorized by field of study, language and preprossed in to feature rich dense articles using larger models
to format in to concise, detailed markdown articles.
The millions of articles provided where suffled in languages making sure larger domain or lingustic features did not overshadow
or any low resource language impact happened due to catastrophic forgetting.
Training Hyperparameters
Parameters: 285 million
Precision: bfloat16 mixed precision
Optimizer: AdamW with weight decay
Batch size: 2048 tokens per GPU
Learning rate schedule: Cosine decay with warmup
Context length: 2048 tokens
Speeds, Sizes, Times
Training hardware: NVIDIA RTX 5090
Training duration: ~50 hours
Checkpoint size: ~285M parameters
Evaluation
Bantam is a pretrained base model, not fine-tuned or benchmarked with external metrics. Qualitatively, it exhibits:
Strong multilingual understanding and generation across 55 languages.
Coherent reasoning and informative responses.
Expected hallucinations due to small model size.
No quantitative metrics or interpretability visualizations (e.g., heatmaps, probing, or evaluation suites) have been produced yet.
Environmental Impact
Hardware Type: NVIDIA RTX 5090
Hours used: ~50
Cloud Provider: Local compute
Compute Region: N/A (local training)
Carbon Emitted: Estimated <0.05 tCO₂eq
Technical Specifications
Model Architecture and Objective
Layer
Type
Query Heads
KV Heads
Head Dim
Groups
Intermediate Size
Window
MoE
Notes
0
Dense
12
3
64
1
2304
128
❌
Dense local linguistic encoding
1
Dense
12
3
64
1
2304
128
❌
Dense local linguistic encoding
2
Dense
12
3
64
1
2368
128
❌
Dense local attention
3
Dense
12
3
64
1
2400
–
❌
Transition layer
4
MoE
(6+3)
3
(80/96)
2
2432
256
✅
6 experts, top-2 routing
5
Dense
(6+3)
3
(80/96)
2
2368
256
❌
Hybrid attention
6
Dense
(6+3)
3
(80/96)
2
2432
256
❌
Hybrid attention
7
Dense
(6+3)
3
(80/96)
2
2368
–
❌
Expanding context
8
Dense
9
3
64/128
2
2304
256
❌
Default grouped attention
9
Dense
9
3
64/128
2
2368
256
❌
Default grouped attention
10
Dense
9
3
64/128
2
2400
256
❌
Default grouped attention
11
Dense
9
3
64/128
2
2432
256
❌
Default grouped attention
12
Dense
9
3
64/128
2
2432
–
❌
Expanding context
13
Dense
9
3
64/128
2
2400
512
❌
Logical attention expansion
14
Dense
9
3
64/128
2
2432
512
❌
Logical attention expansion
15
Dense
9
3
64/128
2
2432
512
❌
Logical attention expansion
16
MoE
9
3
64/128
2
2432
512
✅
8 experts, top-2 routing
17
MoE
9
3
64/128
2
2432
–
✅
8 experts, top-2 routing
18
Dense
9
3
64/128
2
2368
512
❌
Output stabilization
19
Dense
9
3
64/128
2
2400
–
❌
Final dense layer
Attention Group Defaults
Group
Query Heads
KV Heads
Head Dim
Group 1
3
1
128
Group 2
6
2
64
These defaults apply to all layers unless explicitly overridden in layer-specific configurations.
Objective: Causal next-token prediction
Routing: Top-2 expert routing with load-balancing loss 0.01
