You can deploy this model and start earning money today!

AxionLab-official/
MiniBot-0.9M-Base

Name: MiniBot-0.9M-Base
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

🧠 MiniBot-0.9M-Base

Ultra-lightweight GPT-2 style language model (~900K parameters) specialized in Portuguese conversational text.

License

📌 Overview

MiniBot-0.9M-Base is a tiny decoder-only Transformer (~0.9M parameters) based on the GPT-2 architecture, designed for efficient text generation in Portuguese.

This is a base (pretrained) model — trained purely for next-token prediction, with no instruction tuning or alignment of any kind. It serves as the foundation for fine-tuned variants such as MiniBot-0.9M-Instruct.

🎯 Key Characteristics

Attribute	Detail
🇧🇷 Language	Portuguese (primary)
🧠 Architecture	GPT-2 style (Transformer decoder-only)
🔤 Embeddings	GPT-2 compatible
📉 Parameters	~900K
⚙️ Objective	Causal Language Modeling (next-token prediction)
🚫 Alignment	None (base model)

🏗️ Architecture

MiniBot-0.9M follows a scaled-down GPT-2 design:

Token embeddings + positional embeddings
Multi-head self-attention
Feed-forward (MLP) layers
Autoregressive decoding

Despite its small size, it preserves the core inductive biases of GPT-2, making it well-suited for experimentation and educational purposes.

📚 Training Dataset

The model was trained on a Portuguese conversational dataset focused on language pattern learning.

Training notes:

Pure next-token prediction objective
No instruction tuning (no SFT, no RLHF, no alignment)
Lightweight training pipeline
Optimized for small-scale experimentation

💡 Capabilities

✅ Strengths

Portuguese text generation
Basic dialogue structure
Simple prompt continuation
Linguistic pattern learning

❌ Limitations

Very limited reasoning ability
Loses context in long conversations
Inconsistent outputs
Prone to repetition or incoherence

⚠️ This model behaves as a statistical language generator, not a reasoning system.

🚀 Getting Started

Installation

pip install transformers torch

Usage with Hugging Face Transformers

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "AxionLab-official/MiniBot-0.9M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "User: Me explique o que é gravidade\nBot:"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    temperature=0.8,
    top_p=0.95,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚙️ Recommended Settings

Parameter	Recommended Value	Description
`temperature`	`0.7 – 1.0`	Controls randomness
`top_p`	`0.9 – 0.95`	Nucleus sampling
`do_sample`	`True`	Enable sampling
`max_new_tokens`	`30 – 80`	Response length

💡 Base models generally benefit from higher temperature values compared to instruct variants, since there is no fine-tuning to constrain the output distribution.

🧪 Intended Use Cases

Use Case	Suitability
🧠 Fine-tuning (chat, instruction, roleplay)	✅ Ideal
🎮 Prompt playground & experimentation	✅ Ideal
🔬 Research on tiny LLMs	✅ Ideal
📉 Benchmarking small architectures	✅ Ideal
⚡ Local / CPU-only applications	✅ Ideal
🏭 Critical production environments	❌ Not recommended

⚠️ Disclaimer

Extremely small model (~900K parameters)
Limited world knowledge and weak generalization
No safety or alignment measures
Not suitable for production use

🔮 Future Work

🎯 Instruction-tuned version → MiniBot-0.9M-Instruct
📚 Larger and more diverse dataset
🔤 Tokenizer improvements
📈 Scaling to 1M–10M parameters
🧠 Experimental reasoning fine-tuning

📜 License

Distributed under the MIT License. See LICENSE for more details.

👤 Author

Developed by AxionLab 🔬

_{MiniBot-0.9M-Base · AxionLab · MIT License}