MiniBot-0.9M-Instruct is the instruction-tuned version of MiniBot-0.9M-Base, designed to follow prompts more accurately, respond to user inputs, and generate more coherent conversational outputs in Portuguese.
Built on a GPT-2 architecture (~0.9M parameters), this model was fine-tuned on conversational and instruction-style data to improve usability in real-world interactions.
🎯 Key Characteristics
Attribute
Detail
🇧🇷 Language
Portuguese (primary)
🧠 Architecture
GPT-2 style (Transformer decoder-only)
🔤 Embeddings
GPT-2 compatible
📉 Parameters
~900K
⚙️ Base Model
MiniBot-0.9M-Base
🎯 Fine-tuning
Instruction tuning (supervised)
✅ Alignment
Basic prompt-following behavior
🧠 What Changed from Base?
Instruction tuning introduced significant behavioral improvements with no architectural changes:
Feature
Base
Instruct
Prompt understanding
❌
✅
Conversational flow
⚠️ Partial
✅
Instruction following
❌
✅
Overall coherence
Low
Improved
Practical usability
Experimental
Functional
💡 The model is now significantly more usable in chat scenarios.
🏗️ Architecture
The core architecture remains identical to the base model:
Decoder-only Transformer (GPT-2 style)
Token embeddings + positional embeddings
Self-attention + MLP blocks
Autoregressive generation
No structural changes were made — only behavioral improvement through fine-tuning.
📚 Fine-Tuning Dataset
The model was fine-tuned on a Portuguese instruction-style conversational dataset composed of:
💬 Questions and answers
📋 Simple instructions
🤖 Assistant-style chat
🎭 Basic roleplay
🗣️ Natural conversations
Expected format:
User: Me explique o que é gravidade
Bot: A gravidade é a força que atrai objetos com massa...
Training strategy:
Supervised Fine-Tuning (SFT)
Pattern learning for instruction-following
No RLHF or preference optimization
💡 Capabilities
✅ Strengths
Following simple instructions
Answering basic questions
Conversing more naturally
Higher coherence in short responses
More consistent dialogue structure
❌ Limitations
Reasoning is still limited
May generate incorrect facts
Does not retain long context
Sensitive to poorly structured prompts
⚠️ Even with instruction tuning, this remains an extremely small model. Adjust expectations accordingly.
🚀 Getting Started
Installation
pip install transformers torch
Usage with Hugging Face Transformers
python
1from transformers import AutoTokenizer, AutoModelForCausalLM
23model_name ="AxionLab-official/MiniBot-0.9M-Instruct"45tokenizer = AutoTokenizer.from_pretrained(model_name)6model = AutoModelForCausalLM.from_pretrained(model_name)78prompt ="User: Me diga uma curiosidade sobre o espaço\nBot:"9inputs = tokenizer(prompt, return_tensors="pt")1011outputs = model.generate(12**inputs,13 max_new_tokens=80,14 temperature=0.7,15 top_p=0.9,16 do_sample=True,17)1819print(tokenizer.decode(outputs[0], skip_special_tokens=True))
⚙️ Recommended Settings
Parameter
Recommended Value
Description
temperature
0.6 – 0.8
Controls randomness
top_p
0.85 – 0.95
Nucleus sampling
do_sample
True
Enable sampling
max_new_tokens
40 – 100
Response length
💡 Instruct models tend to perform better at lower temperatures. Try values around 0.65 for more accurate and focused responses.
🧪 Intended Use Cases
Use Case
Suitability
💬 Lightweight Portuguese chatbots
✅ Ideal
🎮 NPCs and games
✅ Ideal
🧠 Fine-tuning experiments
✅ Ideal
📚 NLP education
✅ Ideal
⚡ Local / CPU-only applications
✅ Ideal
🏭 Critical production environments
❌ Not recommended
⚠️ Disclaimer
Extremely small model (~900K parameters)
No robust alignment (no RLHF)
May generate incorrect or nonsensical responses
Not suitable for critical production environments
🔮 Future Work
🧠 Reasoning-tuned version (MiniBot-Reason)
📈 Scaling to 1M–10M parameters
📚 Larger and more diverse dataset
🤖 Improved response alignment
🧩 Tool-use experiments
📜 License
Distributed under the MIT License. See LICENSE for more details.
