A 270M parameter decoder-only Small Language Model (SLM) pretrained on English Wikipedia and instruction-tuned for general-purpose conversational AI.
Overview
Wiki-SmartBotLM-Instruct is a compact autoregressive language model developed entirely from scratch as an independent research project. The model was designed to explore the complete lifecycle of modern language model development—from pretraining through instruction tuning—using open datasets and modern transformer architectures.
Unlike models that rely on continued pretraining of existing foundation models, Wiki-SmartBotLM-Instruct was initialized with random weights and trained using a multi-stage pipeline.
The primary goals of this project were:
Understand large language model training from first principles.
Build a compact yet capable language model.
Explore modern transformer architectures.
Demonstrate the effectiveness of instruction tuning on a pretrained base model.
Model Details
Property
Value
Model Name
Wiki-SmartBotLM-Instruct
Architecture
Decoder-only Transformer
Parameters
270 Million
Context Length
8192 Tokens
Training Framework
PyTorch
Precision
BF16 / FP16
License
MIT
Architecture
Wiki-SmartBotLM-Instruct incorporates several modern transformer improvements including:
Rotary Positional Embeddings (RoPE)
RMSNorm
SwiGLU Feed Forward Networks
Grouped Query Attention (GQA)
Flash Attention
Causal Self Attention
Mixed Precision Training
Training Pipeline
The model was trained in two stages.
Stage 1 — Pretraining
The model was pretrained on the English Wikipedia corpus using next-token prediction.
Objectives:
Learn English grammar
Learn factual knowledge
Learn encyclopedic writing style
Learn semantic relationships
Learn long-context language modeling
Stage 2 — Supervised Fine-Tuning (SFT)
Following pretraining, the model was instruction tuned using an open instruction-following dataset.
The objective of this stage was to improve:
Instruction following
Conversational ability
Question answering
Technical explanations
General helpfulness
Training Data
Pretraining
English Wikipedia (Hugging Face)
Supervised Fine-Tuning
Alpaca Instruction Dataset
Capabilities
Wiki-SmartBotLM-Instruct can perform a variety of natural language tasks including:
General Question Answering
Technical Explanations
Programming Assistance
Educational Content Generation
Text Summarization
Brainstorming
Long-form Text Generation
Basic Reasoning
Instruction Following
Benchmarks
The model was evaluated on the following text-based benchmarks:
MMLU
ARC Challenge
HellaSwag
Winogrande
BoolQ
Benchmark
WikiSmartBotLM
Gemma 3 270M IT
Gap
BoolQ
37.8
61.4 (PT) ([Google AI for Developers][1])
-23.6
ARC Challenge
24.2
28.2 ([Google AI for Developers][1])
-4.0
HellaSwag
26.3
37.7 ([Google AI for Developers][1])
-11.4
PIQA
53.5
66.2 ([Google AI for Developers][1])
-12.7
Winogrande
50.2
52.3 ([Google AI for Developers][1])
-2.1
It shows despite the training difference between Gemma3 270M and WikiSmartBotLM, It does come close to Gemma 3 in some benchmarks.
Example
Prompt
Explain what artificial intelligence is.
Response
Artificial intelligence (AI) is a branch of computer science focused on building systems capable of performing tasks that typically require human intelligence. These tasks include reasoning, learning, planning, perception, natural language understanding, and decision making. Modern AI encompasses techniques such as machine learning, deep learning, computer vision, and reinforcement learning, and is widely used in applications ranging from recommendation systems to autonomous vehicles.
Intended Uses
Wiki-SmartBotLM-Instruct is intended for:
Educational use
AI research
NLP experimentation
Local inference
Learning transformer architectures
Small-scale deployment
Prototype conversational assistants
Limitations
As a compact language model, Wiki-SmartBotLM-Instruct has several limitations.
May hallucinate factual information.
Limited reasoning compared to larger frontier models.
Knowledge is limited to the pretraining corpus.
May struggle with highly specialized or domain-specific questions.
Performance may degrade on very long conversations.
Ethical Considerations
Wiki-SmartBotLM-Instruct is intended for research and educational purposes.
Although instruction tuning improves response quality, users should independently verify information before relying on generated content in high-stakes domains such as healthcare, finance, or legal advice.
Future Work
Future versions of Wiki-SmartBotLM aim to include:
Improved factual grounding
Larger instruction datasets
Enhanced reasoning performance
Better multilingual support
Retrieval-Augmented Generation (RAG)
Tool calling support
Longer context windows
Citation
If you use Wiki-SmartBotLM-Instruct in your research, please cite:
bibtex
1@misc{wikismartbotlm2026,
2 title={Wiki-SmartBotLM-Instruct: A 270M Parameter Decoder-Only Small Language Model},
3 author={Pranav Upadhyaya},
4 year={2026},
5 howpublished={Hugging Face Model Repository}
6}
Acknowledgements
This project was made possible by the open-source AI community.
Special thanks to:
Hugging Face
PyTorch
English Wikipedia contributors
Alpaca dataset contributors
The open-source transformer research community
About the Project
Wiki-SmartBotLM-Instruct was developed as an independent research initiative to better understand modern language model development. From random initialization to pretraining and instruction tuning, every stage of the model was built to explore the practical engineering challenges of creating compact, efficient, and accessible language models.
Feedback, issues, and contributions are always welcome.
