Model Card for Small Language Model (SLM)
Model Details
Model Description
This is a from-scratch implementation of a GPT-like transformer model designed for generating short stories. The model is trained on the TinyStories dataset and can generate coherent narratives from given prompts.
- Developed by: Vpragadeesh
- Model type: Transformer-based Language Model
- Language(s) (NLP): English
- License: See LICENSE file
- Finetuned from model: None (trained from scratch)
Model Sources
Uses
Direct Use
This model can be used to generate short stories or continue narratives from given prompts. It's particularly suited for creative writing tasks involving children's stories or simple narratives.
Downstream Use
The model can be fine-tuned for other text generation tasks or used as a base for larger language models.
Out-of-Scope Use
Not intended for production use without further validation, or for generating content that requires high factual accuracy.
Bias, Risks, and Limitations
- Biases: Trained on TinyStories dataset, which contains children's stories. May reflect biases present in that dataset.
- Risks: Generated text may contain inappropriate content or inaccuracies.
- Limitations: Limited context window (256 tokens), may not handle complex narratives well.
How to Get Started with the Model
Use the code below to get started with the model.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3tokenizer = AutoTokenizer.from_pretrained("Vpragadeesh/slm")
4model = AutoModelForCausalLM.from_pretrained("Vpragadeesh/slm")
5
6prompt = "Once upon a time there was a pumpkin."
7inputs = tokenizer(prompt, return_tensors="pt")
8outputs = model.generate(**inputs, max_length=200)
9generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
10print(generated_text)
Training Details
Training Data
The model was trained on the TinyStories dataset (roneneldan/TinyStories), which contains short stories for children.
Training Procedure
- Training regime: From scratch
- Hardware: GPU (CUDA)
- Optimizer: AdamW
- Learning rate: 1e-4 with warmup and cosine decay
- Epochs: 20000 iterations
- Batch size: 32 (with gradient accumulation)
- Mixed precision: bfloat16/float16
Evaluation
Testing Data, Factors & Metrics
- Testing Data: Validation split from TinyStories
- Factors: N/A
- Metrics: Cross-entropy loss
Model Architecture and Objective
Architecture
- Architecture Type: Transformer (GPT-style)
- Layers: 8
- Heads: 8
- Embedding Dimension: 512
- Vocabulary Size: 50257
- Context Window: 256 tokens
Objective
Language modeling objective with causal masking.
Technical Specifications
Compute Infrastructure
- Hardware: NVIDIA GPU with CUDA support
- Software: PyTorch 2.0+
Citation
If you use this model, please cite:
1@misc{slm2025,
2 title={Small Language Model (SLM)},
3 author={Vpragadeesh},
4 year={2025},
5 url={https://github.com/Vpragadeesh/slm}
6}
Model Card Authors
Vpragadeesh
Model Card Contact
For questions or issues, please open an issue on the GitHub repository.
