Model Card for SLMa-94M
SLMa-94M is a 94-million parameter causal language model trained entirely from scratch on synthetic educational data. It uses a modern Llama-style transformer architecture and is designed to maximize reasoning capability within an extreme parameter budget. This is a raw base model — no instruction tuning or alignment — intended for research, fine-tuning, and bare-metal edge deployment.
Model Details
Model Description
SLMa-94M was pretrained on the HuggingFaceTB/cosmopedia dataset following the "Phi Strategy": training on highly structured, textbook-quality synthetic data to maximize reasoning per parameter. The entire training run was completed on a Nvidia T4 GPU on Kaggle.
The model uses a Llama-style architecture with Grouped-Query Attention (GQA), SwiGLU activations, RMSNorm, and Rotary Position Embeddings (RoPE). The vocabulary size of 24,576 is deliberately divisible by 128 for Tensor Core alignment.
- Developed by: Amartya Pandey
- Model type: Causal Language Model (decoder-only transformer)
- Language(s) (NLP): English
- License: Apache 2.0
- Finetuned from model: None — trained from scratch
Model Sources
Uses
Direct Use
Text generation and language modeling research. The model can be used as-is for open-ended text generation, continuation tasks, and exploring the capabilities of small pretrained models.
Note: This model requires trust_remote_code=True as the architecture is defined in model.py in this repository.
1# Use a pipeline as a high-level helper
2from transformers import pipeline
3
4pipe = pipeline("text-generation", model="amartya-pandey/SLMa-94M", trust_remote_code=True)
5output = pipe("The history of the universe is", max_new_tokens=200)
6print(output[0]["generated_text"])
1# Load model directly
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4# Always use the custom tokenizer — do not substitute a generic one
5tokenizer = AutoTokenizer.from_pretrained("amartya-pandey/slm-tokenizer-24k")
6model = AutoModelForCausalLM.from_pretrained(
7 "amartya-pandey/SLMa-94M",
8 trust_remote_code=True,
9)
10
11inputs = tokenizer("The history of the universe is", return_tensors="pt")
12outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
13print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Downstream Use
A natural starting point for fine-tuning on instruction-following, question answering, domain-specific text generation, or any task that benefits from a small, fast, and portable language model.
Out-of-Scope Use
- Production systems requiring factual accuracy or safety guarantees — this model has no alignment training
- Tasks requiring context longer than 768 tokens
- Use with any tokenizer other than
amartya-pandey/slm-tokenizer-24k
Bias, Risks, and Limitations
- Trained exclusively on synthetic, educational-style English text; likely to underperform on informal, colloquial, or non-English content
- No instruction tuning, RLHF, or safety training of any kind — outputs are unfiltered
- Small context window (768 tokens) limits use on long-form tasks
- No formal benchmark evaluation has been conducted
Recommendations
Users should treat this as a research artifact. Fine-tune with appropriate safety considerations before any deployment. Do not use for tasks where factual correctness or safe outputs are required.
How to Get Started with the Model
See the Direct Use section above for pipeline and direct-load code snippets.
For bare-metal inference on Android or other resource-constrained devices (no Python, no PyTorch, no ONNX), see the companion repository:
SLMa-94M-baremetal.
Training Details
Training Data
Trained on
HuggingFaceTB/cosmopedia, a large-scale dataset of synthetic textbook-style text. Token sequences are packed using a custom
IsolatedPackedDataset that uses block-diagonal attention matrices to prevent cross-document attention leakage.
Training Procedure
Preprocessing
Documents are tokenized with the custom 24k BPE tokenizer (amartya-pandey/slm-tokenizer-24k) and packed into fixed-length blocks of 768 tokens using IsolatedPackedDataset.
Training Hyperparameters
- Training regime: fp16 mixed precision (
torch.autocast)
- Optimizer: AdamW (
lr=1e-3, weight_decay=0.1)
- Learning Rate Schedule: Linear warmup followed by cosine decay to
min_lr=1e-4
- Gradient Accumulation: 4 steps
- Hardware: Nvidia T4 GPU (Kaggle)
- Experiment Tracking: Weights & Biases
Evaluation
No formal evaluation has been conducted at this time. The model is released as a research artifact for the community to explore, fine-tune, and benchmark.
Testing Data, Factors & Metrics
Not evaluated.
Results
Not available.
Technical Specifications
Model Architecture and Objective
Decoder-only causal language model (next-token prediction). Architecture details:
| Hyperparameter | Value |
|---|
| Parameters | ~94.3 M |
| Transformer Layers | 12 |
| Embedding Dimension | 768 |
| Context Window | 768 tokens |
| Vocabulary Size | 24,576 (divisible by 128 for Tensor Core alignment) |
| Attention | Grouped-Query Attention — 12 Query heads, 4 KV heads |
| Activations | SwiGLU |
| Normalization | RMSNorm (pre-norm) |
| Position Embeddings | RoPE (real-valued) |
Compute Infrastructure
Hardware
Nvidia T4x2 GPU (16 GB VRAM) via Kaggle free tier.
Software
PyTorch with torch.autocast (fp16), Hugging Face transformers and safetensors, Weights & Biases for logging.
Model Card Authors
Amartya Pandey
