NanoMind-SLM-60M
NanoMind-SLM is a compact Llama-style decoder-only Transformer trained
from random initialization on quality-filtered Python source code.
This project was built for learning and research into dataset filtering,
tokenization, Transformer architecture, pretraining, evaluation, and
small-language-model deployment.
Model Details
| Property | Value |
|---|
| Parameters | 18,808,704 |
| Architecture | Llama-style decoder-only Transformer |
| Training tokens | 62,531,072 |
| Validation tokens | 6,277,632 |
| Vocabulary size | 8,000 |
| Context length | 256 |
| Weight size | 71.8 MB |
| Framework | PyTorch |
Evaluation Results
| Metric | Result |
|---|
| Validation loss | 2.1563 |
| Perplexity | 8.64 |
| Syntax validity | 5/10 (50%) |
| Evaluated validation tokens | 1,024,000 |
Research Comparison
| Model | Training tokens | Syntax validity |
|---|
| Baseline model | ~31M | 1/10 |
| Earlier quality model | ~31M | 0/10 |
| Final quality model | 62.5M | 5/10 |
Intended Use
NanoMind-SLM is intended for:
- Educational demonstrations
- Small-language-model research
- Python code-generation experiments
- Studying training and evaluation pipelines
Limitations
This is not a production coding assistant.
The model frequently generates logically incorrect code, incomplete
docstrings, repeated statements, and invalid syntax. It should not be
used for security-sensitive or production code without human review.
It is a custom PyTorch architecture and is not directly compatible with
the standard Transformers AutoModel interface.
Files
model.pt: trained model weightsmodel.yaml: architecture configurationtraining.yaml: training configurationtokenizer/: 8,000-token BPE tokenizernanomind_slm/: model implementationfinal_metrics.json: validation resultssyntax_results.json: generation evaluationrequirements.txt: runtime dependencies
Source Code
Full training pipeline and documentation:
