TinyStoriesChinese-110M is a charmingly small, toy-like language model adept at generating short, straightforward stories in Chinese. This Small Language Model (SLM) is designed to process and create text with the simplicity and innocence of children’s tales. Despite its small size, TinyStoriesChinese-110M can generate consistent stories with robust and cute grammar, showing an emerging understanding of basic concepts such as personal hygiene and illness.
Inspired by the TinyStories research, which explores the effectiveness of small language models with simplified training material, TinyStoriesChinese-110M focuses on a very narrow task. The model uses a synthetic dataset composed of stories that even a three-year-old could understand. This approach highlights the potential of smaller models to produce coherent, consistent text without the need for extensive computational resources.
Model Details:
Parameter Count: 110M.
Architecture: Standard llama2 format Transformer with 12 layers, 12 attention heads, and a hidden size of 768. The model utilizes a 1024 context window with Relative Positional Encodings (RoPE) and a vocabulary size of 5,000.
Training:
For detailed training procedures and configurations, please refer to this GitHub repository.
Hardware: Trained on an NVIDIA RTX 2080 Super with 8 GB RAM—a modest gaming rig.
Duration: 87 hours (just over 3.5 days), covering 20k iterations and processing 2G tokens.
Optimizer: AdamW, with a learning rate (lr) of 5e-4, with 1000 warm-up iterations. gradient clipping at 1.0.
Dropout: no
Batch Size: 4, configured to fit within the 8GB RAM of the 2080; gradient accumulation steps set at 128, achieving an effective 524,288 tokens per iteration as suggested by the Chinchilla paper (Chinchilla study).
Training Iterations: 20k, including a warm-up phase of 1k steps.
Summary:
TinyStoriesChinese-110M is an excellent educational tool for machine learning beginners. Its straightforward and economical nature makes it perfect for those looking to dip their toes into the world of NLP model training without the need for extensive resources.
TinyStoriesChinese-110M demonstrates how small language models can contribute to meaningful AI development with minimal time and hardware investment, making advanced NLP capabilities more accessible and lowering the entry barriers for a broader audience. Focused on simple, narrowly defined tasks, this model can generate fluent and logical text, displaying emerging logic and common sense abilities. However, the consistency and depth of these capabilities vary, indicating areas for potential enhancement. This model serves as an excellent base for hobbyists interested in exploring and expanding the boundaries of what small language models can achieve, providing a foundation for further experimentation and learning in the field of natural language processing.
