A high-performance custom BPE tokenizer built specifically for Assamese Large Language Models (LLMs).
This project was created as part of a larger effort to build a fully native Assamese AI ecosystem — including datasets, tokenization pipelines, and future GPT-style language models trained primarily on Assamese text.
Why I Built This
Most existing multilingual tokenizers do not properly handle Assamese.
Assamese is usually grouped together with Bengali or other Indic languages inside multilingual vocabularies. While this works at a basic level, it creates several problems:
Poor subword segmentation
Fragmented Assamese words
Unnatural token boundaries
Inefficient token compression
Reduced language modeling quality
Weak handling of Assamese morphology and suffix structures
Generic multilingual tokenizers are optimized for many languages simultaneously.
This tokenizer was built specifically for Assamese.
The goal is to:
Preserve Assamese linguistic structure
Improve token efficiency
Reduce fragmentation
Support large-scale Assamese language model training
Create a tokenizer optimized for GPT-style autoregressive transformers
Build foundational infrastructure for future Assamese AI systems
Key Features
1. Custom Assamese BPE Vocabulary
This tokenizer uses Byte Pair Encoding (BPE) trained directly on Assamese text.
Features:
Learns Assamese subwords automatically
Captures common suffixes and morphemes
Handles compound Assamese words efficiently
Reduces vocabulary redundancy
Improves token compression ratio
Vocabulary size:
VOCAB_SIZE = 50_000
2. SQLite Streaming Training Pipeline
One of the most important features of this project is the streaming training architecture.
Instead of:
loading massive text files into RAM
generating temporary files
requiring huge memory usage
this tokenizer streams data directly from SQLite.
Benefits:
Extremely memory efficient
Scales to huge datasets
Faster dataset management
Easier preprocessing workflows
Better handling of terabyte-scale corpora in the future
Streaming occurs in configurable batches:
BATCH_SIZE = 50_000
This makes the tokenizer suitable for large Assamese corpus training.
