Custom Unigram Tokenizer (Trained on WikiText-103 Raw v1)

Model Overview

• Custom <cls> and <sep> special tokens.
• Unigram subword segmentation for compact and efficient tokenization.
• Template-based post-processing for both single and paired sequences.
• Configured decoding using the Unigram model for accurate text reconstruction.

Training Details

Dataset

• Name: WikiText-103 Raw v1
• Source: High-quality, long-form Wikipedia articles.
• Split Used: train
• Size: ~103 million tokens
• Loading Method: Streaming mode for efficient large-scale training without local storage bottlenecks.

Tokenizer Configuration

• Model Type: Unigram
• Vocabulary Size: 25,000 (optimized for balanced coverage and efficiency)
• Lowercasing: Enabled
• Special Tokens:
  • <cls> — Classification token
  • <sep> — Separator token
  • <unk> — Unknown token
  • <pad> — Padding token
  • <mask> — Masking token (MLM tasks)
  • <s> — Start of sequence
  • </s> — End of sequence
• Post-Processing Template:
  • Single Sequence: $A:0 <sep>:0 <cls>:2
  • Paired Sequences: $A:0 <sep>:0 $B:1 <sep>:1 <cls>:2
• Decoder: Unigram decoder for reconstructing original text.

Training Method

• Corpus Source: Streaming iterator from WikiText-103 Raw v1 (train split)
• Batch Size: 1000 lines per batch
• Trainer: UnigramTrainer from Hugging Face tokenizers library
• Special Tokens Added: <cls>, <sep>, <unk>, <pad>, <mask>, <s>, </s>

Intended Uses & Limitations

Intended Uses

• Pre-tokenization for training Transformer-based LLMs.
• Downstream NLP tasks:
  • Language modeling
  • Text classification
  • Question answering
  • Summarization

Limitations

• Trained exclusively on English Wikipedia text — performance may degrade in informal, domain-specific, or multilingual contexts.
• May inherit biases present in Wikipedia data.

License

Citation