Custom BPE Tokenizer (Trained on WikiText-103 Raw v1)

Model Overview

• Custom <cls> and <sep> special tokens.
• BPE subword segmentation for efficient handling of frequent patterns.
• Template-based post-processing for both single and paired sequences.
• Configured decoding using the BPE decoder for accurate reconstruction of original text.

Training Details

Dataset

• Name: WikiText-103 Raw v1
• Source: High-quality, long-form Wikipedia articles.
• Split Used: train
• Size: ~103 million tokens
• Loading Method: Streaming mode for efficient large-scale training without local storage bottlenecks.

Tokenizer Configuration

• Model Type: Byte-Pair Encoding (BPE)
• Vocabulary Size: 60,000 (medium-scale for general-purpose LLMs)
• Lowercasing: Enabled
• Special Tokens:
  • <cls> — Classification token
  • <sep> — Separator token
  • <unk> — Unknown token
  • <pad> — Padding token
  • <mask> — Masking token (MLM tasks)
• Post-Processing Template:
  • Single Sequence: $A:0 <sep>:0 <cls>:2
  • Paired Sequences: $A:0 <sep>:0 $B:1 <sep>:1 <cls>:2
• Decoder: BPE decoder for reconstructing original text.

Training Method

• Corpus Source: Streaming iterator from WikiText-103 Raw v1 (train split)
• Batch Size: 1000 lines per batch
• Trainer: BpeTrainer from Hugging Face tokenizers library
• Special Tokens Added: <cls>, <sep>, <unk>, <pad>, <mask>

Intended Uses & Limitations

Intended Uses

• Pre-tokenization for training Transformer-based LLMs.
• Downstream NLP tasks:
  • Language modeling
  • Text classification
  • Question answering
  • Summarization

Limitations

• Trained exclusively on English Wikipedia text — performance may degrade in informal, domain-specific, or multilingual contexts.
• May inherit biases present in Wikipedia data.

License

Citation