Grand Tokenizer - Cluster 7 (Vocab 128000)

Usage

Files

• final_normalized_tokenizer.model: SentencePiece model file
• final_normalized_tokenizer.vocab: Vocabulary file
• tokenizer.config: Tokenizer configuration
• special_tokens_map.json: Special tokens mapping

Training Details

• Cluster: 7
• Vocabulary Size: 128000
• Model Type: SentencePiece Unigram