Vietnamese GPT-2 Tokenizer
A ByteLevel BPE tokenizer trained on the
wheevu/ct219-vietnamese-raw-400k Vietnamese corpus.
Configuration
- Model: BPE
- Vocabulary size: 30,000
- Pre-tokenizer: ByteLevel
- Decoder: ByteLevel
- Minimum frequency: 2
- Language: Vietnamese
Purpose
This tokenizer is designed to improve Vietnamese text
tokenization for GPT-2 fine-tuning.
The original GPT-2 tokenizer produces inefficient tokenization
for Vietnamese text because GPT-2 was originally trained primarily
on non-Vietnamese text.
This tokenizer is trained specifically on Vietnamese text.
Training data
Dataset:
wheevu/ct219-vietnamese-raw-400k
