Views
No views yet
hóa → hoá, hủy → huỷ.1./spm_train \
2 --input=vien-corpus.txt \
3 --model_prefix=vien \
4 --vocab_size=64000 \
5 --user_defined_symbols_file=user_defined_symbols.txt \
6 --required_chars_file=required_chars.txt \
7 --unk_surface="<unk>" \
8 --byte_fallback=false \
9 --split_by_unicode_script=true \
10 --split_by_number=true \
11 --split_digits=true \
12 --normalization_rule_tsv=nmt_nfkc_vidiacritic.tsvspm_train is the executable file built by following installation guide in https://github.com/google/sentencepiece. Other files (user_defined_symbols.txt, required_chars.txt and nmt_nfkc_vidiacritic.tsv) are provided in this repo.vien.model and vien.vocab.1from transformers import DebertaV2Tokenizer
2
3tokenizer = DebertaV2Tokenizer(
4 vocab_file="assets/spm/vien.model",
5 do_lower_case=False,
6 split_by_punct=False,
7 bos_token="<s>",
8 eos_token="</s>",
9 unk_token="<unk>",
10 sep_token="<sep>",
11 pad_token="<pad>",
12 cls_token="<cls>",
13 mask_token="<mask>"
14)
15tokenizer.save_pretrained("assets/hf-tokenizer")assets/spm/vien.model and assets/hf-tokenizer with the correct path on your local machine.1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("levuloihust/vien-unigram-tokenizer", use_fast=False)
4tokens = tokenizer.tokenize("How are you? Thời tiết hôm nay đẹp wóa trời lun =))")
5print(tokens)
6# ['▁How', '▁are', '▁you', '?', '▁Thời', '▁tiết', '▁hôm', '▁nay', '▁đẹp', '▁wo', 'á', '▁trời', '▁lun', '▁=))']use_fast=False for the tokenizer to properly function. In case use_fast=True (default), the tokenizer cannot perform normalization (Note that in the usage example, wóa was changed to woá)