Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
Nepali_BPE_Tokenizer – AI Model by raygx | AlphaNeural AI
You can deploy this model and start earning money today!
raygx
/
Nepali_BPE_Tokenizer
like
0
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
This is only a tokenizer.
This tokenizer is a PreTrainedTokenizerFast which is trained on raygx/Nepali-Extended-Corpus datasets.
This tokenizer is trained from scratch using Tokenizers library.
This tokenizer uses
Model: BPE(unk_token="[UNK]")
Normalizer: normalizers.Sequence([NFD(),Strip()])
Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
Post-processor: BertProcessing
Code is available here.
