This repo contains a
Byte-Pair Encoding tokenizer trained on the
Amharic subset of the
oscar dataset. It's the same as the GPT-2 tokenizer but trained from scratch on an amharic dataset with a vocabulary size of
24000.
You can load the tokenizer from huggingface hub as follows.
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("rasyosef/gpt2-oscar-amharic-tokenizer")
4tokenizer("አባይን ያላየ የፕሌን ቲኬት እችለዋለው።")
