Views
No views yet
roberta-base (vocab size: 50265)އީދުގެ ހަރަކާތްroberta-base1Tokens: ['Þ', 'ĩ', 'Þ', '©', 'Þ', 'ĭ', 'Þ', 'ª', 'Þ', 'İ', 'Þ', '¬', 'Ġ', ...]
2Token IDs: [49944, 6382, 49944, 15375, 49944, 13859, ...]Dhivehi Extended1Tokens: ['އީދުގެ', 'Ġ', 'ހަރަކާތް']
2Token IDs: [59825, 1437, 53039]1Tokens: ['The', 'Ġquick', 'Ġbrown', 'Ġfox', 'Ġjumps', 'Ġover', 'Ġthe', 'Ġlazy', 'Ġdog', '.']
2Token IDs: [133, 2119, 6219, 23602, 13855, 81, 5, 22414, 2335, 4]roberta-base.1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("alakxender/dhivehi-roberta-tokenizer-extended")
4tokens = tokenizer.tokenize("އީދުގެ ހަރަކާތް")
5print(tokens)