Views
No views yet
YunMin-tokenizer-96k.model - SentencePiece model file (2.0MB)YunMin-tokenizer-96k.vocab - Vocabulary file (2.0MB)tokenizer.json - Hugging Face tokenizer configurationtokenizer_config.json - Tokenizer configuration parametersspecial_tokens_map.json - Special tokens mappingconfig.json - Model configuration1from transformers import PreTrainedTokenizerFast
2
3# Load the tokenizer from Hugging Face Hub
4tokenizer = PreTrainedTokenizerFast.from_pretrained("mrcha033/YunMin-tokenizer-96k")
5
6# Tokenize Korean text
7text = "안녕하세요, 한국어 토크나이저입니다."
8tokens = tokenizer.tokenize(text)
9token_ids = tokenizer.encode(text)
10
11print(f"Tokens: {tokens}")
12print(f"Token IDs: {token_ids}")
13
14# Decode back to text
15decoded_text = tokenizer.decode(token_ids)
16print(f"Decoded: {decoded_text}")<unk> - Unknown token<s> - Beginning of sequence</s> - End of sequence<pad> - Padding token