Views
No views yet
PreTrainedTokenizerFast)<|pad|> <|unk|> <|bos|> <|eos|><|im_start|> <|im_end|>
<|system|> <|user|> <|assistant|><|speech|> <|frame_start|> <|frame_end|><|reserved_1|> ... <|reserved_80|>NFKCByteLevelByteLevelDecoderBpeTrainer50,000pip install transformers tokenizers huggingface_hub1from transformers import PreTrainedTokenizerFast
2
3tokenizer = PreTrainedTokenizerFast.from_pretrained("Humair332/gpt-style-50k-tokenizer")1text = "Hello world! آپ کیسے ہیں؟"
2
3encoded = tokenizer(text)
4print(encoded["input_ids"])1decoded = tokenizer.decode(encoded["input_ids"])
2print(decoded)1<|im_start|><|system|>
2You are a helpful assistant.
3<|im_end|>
4
5<|im_start|><|user|>
6ہیلو، آپ کیسے ہیں؟
7<|im_end|>
8
9<|im_start|><|assistant|>| Metric | English | Urdu |
|---|---|---|
| Chars/Token | ~5.1 | ~3.9 |
| Tokens/Word | ~1.0 | ~1.2 |
| Lossless Decode | ✅ | ✅ |
1{
2 "input_ids": [...],
3 "attention_mask": [...]
4}Humair332/gpt-style-50k-tokenizer