You can deploy this model and start earning money today!

Jiraya/
zoof-tokenizer

Name: zoof-tokenizer
Availability: InStock

Views

No views yet

Zoof Tokenizer

This is the custom Byte-Pair Encoding (BPE) tokenizer built for the zoof language model family. It was trained from scratch to efficiently handle English text and code.

Most recent zoof model: zoof-v1.2-394M-chat.

Model Details

Vocabulary Size: 49152
Type: Byte-Pair Encoding (BPE)
Language: English
Intended Use: Tokenization for the Zoof model family.

Usage

You can load this tokenizer directly with the transformers library:

python

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("Jiraya/zoof-tokenizer")

text = "Hello, world!"
tokens = tokenizer.encode(text)
decoded = tokenizer.decode(tokens)

print(f"Tokens: {tokens}")
print(f"Decoded: {decoded}")