You can deploy this model and start earning money today!

Dosch2301/
tiny-stack-tokenizer_4096

Name: tiny-stack-tokenizer_4096
Rating: 2.5 (7 reviews)

gpt2

Views

No views yet

TinyStack Tokenizer

ByteLevel BPE tokenizer trained on fhswf/tiny-stack dataset.

Usage

python

from tokenizers.implementations import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing

tokenizer = ByteLevelBPETokenizer("./vocab.json", "./merges.txt")
tokenizer._tokenizer.post_processor = BertProcessing(
    ("</s>", tokenizer.token_to_id("</s>")),
    ("<s>", tokenizer.token_to_id("<s>")),
)
tokenizer.enable_truncation(max_length=512)

Vocab size: 4096

Dosch2301/tiny-stack-tokenizer_4096

TinyStack Tokenizer

Usage

Dosch2301/
tiny-stack-tokenizer_4096