You can deploy this model and start earning money today!

vuiseng9/
bpe-10.0k-tinystories

Name: bpe-10.0k-tinystories
Availability: InStock

apache-2.0

Views

No views yet

BPE Tokenizer - 10k Vocabulary (TinyStories)

A Byte-Pair Encoding (BPE) tokenizer trained on the TinyStories dataset with a vocabulary size of 10,000 tokens.

Model Details

Tokenizer Type: BPE (Byte Pair Encoding), GPT-2 style
Vocabulary Size: 10,000 tokens
Training Dataset: roneneldan/TinyStories
Pre-tokenizer: ByteLevel (handles spaces and bytes like GPT-2)
Decoder: ByteLevel
Special Tokens:
- <|endoftext|>: BOS/EOS/UNK token
- <|padding|>: Padding token

Training Configuration

Minimum Frequency: 2
Batch Size: 1000
Training Split: train

Example Output

Training tokenizer on all samples...
[Info] Tokenizer 10000 vocabs saved to './bpe-10.0k-tinystories'

--- Test ---
Input:   Once upon a time, there was a tiny dragon.
Tokens:  [7013, 2402, 247, 673, 13, 627, 369, 247, 5888, 10295, 15]
Decoded: Once upon a time, there was a tiny dragon.

Usage

Loading the Tokenizer

python

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("vuiseng9/bpe-10.0k-tinystories")

# Encode text
text = "Once upon a time, there was a tiny dragon."
tokens = tokenizer.encode(text)
print(tokens)

# Decode tokens
decoded = tokenizer.decode(tokens)
print(decoded)

Reproduce the Tokenizer

bash

# Full training on entire dataset
python train_bpe.py --real 

# see more options
python train_bpe.py -h