AndrewK101/

cs336-owt-32k-bpe

Pre-tokenized OpenWebText sample (Stanford CS336 assignment 1 source) using a custom 32k byte-level BPE tokenizer.

owt_vocab_32k.pkl — vocab dict[int, bytes] (32000 entries; ID 0 = <|endoftext|>, 1..256 = raw bytes, 257..31999 = merges) owt_merges_32k.pkl — list[tuple[bytes, bytes]] in merge order owt_train.npy — uint16 token IDs of 11 GB owt_train.txt (2,729,574,028 tokens) owt_valid.npy — uint16 token IDs of 277 MB owt_valid.txt (66… See the full description on the dataset page: https://huggingface.co/datasets/AndrewK101/cs336-owt-32k-bpe.