You can deploy this model and start earning money today!

phonemetransformers/
babble-tokenizers

Name: babble-tokenizers
Availability: InStock

phonemetransformers/IPA-BabyLM

2410.22906

Views

No views yet

From Babble to Words: Tokenizers

Tokenizers trained for From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes.

This repository contains the eight tokenizers trained for the project, covering the combinations of the three transformations:

Character-based tokenization (CHAR) vs. subword tokenization (BPE)
Tokenizer for phonemic data (PHON) vs. orthographic data (TXT)
Tokenizer removes whitespace (SPACELESS) vs. keeps whitespace

To load a tokenizer:

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/babble-tokenizers', subfolder='BABYLM-TOKENIZER-CHAR-TXT')