You can deploy this model and start earning money today!

phonemetransformers/
ipa-childes-tokenizers

Name: ipa-childes-tokenizers
Availability: InStock

Views

No views yet

CHILDES IPA Tokenizers

Tokenizers for each language in IPA-CHILDES used to train cross-lingual phoneme LLMs in our papers:

IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling
BabyLM's First Words: Word Segmentation as a Phonological Probing Task

Scripts for creating the tokenizers can be found here. Scripts for training models using these tokenizers can be found here.

To load a tokenizer:

python

from transformers import AutoTokenizer
dutch_tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/ipa-childes-tokenizers', subfolder='Dutch')