Multilingual GPT-2 model for the BabyLM 2026 multilingual setting, paired with a custom ParadigmFinder tokenizer trained on English, Dutch, and Chinese.

This export comes from:

tokenizer experiment: toktrain_ParFindFast_ML_BD_budget16k_zhchildes_v1.0
model experiment: model_gpt2_ParFindFast_ML_BD_budget16k_zhchildes_v1.0

The model and tokenizer are packaged together for direct Hugging Face upload. Because the tokenizer uses custom code, loading requires trust_remote_code=True.

Model

architecture: GPT2LMHeadModel
layers: 12
attention heads: 12
hidden size: 768
max positions: 1024
training sequence length: 512
exported model vocab size: 44090

Tokenizer

class: EnhancedParadigmTokenizerWrapper
paradigm family: ParadigmFinder
boundary discovery: true
boundary discovery mode: space_free_only
multilingual budgeting: soft merge
budget per language: 16384
languages: eng, nld, zho

This tokenizer includes the custom runtime files needed for HF loading:

tokenizer.py
boundary_discovery.py
preprocessing.py
paradigm_utils.py
paradigms.json
multilingual_meta.json

Training Data

The model was trained on the BabyLM 2026 multilingual corpus selection used in this repo, with aligned tokenizer training and LM training over the same multilingual data source.

Languages:

English
Dutch
Chinese

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "NeTSlab/gpt2_parfind_en_nl_zh_equal"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "The child looked at"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

If your Hugging Face cache is not writable, set:

export HF_HOME=/tmp/hf_cache

Notes

This is a BabyLM-scale research model, not a production general-purpose LM.
The tokenizer uses custom remote code, so some environments may need trust_remote_code=True.
The multilingual vocabulary budget is equal per language at configuration time, but the final merged vocabulary is derived after deduplication.