You can deploy this model and start earning money today!

pakphum/
babylm2026-pl-ru-ipa-gate

Views

No views yet

babylm2026-pl-ru-ipa-gate

LLaMA-style causal language model trained for BabyLM 2026 with IPA phonological feature augmentation.

Architecture

Base: LLaMA-style transformer (RMSNorm, RoPE, SwiGLU FFN, weight-tied LM head)
IPA variant: ipa_gate — gated fusion of panphon IPA feature vectors at the embedding layer
Hidden size: 512 | Layers: 6 | Heads: 8
Vocab size: 32000 | Context: 512 tokens
IPA dim: 24 (panphon binary articulatory features)
Training split: pl_ru

IPA Fusion Variants

Variant	Description
`baseline`	Token embeddings only
`ipa_add`	`embed + W·ipa`
`ipa_gate`	`embed + σ(W_g·ipa) ⊙ (W·ipa)`
`ipa_full`	Gated fusion + MSE auxiliary reconstruction loss

Usage

python

from transformers import AutoConfig, AutoModelForCausalLM
import torch

config = AutoConfig.from_pretrained("pakphum/babylm2026-pl-ru-ipa-gate", trust_remote_code=True)
model  = AutoModelForCausalLM.from_pretrained("pakphum/babylm2026-pl-ru-ipa-gate", trust_remote_code=True)
model.eval()

# input_ids from your tokenizer
input_ids = torch.tensor([[1, 2, 3, 4]])

# Optional: supply IPA vectors [B, T, ipa_dim=24]. Zeros = no phonological signal.
ipa_vectors = torch.zeros(1, 4, config.ipa_dim)

with torch.no_grad():
    out = model(input_ids, ipa_vectors=ipa_vectors)
    # out.logits: [B, T, vocab_size]

Citation

BabyLM 2026 shared task.