You can deploy this model and start earning money today!

NeTS-lab/
babylm-mop-10m-gpt2

Name: babylm-mop-10m-gpt2
Availability: InStock

pytorch

gpt2

mit

Views

No views yet

BabyLM 2025 GPT-2 with MorPiece Tokenizer (Strict Small Track)

Model Description

This is a GPT-2 language model trained as part of the BabyLM 2025 Challenge on the strict small track, using the innovative MorPiece tokenizer. The model demonstrates how morphologically-aware tokenization can improve language modeling performance when training on limited data (10M words).

Developed by: NeTS Lab
Model type: Autoregressive Language Model (GPT-2 architecture)
Language(s): English
License: MIT
Parent Model: GPT-2
Tokenizer: MorPiece (cristianochesi/morpiece)

Key Features

Morphologically-aware tokenization via MorPiece for better handling of word structure (hyperparameter used: min_freq=10, bf=2, cutoff=100)
Strict data constraints (10M words) following BabyLM 2025 Strict Small track
Optimized for data efficiency default BabyLM 2025 baseline hyperparameter tuning
768-dimensional embeddings with 12 attention heads and 12 layers

Model Details

Architecture

Base Architecture: GPT-2 (12 layers, 12 attention heads)
Hidden Size: 768
Vocabulary Size: 23,405 (MorPiece tokens)
Context Length: 1,024 tokens
Parameters: ~~104M (estimated)

Training Configuration

Training Type: Strict (BabyLM 2025 guidelines)
Dataset Size: 10M words maximum
Sequence Length: 512 tokens
Batch Size: 16
Learning Rate: 5e-5
Training Steps: 200,000
Warmup Steps: 2,000
Epochs: 10
Weight Decay: 0.0
Gradient Clipping: 1.0

Tokenization

This model uses the MorPiece tokenizer, a split-based tokenizer (cristianochesi/morpiece)

Training Data

The model was trained on the BabyLM 2025 strict small track dataset, which includes:

Size: 10M words maximum
Sources: Child-directed speech and age-appropriate text
Language: English
Preprocessing: Tokenized using MorPiece tokenizer

Intended Uses

Primary Use Cases

Research into data-efficient language modeling
Comparative studies of tokenization methods in low-resource settings
Baseline model for BabyLM 2025 Challenge participants
Educational purposes for understanding morphological tokenization

Out-of-Scope Uses

Production deployments requiring robust, general-purpose language understanding
Safety-critical applications
Tasks requiring knowledge beyond the training data scope

Performance

The model was trained following BabyLM 2025 Challenge protocols:

Training loss: 3.28548
Convergence: Achieved after 200,000 training steps

Usage

Loading the Model

python

from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained("NeTS-lab/babylm-mop-10m-gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("NeTS-lab/babylm-mop-10m-gpt2")

# Generate text
input_text = "The child played with"
inputs = tokenizer.encode(input_text, return_tensors="pt")
outputs = model.generate(inputs, max_length=50, do_sample=True, temperature=0.8)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Text Generation Parameters

Max Length: 50 tokens (default)
Sampling: Enabled by default
Temperature: Adjustable (0.8 recommended)

Limitations and Biases

Known Limitations

Limited training data (10M words) may result in knowledge gaps
Domain specificity due to child-directed speech focus
Vocabulary constraints from MorPiece tokenization
Context window limited to 1,024 tokens

Potential Biases

Age-appropriate content bias from training data selection
English language bias (monolingual training)
Morphological bias toward Indo-European language patterns
Dataset composition bias inherent in BabyLM data curation

Technical Specifications

Training Infrastructure

Framework: PyTorch + Transformers
Precision: float32
Gradient Accumulation: Configured for effective batch size
Monitoring: Weights & Biases integration

Model Configuration

json

{
  "activation_function": "gelu_new",
  "architectures": ["GPT2LMHeadModel"],
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "vocab_size": 23405
}

Citation

If you use this model in your research, please cite:

bibtex

@misc{babylm2025-gpt2-morpiece,
  title={undereview},
  author={[Your Name]},
  year={2025},
  url={https://huggingface.co/NeTS-lab/babylm-mop-10m-gpt2}
}

Also consider citing the original BabyLM Challenge and MorPiece tokenizer:

bibtex

@misc{morpiece2024,
  title={MorPiece: Morphologically-aware Piece Tokenization},
  author={Cristiano Chesi & NeTS Lab},
  year={2024},
  url={https://github.com/cristianochesi/morpiece}
}

Acknowledgments

BabyLM 2025 Challenge organizers for providing the framework and dataset
MorPiece developers for split-based tokenization approach
Hugging Face Transformers team for the modeling infrastructure

Contact

For questions about this model or the training process, please [cristiano.chesi@iusspavia.it].

This model was developed as part of research into data-efficient language modeling and morphologically-aware tokenization techniques.