The model is trained on original FineWeb sentences paired with simplified rewrites. During masked language model pretraining, these paired examples expose the model to two aligned ways of expressing similar content. The goal is to improve sample efficiency under the 10M-word BabyLM strict-small budget.

Developed by: Shaoxiang Wu
Model type: masked language model
Architecture: DeBERTa-v2 style encoder
Language(s): English
Model repo: go76dof/wwm_curriculum_simplification_40k
Training data: go76dof/Fineweb_simplification_pairs
License: other

Uses

This is a pretrained masked language model. It can be used for:

BabyLM-style zero-shot evaluation with masked-token scoring.
Fine-tuning on classification or sentence-pair understanding tasks by adding a task head.
Research on small-data language model pretraining.
Research on meaning-preserving rewrite pairing, simplification, and whole-word-masking curricula.

This model is not intended as a general-purpose production language model. It is small and trained on only 10M words.

Example loading code:

python

from transformers import AutoModelForMaskedLM, AutoTokenizer

model_name = "go76dof/wwm_curriculum_simplification_40k"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

Training Details

Training Data

The model was trained on FineWeb sentence-level simplification pairs:

Dataset: go76dof/Fineweb_simplification_pairs
Training file: FineWeb_simplification_pairs.train
Size: 9,999,969 whitespace-counted words
Format: original FineWeb sentence followed by simplified rewrite

Each pair is stored as an original sentence followed by its simplified rewrite, with blank lines separating pairs.

Example:

text

Enlightenment thinkers proposed that human reason coupled with empirical study of the physical world would lead to progress---namely, the advancement of science and the improvement of the human condition.
Enlightenment thinkers believed that using reason and studying the world would lead to scientific progress and better living conditions.

The tokenizer is a 40k SentencePiece BPE tokenizer trained for this data condition:

tokenizer/FineWeb_simplification_pairs_40k.model
tokenizer/FineWeb_simplification_pairs_40k.vocab

Hyperparameters

Hyperparameter	Value
Architecture	DeBERTa-v2 style encoder
Hidden size	384
Intermediate size	1280
Number of layers	12
Number of attention heads	12
Dropout	0.1
Vocabulary size	40,000
Objective	Masked language modeling
Optimizer	LAMB
Learning rate schedule	cosine
Max learning rate	0.007
Training epochs	10
Sequence length curriculum	64 -> 256
Masking curriculum	WWM7 -> Token3

Training Procedure

The model uses a whole-word-masking curriculum:

Epochs 1-7: whole-word masking, where all subword pieces belonging to a selected word are masked together.
Epochs 8-10: standard token-level masking.

The sequence length also follows a curriculum. Training starts with shorter sequences and later switches to longer sequences, while batch size is scaled inversely with sequence length.

Size and Checkpoints

The model has 34,677,952 parameters. The final checkpoint corresponds to the end of the 10-epoch training run on the 10M-word strict-small training budget.

Evaluation

This model was evaluated with the BabyLM 2026 strict-small evaluation pipeline. The reported submission scores are leaderboard scores for the strict-small track.

Testing Data and Metrics

The evaluation includes:

BLiMP and BLiMP Supplement: linguistic minimal pair accuracy.
EWoK: world-knowledge minimal pair accuracy.
Entity Tracking: accuracy on tracking entity states through text.
COMPS: compositional generalization.
GlobalPIQA: average of parallel and non-parallel GlobalPIQA splits.
Reading: average of eye-tracking and self-paced reading scores.
(Super)GLUE: mean fine-tuning score across the BabyLM GLUE-style tasks.

Results

Metric	Score
Overall Average	41.80
NLP Average	52.97
Human-like Average	2.71
BLiMP	67.20
BLiMP Supplement	56.01
EWoK	56.07
Entity Tracking	28.45
COMPS	53.57
GlobalPIQA	39.67
(Super)GLUE	69.79
Reading	5.42
AoA	0.00

The strongest gains in our experiments were observed on EWoK, Entity Tracking, and GLUE-style fine-tuning. In our BabyLM 2026 strict-small submission, this 10M-word model exceeded the official 100M-word GPT-2 baseline on EWoK, Entity Tracking, GlobalPIQA, and (Super)GLUE.

Technical Specifications

Model Architecture and Objective

The model is a DeBERTa-v2 style encoder trained with the masked language modeling objective.

Important configuration values:

architectures: DebertaV2ForMaskedLM
model_type: deberta-v2
hidden_size: 384
intermediate_size: 1280
num_hidden_layers: 12
num_attention_heads: 12
vocab_size: 40000
relative_attention: true
pos_att_type: p2c, c2p

Software

The model is compatible with Hugging Face Transformers and PyTorch.

Limitations

This model is trained only on English text and only under a 10M-word pretraining budget. It is intended for research use rather than production deployment.

The training data is derived from web text and automatically generated simplifications. It may contain noise, omissions, or changes in nuance inherited from either the source text or the rewrite process.

The model card reports BabyLM evaluation results, but some hidden-task results, especially GlobalPIQA and AoA, can be unstable across seeds or evaluation settings.

Citation

If you use this model, please cite the corresponding BabyLM paper or repository once available.

Model Card Authors

Shaoxiang Wu

go76dof/wwm_curriculum_simplification_40k

Model Card for WWM Curriculum Simplification 40k

Table of Contents

Model Details

Model Description