from transformers import AutoModel, AutoTokenizer

repo_id = "zondaxyz/babyconceptLM-baike102M_bpe60"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)

Model Details

Architecture: ConceptDominantGPTBertForPreTraining
Hidden size: 768
Vocabulary size: 16,384
Max position embeddings: 512
Pretraining sequence length: 256
Token layers: 2
Concept layers: 4
Readout layers: 2
Attention heads: 12 token heads, 14 concept heads, 12 readout heads
Concept size: 896
Intermediate sizes: 3,072 token/readout, 3,584 concept
Model variant: m18_2tok_4concept_2readout_mup_concept_main_gate_baike102m_bpe60
Main gate: concept_main_shell_residual
Segment target: 4.0 tokens per concept
Training data recorded in local metadata: train_zho_baike102m_m15_bpe_tokenized.bin
Tokenizer recorded in local metadata: tokenizer_zho_baike102m_m15_bpe.json
Planned training steps: 64,277
Training epochs: 60

Evaluation

The following numbers were collected from local Chinese BabyLM evaluation artifacts on 2026-06-11. For MRI tasks, the reported best result is selected by the highest report-level mean score across chinese-babylm-eval-pipeline/results and chinese-babylm-eval-pipeline/results_mri_layer_sweep.

For CogBench fMRI tasks, the local cogbench-fmri-0415 data contains train and dev splits but no released test split. The word_fMRI and fMRI numbers below are therefore dev-set scores, not held-out test scores.

Task	Best score	Metric	Selection
zhoBLiMP	72.29	average accuracy	temperature 1.00
Hanzi structure	57.75	average accuracy	temperature 1.00
Hanzi pinyin	50.05	average accuracy	temperature 1.00
word_fMRI	0.563524	mean Pearson r over 6 subjects	best layers: 03/04/05/06/08 tied; max subject score: 0.6168 on `S31_score.mat`
fMRI	0.100538	mean Pearson r over 36 domain/subject files	best layer: 09; max file score: 0.2269 on `Language/03_average.mat`
OCNLI	63.19	accuracy	fine-tuned dev score
TNEWS	53.73	accuracy	fine-tuned dev score
CLUEWSC2020	63.49	accuracy	fine-tuned dev score
AFQMC	68.98	accuracy	fine-tuned dev score

zhoBLiMP Field Accuracy

Field	Accuracy
ellipsis	67.00
control_raising	79.75
BA	78.41
question	82.54
npi_licensing	48.26
nominal_expression	82.48
relativization	86.25
verb_phrase	85.14
anaphor	34.83
fci_licensing	96.87
topicalization	96.08
passive	48.97
quantifiers	41.33
argument_structure	62.38
classifier	64.56

MRI Layer Sweep

Layer	word_fMRI mean	fMRI mean
00	0.558891	0.085080
01	0.560022	0.087955
02	0.561925	0.093293
03	0.563524	0.091392
04	0.563524	0.091048
05	0.563524	0.089261
06	0.563524	0.090662
07	0.563463	0.100302
08	0.563524	0.100186
09	0.560945	0.100538

Intended Use

This model is intended for Chinese BabyLM challenge. It is not intended as a production conversational assistant.