solution-01-minimum-viable-checkpoint-4500
This repository contains the checkpoint-4500 model for solution_01_minimum_viable from the Chinese BabyLM project.
Model summary
- Model type: decoder-only
- Base architecture: GPT-2 style causal language model
- Tokenizer: character-level tokenizer trained on the official corpus
- Number of layers: 6
- Number of attention heads: 8
- Hidden size: 512
- Max sequence length: 256
- Total parameters: 23,240,704
Training setup
- Training data: official Chinese BabyLM corpus (100M)
- Objective: causal language modeling
- Optimizer: AdamW
- Max learning rate: 5e-4
- Batch size in tokens: 16384
- Training steps: 4500
Evaluation snapshot
This checkpoint was evaluated with the official Chinese BabyLM evaluation pipeline.
- zhoblimp: 0.7132
- hanzi_structure: 0.5595
- hanzi_pinyin: 0.4790
- afqmc: 0.6899907321594069
- ocnli: 0.6501694915254237
- tnews: 0.5395
- cluewsc2020: 0.6348684210526315
- word_fmri mean: 0.5595193525625657
- fmri mean: 0.09100095066912078
Notes
This is a training checkpoint rather than a final polished release. It is uploaded to support leaderboard submission and reproducibility.
