Chinese BabyLM Cog A197 Strict Best
This repository contains the current local strict Cog-track candidate:
C0611-A197-strict-a186-plus-a129-s1000-mlm-uncertainty-scale1p25
Scores from the local official CogBench evaluation pipeline:
| task | score |
|---|
word_fmri | 0.5627605401691911 |
fmri | 0.10338260071025865 |
mean(word_fmri, fmri) | 0.33307157043972485 |
Full nine-task scores from the fixed local official pipeline rerun:
| Task | Score |
|---|
| zhoblimp | 0.7295 |
| hanzi_structure | 0.2655 |
| hanzi_pinyin | 0.4750 |
| word_fmri | 0.562760540 |
| fmri | 0.103382601 |
| afqmc | 0.69045412 |
| ocnli | 0.62576271 |
| tnews | 0.5253 |
| cluewsc2020 | 0.63486842 |
| simple_9task_mean | 0.51250316 |
Model path before upload:
/workspace/air/DeepScientist-1.5.17-home/quests/chinese-babylm-all-tracks/experiments/main/official_0611_a197_strict_a129_uncertainty_scale_cog_20260611/outputs/a197_a129_uncertainty_scale_gate/a197_gammam5p1_l678w000_098_002_word234_wordri_d256w4n200kfreq1_s3p05_raw384w8s1p45_freqs5p1_uncsrc_a129_s1000_s1p25
Compliance
This candidate is intended for the Chinese BabyLM strict from-scratch setting.
- Source weights are from the internal random-initialized strict lineage only.
- Training and static features use the official Chinese BabyLM corpus artifacts only.
- No official baseline checkpoint, external pretrained checkpoint, teacher model, external linguistic resource, or external data is used.
- No CogBench stimulus text, item text, labels, hidden labels, predictions, or item-level feedback is used for training or feature construction.
- Public aggregate Cog scores are used only for wrapper/checkpoint/source/scale selection and should be disclosed as public-dev model selection.
- The exported wrapper does not train or modify the underlying BERT weights; it combines internal strict branches, official-corpus static features, and scalar uncertainty values from an internal MLM forward pass.
Architecture Summary
- Backend:
mlm
- Word branch: A180 strict word branch, word layers
[2, 3, 4], concat, word RI scale 3.05.
- Sentence branch: A130 plus
gamma=-5.1 * (A158_s0250 - A122_s1500), layers [6, 7, 8] with weights [0.0, 0.98, 0.02].
- Sentence static features: official-corpus raw384 window8 scale
1.45 plus log-frequency scale 5.10.
- Uncertainty source: internal strict A129
step_01000 MLM checkpoint.
- Uncertainty scalar channels: observed logprob, observed surprisal, entropy, max logprob, top1-top2 margin, observed probability, max probability.
- Uncertainty scale:
1.25.
Use with trust_remote_code=True.
