Architecture: BiRWKV-7 (~28M parameters) — bidirectional RWKV-7 with dual-pass fused inference
Task: BabyLM 2026 Strict-Small track — primary BiRWKV-7 submission (seed 42, step 18k)
Training corpus: 14.87M tokens (BabyLM Strict-Small 10M-word corpus, 32K BPE)
Inference mode: CLM-only forward pass (BabyLM official eval)

Results (BabyLM 2026 official evaluation)

Task	Score
BLiMP (filtered)	68.55%
Entity Tracking (CLM)	22.45%
EWoK	50.46%
COMPS	52.55%

Architecture

8 layers, hidden dim 384, 6 attention heads (head size 64), 32K BPE vocabulary
Bidirectional: forward RWKV-7 pass + reversed backward pass, fused with sigmoid gate
N-gram embeddings (MD5 hash mod 2048), additive n-gram fusion gate
Chunked WKV-7 kernel (chunk_size=64, diagonal-only approximation)
Auxiliary objectives: MLM + MTP (backward pass supervises forward representations)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "yashmahe2018/birwkv7-strict-small-babylm2026",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "yashmahe2018/birwkv7-strict-small-babylm2026",
    trust_remote_code=True
)

Citation

bibtex

@misc{maheshwari2026halved,
  title={Halved CLM Exposure Mitigates Late-Training Collapse in Small Recurrent Language Models},
  author={Maheshwari, Yash},
  year={2026},
  note={BabyLM 2026 submission}
}