You can deploy this model and start earning money today!

anonym5035/
router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small

Name: router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small
Availability: InStock

Views

No views yet

XpertGPT Strict-Small

XpertGPT (Mixture of Experts with Parallelized Multi-Scale Information Transmission) is a sparse, data-efficient recurrent language model for the BabyLM 2026 challenge (Strict-Small (10M) track, 10M words). It combines sliding window attention global streams with sparse parallel expert blocks using Expert Choice routing. ~48.4M non-embedding parameters / ~52.6M total parameters. Custom code (trust_remote_code=True).

Architecture: 6 layers of MoEP-MSIT blocks. Each block combines a lower-dimensional dense global sliding window attention layer (dim = 256) with 4 parallel high-dimensional sparse expert blocks (dim = 384) routed via Expert Choice gating.
Track: BabyLM 2026 Strict-Small (10M) (10M words).
Tokenizer: Custom BPE tokenizer (vocab size: 16384).
Revision / Checkpoint: main

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("SRJ5035/router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small", revision="main", trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained("SRJ5035/router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small", revision="main")
ids = tok("The quick brown fox", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits

Intermediate checkpoints

Intermediate training checkpoints are provided as git revisions named chck_<N>M for the BabyLM challenge fast-eval.

License and citation

Released under CC BY-NC 4.0 (attribution required, non-commercial only). If you use this model or code, please cite (see CITATION.cff):

bibtex

@misc{jain2026xpertgpt,
  title        = {XpertGPT: Mixture of Experts with Parallelized Multi-Scale Information Transmission for Data-Constrained Pretraining},
  author       = {Jain, Soham and Singh, Harsh and Dewan, Divija and Dev, Atul},
  year         = {2026},
  howpublished = {Hugging Face Repository},
  note         = {XpertGPT MoE language model, BabyLM 2026}
}

Provenance and integrity fingerprints are documented in PROVENANCE.md.

anonym5035/router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small

XpertGPT Strict-Small

Usage

Intermediate checkpoints

License and citation

anonym5035/
router_rope_16_16_4_4_ln_swiglu_residuls_xpert_strcit_small