Theskywalker07/seed_0

XpertGPT Strict-Small

XpertGPT (Mixture of Experts with Parallelized Multi-Scale Information Transmission) is a sparse, data-efficient recurrent language model for the BabyLM 2026 challenge (Strict-Small (10M) track, 10M words). It combines sliding window attention global streams with sparse parallel expert blocks using Expert Choice routing. ~48.4M non-embedding parameters / ~52.6M total parameters. Custom code (trust_remote_code=True).

• Architecture: 6 layers of MoEP-MSIT blocks. Each block combines a lower-dimensional dense global sliding window attention layer (dim = 256) with 4 parallel high-dimensional sparse expert blocks (dim = 384) routed via Expert Choice gating.
• Track: BabyLM 2026 Strict-Small (10M) (10M words).
• Tokenizer: Custom BPE tokenizer (vocab size: 16384).
• Revision / Checkpoint: main

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Theskywalker07/seed_0", revision="main", trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained("Theskywalker07/seed_0", revision="main")
ids = tok("The quick brown fox", return_tensors="pt").input_ids
with torch.no_grad():
    logits = model(ids).logits

Intermediate checkpoints

Intermediate training checkpoints are provided as git revisions named chck_<N>M for the BabyLM challenge fast-eval.

License and citation

Released under CC BY-NC 4.0 (attribution required, non-commercial only). If you use this model or code, please cite (see CITATION.cff):

@misc{jain2026xpertgpt,
  title        = {XpertGPT: Mixture of Experts with Parallelized Multi-Scale Information Transmission for Data-Constrained Pretraining},
  author       = {Jain, Soham and Singh, Harsh and Dewan, Divija and Dev, Atul},
  year         = {2026},
  howpublished = {Hugging Face Repository},
  note         = {XpertGPT MoE language model, BabyLM 2026}
}

Provenance and integrity fingerprints are documented in PROVENANCE.md.