babylm-zho-pinyin-code-33M

A compact GPT-style causal language model for the 2026 Chinese BabyLM Challenge, trained on a pinyin-code representation of Mandarin instead of Hanzi.

Each Mandarin word is segmented with jieba and every syllable is encoded as a compact initial+digit token that preserves the pinyin initial, a tone group, and a syllable-length bucket — e.g. 我们 → W6M7. It is a lossy, phonology-first encoding.

• Architecture: 8 layers · 8 heads · 512 hidden · 512 context · 16k SentencePiece-BPE vocab (~33.4M params)
• Training data: BabyLM-zho (~73M pinyin-code tokens), 5 epochs
• Bigger sibling: CPSPX/babylm-zho-pinyin-code-97M

Evaluation — chinese-babylm-eval-pipeline

Task	Score
ZhoBLiMP (acc)	68.79
Hanzi-structure (acc)	55.10
Hanzi-pinyin (acc)	95.35
AFQMC (acc)	69.00
OCNLI (acc)	57.63
TNEWS (acc)	50.73
CLUEWSC (acc)	62.50
word_fmri (corr)	0.556
fmri (corr)	0.089

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "CPSPX/babylm-zho-pinyin-code-33M"
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)

# The tokenizer auto-converts raw Mandarin into the pinyin-code representation:
inputs = tokenizer("我们一起来看电影吧", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(out[0]))

trust_remote_code=True is required — the architecture and the transliterating tokenizer are custom. Source: https://github.com/tbhrobrecht/babylm-pinyin-abbreviations