babylm-zho-pinyin-code-97M

A GPT-style causal language model for the 2026 Chinese BabyLM Challenge, trained on a pinyin-code representation of Mandarin instead of Hanzi.

Each Mandarin word is segmented with jieba and every syllable is encoded as a compact initial+digit token that preserves the pinyin initial, a tone group, and a syllable-length bucket — e.g. 我们 → W6M7. It is a lossy, phonology-first encoding.

This is the larger, better-performing model (the 33.4M version is the small baseline).

• Architecture: 12 layers · 12 heads · 768 hidden · 512 context · 16k SentencePiece-BPE vocab (~97.7M params)
• Training data: BabyLM-zho (~73M pinyin-code tokens), 20 epochs

Evaluation — chinese-babylm-eval-pipeline

Task	Score	vs 33.4M
ZhoBLiMP (acc)	72.68	+3.9
Hanzi-structure (acc)	55.00	~
Hanzi-pinyin (acc)	95.80	+0.4
AFQMC (acc)	68.54	~
OCNLI (acc)	61.80	+4.2
TNEWS (acc)	52.26	+1.5
CLUEWSC (acc)	63.49	+1.0
word_fmri (corr)	0.554	~
fmri (corr)	0.086	~

For reference, the eval pipeline's Qwen3-0.6B baseline scores 71.67 on ZhoBLiMP — this 97.7M pinyin-code model reaches 72.68.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

name = "CPSPX/babylm-zho-pinyin-code-97M"
model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)

# The tokenizer auto-converts raw Mandarin into the pinyin-code representation:
inputs = tokenizer("我们一起来看电影吧", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(out[0]))

trust_remote_code=True is required — the architecture and the transliterating tokenizer are custom. Source: https://github.com/tbhrobrecht/babylm-pinyin-abbreviations