achille-fusco/gpt2_ParFindFast_eng_nl_zho

BabyLM 2026 - Multilingual GPT-2 (ParadigmFinder)

Track: BabyLM 2026 Multilingual
Architecture: GPT-2
Tokenizer: ParadigmFinder with boundary discovery support

A multilingual GPT-2 language model trained for the BabyLM 2026 multilingual setting on English, Dutch, and Chinese. This release is intended as the ParadigmFinder counterpart to the multilingual MoP baseline: it uses the same model family and the same multilingual_selection training data for language-model training, while replacing the tokenizer with a ParadigmFinder-based segmentation pipeline.

The tokenizer and model were exported together and can be loaded directly from the Hub. Since the tokenizer implementation is custom, loading requires trust_remote_code=True.

---

Model Details

Architecture

Hyperparameter	Value
Architecture	GPT-2 (GPT2LMHeadModel)
Hidden size (n_embd)	768
Layers (n_layer)	12
Attention heads (n_head)	12
Positional capacity (n_positions)	1024
Training window length	512
Dropout	0.1
Parameters	119,078,400
Exported vocabulary size	43,276

Tokenizer

The model uses a custom ParadigmFinder tokenizer exported as EnhancedParadigmTokenizerWrapper. In this multilingual configuration, tokenizer training uses a soft per-language vocabulary budget and boundary discovery support designed to remain compatible with space-free scripts such as Chinese.

Tokenizer highlights:

• Boundary discovery enabled: true
• Boundary discovery mode: space_free_only
• Shorter-first sentence ordering: true
• Explicit word-boundary insertion during tokenization: disabled (use_word_boundaries=false)
• Vocabulary budget: 16,384 token slots per language
• Languages: English (eng), Dutch (nld), Chinese (zho)

Special tokens:

• <pad> = 0
• <unk> = 1
• <s> = 2
• </s> = 3

This repo includes the tokenizer code and supporting files needed for loading:

• tokenizer.py
• boundary_discovery.py
• preprocessing.py
• paradigm_utils.py
• paradigms.json
• multilingual_meta.json

Training Data

This model was trained on the BabyLM multilingual data selection stored locally as multilingual_selection, with the three languages:

• English (eng)
• Dutch (nld)
• Chinese (zho)

For comparability with the multilingual MoP baseline, the GPT-2 model training uses the same multilingual corpus selection. In this experiment line, tokenizer training and model training were aligned to the same multilingual data source.

Training Procedure

Hyperparameter	Value
Training type	strict
Epochs	10
Maximum training steps	200,000
Batch size	16
Learning rate	5e-5
Warmup steps	2,000
Weight decay	0
Gradient clipping	1.0

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "achille-fusco/gpt2_ParFindFast_eng_nl_zho"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

prompt_en = "The child looked at"
inputs = tokenizer(prompt_en, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_nl = "Het kind keek naar"
inputs = tokenizer(prompt_nl, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

prompt_zh = "孩子看着"
inputs = tokenizer(prompt_zh, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0]))

If the environment has a read-only Hugging Face cache, set a writable cache first:

export HF_HOME=/tmp/hf_cache

---

Evaluation

This model was evaluated with the BabyLM 2026 multilingual evaluation pipeline across zero-shot and fine-tuned tasks.

Summary scores:

Metric	Score
Zero-shot mean	0.5746
Finetune mean	0.2613
Overall mean	0.3932

Selected zero-shot results:

Task	Score
blimp	0.6176
blimp_nl	0.6759
multiblimp_eng	0.7675
multiblimp_nld	0.7336
zhoblimp	0.9830
xcomps_nl	0.7916
xcomps_zh	0.9733

These numbers correspond to the collated submission artifact produced for this export.

---

Limitations

• This is a small multilingual BabyLM-scale model and is not intended as a production general-purpose LM.
• The tokenizer relies on custom remote code, so some environments may require extra care when loading.
• No explicit language identifier is used; mixed-language prompting may lead to unstable behavior.
• Compared with the multilingual MoP baseline, this model is stronger on some Chinese and compositional zero-shot evaluations, but weaker overall after finetuning.

---

Citation

If you use this model, please cite the model repository and the BabyLM shared task.

@misc{fusco2026parfindfast,
  author       = {Fusco, Achille},
  title        = {{BabyLM 2026 Multilingual GPT-2 (ParadigmFinder)}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/achille-fusco/gpt2_ParFindFast_eng_nl_zho}},
  note         = {BabyLM 2026 multilingual model release with ParadigmFinder tokenizer.}
}