---
language: [en, nl, zh]
license: mit
library_name: transformers
tags: [babylm, babylm-2026, multilingual, code-switching, gpt2]
---

# multilingual-macaroni-models

All 24 models from our BabyLM 2026 Multilingual-track study of **code-switched pretraining
curricula** (English / Dutch / Chinese): **8 training conditions × 3 seeds (42/43/44)**, each a
12-layer GPT-2 (~98M params, 16k vocab, 1024 context) trained from scratch on the 100M
byte-premium-adjusted-word budget.

Every model lives on its **own branch (revision)**. The `main` branch holds only this card.

| Condition (branch) | Trained on | Description |
|---|---|---|
| `switch` | `switch` corpus | Full mixed code-switching, shuffled ordering |
| `noswitch` | `noswitch` | Monolingual twin |
| `word` | `word` | Intrasentential (word-level) CS only |
| `sent` | `sent` | Sentence-level CS only |
| `par` | `par` | Parallel CS |
| `salad` | `salad` | Word-salad control |
| `curriculum` | `curriculum` | Staged CS curriculum (intra → sentence → mono) — the leaderboard submission |
| `curriculum_noswitch` | `curriculum_noswitch` | Monolingual twin, staged |

Seeds: branch `<condition>` = seed 42; `<condition>-s43`, `<condition>-s44` = seeds 43, 44.

Training data: [`drooryck/multilingual-macaroni-corpus`](https://huggingface.co/datasets/drooryck/multilingual-macaroni-corpus).
The `curriculum` (CS) model is the same weights as the leaderboard submission
[`drooryck/babylm-macaroni`](https://huggingface.co/drooryck/babylm-macaroni), which additionally
carries the 28 `chck_*M` learning-curve revisions.

## Load a model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "drooryck/multilingual-macaroni-models"
tok   = AutoTokenizer.from_pretrained(REPO, revision="curriculum")
model = AutoModelForCausalLM.from_pretrained(REPO, revision="curriculum-s43")  # seed 43
```

## Recipe

GPT-2 (12L, 768d, 12 heads, 1024 ctx, 16k vocab); LR 5e-5, cosine-with-min-lr, warmup 0.01,
batch 16, AdamW, 10 epochs over the corpus. See our paper for full details.