| --- |
| language: [en, nl, zh] |
| license: mit |
| library_name: transformers |
| tags: [babylm, babylm-2026, multilingual, code-switching, gpt2] |
| --- |
| |
| # multilingual-macaroni-models |
|
|
| All 24 models from our BabyLM 2026 Multilingual-track study of **code-switched pretraining |
| curricula** (English / Dutch / Chinese): **8 training conditions × 3 seeds (42/43/44)**, each a |
| 12-layer GPT-2 (~98M params, 16k vocab, 1024 context) trained from scratch on the 100M |
| byte-premium-adjusted-word budget. |
|
|
| Every model lives on its **own branch (revision)**. The `main` branch holds only this card. |
|
|
| | Condition (branch) | Trained on | Description | |
| |---|---|---| |
| | `switch` | `switch` corpus | Full mixed code-switching, shuffled ordering | |
| | `noswitch` | `noswitch` | Monolingual twin | |
| | `word` | `word` | Intrasentential (word-level) CS only | |
| | `sent` | `sent` | Sentence-level CS only | |
| | `par` | `par` | Parallel CS | |
| | `salad` | `salad` | Word-salad control | |
| | `curriculum` | `curriculum` | Staged CS curriculum (intra → sentence → mono) — the leaderboard submission | |
| | `curriculum_noswitch` | `curriculum_noswitch` | Monolingual twin, staged | |
|
|
| Seeds: branch `<condition>` = seed 42; `<condition>-s43`, `<condition>-s44` = seeds 43, 44. |
|
|
| Training data: [`drooryck/multilingual-macaroni-corpus`](https://huggingface.co/datasets/drooryck/multilingual-macaroni-corpus). |
| The `curriculum` (CS) model is the same weights as the leaderboard submission |
| [`drooryck/babylm-macaroni`](https://huggingface.co/drooryck/babylm-macaroni), which additionally |
| carries the 28 `chck_*M` learning-curve revisions. |
|
|
| ## Load a model |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| REPO = "drooryck/multilingual-macaroni-models" |
| tok = AutoTokenizer.from_pretrained(REPO, revision="curriculum") |
| model = AutoModelForCausalLM.from_pretrained(REPO, revision="curriculum-s43") # seed 43 |
| ``` |
|
|
| ## Recipe |
|
|
| GPT-2 (12L, 768d, 12 heads, 1024 ctx, 16k vocab); LR 5e-5, cosine-with-min-lr, warmup 0.01, |
| batch 16, AdamW, 10 epochs over the corpus. See our paper for full details. |
|
|
