| | --- |
| | library_name: transformers |
| | license: cc-by-4.0 |
| | language: |
| | - en |
| | datasets: |
| | - babylm-anon/stratified_10m_curriculum |
| | --- |
| | |
| | # Model Card for TICL |
| | A RoBERTa model pre-trained on a dataset of 10M words using (**T**raining Data) **I**nfluence-driven **C**urriculum **L**earning. |
| |
|
| |
|
| |
|
| |
|
| | ## Model Details |
| | See our paper at REDACTED for details on our method. |
| | ### Model Description |
| |
|
| | This is a model submitted to the strict-small track of the 2025 BabyLM challenge. |
| |
|
| | - **Developed by:** REDACTED |
| | - **Funded by [optional]:** REDACTED |
| | - **Model type:** Language model (Masked) |
| | - **Language(s) (NLP):** eng |
| | - **License:** CC-By-4.0 |
| |
|
| |
|
| | ### Model Sources |
| |
|
| | <!-- Provide the basic links for the model. --> |
| |
|
| | - **Repository:** [https://anonymous.4open.science/r/cl-4B5C](https://anonymous.4open.science/r/cl-4B5C) |
| |
|
| |
|
| | ## Uses |
| |
|
| | This model was trained to demonstrate the effectiveness of a novel curriculum learning method over training in random order. |
| |
|
| |
|
| |
|
| | ## Training Details |
| |
|
| | ### Training Data |
| |
|
| | We utilize [this](https://huggingface.co/datasets/babylm-anon/stratified_10m_curriculum) dataset built from the following existing ones: |
| |
|
| | - C1: Child Directed Speech |
| | - CHILDES [(MacWhinney 2000)](https://doi.org/10.4324/9781315805641) |
| | - C2: Children's Books |
| | - [Children Stories Text Corpus](https://www.kaggle.com/datasets/edenbd/children-stories-text-corpus) [(Bensaid et al. 2021)]( |
| | https://doi.org/10.48550/arXiv.2108.04324 |
| | ) |
| | - Children's Book Test [(Hill et al. 2016)]( |
| | https://doi.org/10.48550/arXiv.1511.02301) |
| | - C3: Dialogue |
| | - OpenSubtitles [(Lison and Tiedemann 2016)](https://aclanthology.org/L16-1147) |
| | - Switchboard Dialog Act Corpus [(Stolcke et al. 2000)](https://aclanthology.org/J00-3003) |
| | - [British NationalCorpus (BNC), dialogue portion](http://hdl.handle.net/20.500.14106/2554) |
| | - C4: Educational |
| | - Simple Wiki [(Warstadt et al. 2023)](https://doi.org/10.48550/arXiv.2301.11796) |
| | - [QED](https://opus.nlpl.eu/download.php?f=QED/v2.0a/xml/en.zip) [(Abdelali et al. 2014)](https://aclanthology.org/L14-1675/) |
| | - C5: Written English |
| | - Standardized Project Gutenberg Corpus [(Gerlach and Font-Clos 2018)](https://arxiv.org/abs/1812.08092) |
| | - Wikipedia [(Warstadt et al. 2023)](https://arxiv.org/abs/2301.11796) |
| |
|
| |
|
| | ### Data mix |
| | | | Words | | Documents | | |
| | |:--------------------------|--------:|:-------|------------:|:-------| |
| | | C1: Child Directed Speech | 1999999 | 20.00% | 360533 | 33.68% | |
| | | C2: Children's Books | 1999995 | 20.00% | 77384 | 7.23% | |
| | | C3: Dialogue | 1999987 | 20.00% | 349650 | 32.67% | |
| | | C4: Educational | 1999999 | 20.00% | 161554 | 15.09% | |
| | | C5: Written English | 1999945 | 20.00% | 121200 | 11.32% | |
| |
|
| | ### Training Procedure |
| | We extract training data influence estimates from models trained in random order, and sort the training data based on that information with various strategies detailed in the paper. |
| | This is the overall best performing model in our experiments, trained in order of increasing influence and re-weighted with lognormal filter, see the paper for details. |
| |
|
| |
|
| |
|
| | #### Training Hyperparameters |
| | We employ a novel curriculum learning strategy in which the model is trained in non-random order with a total of 100M words. |
| |
|
| | | Parameter | | |
| | |------------------------------|-------------| |
| | | **Shared Hyperparameters** | | |
| | | Vocabulary size | 52k | |
| | | Hidden size | 768 | |
| | | Number of layers | 12 | |
| | | Number of attention heads | 12 | |
| | | Initializer range | 0.02 | |
| | | Tie word embeddings | True | |
| | | **Model-Specific Settings** | | |
| | | Max position embeddings | 514 | |
| | | Intermediate (FFN) size | 3072 | |
| | | Norm epsilon | 1e-5 | |
| | | Attention dropout | 0.1 | |
| | | Activation function | gelu | |
| | | Hidden dropout | 0.1 | |
| | | **Training Setup** | | |
| | | FP16 | False | |
| | | Per Device Batch Size | 32 | |
| | | Gradient Accumulation Steps | 16 | |
| | | GPUs | 4 | |
| | | Adam β₁ | 0.9 | |
| | | Adam β₂ | 0.98 | |
| | | Adam ε | 1e-6 | |
| | | Weight Decay ε | 0.01 | |
| | | Learning rate | 5e-4 | |
| | | Scheduler | polynomial | |
| |
|
| |
|
| |
|
| | ## Evaluation |
| |
|
| | We use [this](https://github.com/babylm/evaluation-pipeline-2025) evaluation pipeline of the 2025 BabyLM challange |
| |
|
| |
|
| | ### Results |
| |
|
| |
|
| | | Task | Metric | |
| | |---------------------------------|--------------| |
| | | (Super) GLUE | 0.579 | |
| | | blimp_filtered | 0.688 | |
| | | supplement_filtered | 0.559 | |
| | | entity_tracking | 0.302 | |
| | | ewok_filtered | 0.509 | |
| | | wug_adj_nominalization | 0.570 | |
| | | **Macro acc ** | **0.584** | |
| |
|
| |
|
| |
|
| |
|
| |
|
| | ## Model Card Contact |
| |
|
| | REDACTED |
