README.md

# LittleLearner

Research checkpoints for **LittleLearner: Language Models Under Pedagogically
Controlled Knowledge Exposure** — a study of how filtering the
pretraining corpus to K–5 (kindergarten through grade 5) educational content
affects a language model's downstream capabilities.

This organisation hosts **LittleCurriculum**, the K–5-filtered pretraining dataset, as well as the paired **LittleLearner** (K–5-filtered) and
**Unfiltered** (baseline general corpus) checkpoints at three sizes, plus
matched post-trained variants. All checkpoints share the same architecture
and tokenizer, so a LittleLearner *X* B and Unfiltered *X* B differ only in
what data they saw during pretraining — the controlled comparison the paper
is built around.

📄 Paper: **[LittleLearner: Language Models Under Pedagogically
Controlled Knowledge Exposure](https://arxiv.org/abs/2608.13545)** · ✉️ Contact: **fanfei.li@tuebingen.mpg.de**

---

## Naming conventions

Two model families:

- **LittleLearner** — pretrained on the K–5-filtered corpus.
- **Unfiltered** — pretrained on the same corpus without the K–5 filter (baseline general corpus).

---

## Important inference notes

All chatty checkpoints share two non-standard defaults. Please read the
individual model cards for the exact runnable snippet, but at a glance:

1. **No system prompt.** These models were SFT'd without a system message;
   leaving transformers' default system prompt on, or adding your own,
   measurably degrades on-task accuracy. Build the conversation as
   `messages = [{"role": "user", "content": ...}]` only — no system role.
2. **Custom stop tokens.** vLLM's default stop-token list derived from the
   tokenizer is **not** sufficient — generation can run past the assistant
   turn. Pass the `stop_token_ids=[…]` documented on each model card.

Base checkpoints do not carry a chat template and should be used in
completion mode.

---

## Citation

```bibtex
@article{littlelearner2026,
  title   = {LittleLearner: Language Models Under Pedagogically-Controlled Knowledge Exposure},
  author  = {Fanfei Li and Jana Zeller and Manuel Prada-Corral and Thadd{\"a}us Wiedemer
             and Prasanna Mayilvahanan and Ryan Cotterell and Wieland Brendel},
  journal = {arXiv preprint arXiv:26xx.xxxxx},
  year    = {2026}
}
```