README.md

---
license: mit
language:
  - en
  - nl
  - zh
library_name: transformers
tags:
  - babylm
  - babylm-2026
  - multilingual
  - hawk
  - griffin
  - rg-lru
  - recurrent-lm
  - morpiece
  - cognitively-plausible
pipeline_tag: text-generation
---

# babylm26_multiling_hawk_MoP16K_baseline

A multilingual (English / Dutch / Chinese) **Hawk** language model trained for the
**BabyLM 2026 Multilingual track** (EMNLP 2026). This is a *baseline* model in the
NeTS-lab BabyLM-2026 series: it pairs a recurrent **Hawk (RG-LRU)** backbone with the
morphologically-aware **MorPiece (MoP)** tokenizer, trained on the three target
languages (`eng / nld / zho`) with a byte-premium-balanced word budget under the
`baseline` regimen.

It is the Hawk counterpart to the Transformer baseline
[`NeTS-lab/babylm26_multiling_gpt2_MoP16K_baseline`](https://huggingface.co/NeTS-lab/babylm26_multiling_gpt2_MoP16K_baseline);
the two share the **same tokenizer and the same training data**, so the comparison
isolates the architecture (linear-recurrent vs. attention).

> **Note on baselines.** These models are released as controlled reference points for
> the NeTS-lab BabyLM-2026 study. Their purpose is a clean, matched comparison across
> architectures and tokenizers — not leaderboard maximisation.

---

## Model details

- **Developed by:** Cristiano Chesi and NeTS Lab, IUSS Pavia (with Claude Opus 4.8 fixes and optimizations for our HPC)
- **Model type:** Decoder-only causal LM, **recurrent (Hawk / RG-LRU)** — no global self-attention
- **Languages:** English (`eng`), Dutch (`nld`), Chinese (`zho`)
- **Tokenizer:** MorPiece (MoP), ~39.7K vocabulary, shared multilingual
- **License:** MIT
- **Sibling models:** `*_gpt2_MoP16K_baseline` (Transformer), and the eMG / SSM-eMG variants

### Architecture

Hawk is the gated linear-recurrent backbone from the Griffin family (De et al., 2024).
Each block uses a **Real-Gated Linear Recurrent Unit (RG-LRU)** in place of attention,
combined with a gated MLP and RMSNorm. Loading requires `trust_remote_code=True`
because the `hawk_rglru` block is provided via custom modelling code.

| Field | Value |
|---|---|
| Backbone | Hawk (RG-LRU recurrent) — **no attention** |
| `model_type` | `hawk_rglru` |
| Layers (`n_layer`) | 12 |
| Hidden size (`n_embd`) | 704 |
| Recurrent width (`rnn_width`) | 768 |
| Conv kernel | 4 |
| MLP expansion | 3 |
| RG-LRU `c` | 8.0 |
| RMSNorm eps | 1e-6 |
| Max position embeddings | 1024 |
| Tied input/output embeddings | yes |
| Vocabulary size | **39,697** |
| Total parameters | **≈ 115.3M** (115,270,528) |
| Precision | float32 |

> The model is a *pure* Hawk recurrent stack (RG-LRU + depthwise conv + gated MLP,
> RMSNorm pre-norm). There is no self-attention, so the leaderboard "attention heads"
> field is `-1`. Parameter count is with tied embeddings (`lm_head` shares `wte`).
> Note that Hawk has no learned position embeddings — `max_position_embeddings`
> is a config field, not a hard context limit.

### Tokenizer — MorPiece (MoP)

MorPiece is a split-based tokenizer that incrementally segments words into candidate
**morphemes** by applying Yang's (2016) **Tolerance Principle** at every character as a
word traverses a dual root/inflection trie. Splits are licensed only when the TP holds
**bilaterally** (root trie *and* inflection trie). The result is a morphology-aware
vocabulary motivated by developmental linguistics rather than pure frequency. For this multilingual 
experiment we used the `--boundary-discovery` option to ignore whitespaces and process zho the same way of eng and nld.

- Shared across all three languages (single multilingual MoP tokenizer)
- Exported in HuggingFace `WordPiece` format with `++` continuing-subword prefix
- **Actual vocabulary: 39,697 tokens** (the `MoP16K` in the repo name is a nominal
  per-language label; the merged multilingual vocabulary is ~39.7K)

See the MorPiece repository for details: <https://github.com/cristianochesi/morpiece>

---

## Training data

Official **BabyLM 2026 Multilingual** data for `eng / nld / zho`. Languages are
drawn in a **byte-premium round-robin** during training, and the save-point
milestones are denominated in **byte-premium-adjusted English-equivalent words**
(BP: eng 1.000, nld 1.0516, zho 0.9360), per the multilingual track's word budget.
Training **regimen: `baseline`**. Per-corpus sizes: eng 56.2M / nld 57.0M / zho
50.0M model tokens (≈34.2M English-equivalent words each ≈ 102.6M total/epoch).

No custom corpus, no synthetic augmentation, no human-annotated preference data.
A cleaning procedure stripped metalinguistic information (e.g. `tiers`).
Preprocessing routine can be found here:
<https://github.com/cristianochesi/babylm-2026/tree/main/01-preprocess>

## Training procedure

| Field | Value |
|---|---|
| Optimizer | AdamW (β1=0.9, β2=0.999, weight decay 0.1, fused), no-decay on norms/biases/embeddings |
| LR scheduler | cosine decay with linear warmup (warmup = 1% of steps) |
| Max learning rate | 5e-4 |
| Min learning rate | 5e-5 |
| Epochs | ~3.1 of 10 planned (run cut by a 24h cluster time limit) |
| Per-device batch | 16 sequences × 4 grad-accum = **32,768 tokens / optimizer step** |
| Training sequence length | 512 (config `max_position_embeddings` = 1024) |
| Gradient clip | 1.0 (with a non-finite-grad firewall) |
| Random seed | 42 |
| Precision | bf16 mixed precision (no GradScaler); weights stored as float32 |
| Tokens processed | ~504M model tokens (~317M eng-equiv words) at stop |
| Hardware | 1 GPU + 8 CPUs, IUSS SLURM cluster (`gp02`, `gpuq`, conda `env_py3_12_torch2_91_CUDA_12_8`) |
| GPU-hours (training) | ~24 (single GPU; throughput ~2.6k steps/h) |
| Training FLOPs (approx.) | ~1.4 × 10¹⁸ (6·N·D over executed positions) |

---

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "NeTS-lab/babylm26_multiling_hawk_MoP16K_baseline"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

text = "the cats are"
inputs = tokenizer(text, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

> `trust_remote_code=True` is required to load the custom `hawk_rglru` block.
> The repo must contain `modeling_hawk.py` alongside `config.json` (the `auto_map`
> points to it). Generation is correct but has **no KV cache** — the recurrent
> backbone recomputes the full prefix each step, so `generate()` is O(T) per token.

---

## Evaluation

Evaluated with the BabyLM 2026 multilingual harness (lm-eval-style), including
**BLiMP**, **MultiBLiMP** (Dutch), and **SIGMORPHON 2022** morphology, alongside the
official multilingual benchmarks.

| Benchmark | Score |
|---|---|
| BLiMP (filtered) | `0.724` |
| BLiMP-nld (nld) | `0.803` |
| BLiMP-zho (zho) | `0.803` |

---

## Intended use & limitations

A small, sample-efficient research LM for studying cognitively-plausible language
modelling under a constrained (developmentally motivated) data budget. It is **not**
intended for production use. As a baseline trained on a limited multilingual corpus,
outputs are not reliable for downstream generation and may reflect biases in the
training data.

## Citation

```bibtex
@misc{chesi2026babylm_hawk_mop,
  title  = {Multilingual Hawk + MorPiece baseline for BabyLM 2026},
  author = {Chesi, Cristiano and {NeTS Lab, IUSS Pavia}},
  year   = {2026},
  note   = {BabyLM 2026 Multilingual track baseline},
  howpublished = {\url{https://huggingface.co/NeTS-lab/babylm26_multiling_hawk_MoP16K_baseline}}
}
```

**Contact:** cristiano.chesi@iusspavia.it · NeTS Lab, IUSS Pavia