# Kaikki vs lukeslp/etymology-atlas

## What we did

Rebuilt `etymologies.parquet` from a **single pinned English Wiktionary dump extract**:

| Field | Value |
| --- | --- |
| Dump | `enwiktionary` **2026-08-05** |
| Extract | Kaikki / wiktextract on **2026-08-16** (`872fc7b` + `4deed51`) |
| URL | `https://kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz` |
| Input SHA-256 | `4514050b4e4f72cdcfdb9fd79e7eecf95c7fbbb9b0cb675f6f536334bc7b6691` |

Pipeline (deterministic):

1. `pipeline/download_kaikki.py` — fetch the pinned gz  
2. `pipeline/kaikki_etymology.py` — stream JSONL → atlas-shaped edges (sorted, deduped)  
3. Merge Lexibank IE-CoR cognates from `etymologies_atlas_backup.parquet`  
4. Keep Glottolog / WALS / PHOIBLE / cognate_sets companions from the atlas release  
5. `pipeline/build_index.py` — rebuild SQLite + mmap index  

Edge typing uses the same template→relation collapse and fixed confidence table as the atlas packaging (see `TEMPLATE_MAP` / `CONF` in `kaikki_etymology.py`). Descendants default to `inherited` (or `borrowed` when tags/section say so).

## Aggregate diff

| Metric | Kaikki-built | Atlas (etymology-db + IE-CoR) |
| --- | ---: | ---: |
| Edges | **5,968,098** | 4,173,462 |
| Shared typed edges | 2,927,026 | — |
| Only Kaikki (typed) | 3,041,072 | — |
| Only Atlas (typed) | — | 1,215,087 |
| Shared endpoints (ignore type) | 3,009,731 | — |

By source in the new graph:

- `kaikki:etymology_templates` 4,229,316  
- `kaikki:descendants` 1,349,720  
- `lexibank:iecor` 389,062 (some IE-CoR rows dedupe against Kaikki cognates)

Notable type shifts vs atlas:

| Type | Kaikki | Atlas | Delta |
| --- | ---: | ---: | ---: |
| inherited | 1,380,882 | 327,261 | +1,053,621 (mostly descendants) |
| borrowed | 703,727 | 231,814 | +471,913 |
| cognate | 1,047,152 | 790,074 | +257,078 |
| derived | 440,317 | 325,776 | +114,541 |
| other | 1,970,823 | 2,162,894 | −192,071 |
| clipping / back_formation / abbreviation | lower | higher | Kaikki has fewer of these templates in etymology sections |

## What Kaikki adds

- **Descendants / Reconstruction coverage** — e.g. Latin `viridis` → Vulgar Latin `*viridia` (missing from etymology-db).  
- Fresher dump (2026-08 vs etymology-db 2023-12).  
- More inherited/borrowed reverse fan-out for Classical etyma.

## Language-key canonicalization

Kaikki/wiktextract language labels are noisy (alternate English names, HTML/arrow debris in
descendant headers). The pipeline maps **true synonyms** onto canonical graph keys via
``canonicalize_lang`` / ``LANG_ALIASES`` in `kaikki_etymology.py` (e.g. ``Old Portuguese`` →
``old galician-portuguese``, ``Venetian`` → ``venetan``), and maps bare ISO codes to names.

**Historical stages are not collapsed.** ``new latin``, ``medieval latin``, ``late latin``,
``classical persian``, etc. stay distinct from ``latin`` / ``persian``. The same spelling may
legitimately appear as separate nodes in more than one stage.

This is **Kaikki-only** — we do not import etymology-db atlas edges into the live graph.

## What is still missing or different vs atlas

1. **etymology-db-only endpoints (~1.1M / ~27%)** — **not** a small fraction. Most of the typed miss is `other` (~0.87M: mention/link/affix-style and older free-text parses), plus language-key / orthography / dump-age gaps (2023 etymology-db vs 2026-08 Kaikki). Skipping droher `from-parsed` group scaffolding is incidental; it was never the main hole.
2. **Reconstruction redirects (now handled)** — Wiktionary `#REDIRECT` pages such as `Reconstruction:Latin/virdia` → `Reconstruction:Latin/viridia` are present in Kaikki (`pos: hard-redirect`, ~49k total / ~5.5k Reconstruction). We collapse Reconstruction redirect lemmas (`*virdia` → `*viridia`) and restore starred heads / Proto-Romance lang on reconstruction entries so Spanish `berza` and Latin `viridis` descendants share one Vulgar Latin node.
3. **Fewer clipping / back-formation / abbreviation edges** — those templates are rarer or shaped differently in current Wiktionary vs the 2023 etymology-db snapshot.
4. **Companion data unchanged** — Glottolog/WALS/PHOIBLE/IE-CoR sets still come from the atlas release; only the Wiktionary word graph was replaced.

## Memory caps

Heavy jobs install a hard address-space limit (default **10 GiB** via `RLIMIT_AS` + `ulimit -v`) so a runaway cannot OOM-kill a 16 GiB VM. Override with `ETYMOLOGY_MEMLIMIT_GIB` (set `0` to disable). Prefer `scripts/run_capped.sh` / `scripts/run.sh`.

## Local index build

After redirect-aware conversion: **5,968,098** etymology edges (templates + descendants + Lexibank). Rebuild the SQLite/mmap index with `build_index.py` as usual.

Reproduce:

```bash
export ETYMOLOGY_MEMLIMIT_GIB=10
./scripts/run_capped.sh python3 pipeline/download.py --with-kaikki
./scripts/run_capped.sh python3 pipeline/kaikki_etymology.py
./scripts/run_capped.sh python3 pipeline/diff_kaikki_atlas.py
./scripts/run_capped.sh python3 pipeline/build_index.py --force
```

Verified path after redirect collapse: Spanish `berza` → Vulgar Latin `*viridia` → Latin `viridis` (no leftover `*virdia` nodes).