docs/KAIKKI_MIGRATION.md

Kaikki vs lukeslp/etymology-atlas

What we did

Rebuilt etymologies.parquet from a single pinned English Wiktionary dump extract:

Field	Value
Dump	enwiktionary 2026-08-05
Extract	Kaikki / wiktextract on 2026-08-16 (872fc7b + 4deed51)
URL	https://kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz
Input SHA-256	4514050b4e4f72cdcfdb9fd79e7eecf95c7fbbb9b0cb675f6f536334bc7b6691

Pipeline (deterministic):

1. pipeline/download_kaikki.py — fetch the pinned gz
2. pipeline/kaikki_etymology.py — stream JSONL → atlas-shaped edges (sorted, deduped)
3. Merge Lexibank IE-CoR cognates from etymologies_atlas_backup.parquet
4. Keep Glottolog / WALS / PHOIBLE / cognate_sets companions from the atlas release
5. pipeline/build_index.py — rebuild SQLite + mmap index

Edge typing uses the same template→relation collapse and fixed confidence table as the atlas packaging (see TEMPLATE_MAP / CONF in kaikki_etymology.py). Descendants default to inherited (or borrowed when tags/section say so).

Aggregate diff

Metric	Kaikki-built	Atlas (etymology-db + IE-CoR)
Edges	5,968,098	4,173,462
Shared typed edges	2,927,026	—
Only Kaikki (typed)	3,041,072	—
Only Atlas (typed)	—	1,215,087
Shared endpoints (ignore type)	3,009,731	—

By source in the new graph:

• kaikki:etymology_templates 4,229,316
• kaikki:descendants 1,349,720
• lexibank:iecor 389,062 (some IE-CoR rows dedupe against Kaikki cognates)

Notable type shifts vs atlas:

Type	Kaikki	Atlas	Delta
inherited	1,380,882	327,261	+1,053,621 (mostly descendants)
borrowed	703,727	231,814	+471,913
cognate	1,047,152	790,074	+257,078
derived	440,317	325,776	+114,541
other	1,970,823	2,162,894	−192,071
clipping / back_formation / abbreviation	lower	higher	Kaikki has fewer of these templates in etymology sections

What Kaikki adds

• Descendants / Reconstruction coverage — e.g. Latin viridis → Vulgar Latin *viridia (missing from etymology-db).
• Fresher dump (2026-08 vs etymology-db 2023-12).
• More inherited/borrowed reverse fan-out for Classical etyma.

Language-key canonicalization

Kaikki/wiktextract language labels are noisy (alternate English names, HTML/arrow debris in descendant headers). The pipeline maps true synonyms onto canonical graph keys via canonicalize_lang / LANG_ALIASES in kaikki_etymology.py (e.g. Old Portuguese → old galician-portuguese, Venetian → venetan), and maps bare ISO codes to names.

Historical stages are not collapsed. new latin, medieval latin, late latin, classical persian, etc. stay distinct from latin / persian. The same spelling may legitimately appear as separate nodes in more than one stage.

This is Kaikki-only — we do not import etymology-db atlas edges into the live graph.

What is still missing or different vs atlas

1. etymology-db-only endpoints (~1.1M / ~27%) — not a small fraction. Most of the typed miss is other (~0.87M: mention/link/affix-style and older free-text parses), plus language-key / orthography / dump-age gaps (2023 etymology-db vs 2026-08 Kaikki). Skipping droher from-parsed group scaffolding is incidental; it was never the main hole.
2. Reconstruction redirects (now handled) — Wiktionary #REDIRECT pages such as Reconstruction:Latin/virdia → Reconstruction:Latin/viridia are present in Kaikki (pos: hard-redirect, ~49k total / ~5.5k Reconstruction). We collapse Reconstruction redirect lemmas (*virdia → *viridia) and restore starred heads / Proto-Romance lang on reconstruction entries so Spanish berza and Latin viridis descendants share one Vulgar Latin node.
3. Fewer clipping / back-formation / abbreviation edges — those templates are rarer or shaped differently in current Wiktionary vs the 2023 etymology-db snapshot.
4. Companion data unchanged — Glottolog/WALS/PHOIBLE/IE-CoR sets still come from the atlas release; only the Wiktionary word graph was replaced.

Memory caps

Heavy jobs install a hard address-space limit (default 10 GiB via RLIMIT_AS + ulimit -v) so a runaway cannot OOM-kill a 16 GiB VM. Override with ETYMOLOGY_MEMLIMIT_GIB (set 0 to disable). Prefer scripts/run_capped.sh / scripts/run.sh.

Local index build

After redirect-aware conversion: 5,968,098 etymology edges (templates + descendants + Lexibank). Rebuild the SQLite/mmap index with build_index.py as usual.

Reproduce:

export ETYMOLOGY_MEMLIMIT_GIB=10
./scripts/run_capped.sh python3 pipeline/download.py --with-kaikki
./scripts/run_capped.sh python3 pipeline/kaikki_etymology.py
./scripts/run_capped.sh python3 pipeline/diff_kaikki_atlas.py
./scripts/run_capped.sh python3 pipeline/build_index.py --force

Verified path after redirect collapse: Spanish berza → Vulgar Latin *viridia → Latin viridis (no leftover *virdia nodes).