Spaces:
Sleeping
Sleeping
| # Kaikki vs lukeslp/etymology-atlas | |
| ## What we did | |
| Rebuilt `etymologies.parquet` from a **single pinned English Wiktionary dump extract**: | |
| | Field | Value | | |
| | --- | --- | | |
| | Dump | `enwiktionary` **2026-08-05** | | |
| | Extract | Kaikki / wiktextract on **2026-08-16** (`872fc7b` + `4deed51`) | | |
| | URL | `https://kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz` | | |
| | Input SHA-256 | `4514050b4e4f72cdcfdb9fd79e7eecf95c7fbbb9b0cb675f6f536334bc7b6691` | | |
| Pipeline (deterministic): | |
| 1. `pipeline/download_kaikki.py` — fetch the pinned gz | |
| 2. `pipeline/kaikki_etymology.py` — stream JSONL → atlas-shaped edges (sorted, deduped) | |
| 3. Merge Lexibank IE-CoR cognates from `etymologies_atlas_backup.parquet` | |
| 4. Keep Glottolog / WALS / PHOIBLE / cognate_sets companions from the atlas release | |
| 5. `pipeline/build_index.py` — rebuild SQLite + mmap index | |
| Edge typing uses the same template→relation collapse and fixed confidence table as the atlas packaging (see `TEMPLATE_MAP` / `CONF` in `kaikki_etymology.py`). Descendants default to `inherited` (or `borrowed` when tags/section say so). | |
| ## Aggregate diff | |
| | Metric | Kaikki-built | Atlas (etymology-db + IE-CoR) | | |
| | --- | ---: | ---: | | |
| | Edges | **5,968,098** | 4,173,462 | | |
| | Shared typed edges | 2,927,026 | — | | |
| | Only Kaikki (typed) | 3,041,072 | — | | |
| | Only Atlas (typed) | — | 1,215,087 | | |
| | Shared endpoints (ignore type) | 3,009,731 | — | | |
| By source in the new graph: | |
| - `kaikki:etymology_templates` 4,229,316 | |
| - `kaikki:descendants` 1,349,720 | |
| - `lexibank:iecor` 389,062 (some IE-CoR rows dedupe against Kaikki cognates) | |
| Notable type shifts vs atlas: | |
| | Type | Kaikki | Atlas | Delta | | |
| | --- | ---: | ---: | ---: | | |
| | inherited | 1,380,882 | 327,261 | +1,053,621 (mostly descendants) | | |
| | borrowed | 703,727 | 231,814 | +471,913 | | |
| | cognate | 1,047,152 | 790,074 | +257,078 | | |
| | derived | 440,317 | 325,776 | +114,541 | | |
| | other | 1,970,823 | 2,162,894 | −192,071 | | |
| | clipping / back_formation / abbreviation | lower | higher | Kaikki has fewer of these templates in etymology sections | | |
| ## What Kaikki adds | |
| - **Descendants / Reconstruction coverage** — e.g. Latin `viridis` → Vulgar Latin `*viridia` (missing from etymology-db). | |
| - Fresher dump (2026-08 vs etymology-db 2023-12). | |
| - More inherited/borrowed reverse fan-out for Classical etyma. | |
| ## Language-key canonicalization | |
| Kaikki/wiktextract language labels are noisy (alternate English names, HTML/arrow debris in | |
| descendant headers). The pipeline maps **true synonyms** onto canonical graph keys via | |
| ``canonicalize_lang`` / ``LANG_ALIASES`` in `kaikki_etymology.py` (e.g. ``Old Portuguese`` → | |
| ``old galician-portuguese``, ``Venetian`` → ``venetan``), and maps bare ISO codes to names. | |
| **Historical stages are not collapsed.** ``new latin``, ``medieval latin``, ``late latin``, | |
| ``classical persian``, etc. stay distinct from ``latin`` / ``persian``. The same spelling may | |
| legitimately appear as separate nodes in more than one stage. | |
| This is **Kaikki-only** — we do not import etymology-db atlas edges into the live graph. | |
| ## What is still missing or different vs atlas | |
| 1. **etymology-db-only endpoints (~1.1M / ~27%)** — **not** a small fraction. Most of the typed miss is `other` (~0.87M: mention/link/affix-style and older free-text parses), plus language-key / orthography / dump-age gaps (2023 etymology-db vs 2026-08 Kaikki). Skipping droher `from-parsed` group scaffolding is incidental; it was never the main hole. | |
| 2. **Reconstruction redirects (now handled)** — Wiktionary `#REDIRECT` pages such as `Reconstruction:Latin/virdia` → `Reconstruction:Latin/viridia` are present in Kaikki (`pos: hard-redirect`, ~49k total / ~5.5k Reconstruction). We collapse Reconstruction redirect lemmas (`*virdia` → `*viridia`) and restore starred heads / Proto-Romance lang on reconstruction entries so Spanish `berza` and Latin `viridis` descendants share one Vulgar Latin node. | |
| 3. **Fewer clipping / back-formation / abbreviation edges** — those templates are rarer or shaped differently in current Wiktionary vs the 2023 etymology-db snapshot. | |
| 4. **Companion data unchanged** — Glottolog/WALS/PHOIBLE/IE-CoR sets still come from the atlas release; only the Wiktionary word graph was replaced. | |
| ## Memory caps | |
| Heavy jobs install a hard address-space limit (default **10 GiB** via `RLIMIT_AS` + `ulimit -v`) so a runaway cannot OOM-kill a 16 GiB VM. Override with `ETYMOLOGY_MEMLIMIT_GIB` (set `0` to disable). Prefer `scripts/run_capped.sh` / `scripts/run.sh`. | |
| ## Local index build | |
| After redirect-aware conversion: **5,968,098** etymology edges (templates + descendants + Lexibank). Rebuild the SQLite/mmap index with `build_index.py` as usual. | |
| Reproduce: | |
| ```bash | |
| export ETYMOLOGY_MEMLIMIT_GIB=10 | |
| ./scripts/run_capped.sh python3 pipeline/download.py --with-kaikki | |
| ./scripts/run_capped.sh python3 pipeline/kaikki_etymology.py | |
| ./scripts/run_capped.sh python3 pipeline/diff_kaikki_atlas.py | |
| ./scripts/run_capped.sh python3 pipeline/build_index.py --force | |
| ``` | |
| Verified path after redirect collapse: Spanish `berza` → Vulgar Latin `*viridia` → Latin `viridis` (no leftover `*virdia` nodes). | |