docs/DESIGN.md

Reverse Etymology Atlas

Purpose

Ordinary etymology asks "where did this word come from?" Reverse etymology asks the opposite: given a base word, what was derived from it?

This tool turns the lukeslp/etymology-atlas graph (4.17 million relationships across 19,401 languages) into an interactive, mobile-capable explorer. A user picks an etymon such as Latin mater or Proto-Indo-European meh2ter and immediately sees the descendant cloud: inherited daughters, later borrowings, compounds, clippings, and (optionally) cognates.

The product is a local web application: a FastAPI service over an in-memory compressed graph, plus a canvas-rendered frontend. It is designed to feel like a living atlas, not a spreadsheet with circles.

Data sources

The Hugging Face dataset is the system of record. Five parquet tables are ingested, then enriched with two CLDF catalogs so that empty fields in the published atlas (notably lang*_family and most ISO codes) do not cripple filtering.

Source	Role
etymologies.parquet (4,173,462 rows)	Directed word graph
languages.parquet (19,401 rows)	Glottolog skeleton: coordinates, macroarea, vitality, phoneme counts
cognate_sets.parquet (4,981 IE-CoR sets)	Expert cognate groups, shown on node detail
phonemes.parquet (105,484 rows)	PHOIBLE inventories by glottocode
linguistic_features.parquet (76,475 rows)	WALS typological values
Glottolog CLDF languages.csv	Family names, ISO 639-3, coordinates, language vs family vs dialect
WALS CLDF languages.csv + codes.csv	Join WALS codes to glottocodes and human-readable feature values

License of the atlas and Wiktionary-derived etymology-db content is CC BY-SA 3.0. The UI cites Steuber 2026 and the upstream sources.

Graph semantics

Edge direction

Each etymology row is:

(term1, lang1)  relationship_type, confidence, source  (term2, lang2)

Empirical inspection of the atlas matches Wiktionary / etymology-db convention:

• term1 is the derived or attested form.
• term2 is the etymon, stem, or related form.

Examples:

• English portmanteau borrowed Middle French portemanteau
• English mother inherited Middle English moder, Old English modor, Proto-Germanic moder, PIE meh2ter

Reverse etymology therefore walks edges from term2 toward term1: "everything that lists this node as a source."

Forward etymology (etymons of the current node) is still stored and shown in the inspector, but it is not the primary layout.

Node identity

A node is the pair (term, lang) after trimming whitespace and dropping empty terms (a few dozen nulls exist in the parquet file). lang is the atlas language key: a Wiktionary language name (latin, proto-indo-european) for etymology-db rows, or an ISO 639-3 code (ben, hin) for Lexibank IE-CoR rows.

Display names prefer, in order: atlas lang*_name when non-empty, Glottolog Name, then title-cased lang.

Relation taxonomy

The atlas already normalizes Wiktionary's ~31 templates into 11 types:

Type	Count	Default in reverse walk	Notes
inherited	327,261	on	Unbroken descent. Highest-signal descendants.
derived	325,776	on	Derivation that is not specifically inherited/borrowed.
borrowed	231,814	on	Loanwords.
calque	10,318	on	Loan translations.
clipping	2,482	on	Spoken shortening.
back_formation	2,173	on	Affix removal.
abbreviation	404	on	Written shortening.
compound	304,470	on	term1 contains term2 as a part. Useful for affix/root exploration; noisier for strict lexical descent — uncheck if a root gets bushy.
blend	15,796	on	Portmanteau parts.
other	2,162,894	on	Catch-all (prefixes, mentions, mixed Wiktionary templates).
cognate	790,074	on	Lateral, not descent. Includes Lexibank pairwise cognates; can expand into cliques — uncheck for stricter trees.

Global edge filters (allowed types + minimum confidence) apply before traversal. They define which graph is being searched, not which nodes survive afterwards.

Confidence in the atlas is discrete: 0.70 (other), 0.75, 0.85, 0.90, 0.92 (inherited), 0.95 (derived), 1.00 (Lexibank). Default floor is 0.70 (everything). A slider exposes the steps.

Cycles, multi-parents, and depth

The graph is not a tree. A word can list several etymons; walking reverse edges can revisit nodes through different parents. Traversal is a bounded BFS:

1. Start at the selected root.
2. Follow reverse edges whose type/confidence pass the global edge filters.
3. Ignore edges back to an already-seen node (cycle cut).
4. Record the BFS parent so the visible structure is a tree even when the underlying graph is a DAG with extra links.
5. Stop at max_depth (default 3, max 8) or max_visit (50,000 nodes).

Extra parents that were not chosen as the BFS parent are attached as also_from on the node payload so the inspector can show them without breaking layout.

Filtering model (the core requirement)

Filters are not a single global predicate applied to every visible node. That would be wrong for reverse etymology. A user who asks "English words that came from Latin caput by way of French" wants:

• Leaves (terminal descendants, or nodes at max depth treated as terminals for this query) to be English.
• Some intermediate hop on the unique BFS path to be French.
• The Latin root itself to remain visible even though it is not English.

Three independent layers are applied in order.

Layer A — graph construction (edges)

Applies to every hop, including those that will later be pruned.

• Allowed relationship_type set.
• min_confidence.
• Optional max_depth.

Layer B — leaf predicates

When no leaf constraints are set, candidate leaves are structural sinks under the construction graph (or nodes at max_depth). When leaf constraints are set, every visited non-root node that matches those constraints is a candidate — including productive words that still have further descendants (Spanish verde under Latin viridis must survive a Spanish leaf filter even though Tagalog berde was borrowed from it). A candidate is kept only if it matches all specified leaf constraints:

• Term contains / regex (case-insensitive, Unicode-aware).
• Language key or display-name set (multi-select).
• Language family (Glottolog Family_ID resolved to a name such as Indo-European).
• Macroarea (Africa, Eurasia, Papunesia, Australia, North America, South America).
• Vitality / Glottolog endangerment when known.
• Geographic: has coordinates; optional lat/lon bounding box.
• WALS: one or more (feature_id, allowed_values) clauses, AND-combined. Example: 81A in {SOV, SVO}.
• Phonology: inventory contains these IPA segments; inventory lacks these segments; has tone; segment class present (click, implosive approximated via IPA).
• Source dataset: etymology-db, lexibank, or both (inherited from incident edges).
• Minimum descendant fan-out of the leaf itself (rarely used; more often a search ranking feature).

Unknown metadata: by default a positive constraint fails closed (a language with no WALS 81A value does not count as SOV). A per-layer toggle keep_unknown lets sparse typological data through.

Layer C — path / intermediate predicates

Let path(leaf) = [root, hop1, hop2, ..., leaf].

Intermediates are path[1:-1] (root and leaf excluded). If the path has length 1 (the root is also a leaf), intermediate filters do not apply unless the user opts into apply_path_to_root.

Each intermediate constraint set is evaluated with an explicit quantifier:

Quantifier	Meaning
any (default)	At least one intermediate matches. "Passed through French."
all	Every intermediate matches. "Never left Romance."
none	No intermediate matches. "Did not pass through English."
exactly	Exactly k intermediates match.

Path constraints may include the same trait vocabulary as leaves (language, family, macroarea, WALS, phonemes, term pattern) plus two edge-aware clauses evaluated on the path's edges rather than nodes:

• path_relations_any: at least one hop has this relation (e.g. a borrowing occurred somewhere).
• path_relations_none: no hop has this relation (e.g. no compounds).

Pruning

After BFS:

1. Evaluate every candidate leaf against Layer B.
2. Evaluate path(leaf) against Layer C.
3. Keep a leaf only if both pass.
4. Rebuild the visible tree as the union of surviving root-to-leaf paths.
5. Drop nodes that are not on any surviving path (except the root, always kept).
6. Cluster remaining high-degree sibling lists for display (see Layout).

This is the only definition of "filter correctly ... for the leaf words and intermediate nodes along the path." Applying leaf language = English to every node would hide the French and Latin ancestors the user needs in order to understand the path.

Worked example

Root: Latin caput.

• Layer A: relations {inherited, borrowed, derived}, confidence >= 0.9, depth 4.
• Layer B: leaf language = English.
• Layer C: any intermediate language = French.

Kept: English chief, chef, capital when their BFS paths include a French (or Middle French) hop. Dropped: English head if it arrived only through Germanic, even though it is English; dropped French chef because it is not an English leaf.

Search and ranking

Autocomplete queries SQLite FTS5 on term with a Unicode61 tokenizer, plus a normalized shadow column (nfc, lowercased, leading * stripped, h2/h₂ folded) so PIE reconstructions remain findable.

Ranking for prefix q:

1. Exact term match.
2. Prefix match.
3. Substring / FTS rank.
4. Tie-break: child_count descending (productive etymons float up), then language name.

Each hit returns {term, lang, display_lang, family, child_count, sample_children}. Homographs are separate rows (Latin mater vs English mater).

A curated example list is shipped for first-run exploration: PIE meh2ter, Latin mater, Latin caput, Latin habere, English computer, Arabic kitab, Sanskrit cakra, etc.

Visualization

Views

1. Descent tree (default). Left-to-right tidy tree: etymon on the left, descendants growing right. This matches the reverse-etymology metaphor.
2. Radial. Root at center, depth as radius. Better on tablets for bushy trees.
3. Map. Surviving leaves (and optionally intermediates) plotted by language coordinates. Marker size = number of surviving words in that language; color = family. Clicking a marker lists those words and can isolate the language in the tree.
4. Table. Virtualized, sortable: term, language, family, relation from parent, confidence, depth, full path as a breadcrumb. CSV export.
5. Stats. Relation mix, family mix, depth histogram, borrowing vs inheritance ratio, macroarea breakdown for the current filtered tree.

Canvas renderer

SVG is too slow above a few hundred labeled nodes. The tree and radial views use a single high-DPI canvas:

• Layout via d3-hierarchy (tree / cluster) computed on a web worker when the node count exceeds 800.
• Camera: pan, wheel zoom, pinch zoom, double-tap zoom, keyboard arrows.
• Hit testing: uniform grid hashed by screen cell, rebuilt on camera change.
• Level of detail: labels fade in by zoom and by node importance (child_count, depth, selection).
• Minimap in the corner on desktop; hidden on narrow screens unless toggled.
• Selection: click a node to open the inspector; hover previews the BFS path back to the root in a brighter stroke.
• Re-root: "descend from here" makes the node the new query without leaving the UI.

Clustering (speed and legibility)

Unfiltered reverse walks of productive affixes are huge (ly English has 13,422 direct children). After filtering, if a node still has more than cluster_threshold (default 28) children:

1. Group by language.
2. If a single language still exceeds the threshold, group by initial letter.
3. Emit a cluster stub {kind: "cluster", label, count, relation_mix} that expands on click.

The server never ships more than max_payload_nodes (2,500) in one response. Expansion requests pass expand keys so only the requested cluster is unpacked, still subject to the same filters.

Color and encoding

User-selectable color mode:

• Relation from parent (default): inherited teal, borrowed copper, derived moss, calque violet, clipping/abbreviation slate, compound/blend ochre, other ink-wash gray, cognate muted magenta.
• Language family (hashed qualitative palette, stable per family name).
• Confidence (sequential).
• Depth (sequential).
• Macroarea.

Stroke width encodes child_count on the visible subtree. A legend is always present.

Inspector

Slide-over on desktop, bottom sheet on mobile:

• Term, language, family, coordinates, vitality.
• Incident reverse children (counts by relation).
• Forward etymons ("this word comes from...").
• IE-CoR cognate sets containing the form, when any.
• WALS highlights (word order, case, tone).
• Phoneme inventory summary (counts by class, tone yes/no, searchable list).
• Wiktionary deep link for etymology-db nodes.
• Actions: re-root, isolate this language, copy path, exclude this language from leaves / from path.

Mobile

"Viewable on mobile" is a first-class layout, not a shrunk desktop.

• Responsive shell: below 840px the filter panel and inspector become bottom sheets; the canvas is full-bleed under a compact chrome (search + view switcher + filter chip count).
• Touch: one-finger pan, pinch zoom, tap select, two-finger rotate disabled (it fights tree reading).
• Filter chips summarize active leaf vs path constraints so a phone user can see the query without opening the sheet.
• PWA manifest + service worker cache the app shell (not the 4M-edge index). Add-to-home-screen works when served over HTTPS or localhost.
• The API binds 0.0.0.0 so a phone on the same network can open http://<lan-ip>:8000. A "Open on phone" dialog shows that origin as a QR code.
• Shareable URLs encode the full query (root, layers A/B/C, view, color mode). Opening the URL on a phone restores the same tree.

Architecture

parquet + CLDF CSVs
        |
        v
 pipeline/build_index.py
        |
        +--> data/index/atlas.sqlite   (nodes, FTS, languages, WALS, phonemes, cognates)
        +--> data/index/graph.npz      (CSR: offsets, targets, rel_code, conf_code, src_code)
        +--> data/index/meta.json      (vocabularies, color maps, examples)
        |
        v
 FastAPI  --loads CSR into RAM once--  React canvas SPA

Why this split:

• SQLite FTS is the right tool for autocomplete and inspector joins.
• Pure SQL recursive CTEs cannot apply Layer C quantifiers quickly enough on 13k-way affixes. An in-memory CSR BFS in Python over int32 arrays does this in tens of milliseconds.
• The SPA never downloads the full graph. It only receives the pruned, clustered payload for the current query.

Estimated RAM after load: well under 500 MB (CSR ~30 MB, interned trait arrays ~20 MB, SQLite mmap, Python overhead).

API

All JSON. CORS open. No auth.

Method	Path	Purpose
GET	/api/health	Index stats, load time
GET	/api/suggest?q=&limit=	Autocomplete
GET	/api/meta	Vocabularies for filter widgets
POST	/api/tree	Filtered descendant tree
GET	/api/node?term=&lang=	Inspector payload
GET	/api/cognates?term=&lang=	IE-CoR sets
POST	/api/export	Same tree as CSV or JSON download
GET	/api/examples	Curated roots

POST /api/tree body is the serialized filter document (see Implementation plan). Response:

{
  "root": {"id": 0, "term": "mater", "lang": "latin", "...": ""},
  "nodes": [{"id": 1, "parent": 0, "kind": "word"|"cluster", "...": ""}],
  "stats": {"visited": 1800, "kept_leaves": 42, "elapsed_ms": 18},
  "truncated": false
}

Performance budget

Action	Target
Process start including mmap load	< 3 s on a laptop
Autocomplete	< 30 ms
Typical tree (depth 3, < 2k visit)	< 80 ms
Worst productive affix, filtered	< 400 ms
Canvas pan/zoom	60 fps at 2,500 nodes
Mobile first interaction	usable within one round trip

Techniques: CSR int32, BFS with typed arrays, cluster stubs, canvas LOD, SQLite covering indexes, gzip JSON, immutable filter objects so React does not relayout on every keystroke (search input is debounced 120 ms).

Accessibility and i18n

• Canvas is supplemented by the table view (keyboard reachable, screen-reader friendly).
• Focus states on all controls; / focuses search; f toggles filters; 1-4 switch views.
• Color is never the only encoding (relation also appears as a short label and in the inspector).
• Terms are rendered as stored (IPA, Greek, reconstructions). UI chrome is English.

Non-goals (this version)

• Editing or crowdsourcing etymologies.
• Full phylogenetic tree inference.
• Offline copies of Wiktionary prose.
• Account/sync in the cloud.

Visual language

The UI should read as a night atlas: deep ink navy, warm parchment text, copper and sea-teal relation colors, serif display for headwords, sans for chrome. No generic dashboard cards, no rainbow force-directed hairball as the default.