Spaces:
Running
Match places on their English name as well as their own
Browse filesFuzzy search scored each record on `names.primary` alone, so a place whose
English name differs from the name it is filed under was unreachable by the
English name — and the miss came back as a confident-looking ranked list.
"Copenhagen" returned Congaree, a lake centerline in South Carolina;
"Vienna" returned Vienne in France, ~1000 km from the intended place and
plausible enough to build a bounding box from.
The English name was already on the record: `search_divisions_area` matched
`names.primary` while `fetch_by_id` displayed `names.common.en`, so the same
row answered to two different names depending on the endpoint.
Every record is now scored under both names and keeps the better of the two.
Neither works alone: `names.common.en` is null for 77% of divisions_area rows,
which matching on English alone would drop, and `names.primary` alone is what
this bug is. Rows with no English name match exactly as before. The same
applies to natural_earth, whose conversion writes `names.en`.
- `name` is the English name where the record has one and its own name
otherwise, in search and in `GET /geometry/{id}` alike, so a candidate and a
later lookup of it agree on what the place is called. It was previously null
for the 77% on the lookup side.
- `matched_name` reports which of the two names the query hit, so a caller can
show "Copenhagen" against a record titled "Københavns Kommune".
- `similarity` and `is_substring_match` reach `ids_only` responses. They were
computed, used for ranking and then dropped, and Jaro-Winkler always returns
a best row: without a score a caller cannot tell a hit from a total miss, and
cannot apply a floor of its own.
- A blank name is treated as absent rather than as a name to display.
Verified against the deployed parquet files: Copenhagen, Munich, Vienna and
Denmark each return the intended place as the top hit across both sources,
and Loja still ranks its province and canton first.
Closes #10
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- README.md +2 -2
- src/gazet/api.py +22 -9
- src/gazet/schemas.py +10 -0
- src/gazet/search.py +92 -16
- tests/conftest.py +92 -0
- tests/test_api.py +57 -0
- tests/test_search.py +70 -0
|
@@ -90,8 +90,8 @@ uv run streamlit run gazet_demo.py # demo UI
|
|
| 90 |
## Design notes
|
| 91 |
|
| 92 |
- `api.py` exposes GET `/search?q=<query>`; returns GeoJSON FeatureCollection and logs intermediate output. This path runs two LLM calls (place extraction, then SQL generation for candidate selection/disambiguation and GIS ops).
|
| 93 |
-
- `api.py` also exposes GET `/search/fuzzy?q=<place name>&limit=5&simplify=true&sources=divisions_area,natural_earth` — pure Jaro-Winkler fuzzy match with no LLM involved. `q` is a place-name string, not a natural-language query (there's no place-extraction step). Returns the combined top-`limit` matches across sources as a GeoJSON FeatureCollection with raw (or simplified, if `simplify=true`) geometry attached. Pass `ids_only=true` to skip full geometry and get back `{"ids": [{"source", "id", "name", "country", "subtype", "admin_level", "bbox"}, ...]}` — `country`/`subtype`/`admin_level` disambiguate same-named places (e.g. multiple real "Loja"s across Ecuador and Spain, or Ecuador's "Loja" region vs. its nested "Loja" county),
|
| 94 |
-
- Fuzzy matching (`search.py`) ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity.
|
| 95 |
- `api.py` also exposes GET `/geometry/{id}?source=divisions_area&simplify=true` — fetch a single feature's geometry directly by ID, no fuzzy matching or LLM involved. `source` is inferred from the ID if omitted (Natural Earth IDs are prefixed `ne_`). Returns a single GeoJSON Feature, 404 if not found.
|
| 96 |
- The spatial extension is loaded once at API startup (FastAPI `lifespan`) onto a shared DuckDB connection stored on `app.state`; each request gets a cheap `cursor()` off it instead of paying DuckDB's ~90ms `LOAD spatial` cost per call.
|
| 97 |
- LM is initialised at import time in `lm.py`, suitable for a long-lived server process.
|
|
|
|
| 90 |
## Design notes
|
| 91 |
|
| 92 |
- `api.py` exposes GET `/search?q=<query>`; returns GeoJSON FeatureCollection and logs intermediate output. This path runs two LLM calls (place extraction, then SQL generation for candidate selection/disambiguation and GIS ops).
|
| 93 |
+
- `api.py` also exposes GET `/search/fuzzy?q=<place name>&limit=5&simplify=true&sources=divisions_area,natural_earth` — pure Jaro-Winkler fuzzy match with no LLM involved. `q` is a place-name string, not a natural-language query (there's no place-extraction step). Returns the combined top-`limit` matches across sources as a GeoJSON FeatureCollection with raw (or simplified, if `simplify=true`) geometry attached. Pass `ids_only=true` to skip full geometry and get back `{"ids": [{"source", "id", "name", "matched_name", "country", "subtype", "admin_level", "bbox", "similarity", "is_substring_match"}, ...]}` — `country`/`subtype`/`admin_level` disambiguate same-named places (e.g. multiple real "Loja"s across Ecuador and Spain, or Ecuador's "Loja" region vs. its nested "Loja" county), `bbox` is `[minx, miny, maxx, maxy]`, a much smaller payload than full geometry, for picking a candidate before fetching its full geometry, and `similarity` is the score the list was ranked on — Jaro-Winkler always returns a best row, so without it a caller cannot tell a hit from the closest thing to a miss.
|
| 94 |
+
- Fuzzy matching (`search.py`) ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity. Each record is matched under two names and keeps the better score: its own (`names.primary`) and its English one (`names.common.en` for `divisions_area`, `names.en` for `natural_earth`). Neither works alone — `names.common.en` is null for 77% of `divisions_area` rows, which matching on English alone would drop, while matching on `names.primary` alone leaves a place unreachable by the English name it is usually asked for ("Copenhagen" for `Københavns Kommune`). `matched_name` reports which of the two the query hit; `name` is the English one where the record has one, the same expression `/geometry/{id}` uses, so a candidate and a later lookup of it agree on what the place is called.
|
| 95 |
- `api.py` also exposes GET `/geometry/{id}?source=divisions_area&simplify=true` — fetch a single feature's geometry directly by ID, no fuzzy matching or LLM involved. `source` is inferred from the ID if omitted (Natural Earth IDs are prefixed `ne_`). Returns a single GeoJSON Feature, 404 if not found.
|
| 96 |
- The spatial extension is loaded once at API startup (FastAPI `lifespan`) onto a shared DuckDB connection stored on `app.state`; each request gets a cheap `cursor()` off it instead of paying DuckDB's ~90ms `LOAD spatial` cost per call.
|
| 97 |
- LM is initialised at import time in `lm.py`, suitable for a long-lived server process.
|
|
@@ -62,8 +62,8 @@ _Q_SOURCES = Query(
|
|
| 62 |
_Q_IDS_ONLY = Query(
|
| 63 |
False,
|
| 64 |
description=(
|
| 65 |
-
"For mode=fuzzy: return lightweight id/bbox records
|
| 66 |
-
"geometry; ignored for mode=nl."
|
| 67 |
),
|
| 68 |
)
|
| 69 |
|
|
@@ -74,7 +74,8 @@ _TAGS_METADATA = [
|
|
| 74 |
"`mode=nl` (default) runs the full natural-language pipeline: "
|
| 75 |
"LLM place-extraction, fuzzy candidate matching, then LLM-generated "
|
| 76 |
"SQL. `mode=fuzzy` skips the LLM entirely and does a direct "
|
| 77 |
-
"Jaro-Winkler name match
|
|
|
|
| 78 |
),
|
| 79 |
},
|
| 80 |
{"name": "geometry", "description": "Direct, non-fuzzy geometry lookups by ID."},
|
|
@@ -356,13 +357,15 @@ def _fuzzy_search(
|
|
| 356 |
place-extraction step. Matches are ranked by Jaro-Winkler similarity
|
| 357 |
across the requested ``sources`` (comma-separated subset of
|
| 358 |
divisions_area/natural_earth; defaults to both), combined and truncated
|
| 359 |
-
to the top ``limit``.
|
|
|
|
| 360 |
|
| 361 |
Pass ``ids_only=true`` to skip fetching full geometry and get back
|
| 362 |
-
lightweight candidates (id/name/country/subtype/admin_level/bbox
|
| 363 |
-
instead — enough to disambiguate
|
| 364 |
-
real-world "Loja"s across Ecuador and
|
| 365 |
-
|
|
|
|
| 366 |
"""
|
| 367 |
requested_sources = (
|
| 368 |
tuple(s.strip() for s in sources.split(",")) if sources else _FUZZY_SOURCES
|
|
@@ -399,7 +402,17 @@ def _fuzzy_search(
|
|
| 399 |
)
|
| 400 |
|
| 401 |
if ids_only:
|
| 402 |
-
scalar_cols = [
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 403 |
ids_df = candidates_df[scalar_cols].copy()
|
| 404 |
ids_df = ids_df.astype(object).where(ids_df.notna(), None)
|
| 405 |
ids_df["bbox"] = candidates_df["bbox"].apply(
|
|
|
|
| 62 |
_Q_IDS_ONLY = Query(
|
| 63 |
False,
|
| 64 |
description=(
|
| 65 |
+
"For mode=fuzzy: return lightweight id/bbox/similarity records "
|
| 66 |
+
"instead of full geometry; ignored for mode=nl."
|
| 67 |
),
|
| 68 |
)
|
| 69 |
|
|
|
|
| 74 |
"`mode=nl` (default) runs the full natural-language pipeline: "
|
| 75 |
"LLM place-extraction, fuzzy candidate matching, then LLM-generated "
|
| 76 |
"SQL. `mode=fuzzy` skips the LLM entirely and does a direct "
|
| 77 |
+
"Jaro-Winkler name match, against each record's own name and its "
|
| 78 |
+
"English one — `q` is a place name, not a sentence."
|
| 79 |
),
|
| 80 |
},
|
| 81 |
{"name": "geometry", "description": "Direct, non-fuzzy geometry lookups by ID."},
|
|
|
|
| 357 |
place-extraction step. Matches are ranked by Jaro-Winkler similarity
|
| 358 |
across the requested ``sources`` (comma-separated subset of
|
| 359 |
divisions_area/natural_earth; defaults to both), combined and truncated
|
| 360 |
+
to the top ``limit``. Each record is matched under both its own name and
|
| 361 |
+
its English one, so "Copenhagen" and "København" find the same place.
|
| 362 |
|
| 363 |
Pass ``ids_only=true`` to skip fetching full geometry and get back
|
| 364 |
+
lightweight candidates (id/name/country/subtype/admin_level/bbox, plus
|
| 365 |
+
the ``similarity`` they were ranked on) instead — enough to disambiguate
|
| 366 |
+
same-named places (e.g. multiple real-world "Loja"s across Ecuador and
|
| 367 |
+
Spain), and to reject the lot, before fetching the full geometry for one
|
| 368 |
+
candidate via ``GET /geometry/{id}``.
|
| 369 |
"""
|
| 370 |
requested_sources = (
|
| 371 |
tuple(s.strip() for s in sources.split(",")) if sources else _FUZZY_SOURCES
|
|
|
|
| 402 |
)
|
| 403 |
|
| 404 |
if ids_only:
|
| 405 |
+
scalar_cols = [
|
| 406 |
+
"source",
|
| 407 |
+
"id",
|
| 408 |
+
"name",
|
| 409 |
+
"matched_name",
|
| 410 |
+
"country",
|
| 411 |
+
"subtype",
|
| 412 |
+
"admin_level",
|
| 413 |
+
"similarity",
|
| 414 |
+
"is_substring_match",
|
| 415 |
+
]
|
| 416 |
ids_df = candidates_df[scalar_cols].copy()
|
| 417 |
ids_df = ids_df.astype(object).where(ids_df.notna(), None)
|
| 418 |
ids_df["bbox"] = candidates_df["bbox"].apply(
|
|
@@ -40,15 +40,25 @@ class FuzzyIdItem(BaseModel):
|
|
| 40 |
"Loja" region vs. its nested "Loja" county). ``bbox`` is
|
| 41 |
``[minx, miny, maxx, maxy]``, a much smaller payload than full geometry —
|
| 42 |
fetch the full geometry for one candidate via ``GET /geometry/{id}``.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
"""
|
| 44 |
|
| 45 |
source: str
|
| 46 |
id: str
|
| 47 |
name: str | None = None
|
|
|
|
| 48 |
country: str | None = None
|
| 49 |
subtype: str | None = None
|
| 50 |
admin_level: int | None = None
|
| 51 |
bbox: list[float] | None = None
|
|
|
|
|
|
|
| 52 |
|
| 53 |
|
| 54 |
class FuzzyIdsResult(BaseModel):
|
|
|
|
| 40 |
"Loja" region vs. its nested "Loja" county). ``bbox`` is
|
| 41 |
``[minx, miny, maxx, maxy]``, a much smaller payload than full geometry —
|
| 42 |
fetch the full geometry for one candidate via ``GET /geometry/{id}``.
|
| 43 |
+
|
| 44 |
+
``name`` is the record's English name where it has one and its own name
|
| 45 |
+
otherwise; ``matched_name`` is the one the query actually matched, which
|
| 46 |
+
is how a search for "Copenhagen" can return a record titled "Københavns
|
| 47 |
+
Kommune". ``similarity`` and ``is_substring_match`` are the ranking
|
| 48 |
+
itself: Jaro-Winkler always returns a best row, so a caller with no score
|
| 49 |
+
cannot tell a hit from the closest thing to a miss.
|
| 50 |
"""
|
| 51 |
|
| 52 |
source: str
|
| 53 |
id: str
|
| 54 |
name: str | None = None
|
| 55 |
+
matched_name: str | None = None
|
| 56 |
country: str | None = None
|
| 57 |
subtype: str | None = None
|
| 58 |
admin_level: int | None = None
|
| 59 |
bbox: list[float] | None = None
|
| 60 |
+
similarity: float | None = None
|
| 61 |
+
is_substring_match: bool | None = None
|
| 62 |
|
| 63 |
|
| 64 |
class FuzzyIdsResult(BaseModel):
|
|
@@ -8,13 +8,26 @@ from .schemas import Place
|
|
| 8 |
|
| 9 |
logger = logging.getLogger(__name__)
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
def simple_fuzzy_search(
|
| 13 |
con: duckdb.DuckDBPyConnection,
|
| 14 |
path: str,
|
| 15 |
source: str,
|
| 16 |
place: Place,
|
| 17 |
-
name_expr: str =
|
|
|
|
| 18 |
extra_select: str = "",
|
| 19 |
limit: int = 5,
|
| 20 |
include_geometry: bool = False,
|
|
@@ -22,6 +35,17 @@ def simple_fuzzy_search(
|
|
| 22 |
) -> pd.DataFrame:
|
| 23 |
"""Jaro-Winkler similarity search using only the place name.
|
| 24 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
Ranks exact-substring matches (query literally contained in the name,
|
| 26 |
e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses
|
| 27 |
(e.g. "Lodja"), even when the near-miss scores a slightly higher raw
|
|
@@ -35,8 +59,9 @@ def simple_fuzzy_search(
|
|
| 35 |
(no coordinate arrays or GeoJSON serialization), for lightweight
|
| 36 |
context (e.g. ``ids_only`` responses).
|
| 37 |
"""
|
| 38 |
-
params = [place.place, place.place,
|
| 39 |
|
|
|
|
| 40 |
extra_clause = f", {extra_select}" if extra_select else ""
|
| 41 |
geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
|
| 42 |
bbox_clause = (
|
|
@@ -44,22 +69,64 @@ def simple_fuzzy_search(
|
|
| 44 |
if include_bbox
|
| 45 |
else ""
|
| 46 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
rel = con.execute(
|
| 48 |
f"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
SELECT
|
| 50 |
id,
|
| 51 |
-
|
| 52 |
country,
|
| 53 |
subtype,
|
| 54 |
class,
|
| 55 |
region,
|
| 56 |
admin_level,
|
| 57 |
is_land,
|
| 58 |
-
is_territorial{extra_clause}{
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
ORDER BY is_substring_match DESC, similarity DESC, admin_level ASC
|
| 64 |
LIMIT ?
|
| 65 |
""",
|
|
@@ -83,16 +150,18 @@ def search_divisions_area(
|
|
| 83 |
) -> pd.DataFrame:
|
| 84 |
"""Fuzzy-match a place against divisions_area (Overture admin boundaries).
|
| 85 |
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
|
|
|
|
|
|
| 89 |
"""
|
| 90 |
return simple_fuzzy_search(
|
| 91 |
con,
|
| 92 |
DIVISIONS_AREA_PATH,
|
| 93 |
"divisions_area",
|
| 94 |
place,
|
| 95 |
-
|
| 96 |
extra_select="division_id",
|
| 97 |
limit=limit,
|
| 98 |
include_geometry=include_geometry,
|
|
@@ -113,7 +182,7 @@ def search_natural_earth(
|
|
| 113 |
NATURAL_EARTH_PATH,
|
| 114 |
"natural_earth",
|
| 115 |
place,
|
| 116 |
-
|
| 117 |
limit=limit,
|
| 118 |
include_geometry=include_geometry,
|
| 119 |
include_bbox=include_bbox,
|
|
@@ -125,22 +194,28 @@ def fetch_by_id(
|
|
| 125 |
path: str,
|
| 126 |
source: str,
|
| 127 |
id: str,
|
| 128 |
-
name_expr: str =
|
|
|
|
| 129 |
extra_select: str = "",
|
| 130 |
include_geometry: bool = True,
|
| 131 |
) -> pd.DataFrame:
|
| 132 |
"""Look up a single record by exact ID — no fuzzy matching.
|
| 133 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
``include_geometry`` requires the spatial extension to already be
|
| 135 |
loaded on ``con`` (``INSTALL spatial; LOAD spatial;``).
|
| 136 |
"""
|
|
|
|
| 137 |
extra_clause = f", {extra_select}" if extra_select else ""
|
| 138 |
geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
|
| 139 |
rel = con.execute(
|
| 140 |
f"""
|
| 141 |
SELECT
|
| 142 |
id,
|
| 143 |
-
{name_expr} AS name,
|
| 144 |
country,
|
| 145 |
subtype,
|
| 146 |
class,
|
|
@@ -168,6 +243,7 @@ def get_division_by_id(
|
|
| 168 |
DIVISIONS_AREA_PATH,
|
| 169 |
"divisions_area",
|
| 170 |
id,
|
|
|
|
| 171 |
extra_select="division_id",
|
| 172 |
include_geometry=include_geometry,
|
| 173 |
)
|
|
@@ -182,7 +258,7 @@ def get_natural_earth_by_id(
|
|
| 182 |
NATURAL_EARTH_PATH,
|
| 183 |
"natural_earth",
|
| 184 |
id,
|
| 185 |
-
|
| 186 |
include_geometry=include_geometry,
|
| 187 |
)
|
| 188 |
|
|
|
|
| 8 |
|
| 9 |
logger = logging.getLogger(__name__)
|
| 10 |
|
| 11 |
+
#: Where each source keeps a record's own name and its English one. Overture
|
| 12 |
+
#: holds translations in a ``names.common`` map; the Natural Earth conversion
|
| 13 |
+
#: gives every language a struct field of its own (``ingest/convert_natural_earth.py``).
|
| 14 |
+
PRIMARY_NAME = "names.primary"
|
| 15 |
+
DIVISIONS_AREA_ENGLISH_NAME = "names.common.en"
|
| 16 |
+
NATURAL_EARTH_ENGLISH_NAME = "names.en"
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
def _readable(expr: str) -> str:
|
| 20 |
+
"""SQL that reads a name, treating a blank one as absent."""
|
| 21 |
+
return f"NULLIF(trim({expr}), '')"
|
| 22 |
+
|
| 23 |
|
| 24 |
def simple_fuzzy_search(
|
| 25 |
con: duckdb.DuckDBPyConnection,
|
| 26 |
path: str,
|
| 27 |
source: str,
|
| 28 |
place: Place,
|
| 29 |
+
name_expr: str = PRIMARY_NAME,
|
| 30 |
+
english_name_expr: str | None = None,
|
| 31 |
extra_select: str = "",
|
| 32 |
limit: int = 5,
|
| 33 |
include_geometry: bool = False,
|
|
|
|
| 35 |
) -> pd.DataFrame:
|
| 36 |
"""Jaro-Winkler similarity search using only the place name.
|
| 37 |
|
| 38 |
+
Each record is matched under two names: ``name_expr``, its own, and
|
| 39 |
+
``english_name_expr``, its English one where the source carries it. A
|
| 40 |
+
record scores the better of the two, so a place is reachable by its
|
| 41 |
+
English exonym ("Copenhagen") as well as by its local name ("Københavns
|
| 42 |
+
Kommune"), and a record with no English name matches on its own name
|
| 43 |
+
alone. ``matched_name`` is whichever of the two produced the score.
|
| 44 |
+
|
| 45 |
+
``name`` is the English name where there is one and the record's own name
|
| 46 |
+
otherwise — the expression :func:`fetch_by_id` uses too, so a candidate
|
| 47 |
+
and a later lookup of that same candidate agree on what it is called.
|
| 48 |
+
|
| 49 |
Ranks exact-substring matches (query literally contained in the name,
|
| 50 |
e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses
|
| 51 |
(e.g. "Lodja"), even when the near-miss scores a slightly higher raw
|
|
|
|
| 59 |
(no coordinate arrays or GeoJSON serialization), for lightweight
|
| 60 |
context (e.g. ``ids_only`` responses).
|
| 61 |
"""
|
| 62 |
+
params = [path, place.place, place.place, place.place, place.place, limit]
|
| 63 |
|
| 64 |
+
english_expr = english_name_expr or "CAST(NULL AS VARCHAR)"
|
| 65 |
extra_clause = f", {extra_select}" if extra_select else ""
|
| 66 |
geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
|
| 67 |
bbox_clause = (
|
|
|
|
| 69 |
if include_bbox
|
| 70 |
else ""
|
| 71 |
)
|
| 72 |
+
# Geometry and bbox are computed in the first stage, so the final select
|
| 73 |
+
# only has to name them.
|
| 74 |
+
carried_clause = (", geometry" if include_geometry else "") + (
|
| 75 |
+
", bbox" if include_bbox else ""
|
| 76 |
+
)
|
| 77 |
rel = con.execute(
|
| 78 |
f"""
|
| 79 |
+
WITH named AS (
|
| 80 |
+
SELECT
|
| 81 |
+
id,
|
| 82 |
+
country,
|
| 83 |
+
subtype,
|
| 84 |
+
class,
|
| 85 |
+
region,
|
| 86 |
+
admin_level,
|
| 87 |
+
is_land,
|
| 88 |
+
is_territorial{extra_clause}{geometry_clause}{bbox_clause},
|
| 89 |
+
{_readable(name_expr)} AS primary_name,
|
| 90 |
+
{_readable(english_expr)} AS english_name
|
| 91 |
+
FROM read_parquet(?)
|
| 92 |
+
),
|
| 93 |
+
scored AS (
|
| 94 |
+
SELECT
|
| 95 |
+
*,
|
| 96 |
+
COALESCE(
|
| 97 |
+
jaro_winkler_similarity(lower(primary_name), lower(?)), 0.0
|
| 98 |
+
) AS primary_similarity,
|
| 99 |
+
COALESCE(
|
| 100 |
+
jaro_winkler_similarity(lower(english_name), lower(?)), 0.0
|
| 101 |
+
) AS english_similarity,
|
| 102 |
+
COALESCE(contains(lower(primary_name), lower(?)), false)
|
| 103 |
+
AS primary_substring,
|
| 104 |
+
COALESCE(contains(lower(english_name), lower(?)), false)
|
| 105 |
+
AS english_substring
|
| 106 |
+
FROM named
|
| 107 |
+
WHERE primary_name IS NOT NULL OR english_name IS NOT NULL
|
| 108 |
+
)
|
| 109 |
SELECT
|
| 110 |
id,
|
| 111 |
+
COALESCE(english_name, primary_name) AS name,
|
| 112 |
country,
|
| 113 |
subtype,
|
| 114 |
class,
|
| 115 |
region,
|
| 116 |
admin_level,
|
| 117 |
is_land,
|
| 118 |
+
is_territorial{extra_clause}{carried_clause},
|
| 119 |
+
-- Compared as a pair so the winning name is the one the ranking
|
| 120 |
+
-- below would pick: a substring hit beats a higher raw score.
|
| 121 |
+
CASE
|
| 122 |
+
WHEN (english_substring, english_similarity)
|
| 123 |
+
> (primary_substring, primary_similarity)
|
| 124 |
+
THEN english_name
|
| 125 |
+
ELSE COALESCE(primary_name, english_name)
|
| 126 |
+
END AS matched_name,
|
| 127 |
+
GREATEST(primary_similarity, english_similarity) AS similarity,
|
| 128 |
+
primary_substring OR english_substring AS is_substring_match
|
| 129 |
+
FROM scored
|
| 130 |
ORDER BY is_substring_match DESC, similarity DESC, admin_level ASC
|
| 131 |
LIMIT ?
|
| 132 |
""",
|
|
|
|
| 150 |
) -> pd.DataFrame:
|
| 151 |
"""Fuzzy-match a place against divisions_area (Overture admin boundaries).
|
| 152 |
|
| 153 |
+
Matched on ``names.primary`` and on ``names.common.en`` together: 77% of
|
| 154 |
+
divisions_area rows have no English common name and would drop out of
|
| 155 |
+
every search if English were the only match field, while ``names.primary``
|
| 156 |
+
on its own leaves a place unreachable by the English name it is usually
|
| 157 |
+
asked for.
|
| 158 |
"""
|
| 159 |
return simple_fuzzy_search(
|
| 160 |
con,
|
| 161 |
DIVISIONS_AREA_PATH,
|
| 162 |
"divisions_area",
|
| 163 |
place,
|
| 164 |
+
english_name_expr=DIVISIONS_AREA_ENGLISH_NAME,
|
| 165 |
extra_select="division_id",
|
| 166 |
limit=limit,
|
| 167 |
include_geometry=include_geometry,
|
|
|
|
| 182 |
NATURAL_EARTH_PATH,
|
| 183 |
"natural_earth",
|
| 184 |
place,
|
| 185 |
+
english_name_expr=NATURAL_EARTH_ENGLISH_NAME,
|
| 186 |
limit=limit,
|
| 187 |
include_geometry=include_geometry,
|
| 188 |
include_bbox=include_bbox,
|
|
|
|
| 194 |
path: str,
|
| 195 |
source: str,
|
| 196 |
id: str,
|
| 197 |
+
name_expr: str = PRIMARY_NAME,
|
| 198 |
+
english_name_expr: str | None = None,
|
| 199 |
extra_select: str = "",
|
| 200 |
include_geometry: bool = True,
|
| 201 |
) -> pd.DataFrame:
|
| 202 |
"""Look up a single record by exact ID — no fuzzy matching.
|
| 203 |
|
| 204 |
+
``name`` is built as :func:`simple_fuzzy_search` builds it — the English
|
| 205 |
+
name where the record has one, its own name otherwise — so a record
|
| 206 |
+
answers to a single name whichever way it is reached.
|
| 207 |
+
|
| 208 |
``include_geometry`` requires the spatial extension to already be
|
| 209 |
loaded on ``con`` (``INSTALL spatial; LOAD spatial;``).
|
| 210 |
"""
|
| 211 |
+
english_expr = english_name_expr or "CAST(NULL AS VARCHAR)"
|
| 212 |
extra_clause = f", {extra_select}" if extra_select else ""
|
| 213 |
geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
|
| 214 |
rel = con.execute(
|
| 215 |
f"""
|
| 216 |
SELECT
|
| 217 |
id,
|
| 218 |
+
COALESCE({_readable(english_expr)}, {_readable(name_expr)}) AS name,
|
| 219 |
country,
|
| 220 |
subtype,
|
| 221 |
class,
|
|
|
|
| 243 |
DIVISIONS_AREA_PATH,
|
| 244 |
"divisions_area",
|
| 245 |
id,
|
| 246 |
+
english_name_expr=DIVISIONS_AREA_ENGLISH_NAME,
|
| 247 |
extra_select="division_id",
|
| 248 |
include_geometry=include_geometry,
|
| 249 |
)
|
|
|
|
| 258 |
NATURAL_EARTH_PATH,
|
| 259 |
"natural_earth",
|
| 260 |
id,
|
| 261 |
+
english_name_expr=NATURAL_EARTH_ENGLISH_NAME,
|
| 262 |
include_geometry=include_geometry,
|
| 263 |
)
|
| 264 |
|
|
@@ -35,3 +35,95 @@ def con_no_spatial():
|
|
| 35 |
c = duckdb.connect()
|
| 36 |
yield c
|
| 37 |
c.close()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
c = duckdb.connect()
|
| 36 |
yield c
|
| 37 |
c.close()
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
# ---------------------------------------------------------------------------
|
| 41 |
+
# English-exonym fixture
|
| 42 |
+
# ---------------------------------------------------------------------------
|
| 43 |
+
|
| 44 |
+
#: Places whose English name differs from the name the record is filed under,
|
| 45 |
+
#: shaped like Overture divisions_area rows: ``names.primary`` is the local
|
| 46 |
+
#: name and ``names.common`` is a translation map that may or may not carry
|
| 47 |
+
#: ``en``. ``Coper`` is the Colombian county a search for "Copenhagen" used to
|
| 48 |
+
#: land on, and it has no English name — the 77% of the real table that a
|
| 49 |
+
#: match on English alone would drop. ``Genève`` carries a blank one, which is
|
| 50 |
+
#: absent rather than a name the record can be shown under.
|
| 51 |
+
_EXONYM_ROWS = [
|
| 52 |
+
(
|
| 53 |
+
"cph",
|
| 54 |
+
"Københavns Kommune",
|
| 55 |
+
"Copenhagen Municipality",
|
| 56 |
+
"DK",
|
| 57 |
+
"county",
|
| 58 |
+
2,
|
| 59 |
+
12.5,
|
| 60 |
+
55.6,
|
| 61 |
+
),
|
| 62 |
+
("dnk", "Danmark", "Denmark", "DK", "country", 0, 9.5, 55.5),
|
| 63 |
+
("muc", "München", "Munich", "DE", "county", 2, 11.5, 48.1),
|
| 64 |
+
("vie", "Wien", "Vienna", "AT", "region", 1, 16.3, 48.2),
|
| 65 |
+
("coper", "Coper", None, "CO", "county", 2, -74.0, 5.4),
|
| 66 |
+
("ratnapura", None, "Ratnapura District", "LK", "county", 2, 80.4, 6.7),
|
| 67 |
+
("gva", "Genève", "", "CH", "county", 2, 6.1, 46.2),
|
| 68 |
+
]
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
@pytest.fixture(scope="session")
|
| 72 |
+
def exonym_parquet(tmp_path_factory):
|
| 73 |
+
"""Write the exonym rows to a parquet with the divisions_area schema."""
|
| 74 |
+
path = tmp_path_factory.mktemp("exonym") / "divisions_area.parquet"
|
| 75 |
+
c = duckdb.connect()
|
| 76 |
+
c.execute("INSTALL spatial")
|
| 77 |
+
c.execute("LOAD spatial")
|
| 78 |
+
rows = ",\n".join(
|
| 79 |
+
"(%s, %s, %s, '%s', '%s', %d, %f, %f)"
|
| 80 |
+
% (
|
| 81 |
+
f"'{id}'",
|
| 82 |
+
"NULL" if primary is None else f"'{primary}'",
|
| 83 |
+
"MAP([], [])" if english is None else f"MAP(['en'], ['{english}'])",
|
| 84 |
+
country,
|
| 85 |
+
subtype,
|
| 86 |
+
admin_level,
|
| 87 |
+
lon,
|
| 88 |
+
lat,
|
| 89 |
+
)
|
| 90 |
+
for id, primary, english, country, subtype, admin_level, lon, lat in _EXONYM_ROWS
|
| 91 |
+
)
|
| 92 |
+
c.execute(
|
| 93 |
+
f"""
|
| 94 |
+
COPY (
|
| 95 |
+
SELECT
|
| 96 |
+
id,
|
| 97 |
+
ST_AsWKB(
|
| 98 |
+
ST_GeomFromText(
|
| 99 |
+
'POLYGON((' || lon || ' ' || lat || ', '
|
| 100 |
+
|| (lon + 0.1) || ' ' || lat || ', '
|
| 101 |
+
|| (lon + 0.1) || ' ' || (lat + 0.1) || ', '
|
| 102 |
+
|| lon || ' ' || (lat + 0.1) || ', '
|
| 103 |
+
|| lon || ' ' || lat || '))'
|
| 104 |
+
)
|
| 105 |
+
) AS geometry,
|
| 106 |
+
country,
|
| 107 |
+
subtype,
|
| 108 |
+
'land' AS class,
|
| 109 |
+
{{'primary': primary_name, 'common': common}} AS names,
|
| 110 |
+
NULL AS region,
|
| 111 |
+
admin_level,
|
| 112 |
+
true AS is_land,
|
| 113 |
+
false AS is_territorial,
|
| 114 |
+
id AS division_id
|
| 115 |
+
FROM (VALUES
|
| 116 |
+
{rows}
|
| 117 |
+
) AS t(id, primary_name, common, country, subtype, admin_level, lon, lat)
|
| 118 |
+
) TO '{path}' (FORMAT PARQUET)
|
| 119 |
+
"""
|
| 120 |
+
)
|
| 121 |
+
c.close()
|
| 122 |
+
return str(path)
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
@pytest.fixture()
|
| 126 |
+
def exonym_source(monkeypatch, exonym_parquet):
|
| 127 |
+
"""Point divisions_area at the exonym parquet for the duration of a test."""
|
| 128 |
+
monkeypatch.setattr("gazet.search.DIVISIONS_AREA_PATH", exonym_parquet)
|
| 129 |
+
return exonym_parquet
|
|
@@ -114,6 +114,63 @@ class TestSearchFuzzy:
|
|
| 114 |
# Known limitation: masked bbox arrays + nan in JSON encoding
|
| 115 |
pass
|
| 116 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
def test_fuzzy_search_with_sources(self, client):
|
| 118 |
try:
|
| 119 |
resp = client.get(
|
|
|
|
| 114 |
# Known limitation: masked bbox arrays + nan in JSON encoding
|
| 115 |
pass
|
| 116 |
|
| 117 |
+
def test_ids_carry_the_score_they_were_ranked_on(self, client, exonym_source):
|
| 118 |
+
resp = client.get(
|
| 119 |
+
"/search",
|
| 120 |
+
params={"q": "Munich", "mode": "fuzzy", "ids_only": "true", "limit": 1},
|
| 121 |
+
)
|
| 122 |
+
assert resp.status_code == 200
|
| 123 |
+
item = resp.json()["ids"][0]
|
| 124 |
+
assert item["similarity"] == 1.0
|
| 125 |
+
assert item["is_substring_match"] is True
|
| 126 |
+
|
| 127 |
+
def test_a_miss_is_visible_as_a_low_score(self, client, exonym_source):
|
| 128 |
+
resp = client.get(
|
| 129 |
+
"/search",
|
| 130 |
+
params={"q": "Xyzz98765", "mode": "fuzzy", "ids_only": "true", "limit": 1},
|
| 131 |
+
)
|
| 132 |
+
assert resp.status_code == 200
|
| 133 |
+
item = resp.json()["ids"][0]
|
| 134 |
+
# Jaro-Winkler always returns a best row, so this is a full miss
|
| 135 |
+
# returned as a ranked list — only the score says so.
|
| 136 |
+
assert item["similarity"] < 0.5
|
| 137 |
+
assert item["is_substring_match"] is False
|
| 138 |
+
|
| 139 |
+
def test_english_name_resolves_over_the_wire(self, client, exonym_source):
|
| 140 |
+
resp = client.get(
|
| 141 |
+
"/search",
|
| 142 |
+
params={"q": "Copenhagen", "mode": "fuzzy", "ids_only": "true", "limit": 3},
|
| 143 |
+
)
|
| 144 |
+
assert resp.status_code == 200
|
| 145 |
+
top = resp.json()["ids"][0]
|
| 146 |
+
assert top["id"] == "cph"
|
| 147 |
+
assert top["name"] == "Copenhagen Municipality"
|
| 148 |
+
|
| 149 |
+
def test_local_name_returns_the_same_record(self, client, exonym_source):
|
| 150 |
+
resp = client.get(
|
| 151 |
+
"/search",
|
| 152 |
+
params={
|
| 153 |
+
"q": "K\u00f8benhavn",
|
| 154 |
+
"mode": "fuzzy",
|
| 155 |
+
"ids_only": "true",
|
| 156 |
+
"limit": 3,
|
| 157 |
+
},
|
| 158 |
+
)
|
| 159 |
+
assert resp.status_code == 200
|
| 160 |
+
top = resp.json()["ids"][0]
|
| 161 |
+
assert top["id"] == "cph"
|
| 162 |
+
assert top["matched_name"] == "K\u00f8benhavns Kommune"
|
| 163 |
+
|
| 164 |
+
def test_geometry_response_carries_the_score(self, client, exonym_source):
|
| 165 |
+
resp = client.get(
|
| 166 |
+
"/search",
|
| 167 |
+
params={"q": "Copenhagen", "mode": "fuzzy", "limit": 1},
|
| 168 |
+
)
|
| 169 |
+
assert resp.status_code == 200
|
| 170 |
+
properties = resp.json()["features"][0]["properties"]
|
| 171 |
+
assert properties["id"] == "cph"
|
| 172 |
+
assert properties["similarity"] > 0.8
|
| 173 |
+
|
| 174 |
def test_fuzzy_search_with_sources(self, client):
|
| 175 |
try:
|
| 176 |
resp = client.get(
|
|
@@ -194,3 +194,73 @@ class TestGetById:
|
|
| 194 |
result = get_by_id(con, rid, source="divisions_area")
|
| 195 |
assert not result.empty
|
| 196 |
assert result["source"].iloc[0] == "divisions_area"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 194 |
result = get_by_id(con, rid, source="divisions_area")
|
| 195 |
assert not result.empty
|
| 196 |
assert result["source"].iloc[0] == "divisions_area"
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
class TestEnglishNames:
|
| 200 |
+
"""A place is reachable by its English name as well as its own.
|
| 201 |
+
|
| 202 |
+
Fuzzy search matched only ``names.primary``, so "Copenhagen" scored
|
| 203 |
+
against "Københavns Kommune" and lost to unrelated places that happen to
|
| 204 |
+
start with "Cop-"; the English name was on the record the whole time.
|
| 205 |
+
"""
|
| 206 |
+
|
| 207 |
+
def test_english_exonym_finds_the_place(self, con, exonym_source):
|
| 208 |
+
df = search_divisions_area(con, Place(place="Copenhagen"), limit=3)
|
| 209 |
+
assert df.iloc[0]["id"] == "cph"
|
| 210 |
+
|
| 211 |
+
def test_local_name_still_finds_the_place(self, con, exonym_source):
|
| 212 |
+
df = search_divisions_area(con, Place(place="København"), limit=3)
|
| 213 |
+
assert df.iloc[0]["id"] == "cph"
|
| 214 |
+
|
| 215 |
+
def test_the_two_names_agree_on_one_record(self, con, exonym_source):
|
| 216 |
+
english = search_divisions_area(con, Place(place="Copenhagen"), limit=1)
|
| 217 |
+
local = search_divisions_area(con, Place(place="København"), limit=1)
|
| 218 |
+
assert english.iloc[0]["id"] == local.iloc[0]["id"]
|
| 219 |
+
|
| 220 |
+
def test_the_wrong_place_no_longer_wins(self, con, exonym_source):
|
| 221 |
+
# "Coper" (Colombia) is what the top hit for "Copenhagen" used to be.
|
| 222 |
+
df = search_divisions_area(con, Place(place="Copenhagen"), limit=5)
|
| 223 |
+
assert df.iloc[0]["id"] != "coper"
|
| 224 |
+
|
| 225 |
+
def test_geometry_is_the_right_one(self, con, exonym_source):
|
| 226 |
+
df = search_divisions_area(
|
| 227 |
+
con, Place(place="Copenhagen"), limit=1, include_bbox=True
|
| 228 |
+
)
|
| 229 |
+
# Denmark is east of Greenwich; the county this used to resolve to is
|
| 230 |
+
# in the Americas, and a bbox is the only part of the answer a caller
|
| 231 |
+
# passing it on to a data request ever reads.
|
| 232 |
+
assert df.iloc[0]["bbox"][0] > 0
|
| 233 |
+
|
| 234 |
+
def test_matched_name_reports_which_name_hit(self, con, exonym_source):
|
| 235 |
+
df = search_divisions_area(con, Place(place="København"), limit=1)
|
| 236 |
+
assert df.iloc[0]["matched_name"] == "Københavns Kommune"
|
| 237 |
+
assert df.iloc[0]["name"] == "Copenhagen Municipality"
|
| 238 |
+
|
| 239 |
+
def test_record_without_an_english_name_still_matches(self, con, exonym_source):
|
| 240 |
+
df = search_divisions_area(con, Place(place="Coper"), limit=1)
|
| 241 |
+
assert df.iloc[0]["id"] == "coper"
|
| 242 |
+
assert df.iloc[0]["name"] == "Coper"
|
| 243 |
+
assert df.iloc[0]["similarity"] == 1.0
|
| 244 |
+
|
| 245 |
+
def test_record_with_only_an_english_name_is_reachable(self, con, exonym_source):
|
| 246 |
+
df = search_divisions_area(con, Place(place="Ratnapura"), limit=1)
|
| 247 |
+
assert df.iloc[0]["id"] == "ratnapura"
|
| 248 |
+
|
| 249 |
+
def test_blank_english_name_is_not_a_name(self, con, exonym_source):
|
| 250 |
+
df = search_divisions_area(con, Place(place="Genève"), limit=1)
|
| 251 |
+
assert df.iloc[0]["name"] == "Genève"
|
| 252 |
+
|
| 253 |
+
def test_search_and_fetch_agree_on_the_name(self, con, exonym_source):
|
| 254 |
+
found = search_divisions_area(con, Place(place="Copenhagen"), limit=1)
|
| 255 |
+
fetched = get_division_by_id(con, found.iloc[0]["id"], include_geometry=False)
|
| 256 |
+
assert fetched.iloc[0]["name"] == found.iloc[0]["name"]
|
| 257 |
+
|
| 258 |
+
def test_exact_english_match_scores_one(self, con, exonym_source):
|
| 259 |
+
df = search_divisions_area(con, Place(place="Munich"), limit=1)
|
| 260 |
+
assert df.iloc[0]["similarity"] == 1.0
|
| 261 |
+
assert bool(df.iloc[0]["is_substring_match"]) is True
|
| 262 |
+
|
| 263 |
+
def test_a_miss_scores_low(self, con, exonym_source):
|
| 264 |
+
df = search_divisions_area(con, Place(place="Xyzz98765"), limit=1)
|
| 265 |
+
assert df.iloc[0]["similarity"] < 0.5
|
| 266 |
+
assert bool(df.iloc[0]["is_substring_match"]) is False
|