CiaranDevSeed Claude Opus 5 commited on
Commit
31ff03e
·
1 Parent(s): 02b8f92

Match places on their English name as well as their own

Browse files

Fuzzy search scored each record on `names.primary` alone, so a place whose
English name differs from the name it is filed under was unreachable by the
English name — and the miss came back as a confident-looking ranked list.
"Copenhagen" returned Congaree, a lake centerline in South Carolina;
"Vienna" returned Vienne in France, ~1000 km from the intended place and
plausible enough to build a bounding box from.

The English name was already on the record: `search_divisions_area` matched
`names.primary` while `fetch_by_id` displayed `names.common.en`, so the same
row answered to two different names depending on the endpoint.

Every record is now scored under both names and keeps the better of the two.
Neither works alone: `names.common.en` is null for 77% of divisions_area rows,
which matching on English alone would drop, and `names.primary` alone is what
this bug is. Rows with no English name match exactly as before. The same
applies to natural_earth, whose conversion writes `names.en`.

- `name` is the English name where the record has one and its own name
otherwise, in search and in `GET /geometry/{id}` alike, so a candidate and a
later lookup of it agree on what the place is called. It was previously null
for the 77% on the lookup side.
- `matched_name` reports which of the two names the query hit, so a caller can
show "Copenhagen" against a record titled "Københavns Kommune".
- `similarity` and `is_substring_match` reach `ids_only` responses. They were
computed, used for ranking and then dropped, and Jaro-Winkler always returns
a best row: without a score a caller cannot tell a hit from a total miss, and
cannot apply a floor of its own.
- A blank name is treated as absent rather than as a name to display.

Verified against the deployed parquet files: Copenhagen, Munich, Vienna and
Denmark each return the intended place as the top hit across both sources,
and Loja still ranks its province and canton first.

Closes #10

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md CHANGED
@@ -90,8 +90,8 @@ uv run streamlit run gazet_demo.py # demo UI
90
  ## Design notes
91
 
92
  - `api.py` exposes GET `/search?q=<query>`; returns GeoJSON FeatureCollection and logs intermediate output. This path runs two LLM calls (place extraction, then SQL generation for candidate selection/disambiguation and GIS ops).
93
- - `api.py` also exposes GET `/search/fuzzy?q=<place name>&limit=5&simplify=true&sources=divisions_area,natural_earth` — pure Jaro-Winkler fuzzy match with no LLM involved. `q` is a place-name string, not a natural-language query (there's no place-extraction step). Returns the combined top-`limit` matches across sources as a GeoJSON FeatureCollection with raw (or simplified, if `simplify=true`) geometry attached. Pass `ids_only=true` to skip full geometry and get back `{"ids": [{"source", "id", "name", "country", "subtype", "admin_level", "bbox"}, ...]}` — `country`/`subtype`/`admin_level` disambiguate same-named places (e.g. multiple real "Loja"s across Ecuador and Spain, or Ecuador's "Loja" region vs. its nested "Loja" county), and `bbox` is `[minx, miny, maxx, maxy]`, a much smaller payload than full geometry, for picking a candidate before fetching its full geometry.
94
- - Fuzzy matching (`search.py`) ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity. `divisions_area` matches against `names.primary` (not `names.common.en`, which is null for 77% of rows and would otherwise exclude most of the dataset from every fuzzy search).
95
  - `api.py` also exposes GET `/geometry/{id}?source=divisions_area&simplify=true` — fetch a single feature's geometry directly by ID, no fuzzy matching or LLM involved. `source` is inferred from the ID if omitted (Natural Earth IDs are prefixed `ne_`). Returns a single GeoJSON Feature, 404 if not found.
96
  - The spatial extension is loaded once at API startup (FastAPI `lifespan`) onto a shared DuckDB connection stored on `app.state`; each request gets a cheap `cursor()` off it instead of paying DuckDB's ~90ms `LOAD spatial` cost per call.
97
  - LM is initialised at import time in `lm.py`, suitable for a long-lived server process.
 
90
  ## Design notes
91
 
92
  - `api.py` exposes GET `/search?q=<query>`; returns GeoJSON FeatureCollection and logs intermediate output. This path runs two LLM calls (place extraction, then SQL generation for candidate selection/disambiguation and GIS ops).
93
+ - `api.py` also exposes GET `/search/fuzzy?q=<place name>&limit=5&simplify=true&sources=divisions_area,natural_earth` — pure Jaro-Winkler fuzzy match with no LLM involved. `q` is a place-name string, not a natural-language query (there's no place-extraction step). Returns the combined top-`limit` matches across sources as a GeoJSON FeatureCollection with raw (or simplified, if `simplify=true`) geometry attached. Pass `ids_only=true` to skip full geometry and get back `{"ids": [{"source", "id", "name", "matched_name", "country", "subtype", "admin_level", "bbox", "similarity", "is_substring_match"}, ...]}` — `country`/`subtype`/`admin_level` disambiguate same-named places (e.g. multiple real "Loja"s across Ecuador and Spain, or Ecuador's "Loja" region vs. its nested "Loja" county), `bbox` is `[minx, miny, maxx, maxy]`, a much smaller payload than full geometry, for picking a candidate before fetching its full geometry, and `similarity` is the score the list was ranked on — Jaro-Winkler always returns a best row, so without it a caller cannot tell a hit from the closest thing to a miss.
94
+ - Fuzzy matching (`search.py`) ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity. Each record is matched under two names and keeps the better score: its own (`names.primary`) and its English one (`names.common.en` for `divisions_area`, `names.en` for `natural_earth`). Neither works alone — `names.common.en` is null for 77% of `divisions_area` rows, which matching on English alone would drop, while matching on `names.primary` alone leaves a place unreachable by the English name it is usually asked for ("Copenhagen" for `Københavns Kommune`). `matched_name` reports which of the two the query hit; `name` is the English one where the record has one, the same expression `/geometry/{id}` uses, so a candidate and a later lookup of it agree on what the place is called.
95
  - `api.py` also exposes GET `/geometry/{id}?source=divisions_area&simplify=true` — fetch a single feature's geometry directly by ID, no fuzzy matching or LLM involved. `source` is inferred from the ID if omitted (Natural Earth IDs are prefixed `ne_`). Returns a single GeoJSON Feature, 404 if not found.
96
  - The spatial extension is loaded once at API startup (FastAPI `lifespan`) onto a shared DuckDB connection stored on `app.state`; each request gets a cheap `cursor()` off it instead of paying DuckDB's ~90ms `LOAD spatial` cost per call.
97
  - LM is initialised at import time in `lm.py`, suitable for a long-lived server process.
src/gazet/api.py CHANGED
@@ -62,8 +62,8 @@ _Q_SOURCES = Query(
62
  _Q_IDS_ONLY = Query(
63
  False,
64
  description=(
65
- "For mode=fuzzy: return lightweight id/bbox records instead of full "
66
- "geometry; ignored for mode=nl."
67
  ),
68
  )
69
 
@@ -74,7 +74,8 @@ _TAGS_METADATA = [
74
  "`mode=nl` (default) runs the full natural-language pipeline: "
75
  "LLM place-extraction, fuzzy candidate matching, then LLM-generated "
76
  "SQL. `mode=fuzzy` skips the LLM entirely and does a direct "
77
- "Jaro-Winkler name match — `q` is a place name, not a sentence."
 
78
  ),
79
  },
80
  {"name": "geometry", "description": "Direct, non-fuzzy geometry lookups by ID."},
@@ -356,13 +357,15 @@ def _fuzzy_search(
356
  place-extraction step. Matches are ranked by Jaro-Winkler similarity
357
  across the requested ``sources`` (comma-separated subset of
358
  divisions_area/natural_earth; defaults to both), combined and truncated
359
- to the top ``limit``.
 
360
 
361
  Pass ``ids_only=true`` to skip fetching full geometry and get back
362
- lightweight candidates (id/name/country/subtype/admin_level/bbox)
363
- instead — enough to disambiguate same-named places (e.g. multiple
364
- real-world "Loja"s across Ecuador and Spain) before fetching the full
365
- geometry for one candidate via ``GET /geometry/{id}``.
 
366
  """
367
  requested_sources = (
368
  tuple(s.strip() for s in sources.split(",")) if sources else _FUZZY_SOURCES
@@ -399,7 +402,17 @@ def _fuzzy_search(
399
  )
400
 
401
  if ids_only:
402
- scalar_cols = ["source", "id", "name", "country", "subtype", "admin_level"]
 
 
 
 
 
 
 
 
 
 
403
  ids_df = candidates_df[scalar_cols].copy()
404
  ids_df = ids_df.astype(object).where(ids_df.notna(), None)
405
  ids_df["bbox"] = candidates_df["bbox"].apply(
 
62
  _Q_IDS_ONLY = Query(
63
  False,
64
  description=(
65
+ "For mode=fuzzy: return lightweight id/bbox/similarity records "
66
+ "instead of full geometry; ignored for mode=nl."
67
  ),
68
  )
69
 
 
74
  "`mode=nl` (default) runs the full natural-language pipeline: "
75
  "LLM place-extraction, fuzzy candidate matching, then LLM-generated "
76
  "SQL. `mode=fuzzy` skips the LLM entirely and does a direct "
77
+ "Jaro-Winkler name match, against each record's own name and its "
78
+ "English one — `q` is a place name, not a sentence."
79
  ),
80
  },
81
  {"name": "geometry", "description": "Direct, non-fuzzy geometry lookups by ID."},
 
357
  place-extraction step. Matches are ranked by Jaro-Winkler similarity
358
  across the requested ``sources`` (comma-separated subset of
359
  divisions_area/natural_earth; defaults to both), combined and truncated
360
+ to the top ``limit``. Each record is matched under both its own name and
361
+ its English one, so "Copenhagen" and "København" find the same place.
362
 
363
  Pass ``ids_only=true`` to skip fetching full geometry and get back
364
+ lightweight candidates (id/name/country/subtype/admin_level/bbox, plus
365
+ the ``similarity`` they were ranked on) instead — enough to disambiguate
366
+ same-named places (e.g. multiple real-world "Loja"s across Ecuador and
367
+ Spain), and to reject the lot, before fetching the full geometry for one
368
+ candidate via ``GET /geometry/{id}``.
369
  """
370
  requested_sources = (
371
  tuple(s.strip() for s in sources.split(",")) if sources else _FUZZY_SOURCES
 
402
  )
403
 
404
  if ids_only:
405
+ scalar_cols = [
406
+ "source",
407
+ "id",
408
+ "name",
409
+ "matched_name",
410
+ "country",
411
+ "subtype",
412
+ "admin_level",
413
+ "similarity",
414
+ "is_substring_match",
415
+ ]
416
  ids_df = candidates_df[scalar_cols].copy()
417
  ids_df = ids_df.astype(object).where(ids_df.notna(), None)
418
  ids_df["bbox"] = candidates_df["bbox"].apply(
src/gazet/schemas.py CHANGED
@@ -40,15 +40,25 @@ class FuzzyIdItem(BaseModel):
40
  "Loja" region vs. its nested "Loja" county). ``bbox`` is
41
  ``[minx, miny, maxx, maxy]``, a much smaller payload than full geometry —
42
  fetch the full geometry for one candidate via ``GET /geometry/{id}``.
 
 
 
 
 
 
 
43
  """
44
 
45
  source: str
46
  id: str
47
  name: str | None = None
 
48
  country: str | None = None
49
  subtype: str | None = None
50
  admin_level: int | None = None
51
  bbox: list[float] | None = None
 
 
52
 
53
 
54
  class FuzzyIdsResult(BaseModel):
 
40
  "Loja" region vs. its nested "Loja" county). ``bbox`` is
41
  ``[minx, miny, maxx, maxy]``, a much smaller payload than full geometry —
42
  fetch the full geometry for one candidate via ``GET /geometry/{id}``.
43
+
44
+ ``name`` is the record's English name where it has one and its own name
45
+ otherwise; ``matched_name`` is the one the query actually matched, which
46
+ is how a search for "Copenhagen" can return a record titled "Københavns
47
+ Kommune". ``similarity`` and ``is_substring_match`` are the ranking
48
+ itself: Jaro-Winkler always returns a best row, so a caller with no score
49
+ cannot tell a hit from the closest thing to a miss.
50
  """
51
 
52
  source: str
53
  id: str
54
  name: str | None = None
55
+ matched_name: str | None = None
56
  country: str | None = None
57
  subtype: str | None = None
58
  admin_level: int | None = None
59
  bbox: list[float] | None = None
60
+ similarity: float | None = None
61
+ is_substring_match: bool | None = None
62
 
63
 
64
  class FuzzyIdsResult(BaseModel):
src/gazet/search.py CHANGED
@@ -8,13 +8,26 @@ from .schemas import Place
8
 
9
  logger = logging.getLogger(__name__)
10
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
12
  def simple_fuzzy_search(
13
  con: duckdb.DuckDBPyConnection,
14
  path: str,
15
  source: str,
16
  place: Place,
17
- name_expr: str = "names.common.en",
 
18
  extra_select: str = "",
19
  limit: int = 5,
20
  include_geometry: bool = False,
@@ -22,6 +35,17 @@ def simple_fuzzy_search(
22
  ) -> pd.DataFrame:
23
  """Jaro-Winkler similarity search using only the place name.
24
 
 
 
 
 
 
 
 
 
 
 
 
25
  Ranks exact-substring matches (query literally contained in the name,
26
  e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses
27
  (e.g. "Lodja"), even when the near-miss scores a slightly higher raw
@@ -35,8 +59,9 @@ def simple_fuzzy_search(
35
  (no coordinate arrays or GeoJSON serialization), for lightweight
36
  context (e.g. ``ids_only`` responses).
37
  """
38
- params = [place.place, place.place, path, limit]
39
 
 
40
  extra_clause = f", {extra_select}" if extra_select else ""
41
  geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
42
  bbox_clause = (
@@ -44,22 +69,64 @@ def simple_fuzzy_search(
44
  if include_bbox
45
  else ""
46
  )
 
 
 
 
 
47
  rel = con.execute(
48
  f"""
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  SELECT
50
  id,
51
- {name_expr} AS name,
52
  country,
53
  subtype,
54
  class,
55
  region,
56
  admin_level,
57
  is_land,
58
- is_territorial{extra_clause}{geometry_clause}{bbox_clause},
59
- jaro_winkler_similarity(lower({name_expr}), lower(?)) AS similarity,
60
- contains(lower({name_expr}), lower(?)) AS is_substring_match
61
- FROM read_parquet(?)
62
- WHERE {name_expr} IS NOT NULL AND trim({name_expr}) != ''
 
 
 
 
 
 
 
63
  ORDER BY is_substring_match DESC, similarity DESC, admin_level ASC
64
  LIMIT ?
65
  """,
@@ -83,16 +150,18 @@ def search_divisions_area(
83
  ) -> pd.DataFrame:
84
  """Fuzzy-match a place against divisions_area (Overture admin boundaries).
85
 
86
- Matches against ``names.primary`` rather than ``names.common.en``:
87
- 77% of divisions_area rows have no English common name, and would be
88
- silently excluded from every fuzzy search if that were the match field.
 
 
89
  """
90
  return simple_fuzzy_search(
91
  con,
92
  DIVISIONS_AREA_PATH,
93
  "divisions_area",
94
  place,
95
- name_expr="names.primary",
96
  extra_select="division_id",
97
  limit=limit,
98
  include_geometry=include_geometry,
@@ -113,7 +182,7 @@ def search_natural_earth(
113
  NATURAL_EARTH_PATH,
114
  "natural_earth",
115
  place,
116
- name_expr="names.primary",
117
  limit=limit,
118
  include_geometry=include_geometry,
119
  include_bbox=include_bbox,
@@ -125,22 +194,28 @@ def fetch_by_id(
125
  path: str,
126
  source: str,
127
  id: str,
128
- name_expr: str = "names.common.en",
 
129
  extra_select: str = "",
130
  include_geometry: bool = True,
131
  ) -> pd.DataFrame:
132
  """Look up a single record by exact ID — no fuzzy matching.
133
 
 
 
 
 
134
  ``include_geometry`` requires the spatial extension to already be
135
  loaded on ``con`` (``INSTALL spatial; LOAD spatial;``).
136
  """
 
137
  extra_clause = f", {extra_select}" if extra_select else ""
138
  geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
139
  rel = con.execute(
140
  f"""
141
  SELECT
142
  id,
143
- {name_expr} AS name,
144
  country,
145
  subtype,
146
  class,
@@ -168,6 +243,7 @@ def get_division_by_id(
168
  DIVISIONS_AREA_PATH,
169
  "divisions_area",
170
  id,
 
171
  extra_select="division_id",
172
  include_geometry=include_geometry,
173
  )
@@ -182,7 +258,7 @@ def get_natural_earth_by_id(
182
  NATURAL_EARTH_PATH,
183
  "natural_earth",
184
  id,
185
- name_expr="names.primary",
186
  include_geometry=include_geometry,
187
  )
188
 
 
8
 
9
  logger = logging.getLogger(__name__)
10
 
11
+ #: Where each source keeps a record's own name and its English one. Overture
12
+ #: holds translations in a ``names.common`` map; the Natural Earth conversion
13
+ #: gives every language a struct field of its own (``ingest/convert_natural_earth.py``).
14
+ PRIMARY_NAME = "names.primary"
15
+ DIVISIONS_AREA_ENGLISH_NAME = "names.common.en"
16
+ NATURAL_EARTH_ENGLISH_NAME = "names.en"
17
+
18
+
19
+ def _readable(expr: str) -> str:
20
+ """SQL that reads a name, treating a blank one as absent."""
21
+ return f"NULLIF(trim({expr}), '')"
22
+
23
 
24
  def simple_fuzzy_search(
25
  con: duckdb.DuckDBPyConnection,
26
  path: str,
27
  source: str,
28
  place: Place,
29
+ name_expr: str = PRIMARY_NAME,
30
+ english_name_expr: str | None = None,
31
  extra_select: str = "",
32
  limit: int = 5,
33
  include_geometry: bool = False,
 
35
  ) -> pd.DataFrame:
36
  """Jaro-Winkler similarity search using only the place name.
37
 
38
+ Each record is matched under two names: ``name_expr``, its own, and
39
+ ``english_name_expr``, its English one where the source carries it. A
40
+ record scores the better of the two, so a place is reachable by its
41
+ English exonym ("Copenhagen") as well as by its local name ("Københavns
42
+ Kommune"), and a record with no English name matches on its own name
43
+ alone. ``matched_name`` is whichever of the two produced the score.
44
+
45
+ ``name`` is the English name where there is one and the record's own name
46
+ otherwise — the expression :func:`fetch_by_id` uses too, so a candidate
47
+ and a later lookup of that same candidate agree on what it is called.
48
+
49
  Ranks exact-substring matches (query literally contained in the name,
50
  e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses
51
  (e.g. "Lodja"), even when the near-miss scores a slightly higher raw
 
59
  (no coordinate arrays or GeoJSON serialization), for lightweight
60
  context (e.g. ``ids_only`` responses).
61
  """
62
+ params = [path, place.place, place.place, place.place, place.place, limit]
63
 
64
+ english_expr = english_name_expr or "CAST(NULL AS VARCHAR)"
65
  extra_clause = f", {extra_select}" if extra_select else ""
66
  geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
67
  bbox_clause = (
 
69
  if include_bbox
70
  else ""
71
  )
72
+ # Geometry and bbox are computed in the first stage, so the final select
73
+ # only has to name them.
74
+ carried_clause = (", geometry" if include_geometry else "") + (
75
+ ", bbox" if include_bbox else ""
76
+ )
77
  rel = con.execute(
78
  f"""
79
+ WITH named AS (
80
+ SELECT
81
+ id,
82
+ country,
83
+ subtype,
84
+ class,
85
+ region,
86
+ admin_level,
87
+ is_land,
88
+ is_territorial{extra_clause}{geometry_clause}{bbox_clause},
89
+ {_readable(name_expr)} AS primary_name,
90
+ {_readable(english_expr)} AS english_name
91
+ FROM read_parquet(?)
92
+ ),
93
+ scored AS (
94
+ SELECT
95
+ *,
96
+ COALESCE(
97
+ jaro_winkler_similarity(lower(primary_name), lower(?)), 0.0
98
+ ) AS primary_similarity,
99
+ COALESCE(
100
+ jaro_winkler_similarity(lower(english_name), lower(?)), 0.0
101
+ ) AS english_similarity,
102
+ COALESCE(contains(lower(primary_name), lower(?)), false)
103
+ AS primary_substring,
104
+ COALESCE(contains(lower(english_name), lower(?)), false)
105
+ AS english_substring
106
+ FROM named
107
+ WHERE primary_name IS NOT NULL OR english_name IS NOT NULL
108
+ )
109
  SELECT
110
  id,
111
+ COALESCE(english_name, primary_name) AS name,
112
  country,
113
  subtype,
114
  class,
115
  region,
116
  admin_level,
117
  is_land,
118
+ is_territorial{extra_clause}{carried_clause},
119
+ -- Compared as a pair so the winning name is the one the ranking
120
+ -- below would pick: a substring hit beats a higher raw score.
121
+ CASE
122
+ WHEN (english_substring, english_similarity)
123
+ > (primary_substring, primary_similarity)
124
+ THEN english_name
125
+ ELSE COALESCE(primary_name, english_name)
126
+ END AS matched_name,
127
+ GREATEST(primary_similarity, english_similarity) AS similarity,
128
+ primary_substring OR english_substring AS is_substring_match
129
+ FROM scored
130
  ORDER BY is_substring_match DESC, similarity DESC, admin_level ASC
131
  LIMIT ?
132
  """,
 
150
  ) -> pd.DataFrame:
151
  """Fuzzy-match a place against divisions_area (Overture admin boundaries).
152
 
153
+ Matched on ``names.primary`` and on ``names.common.en`` together: 77% of
154
+ divisions_area rows have no English common name and would drop out of
155
+ every search if English were the only match field, while ``names.primary``
156
+ on its own leaves a place unreachable by the English name it is usually
157
+ asked for.
158
  """
159
  return simple_fuzzy_search(
160
  con,
161
  DIVISIONS_AREA_PATH,
162
  "divisions_area",
163
  place,
164
+ english_name_expr=DIVISIONS_AREA_ENGLISH_NAME,
165
  extra_select="division_id",
166
  limit=limit,
167
  include_geometry=include_geometry,
 
182
  NATURAL_EARTH_PATH,
183
  "natural_earth",
184
  place,
185
+ english_name_expr=NATURAL_EARTH_ENGLISH_NAME,
186
  limit=limit,
187
  include_geometry=include_geometry,
188
  include_bbox=include_bbox,
 
194
  path: str,
195
  source: str,
196
  id: str,
197
+ name_expr: str = PRIMARY_NAME,
198
+ english_name_expr: str | None = None,
199
  extra_select: str = "",
200
  include_geometry: bool = True,
201
  ) -> pd.DataFrame:
202
  """Look up a single record by exact ID — no fuzzy matching.
203
 
204
+ ``name`` is built as :func:`simple_fuzzy_search` builds it — the English
205
+ name where the record has one, its own name otherwise — so a record
206
+ answers to a single name whichever way it is reached.
207
+
208
  ``include_geometry`` requires the spatial extension to already be
209
  loaded on ``con`` (``INSTALL spatial; LOAD spatial;``).
210
  """
211
+ english_expr = english_name_expr or "CAST(NULL AS VARCHAR)"
212
  extra_clause = f", {extra_select}" if extra_select else ""
213
  geometry_clause = ", ST_AsGeoJSON(geometry) AS geometry" if include_geometry else ""
214
  rel = con.execute(
215
  f"""
216
  SELECT
217
  id,
218
+ COALESCE({_readable(english_expr)}, {_readable(name_expr)}) AS name,
219
  country,
220
  subtype,
221
  class,
 
243
  DIVISIONS_AREA_PATH,
244
  "divisions_area",
245
  id,
246
+ english_name_expr=DIVISIONS_AREA_ENGLISH_NAME,
247
  extra_select="division_id",
248
  include_geometry=include_geometry,
249
  )
 
258
  NATURAL_EARTH_PATH,
259
  "natural_earth",
260
  id,
261
+ english_name_expr=NATURAL_EARTH_ENGLISH_NAME,
262
  include_geometry=include_geometry,
263
  )
264
 
tests/conftest.py CHANGED
@@ -35,3 +35,95 @@ def con_no_spatial():
35
  c = duckdb.connect()
36
  yield c
37
  c.close()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  c = duckdb.connect()
36
  yield c
37
  c.close()
38
+
39
+
40
+ # ---------------------------------------------------------------------------
41
+ # English-exonym fixture
42
+ # ---------------------------------------------------------------------------
43
+
44
+ #: Places whose English name differs from the name the record is filed under,
45
+ #: shaped like Overture divisions_area rows: ``names.primary`` is the local
46
+ #: name and ``names.common`` is a translation map that may or may not carry
47
+ #: ``en``. ``Coper`` is the Colombian county a search for "Copenhagen" used to
48
+ #: land on, and it has no English name — the 77% of the real table that a
49
+ #: match on English alone would drop. ``Genève`` carries a blank one, which is
50
+ #: absent rather than a name the record can be shown under.
51
+ _EXONYM_ROWS = [
52
+ (
53
+ "cph",
54
+ "Københavns Kommune",
55
+ "Copenhagen Municipality",
56
+ "DK",
57
+ "county",
58
+ 2,
59
+ 12.5,
60
+ 55.6,
61
+ ),
62
+ ("dnk", "Danmark", "Denmark", "DK", "country", 0, 9.5, 55.5),
63
+ ("muc", "München", "Munich", "DE", "county", 2, 11.5, 48.1),
64
+ ("vie", "Wien", "Vienna", "AT", "region", 1, 16.3, 48.2),
65
+ ("coper", "Coper", None, "CO", "county", 2, -74.0, 5.4),
66
+ ("ratnapura", None, "Ratnapura District", "LK", "county", 2, 80.4, 6.7),
67
+ ("gva", "Genève", "", "CH", "county", 2, 6.1, 46.2),
68
+ ]
69
+
70
+
71
+ @pytest.fixture(scope="session")
72
+ def exonym_parquet(tmp_path_factory):
73
+ """Write the exonym rows to a parquet with the divisions_area schema."""
74
+ path = tmp_path_factory.mktemp("exonym") / "divisions_area.parquet"
75
+ c = duckdb.connect()
76
+ c.execute("INSTALL spatial")
77
+ c.execute("LOAD spatial")
78
+ rows = ",\n".join(
79
+ "(%s, %s, %s, '%s', '%s', %d, %f, %f)"
80
+ % (
81
+ f"'{id}'",
82
+ "NULL" if primary is None else f"'{primary}'",
83
+ "MAP([], [])" if english is None else f"MAP(['en'], ['{english}'])",
84
+ country,
85
+ subtype,
86
+ admin_level,
87
+ lon,
88
+ lat,
89
+ )
90
+ for id, primary, english, country, subtype, admin_level, lon, lat in _EXONYM_ROWS
91
+ )
92
+ c.execute(
93
+ f"""
94
+ COPY (
95
+ SELECT
96
+ id,
97
+ ST_AsWKB(
98
+ ST_GeomFromText(
99
+ 'POLYGON((' || lon || ' ' || lat || ', '
100
+ || (lon + 0.1) || ' ' || lat || ', '
101
+ || (lon + 0.1) || ' ' || (lat + 0.1) || ', '
102
+ || lon || ' ' || (lat + 0.1) || ', '
103
+ || lon || ' ' || lat || '))'
104
+ )
105
+ ) AS geometry,
106
+ country,
107
+ subtype,
108
+ 'land' AS class,
109
+ {{'primary': primary_name, 'common': common}} AS names,
110
+ NULL AS region,
111
+ admin_level,
112
+ true AS is_land,
113
+ false AS is_territorial,
114
+ id AS division_id
115
+ FROM (VALUES
116
+ {rows}
117
+ ) AS t(id, primary_name, common, country, subtype, admin_level, lon, lat)
118
+ ) TO '{path}' (FORMAT PARQUET)
119
+ """
120
+ )
121
+ c.close()
122
+ return str(path)
123
+
124
+
125
+ @pytest.fixture()
126
+ def exonym_source(monkeypatch, exonym_parquet):
127
+ """Point divisions_area at the exonym parquet for the duration of a test."""
128
+ monkeypatch.setattr("gazet.search.DIVISIONS_AREA_PATH", exonym_parquet)
129
+ return exonym_parquet
tests/test_api.py CHANGED
@@ -114,6 +114,63 @@ class TestSearchFuzzy:
114
  # Known limitation: masked bbox arrays + nan in JSON encoding
115
  pass
116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
117
  def test_fuzzy_search_with_sources(self, client):
118
  try:
119
  resp = client.get(
 
114
  # Known limitation: masked bbox arrays + nan in JSON encoding
115
  pass
116
 
117
+ def test_ids_carry_the_score_they_were_ranked_on(self, client, exonym_source):
118
+ resp = client.get(
119
+ "/search",
120
+ params={"q": "Munich", "mode": "fuzzy", "ids_only": "true", "limit": 1},
121
+ )
122
+ assert resp.status_code == 200
123
+ item = resp.json()["ids"][0]
124
+ assert item["similarity"] == 1.0
125
+ assert item["is_substring_match"] is True
126
+
127
+ def test_a_miss_is_visible_as_a_low_score(self, client, exonym_source):
128
+ resp = client.get(
129
+ "/search",
130
+ params={"q": "Xyzz98765", "mode": "fuzzy", "ids_only": "true", "limit": 1},
131
+ )
132
+ assert resp.status_code == 200
133
+ item = resp.json()["ids"][0]
134
+ # Jaro-Winkler always returns a best row, so this is a full miss
135
+ # returned as a ranked list — only the score says so.
136
+ assert item["similarity"] < 0.5
137
+ assert item["is_substring_match"] is False
138
+
139
+ def test_english_name_resolves_over_the_wire(self, client, exonym_source):
140
+ resp = client.get(
141
+ "/search",
142
+ params={"q": "Copenhagen", "mode": "fuzzy", "ids_only": "true", "limit": 3},
143
+ )
144
+ assert resp.status_code == 200
145
+ top = resp.json()["ids"][0]
146
+ assert top["id"] == "cph"
147
+ assert top["name"] == "Copenhagen Municipality"
148
+
149
+ def test_local_name_returns_the_same_record(self, client, exonym_source):
150
+ resp = client.get(
151
+ "/search",
152
+ params={
153
+ "q": "K\u00f8benhavn",
154
+ "mode": "fuzzy",
155
+ "ids_only": "true",
156
+ "limit": 3,
157
+ },
158
+ )
159
+ assert resp.status_code == 200
160
+ top = resp.json()["ids"][0]
161
+ assert top["id"] == "cph"
162
+ assert top["matched_name"] == "K\u00f8benhavns Kommune"
163
+
164
+ def test_geometry_response_carries_the_score(self, client, exonym_source):
165
+ resp = client.get(
166
+ "/search",
167
+ params={"q": "Copenhagen", "mode": "fuzzy", "limit": 1},
168
+ )
169
+ assert resp.status_code == 200
170
+ properties = resp.json()["features"][0]["properties"]
171
+ assert properties["id"] == "cph"
172
+ assert properties["similarity"] > 0.8
173
+
174
  def test_fuzzy_search_with_sources(self, client):
175
  try:
176
  resp = client.get(
tests/test_search.py CHANGED
@@ -194,3 +194,73 @@ class TestGetById:
194
  result = get_by_id(con, rid, source="divisions_area")
195
  assert not result.empty
196
  assert result["source"].iloc[0] == "divisions_area"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
194
  result = get_by_id(con, rid, source="divisions_area")
195
  assert not result.empty
196
  assert result["source"].iloc[0] == "divisions_area"
197
+
198
+
199
+ class TestEnglishNames:
200
+ """A place is reachable by its English name as well as its own.
201
+
202
+ Fuzzy search matched only ``names.primary``, so "Copenhagen" scored
203
+ against "Københavns Kommune" and lost to unrelated places that happen to
204
+ start with "Cop-"; the English name was on the record the whole time.
205
+ """
206
+
207
+ def test_english_exonym_finds_the_place(self, con, exonym_source):
208
+ df = search_divisions_area(con, Place(place="Copenhagen"), limit=3)
209
+ assert df.iloc[0]["id"] == "cph"
210
+
211
+ def test_local_name_still_finds_the_place(self, con, exonym_source):
212
+ df = search_divisions_area(con, Place(place="København"), limit=3)
213
+ assert df.iloc[0]["id"] == "cph"
214
+
215
+ def test_the_two_names_agree_on_one_record(self, con, exonym_source):
216
+ english = search_divisions_area(con, Place(place="Copenhagen"), limit=1)
217
+ local = search_divisions_area(con, Place(place="København"), limit=1)
218
+ assert english.iloc[0]["id"] == local.iloc[0]["id"]
219
+
220
+ def test_the_wrong_place_no_longer_wins(self, con, exonym_source):
221
+ # "Coper" (Colombia) is what the top hit for "Copenhagen" used to be.
222
+ df = search_divisions_area(con, Place(place="Copenhagen"), limit=5)
223
+ assert df.iloc[0]["id"] != "coper"
224
+
225
+ def test_geometry_is_the_right_one(self, con, exonym_source):
226
+ df = search_divisions_area(
227
+ con, Place(place="Copenhagen"), limit=1, include_bbox=True
228
+ )
229
+ # Denmark is east of Greenwich; the county this used to resolve to is
230
+ # in the Americas, and a bbox is the only part of the answer a caller
231
+ # passing it on to a data request ever reads.
232
+ assert df.iloc[0]["bbox"][0] > 0
233
+
234
+ def test_matched_name_reports_which_name_hit(self, con, exonym_source):
235
+ df = search_divisions_area(con, Place(place="København"), limit=1)
236
+ assert df.iloc[0]["matched_name"] == "Københavns Kommune"
237
+ assert df.iloc[0]["name"] == "Copenhagen Municipality"
238
+
239
+ def test_record_without_an_english_name_still_matches(self, con, exonym_source):
240
+ df = search_divisions_area(con, Place(place="Coper"), limit=1)
241
+ assert df.iloc[0]["id"] == "coper"
242
+ assert df.iloc[0]["name"] == "Coper"
243
+ assert df.iloc[0]["similarity"] == 1.0
244
+
245
+ def test_record_with_only_an_english_name_is_reachable(self, con, exonym_source):
246
+ df = search_divisions_area(con, Place(place="Ratnapura"), limit=1)
247
+ assert df.iloc[0]["id"] == "ratnapura"
248
+
249
+ def test_blank_english_name_is_not_a_name(self, con, exonym_source):
250
+ df = search_divisions_area(con, Place(place="Genève"), limit=1)
251
+ assert df.iloc[0]["name"] == "Genève"
252
+
253
+ def test_search_and_fetch_agree_on_the_name(self, con, exonym_source):
254
+ found = search_divisions_area(con, Place(place="Copenhagen"), limit=1)
255
+ fetched = get_division_by_id(con, found.iloc[0]["id"], include_geometry=False)
256
+ assert fetched.iloc[0]["name"] == found.iloc[0]["name"]
257
+
258
+ def test_exact_english_match_scores_one(self, con, exonym_source):
259
+ df = search_divisions_area(con, Place(place="Munich"), limit=1)
260
+ assert df.iloc[0]["similarity"] == 1.0
261
+ assert bool(df.iloc[0]["is_substring_match"]) is True
262
+
263
+ def test_a_miss_scores_low(self, con, exonym_source):
264
+ df = search_divisions_area(con, Place(place="Xyzz98765"), limit=1)
265
+ assert df.iloc[0]["similarity"] < 0.5
266
+ assert bool(df.iloc[0]["is_substring_match"]) is False