Spaces:
Running
Running
Commit History
-
Find cities Overture has only as a point, and rank the largest first
London, UK has no polygon in any Overture release: its locality is a point with no division_area, and there is no Greater London level between England and its boroughs. A search for "London" returned Canada, US towns, Finland and Kiribati. About 25 cities of 3M+ are in the same position (#14). GHSL's Urban Centre Database (GHS-UCDB R2024A) is now a fuzzy-only source, urban_centres: 11,422 named city polygons, each with alternate names and a population. It stays a source of its own, with ghsl_-prefixed ids, so every result traces back to one dataset. normalize_geodata reprojects it from Mollweide and strips the byte-order mark GHSL puts on every column and value. Equal scores now break by population (#12, item 2), in each source and when the API merges them, so "London" puts the UK's first. Overture's population is joined from its division records at normalize time; a place without one sorts last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b20c32c
-
Break ranking ties by id, so results come back in the same order
Daniel Wiesmann +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>755fa0e
-
Match every name a place has, without case or accents
Three of the misses in #12 were in the index all along, under another spelling: Chittagong is filed as Chattogram District, Da Nang as Đà Nẵng, Bogota as Bogotá, D.c. Search compared the query with the primary and English names only, and Jaro-Winkler rewards a shared prefix, so a near-homograph on another continent won. A record is now matched under every name Overture holds for it: its own, its translations (names.common) and its alternate, short and official names (names.rules), each compared after lowercasing and stripping accents. matched_name reports the spelling that hit. Natural Earth, which has no alternates, is matched on its own and English names as before, folded the same way. Building and folding those names on every query would make search about twice as slow as main, so normalize_geodata stores them in a search_names column, and search reads it where every file has one. A raw Overture download still works; it builds the names on the fly. On the 2026-08-19.0 release a fu…30114d9 -
Read Overture's own bbox rather than computing it from the geometry
An ids_only fuzzy search returned each candidate's bbox by running ST_XMin/YMin/XMax/YMax over its geometry, which meant decoding the geometry column: the largest in the file, for a four-number answer. Overture already stores that bbox beside the geometry, so divisions_area now reads it. On the 2026-08-19.0 release the two agree to within 3e-5 degrees on every row, and none are missing. Natural Earth has no bbox column and keeps computing it; its file is small enough that this costs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>5b2b311 -
Let fuzzy search find cities, and keep them from the model
A well-known city could be missing from every answer: normalisation kept only country, region and county, and whether a city survives that depends on how its country maps onto those levels. Brazil and Vietnam have no county rows, so none of their cities were searchable (#12). normalize_geodata now also writes Overture's locality and localadmin areas, to overture_normalized/localities/ rather than into divisions_area. Only mode=fuzzy searches that file. The natural-language pipeline, the SQL it generates and the training data all read divisions_area alone, since the model was never trained on these subtypes. /geometry/{id} resolves a locality, because its id only ever comes from a fuzzy result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>41e01af
-
Serve the installed version, and bump it to 0.2.0
The number was typed out twice — once in pyproject.toml and once in the FastAPI constructor — with nothing keeping the two in step. api.py now reads it back from the installed distribution, so there is one place to change. 0.2.0 because the fuzzy search gained fields: ids_only responses now carry matched_name, similarity and is_substring_match. uv.lock records the project's own version and the Dockerfile syncs with --frozen, so the lock moves with the bump or the image stops building. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>251c933
-
Match places on their English name as well as their own
Fuzzy search scored each record on `names.primary` alone, so a place whose English name differs from the name it is filed under was unreachable by the English name — and the miss came back as a confident-looking ranked list. "Copenhagen" returned Congaree, a lake centerline in South Carolina; "Vienna" returned Vienne in France, ~1000 km from the intended place and plausible enough to build a bounding box from. The English name was already on the record: `search_divisions_area` matched `names.primary` while `fetch_by_id` displayed `names.common.en`, so the same row answered to two different names depending on the endpoint. Every record is now scored under both names and keeps the better of the two. Neither works alone: `names.common.en` is null for 77% of divisions_area rows, which matching on English alone would drop, and `names.primary` alone is what this bug is. Rows with no English name match exactly as before. The same applies to natural_earth, whose conversion writes `names.en`.…31ff03e
-
Unify /search and /search/fuzzy under a mode param; add OpenAPI response models
Daniel WiesmannGET /search and /search/stream now take mode=nl (default, unchanged LLM pipeline) or mode=fuzzy (the direct name-match logic previously only reachable via GET /search/fuzzy). /search/fuzzy is kept as a deprecated thin alias for backward compatibility - Streamlit is unaffected since it never sets mode and defaults to nl. Also replaces response_model=None / bare dict[str, Any] returns with real Pydantic models (FeatureCollection, NLSearchResult, FuzzyIdsResult, HealthStatus, SourceInfo) and per-parameter Query() descriptions, so the generated OpenAPI schema at /openapi.json is actually useful for agents calling this API without reading the source.34bbe35 -
fix: point NATURAL_EARTH_PATH to test_ne_geography.parquet tracked in git
Daniel Wiesmannne_geography.parquet (52MB) is not in git; only test_ne_geography.parquet (5KB, 3 rows) is tracked. The config referenced the missing file so CI failed with IOException on every natural_earth search. Schemas are identical — test file has 3 rows instead of 31k.059dfce -
Add ruff.toml and fix lint errors (imports, type hints, formatting)
Daniel Wiesmann- Add ruff.toml to configure lint rules (ignore BLE001/S110/B017 which are intentional in health checks and error handling) - Fix import sorting (I001) across api.py, lm.py, sql.py, test_lm.py, test_search.py - Modernize type hints: Generator[str] instead of Generator[str, None, None] (UP043), X | None instead of Optional[X] (UP045), collections.abc.Gen erator instead of typing.Generator (UP035) - Use str.removeprefix() instead of startswith + slicing (FURB188) - Use from-module import instead of aliased import (PLR0402) - Reformat all source and test files with ruff formatad8201c -
Fix ruff formatting across all source and test files
Daniel Wiesmann61f1033 -
Add mypy type checking to CI and fix type errors
Daniel WiesmannAdd mypy to dev dependencies with strict mode config and per-module overrides for untyped deps (dspy, duckdb) and test files. Fix type errors surfaced by the initial run: - Annotate _to_serializable in export.py and wrap _is_geojson_col return - Add return type and param annotations to lifespan and log_request in api.py - Add type: ignore on _df_to_records for pandas-stubs Hashable key mismatch - Fix PlacesResult import in test_lm.py (defined in schemas, not lm) - Add type check step to CI lint job3245c3b -
Fix lint: remove unused math import, move imports before constants in sql.py
Daniel Wiesmann11e0d81 -
Add GET /sources endpoint
Daniel WiesmannLists available data sources with row counts, name ranges, and parquet paths. Helpful for API consumers to understand what data is available without guessing.df4a3f3 -
Optimize to_feature_collection to use to_dict(orient=records)
Daniel WiesmannReplaced iterrows() with to_dict(orient=records) which is significantly faster on large DataFrames. Also added early return for empty DataFrames.5ff62f8 -
Unify /search/fuzzy response shape for ids_only mode
Daniel WiesmannPreviously returned {"ids": [...]} or a FeatureCollection depending on ids_only. Now always returns a FeatureCollection envelope, with ids_only adding an "ids" key alongside an empty geojson for the lightweight case.0e8628d -
Fix _per_source_limit to floor at 3 candidates for complex queries
Daniel WiesmannThe fallback max(1, math.ceil(5/n)) produced only 1 candidate per source for 6+ places, silently degrading match quality. Now floors at 3 for any query with 3+ places.10597d3 -
Add optional country and subtype to Place schema
Daniel WiesmannThe demo renders country/subtype columns from extracted places data, but Place only had the place name. Now extraction prompts can return these optional hints and the demo (and future clients) can use them before fuzzy matching.87ca437 -
Add request ID middleware and request logging
Daniel WiesmannAttaches X-Request-Id header (or generates one) and logs method, path, status code, request ID, and query params on every request. Useful for correlating API logs across streaming and LLM calls.925ffa4 -
Add GET /health endpoint
Daniel WiesmannChecks DuckDB connection alive and llama-server availability. Returns status ok/degraded/unhealthy for use by load balancers and orchestrators (Modal, Docker).42c4cad -
Consolidate schema definition into config.SCHEMA_INFO
Daniel WiesmannThe GGUF _SYSTEM_PROMPT embedded its own copy of the table schema. Now both backends (DSPy and GGUF) reference config.SCHEMA_INFO, so any column or dataset change only needs one edit.ac7489a -
Unify geometry normalization and kill per-row DuckDB connections in export.py
Daniel Wiesmannexport._wkb_to_geojson() created a new DuckDB connection and LOADed spatial per geometry row — the full ~90ms init cost per feature. Now to_feature_collection() expects geometry pre-normalized (as it already is via geometry.normalize_geometry_to_geojson in all API paths). Also replaced the iterrows() loop with a cleaner version that handles only the normalized GeoJSON string case, which is the actual production path. The old WKB and generic geojson column heuristics were dead code in the API flow. Replaced print() in save_geojson with logger.info().5504226 -
Replace print() with logging in sql.py and search.py
Daniel Wiesmannprint() statements polluted stdout in production (Modal, Docker) and leaked into API responses. Use logger.debug() for progress/info, logger.warning() for errors, logger.error() for failures.692e6e8 -
Remove dead COUNTRIES and SUBTYPES literals from schemas.py
Daniel Wiesmann~250-line Literal type listing ISO country codes and a 3-value subtype literal, neither imported or used anywhere.6555d2a
-
Add subtype to /search/fuzzy ids_only responses
Daniel Wiesmannadmin_level alone doesn't disambiguate well: the same subtype (e.g. "county") occurs at different admin_levels, and locality-type subtypes have no admin_level at all. subtype gives the human-readable category (region/county/locality/...) alongside the numeric admin_level.3faf809 -
Fix fuzzy-match ranking and broaden divisions_area coverage
Daniel Wiesmann- divisions_area now matches against names.primary instead of names.common.en, which is null for 77% of rows and silently excluded most of the dataset from every fuzzy search (both /search/fuzzy and the existing generative /search pipeline, which share search_candidates). - Ranks exact-substring matches (query literally contained in the name, e.g. "Loja" in "Loja Province") ahead of pure edit-distance near-misses (e.g. "Lodja"), even when the near-miss scores a slightly higher raw Jaro-Winkler similarity. - ids_only responses now include country and admin_level to disambiguate identically-named places (e.g. multiple real "Loja"s across Ecuador, Spain, and Latvia at different admin levels).34b27af -
Include name and bbox in /search/fuzzy ids_only responses
Daniel WiesmannGives minimal spatial context (bounding box) alongside bare IDs, without paying for full geometry. bbox is computed via ST_XMin/YMin/XMax/YMax directly in SQL rather than fetching and simplifying full geometry.0c6586d -
Add pure fuzzy-search and by-ID geometry endpoints, decoupled from the LLM path
Daniel WiesmannSplits the generative pipeline from fuzzy matching so raw geometry can be fetched without any LLM involvement: - GET /search/fuzzy: Jaro-Winkler match against a place name, returns the combined top-N GeoJSON FeatureCollection across divisions_area/natural_earth. Supports ids_only=true for a cheap candidate-ID-only response. - GET /geometry/{id}: direct single-feature geometry lookup by ID (source inferred from the ne_ prefix if omitted), no fuzzy matching or LLM. - Moves geometry simplification out of sql.py into a standalone geometry.py with no dependency on lm.py, so the fuzzy/ID-lookup paths never import DSPy/LLM machinery. - Loads the spatial extension once via FastAPI lifespan onto a shared connection; requests use cursor() instead of reconnecting, cutting ~90ms of redundant LOAD spatial overhead per call (previously paid twice per request on the simplify path). - gazet_demo.py: backend now configurable via GAZET_BACKEND env var instead of hardcoded "gguf".84b18f8
-
finetune: update the params to run experiment on H200
b4a6b6b
-
fix: reduce templates to country, region & county for mvp
3ba9557
-
enh: Run gazet on modal - llama server on GPU, API & UI on cpu
0836eef
-
Lowercase subtype names, fix question type hints
a5fd57d -
Fix: No pairs are created for mixed queries
- Normalize overture & natural earth to EPSG:4326 - Add better adjancency matrix to increase the overlap in query pairs - Fix titled & lower case subtype names in NE - Reduce number of threads in duckdb to prevent memory issuesdfb9466
-
prompt: Add notes on how to pick places from query better
d237392 -
Fix natural earth geometries by converting wkb geometries to geojson
Daniel Wiesmann4814e70
-
chore: clean sql generation, use conversation format, move prompts from user to system
c77ca5f
-
Add model training script for qwen, update eval & gazet lm to work with qwen template
2bf5583
-
Add finetune instructions for gemma
1104031
-
Add options to run gguf models using llama cpp
8ba44ad -
Revert ollama port to 11434
a6fc067
-
ENH: Add dataset creation on scale using modal clusters
c88725f
-
ENH: use separate models for places & sql generation
5623a00
-
Disable dspy cache
Daniel Wiesmann5e72207 -
Intitial commit
Daniel Wiesmannad5ec6b