Spaces:
Running
Running
enh: Run gazet on modal - llama server on GPU, API & UI on cpu
Browse files- gazet_demo.py +4 -0
- modal_serve/README.md +112 -0
- modal_serve/serve.py +139 -0
- src/gazet/api.py +6 -1
- src/gazet/lm.py +31 -15
- src/gazet/sql.py +45 -17
gazet_demo.py
CHANGED
|
@@ -219,6 +219,10 @@ with col2:
|
|
| 219 |
event = json.loads(raw)
|
| 220 |
t = event["type"]
|
| 221 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
if t == "places":
|
| 223 |
places = event["data"].get("places", [])
|
| 224 |
result["places"] = places
|
|
|
|
| 219 |
event = json.loads(raw)
|
| 220 |
t = event["type"]
|
| 221 |
|
| 222 |
+
if t == "warming_up":
|
| 223 |
+
status_ph.warning(event["data"])
|
| 224 |
+
continue
|
| 225 |
+
|
| 226 |
if t == "places":
|
| 227 |
places = event["data"].get("places", [])
|
| 228 |
result["places"] = places
|
modal_serve/README.md
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Modal Deployment
|
| 2 |
+
|
| 3 |
+
Deploys Gazet to Modal as three independently scaled containers:
|
| 4 |
+
|
| 5 |
+
| Cls | Hardware | Role |
|
| 6 |
+
|---|---|---|
|
| 7 |
+
| `LlamaServer` | GPU T4 | `llama-server` from the official `ghcr.io/ggml-org/llama.cpp:server-cuda` image |
|
| 8 |
+
| `Api` | CPU | `gazet.api` FastAPI app served as native ASGI |
|
| 9 |
+
| `Demo` | CPU | Streamlit UI from `gazet_demo.py` |
|
| 10 |
+
|
| 11 |
+
All three scale to zero independently. The Demo URL is the user-facing entrypoint.
|
| 12 |
+
|
| 13 |
+
## Prerequisites
|
| 14 |
+
|
| 15 |
+
- Modal account + CLI: `uv tool install modal` then `modal token new`
|
| 16 |
+
- Modal volumes already exist:
|
| 17 |
+
- `gazet` (model checkpoints)
|
| 18 |
+
- `gazet-data` (normalized parquet datasets)
|
| 19 |
+
|
| 20 |
+
## 1. Upload the Q4 GGUF model
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
modal volume put gazet \
|
| 24 |
+
finetune/models/ckpt-q4_k_m.gguf \
|
| 25 |
+
checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
Verify:
|
| 29 |
+
|
| 30 |
+
```bash
|
| 31 |
+
modal volume ls gazet checkpoints/qwen35-fientune-v3 | grep ckpt-q4_k_m
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
## 2. Deploy
|
| 35 |
+
|
| 36 |
+
From the repo root:
|
| 37 |
+
|
| 38 |
+
```bash
|
| 39 |
+
modal deploy modal_serve/serve.py
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
The output prints three URLs:
|
| 43 |
+
|
| 44 |
+
- `LlamaServer.serve` -> internal, called by `Api`
|
| 45 |
+
- `Api.fastapi_app` -> public FastAPI (`/search`, `/search/stream`)
|
| 46 |
+
- `Demo.serve` -> public Streamlit UI
|
| 47 |
+
|
| 48 |
+
## 3. Smoke test
|
| 49 |
+
|
| 50 |
+
Streamlit:
|
| 51 |
+
|
| 52 |
+
```bash
|
| 53 |
+
open https://<workspace>--gazet-demo-serve.modal.run
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
API directly:
|
| 57 |
+
|
| 58 |
+
```bash
|
| 59 |
+
curl "https://<workspace>--gazet-api-fastapi-app.modal.run/search?q=Odisha"
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
Expected timings:
|
| 63 |
+
|
| 64 |
+
- First request after idle: ~25-30s (cold start cascade across all three)
|
| 65 |
+
- Warm requests within scaledown windows: ~1-3s
|
| 66 |
+
|
| 67 |
+
## 4. Set the budget cap
|
| 68 |
+
|
| 69 |
+
Modal dashboard -> Settings -> Billing:
|
| 70 |
+
|
| 71 |
+
- Workspace spending limit: **$50/mo**
|
| 72 |
+
- Email alerts at $25 / $40 / $50
|
| 73 |
+
|
| 74 |
+
Modal pauses new container starts when the limit is hit.
|
| 75 |
+
|
| 76 |
+
## Updating
|
| 77 |
+
|
| 78 |
+
Code or dependency changes:
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
modal deploy modal_serve/serve.py
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
Model updates: re-upload to the `gazet` volume; running containers pick up the new file at next cold start.
|
| 85 |
+
|
| 86 |
+
## Architecture notes
|
| 87 |
+
|
| 88 |
+
- **No supervisord.** Each Cls runs one logical service.
|
| 89 |
+
- **`@modal.asgi_app`** serves FastAPI natively, no uvicorn subprocess.
|
| 90 |
+
- **`@modal.web_server`** wraps non-ASGI processes (`llama-server` binary, Streamlit).
|
| 91 |
+
- **Cross-Cls URLs** resolved at runtime via `modal.Cls.from_name(...)`.
|
| 92 |
+
- **`scaledown_window`** tuned per tier: 120s GPU, 300s API, 600s Demo (UI sessions are sticky).
|
| 93 |
+
|
| 94 |
+
## Cost reference
|
| 95 |
+
|
| 96 |
+
| Traffic | Monthly estimate |
|
| 97 |
+
|---|---|
|
| 98 |
+
| 200 queries/day | ~$5-7 |
|
| 99 |
+
| 1000 queries/day | ~$18-25 |
|
| 100 |
+
| Idle | $0 |
|
| 101 |
+
|
| 102 |
+
T4 GPU at ~$0.59/hr is only billed during active inference + brief warmup. CPU containers are negligible.
|
| 103 |
+
|
| 104 |
+
## Troubleshooting
|
| 105 |
+
|
| 106 |
+
**Cold start fails on `LlamaServer`**: check the binary path in `modal_serve/serve.py`. The official image's binary is at `/app/llama-server`; if upstream changes, run `modal shell gazet::LlamaServer` and `which llama-server`.
|
| 107 |
+
|
| 108 |
+
**`Api` cannot reach `LlamaServer`**: confirm `modal.Cls.from_name(...).serve.web_url` returns a non-empty string. The first deploy registers URLs; redeploys keep them stable.
|
| 109 |
+
|
| 110 |
+
**Streamlit websocket errors**: `@modal.web_server` supports websockets natively; if a proxy issue appears, raise `startup_timeout` and check `modal logs gazet`.
|
| 111 |
+
|
| 112 |
+
**Model not found**: the path in `serve.py` is `/models/checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf`. Verify the volume layout matches.
|
modal_serve/serve.py
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Modal deployment for Gazet.
|
| 2 |
+
|
| 3 |
+
Three independently scaled containers:
|
| 4 |
+
- LlamaServer: GPU T4 running the official llama.cpp server image
|
| 5 |
+
- Api: CPU container serving gazet.api as a native ASGI app
|
| 6 |
+
- Demo: CPU container running the Streamlit UI
|
| 7 |
+
|
| 8 |
+
Deploy:
|
| 9 |
+
modal deploy modal_serve/serve.py
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
import modal
|
| 15 |
+
|
| 16 |
+
APP_NAME = "gazet"
|
| 17 |
+
|
| 18 |
+
MODEL_PATH = "/models/checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf"
|
| 19 |
+
DATA_PATH = "/data"
|
| 20 |
+
|
| 21 |
+
app = modal.App(APP_NAME)
|
| 22 |
+
|
| 23 |
+
gazet_vol = modal.Volume.from_name("gazet")
|
| 24 |
+
data_vol = modal.Volume.from_name("gazet-data")
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
# Llama Server
|
| 28 |
+
# The base image's ENTRYPOINT is ["/app/llama-server"], which is incompatible
|
| 29 |
+
# with Modal's runtime: per Modal docs, an image's ENTRYPOINT must exec args
|
| 30 |
+
# passed to it so the Python harness can launch. We override it to a no-op
|
| 31 |
+
# passthrough, relocate libs to a canonical path, and ldconfig them.
|
| 32 |
+
llama_image = (
|
| 33 |
+
modal.Image.from_registry(
|
| 34 |
+
"ghcr.io/ggml-org/llama.cpp:server-cuda",
|
| 35 |
+
add_python="3.11",
|
| 36 |
+
)
|
| 37 |
+
.entrypoint([])
|
| 38 |
+
.run_commands(
|
| 39 |
+
"cp -r /app /usr/local/lib/llama",
|
| 40 |
+
"echo /usr/local/lib/llama > /etc/ld.so.conf.d/llama.conf",
|
| 41 |
+
"ldconfig",
|
| 42 |
+
)
|
| 43 |
+
.env({"LD_LIBRARY_PATH": "/usr/local/lib/llama"})
|
| 44 |
+
)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
@app.cls(
|
| 48 |
+
image=llama_image,
|
| 49 |
+
gpu="T4",
|
| 50 |
+
volumes={"/models": gazet_vol},
|
| 51 |
+
scaledown_window=120,
|
| 52 |
+
max_containers=2,
|
| 53 |
+
timeout=600,
|
| 54 |
+
)
|
| 55 |
+
@modal.concurrent(max_inputs=4)
|
| 56 |
+
class LlamaServer:
|
| 57 |
+
@modal.web_server(port=9000, startup_timeout=120)
|
| 58 |
+
def serve(self):
|
| 59 |
+
import subprocess
|
| 60 |
+
|
| 61 |
+
subprocess.Popen(
|
| 62 |
+
[
|
| 63 |
+
"/usr/local/lib/llama/llama-server",
|
| 64 |
+
"-m", MODEL_PATH,
|
| 65 |
+
"-ngl", "99",
|
| 66 |
+
"--host", "0.0.0.0",
|
| 67 |
+
"--port", "9000",
|
| 68 |
+
"--ctx-size", "2048",
|
| 69 |
+
]
|
| 70 |
+
)
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
# FastAPI app
|
| 74 |
+
api_image = (
|
| 75 |
+
modal.Image.debian_slim(python_version="3.11")
|
| 76 |
+
.pip_install_from_pyproject("pyproject.toml")
|
| 77 |
+
.add_local_python_source("gazet")
|
| 78 |
+
)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
@app.cls(
|
| 82 |
+
image=api_image,
|
| 83 |
+
volumes={"/data": data_vol},
|
| 84 |
+
scaledown_window=300,
|
| 85 |
+
max_containers=3,
|
| 86 |
+
timeout=300,
|
| 87 |
+
)
|
| 88 |
+
@modal.concurrent(max_inputs=10)
|
| 89 |
+
class Api:
|
| 90 |
+
@modal.enter()
|
| 91 |
+
def setup(self):
|
| 92 |
+
import os
|
| 93 |
+
|
| 94 |
+
os.environ.setdefault("GAZET_DATA_DIR", DATA_PATH)
|
| 95 |
+
os.environ.setdefault("GAZET_USE_NORMALIZED_DATA", "1")
|
| 96 |
+
os.environ["LLAMA_SERVER_URL"] = LlamaServer().serve.get_web_url()
|
| 97 |
+
|
| 98 |
+
@modal.asgi_app()
|
| 99 |
+
def fastapi_app(self):
|
| 100 |
+
from gazet.api import app as fastapi_app
|
| 101 |
+
|
| 102 |
+
return fastapi_app
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
# Streamlit Demo
|
| 106 |
+
demo_image = (
|
| 107 |
+
modal.Image.debian_slim(python_version="3.11")
|
| 108 |
+
.pip_install_from_pyproject("pyproject.toml", optional_dependencies=["demo"])
|
| 109 |
+
.add_local_python_source("gazet")
|
| 110 |
+
.add_local_file("gazet_demo.py", "/root/gazet_demo.py")
|
| 111 |
+
)
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
@app.cls(
|
| 115 |
+
image=demo_image,
|
| 116 |
+
scaledown_window=600,
|
| 117 |
+
max_containers=3,
|
| 118 |
+
timeout=600,
|
| 119 |
+
)
|
| 120 |
+
@modal.concurrent(max_inputs=10)
|
| 121 |
+
class Demo:
|
| 122 |
+
@modal.web_server(port=7860, startup_timeout=60)
|
| 123 |
+
def serve(self):
|
| 124 |
+
import os
|
| 125 |
+
import subprocess
|
| 126 |
+
|
| 127 |
+
env = os.environ.copy()
|
| 128 |
+
env["GAZET_API_URL"] = Api().fastapi_app.get_web_url()
|
| 129 |
+
|
| 130 |
+
subprocess.Popen(
|
| 131 |
+
[
|
| 132 |
+
"streamlit", "run", "/root/gazet_demo.py",
|
| 133 |
+
"--server.port", "7860",
|
| 134 |
+
"--server.address", "0.0.0.0",
|
| 135 |
+
"--server.headless", "true",
|
| 136 |
+
"--browser.gatherUsageStats", "false",
|
| 137 |
+
],
|
| 138 |
+
env=env,
|
| 139 |
+
)
|
src/gazet/api.py
CHANGED
|
@@ -46,11 +46,16 @@ def _run_stream(query: str, backend: str = "gguf") -> Generator[str, None, None]
|
|
| 46 |
- ``error`` – fatal error (no result)
|
| 47 |
"""
|
| 48 |
if backend == "gguf":
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
places_result = generate_places(query)
|
| 50 |
else:
|
| 51 |
pred = extract(query=query)
|
| 52 |
places_result = pred.result
|
| 53 |
-
print("places:", places_result)
|
| 54 |
|
| 55 |
yield json.dumps({"type": "places", "data": places_result.model_dump()}) + "\n"
|
| 56 |
|
|
|
|
| 46 |
- ``error`` – fatal error (no result)
|
| 47 |
"""
|
| 48 |
if backend == "gguf":
|
| 49 |
+
from .lm import is_llama_server_available
|
| 50 |
+
if not is_llama_server_available():
|
| 51 |
+
yield json.dumps({
|
| 52 |
+
"type": "warming_up",
|
| 53 |
+
"data": "Starting model server, this takes 30-60s on cold start",
|
| 54 |
+
}) + "\n"
|
| 55 |
places_result = generate_places(query)
|
| 56 |
else:
|
| 57 |
pred = extract(query=query)
|
| 58 |
places_result = pred.result
|
|
|
|
| 59 |
|
| 60 |
yield json.dumps({"type": "places", "data": places_result.model_dump()}) + "\n"
|
| 61 |
|
src/gazet/lm.py
CHANGED
|
@@ -1,5 +1,6 @@
|
|
| 1 |
import json
|
| 2 |
import logging
|
|
|
|
| 3 |
|
| 4 |
import dspy
|
| 5 |
import httpx
|
|
@@ -213,21 +214,36 @@ def is_llama_server_available() -> bool:
|
|
| 213 |
|
| 214 |
|
| 215 |
def _llama_chat_complete(messages: list[dict]) -> str:
|
| 216 |
-
"""Call llama-server /v1/chat/completions
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
)
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 231 |
|
| 232 |
|
| 233 |
_PLACES_SYSTEM_PROMPT = """You are a geographic entity extractor. Extract the place names the user is asking about and return valid JSON only.
|
|
|
|
| 1 |
import json
|
| 2 |
import logging
|
| 3 |
+
import time
|
| 4 |
|
| 5 |
import dspy
|
| 6 |
import httpx
|
|
|
|
| 214 |
|
| 215 |
|
| 216 |
def _llama_chat_complete(messages: list[dict]) -> str:
|
| 217 |
+
"""Call llama-server /v1/chat/completions, waiting through cold starts."""
|
| 218 |
+
payload = {
|
| 219 |
+
"messages": messages,
|
| 220 |
+
"n_predict": LLAMA_MAX_TOKENS,
|
| 221 |
+
"temperature": LLAMA_TEMPERATURE,
|
| 222 |
+
"chat_template_kwargs": {"enable_thinking": False},
|
| 223 |
+
}
|
| 224 |
+
deadline = time.monotonic() + 120.0
|
| 225 |
+
backoff = 2.0
|
| 226 |
+
last_err: Exception | None = None
|
| 227 |
+
while time.monotonic() < deadline:
|
| 228 |
+
try:
|
| 229 |
+
resp = httpx.post(
|
| 230 |
+
f"{LLAMA_SERVER_URL}/v1/chat/completions",
|
| 231 |
+
json=payload,
|
| 232 |
+
timeout=90,
|
| 233 |
+
)
|
| 234 |
+
if resp.status_code == 503:
|
| 235 |
+
time.sleep(backoff)
|
| 236 |
+
backoff = min(backoff * 1.5, 8.0)
|
| 237 |
+
continue
|
| 238 |
+
if resp.status_code != 200:
|
| 239 |
+
logger.error("llama-server %s: %s", resp.status_code, resp.text[:500])
|
| 240 |
+
resp.raise_for_status()
|
| 241 |
+
return resp.json()["choices"][0]["message"]["content"]
|
| 242 |
+
except (httpx.ConnectError, httpx.ReadError, httpx.TimeoutException) as e:
|
| 243 |
+
last_err = e
|
| 244 |
+
time.sleep(backoff)
|
| 245 |
+
backoff = min(backoff * 1.5, 8.0)
|
| 246 |
+
raise RuntimeError(f"llama-server did not become ready within 120s: {last_err}")
|
| 247 |
|
| 248 |
|
| 249 |
_PLACES_SYSTEM_PROMPT = """You are a geographic entity extractor. Extract the place names the user is asking about and return valid JSON only.
|
src/gazet/sql.py
CHANGED
|
@@ -22,34 +22,62 @@ from .config import DIVISIONS_AREA_PATH, MAX_SQL_ITERATIONS, NATURAL_EARTH_PATH,
|
|
| 22 |
from .lm import generate_sql, write_sql
|
| 23 |
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
|
| 35 |
def _normalize_geometry_to_geojson(
|
| 36 |
result_df: pd.DataFrame,
|
| 37 |
) -> pd.DataFrame:
|
| 38 |
-
"""
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
return result_df
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
return val
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
|
|
|
| 49 |
return None
|
|
|
|
| 50 |
|
| 51 |
normalized_df = result_df.copy()
|
| 52 |
-
normalized_df["geometry"] = normalized_df["geometry"].apply(
|
|
|
|
| 53 |
return normalized_df
|
| 54 |
|
| 55 |
|
|
|
|
| 22 |
from .lm import generate_sql, write_sql
|
| 23 |
|
| 24 |
|
| 25 |
+
SIMPLIFY_TOLERANCE = 0.001 # ~100m; adequate for web map display
|
| 26 |
+
COORD_PRECISION = 5 # ~1.1m; sufficient for web maps, shrinks payload 30-50%
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _round_coords(obj: Any, precision: int) -> Any:
|
| 30 |
+
"""Recursively round numeric coordinates in a GeoJSON geometry dict."""
|
| 31 |
+
if isinstance(obj, float):
|
| 32 |
+
return round(obj, precision)
|
| 33 |
+
if isinstance(obj, list):
|
| 34 |
+
return [_round_coords(x, precision) for x in obj]
|
| 35 |
+
if isinstance(obj, dict):
|
| 36 |
+
return {k: _round_coords(v, precision) for k, v in obj.items()}
|
| 37 |
+
return obj
|
| 38 |
|
| 39 |
|
| 40 |
def _normalize_geometry_to_geojson(
|
| 41 |
result_df: pd.DataFrame,
|
| 42 |
) -> pd.DataFrame:
|
| 43 |
+
"""Simplify geometries and convert to compact GeoJSON text.
|
| 44 |
+
|
| 45 |
+
Accepts either binary WKB blobs or GeoJSON strings in the `geometry` column.
|
| 46 |
+
Runs ST_SimplifyPreserveTopology then rounds coordinates to reduce payload.
|
| 47 |
+
"""
|
| 48 |
+
if "geometry" not in result_df.columns or result_df.empty:
|
| 49 |
return result_df
|
| 50 |
|
| 51 |
+
sample = result_df["geometry"].dropna().head(5)
|
| 52 |
+
if sample.empty:
|
| 53 |
+
return result_df
|
| 54 |
+
|
| 55 |
+
con = duckdb.connect()
|
| 56 |
+
con.execute("INSTALL spatial")
|
| 57 |
+
con.execute("LOAD spatial")
|
| 58 |
+
|
| 59 |
+
def _simplify(val: Any) -> Optional[str]:
|
| 60 |
+
if val is None:
|
| 61 |
+
return None
|
| 62 |
+
if isinstance(val, (bytes, bytearray, memoryview)):
|
| 63 |
+
geom_expr = "ST_GeomFromWKB(?::BLOB)"
|
| 64 |
+
arg: Any = bytes(val)
|
| 65 |
+
elif isinstance(val, str) and val.lstrip().startswith('{"'):
|
| 66 |
+
geom_expr = "ST_GeomFromGeoJSON(?)"
|
| 67 |
+
arg = val
|
| 68 |
+
else:
|
| 69 |
return val
|
| 70 |
+
row = con.execute(
|
| 71 |
+
f"SELECT ST_AsGeoJSON(ST_SimplifyPreserveTopology({geom_expr}, ?))",
|
| 72 |
+
[arg, SIMPLIFY_TOLERANCE],
|
| 73 |
+
).fetchone()
|
| 74 |
+
if not row or not row[0]:
|
| 75 |
return None
|
| 76 |
+
return json.dumps(_round_coords(json.loads(row[0]), COORD_PRECISION))
|
| 77 |
|
| 78 |
normalized_df = result_df.copy()
|
| 79 |
+
normalized_df["geometry"] = normalized_df["geometry"].apply(_simplify)
|
| 80 |
+
con.close()
|
| 81 |
return normalized_df
|
| 82 |
|
| 83 |
|