srmsoumya commited on
Commit
0836eef
·
1 Parent(s): d128a38

enh: Run gazet on modal - llama server on GPU, API & UI on cpu

Browse files
gazet_demo.py CHANGED
@@ -219,6 +219,10 @@ with col2:
219
  event = json.loads(raw)
220
  t = event["type"]
221
 
 
 
 
 
222
  if t == "places":
223
  places = event["data"].get("places", [])
224
  result["places"] = places
 
219
  event = json.loads(raw)
220
  t = event["type"]
221
 
222
+ if t == "warming_up":
223
+ status_ph.warning(event["data"])
224
+ continue
225
+
226
  if t == "places":
227
  places = event["data"].get("places", [])
228
  result["places"] = places
modal_serve/README.md ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modal Deployment
2
+
3
+ Deploys Gazet to Modal as three independently scaled containers:
4
+
5
+ | Cls | Hardware | Role |
6
+ |---|---|---|
7
+ | `LlamaServer` | GPU T4 | `llama-server` from the official `ghcr.io/ggml-org/llama.cpp:server-cuda` image |
8
+ | `Api` | CPU | `gazet.api` FastAPI app served as native ASGI |
9
+ | `Demo` | CPU | Streamlit UI from `gazet_demo.py` |
10
+
11
+ All three scale to zero independently. The Demo URL is the user-facing entrypoint.
12
+
13
+ ## Prerequisites
14
+
15
+ - Modal account + CLI: `uv tool install modal` then `modal token new`
16
+ - Modal volumes already exist:
17
+ - `gazet` (model checkpoints)
18
+ - `gazet-data` (normalized parquet datasets)
19
+
20
+ ## 1. Upload the Q4 GGUF model
21
+
22
+ ```bash
23
+ modal volume put gazet \
24
+ finetune/models/ckpt-q4_k_m.gguf \
25
+ checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf
26
+ ```
27
+
28
+ Verify:
29
+
30
+ ```bash
31
+ modal volume ls gazet checkpoints/qwen35-fientune-v3 | grep ckpt-q4_k_m
32
+ ```
33
+
34
+ ## 2. Deploy
35
+
36
+ From the repo root:
37
+
38
+ ```bash
39
+ modal deploy modal_serve/serve.py
40
+ ```
41
+
42
+ The output prints three URLs:
43
+
44
+ - `LlamaServer.serve` -> internal, called by `Api`
45
+ - `Api.fastapi_app` -> public FastAPI (`/search`, `/search/stream`)
46
+ - `Demo.serve` -> public Streamlit UI
47
+
48
+ ## 3. Smoke test
49
+
50
+ Streamlit:
51
+
52
+ ```bash
53
+ open https://<workspace>--gazet-demo-serve.modal.run
54
+ ```
55
+
56
+ API directly:
57
+
58
+ ```bash
59
+ curl "https://<workspace>--gazet-api-fastapi-app.modal.run/search?q=Odisha"
60
+ ```
61
+
62
+ Expected timings:
63
+
64
+ - First request after idle: ~25-30s (cold start cascade across all three)
65
+ - Warm requests within scaledown windows: ~1-3s
66
+
67
+ ## 4. Set the budget cap
68
+
69
+ Modal dashboard -> Settings -> Billing:
70
+
71
+ - Workspace spending limit: **$50/mo**
72
+ - Email alerts at $25 / $40 / $50
73
+
74
+ Modal pauses new container starts when the limit is hit.
75
+
76
+ ## Updating
77
+
78
+ Code or dependency changes:
79
+
80
+ ```bash
81
+ modal deploy modal_serve/serve.py
82
+ ```
83
+
84
+ Model updates: re-upload to the `gazet` volume; running containers pick up the new file at next cold start.
85
+
86
+ ## Architecture notes
87
+
88
+ - **No supervisord.** Each Cls runs one logical service.
89
+ - **`@modal.asgi_app`** serves FastAPI natively, no uvicorn subprocess.
90
+ - **`@modal.web_server`** wraps non-ASGI processes (`llama-server` binary, Streamlit).
91
+ - **Cross-Cls URLs** resolved at runtime via `modal.Cls.from_name(...)`.
92
+ - **`scaledown_window`** tuned per tier: 120s GPU, 300s API, 600s Demo (UI sessions are sticky).
93
+
94
+ ## Cost reference
95
+
96
+ | Traffic | Monthly estimate |
97
+ |---|---|
98
+ | 200 queries/day | ~$5-7 |
99
+ | 1000 queries/day | ~$18-25 |
100
+ | Idle | $0 |
101
+
102
+ T4 GPU at ~$0.59/hr is only billed during active inference + brief warmup. CPU containers are negligible.
103
+
104
+ ## Troubleshooting
105
+
106
+ **Cold start fails on `LlamaServer`**: check the binary path in `modal_serve/serve.py`. The official image's binary is at `/app/llama-server`; if upstream changes, run `modal shell gazet::LlamaServer` and `which llama-server`.
107
+
108
+ **`Api` cannot reach `LlamaServer`**: confirm `modal.Cls.from_name(...).serve.web_url` returns a non-empty string. The first deploy registers URLs; redeploys keep them stable.
109
+
110
+ **Streamlit websocket errors**: `@modal.web_server` supports websockets natively; if a proxy issue appears, raise `startup_timeout` and check `modal logs gazet`.
111
+
112
+ **Model not found**: the path in `serve.py` is `/models/checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf`. Verify the volume layout matches.
modal_serve/serve.py ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Modal deployment for Gazet.
2
+
3
+ Three independently scaled containers:
4
+ - LlamaServer: GPU T4 running the official llama.cpp server image
5
+ - Api: CPU container serving gazet.api as a native ASGI app
6
+ - Demo: CPU container running the Streamlit UI
7
+
8
+ Deploy:
9
+ modal deploy modal_serve/serve.py
10
+ """
11
+
12
+ from __future__ import annotations
13
+
14
+ import modal
15
+
16
+ APP_NAME = "gazet"
17
+
18
+ MODEL_PATH = "/models/checkpoints/qwen35-fientune-v3/ckpt-q4_k_m.gguf"
19
+ DATA_PATH = "/data"
20
+
21
+ app = modal.App(APP_NAME)
22
+
23
+ gazet_vol = modal.Volume.from_name("gazet")
24
+ data_vol = modal.Volume.from_name("gazet-data")
25
+
26
+
27
+ # Llama Server
28
+ # The base image's ENTRYPOINT is ["/app/llama-server"], which is incompatible
29
+ # with Modal's runtime: per Modal docs, an image's ENTRYPOINT must exec args
30
+ # passed to it so the Python harness can launch. We override it to a no-op
31
+ # passthrough, relocate libs to a canonical path, and ldconfig them.
32
+ llama_image = (
33
+ modal.Image.from_registry(
34
+ "ghcr.io/ggml-org/llama.cpp:server-cuda",
35
+ add_python="3.11",
36
+ )
37
+ .entrypoint([])
38
+ .run_commands(
39
+ "cp -r /app /usr/local/lib/llama",
40
+ "echo /usr/local/lib/llama > /etc/ld.so.conf.d/llama.conf",
41
+ "ldconfig",
42
+ )
43
+ .env({"LD_LIBRARY_PATH": "/usr/local/lib/llama"})
44
+ )
45
+
46
+
47
+ @app.cls(
48
+ image=llama_image,
49
+ gpu="T4",
50
+ volumes={"/models": gazet_vol},
51
+ scaledown_window=120,
52
+ max_containers=2,
53
+ timeout=600,
54
+ )
55
+ @modal.concurrent(max_inputs=4)
56
+ class LlamaServer:
57
+ @modal.web_server(port=9000, startup_timeout=120)
58
+ def serve(self):
59
+ import subprocess
60
+
61
+ subprocess.Popen(
62
+ [
63
+ "/usr/local/lib/llama/llama-server",
64
+ "-m", MODEL_PATH,
65
+ "-ngl", "99",
66
+ "--host", "0.0.0.0",
67
+ "--port", "9000",
68
+ "--ctx-size", "2048",
69
+ ]
70
+ )
71
+
72
+
73
+ # FastAPI app
74
+ api_image = (
75
+ modal.Image.debian_slim(python_version="3.11")
76
+ .pip_install_from_pyproject("pyproject.toml")
77
+ .add_local_python_source("gazet")
78
+ )
79
+
80
+
81
+ @app.cls(
82
+ image=api_image,
83
+ volumes={"/data": data_vol},
84
+ scaledown_window=300,
85
+ max_containers=3,
86
+ timeout=300,
87
+ )
88
+ @modal.concurrent(max_inputs=10)
89
+ class Api:
90
+ @modal.enter()
91
+ def setup(self):
92
+ import os
93
+
94
+ os.environ.setdefault("GAZET_DATA_DIR", DATA_PATH)
95
+ os.environ.setdefault("GAZET_USE_NORMALIZED_DATA", "1")
96
+ os.environ["LLAMA_SERVER_URL"] = LlamaServer().serve.get_web_url()
97
+
98
+ @modal.asgi_app()
99
+ def fastapi_app(self):
100
+ from gazet.api import app as fastapi_app
101
+
102
+ return fastapi_app
103
+
104
+
105
+ # Streamlit Demo
106
+ demo_image = (
107
+ modal.Image.debian_slim(python_version="3.11")
108
+ .pip_install_from_pyproject("pyproject.toml", optional_dependencies=["demo"])
109
+ .add_local_python_source("gazet")
110
+ .add_local_file("gazet_demo.py", "/root/gazet_demo.py")
111
+ )
112
+
113
+
114
+ @app.cls(
115
+ image=demo_image,
116
+ scaledown_window=600,
117
+ max_containers=3,
118
+ timeout=600,
119
+ )
120
+ @modal.concurrent(max_inputs=10)
121
+ class Demo:
122
+ @modal.web_server(port=7860, startup_timeout=60)
123
+ def serve(self):
124
+ import os
125
+ import subprocess
126
+
127
+ env = os.environ.copy()
128
+ env["GAZET_API_URL"] = Api().fastapi_app.get_web_url()
129
+
130
+ subprocess.Popen(
131
+ [
132
+ "streamlit", "run", "/root/gazet_demo.py",
133
+ "--server.port", "7860",
134
+ "--server.address", "0.0.0.0",
135
+ "--server.headless", "true",
136
+ "--browser.gatherUsageStats", "false",
137
+ ],
138
+ env=env,
139
+ )
src/gazet/api.py CHANGED
@@ -46,11 +46,16 @@ def _run_stream(query: str, backend: str = "gguf") -> Generator[str, None, None]
46
  - ``error`` – fatal error (no result)
47
  """
48
  if backend == "gguf":
 
 
 
 
 
 
49
  places_result = generate_places(query)
50
  else:
51
  pred = extract(query=query)
52
  places_result = pred.result
53
- print("places:", places_result)
54
 
55
  yield json.dumps({"type": "places", "data": places_result.model_dump()}) + "\n"
56
 
 
46
  - ``error`` – fatal error (no result)
47
  """
48
  if backend == "gguf":
49
+ from .lm import is_llama_server_available
50
+ if not is_llama_server_available():
51
+ yield json.dumps({
52
+ "type": "warming_up",
53
+ "data": "Starting model server, this takes 30-60s on cold start",
54
+ }) + "\n"
55
  places_result = generate_places(query)
56
  else:
57
  pred = extract(query=query)
58
  places_result = pred.result
 
59
 
60
  yield json.dumps({"type": "places", "data": places_result.model_dump()}) + "\n"
61
 
src/gazet/lm.py CHANGED
@@ -1,5 +1,6 @@
1
  import json
2
  import logging
 
3
 
4
  import dspy
5
  import httpx
@@ -213,21 +214,36 @@ def is_llama_server_available() -> bool:
213
 
214
 
215
  def _llama_chat_complete(messages: list[dict]) -> str:
216
- """Call llama-server /v1/chat/completions with a messages list."""
217
- resp = httpx.post(
218
- f"{LLAMA_SERVER_URL}/v1/chat/completions",
219
- json={
220
- "messages": messages,
221
- "n_predict": LLAMA_MAX_TOKENS,
222
- "temperature": LLAMA_TEMPERATURE,
223
- "chat_template_kwargs": {"enable_thinking": False},
224
- },
225
- timeout=60,
226
- )
227
- if resp.status_code != 200:
228
- logger.error("llama-server %s: %s", resp.status_code, resp.text[:500])
229
- resp.raise_for_status()
230
- return resp.json()["choices"][0]["message"]["content"]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
231
 
232
 
233
  _PLACES_SYSTEM_PROMPT = """You are a geographic entity extractor. Extract the place names the user is asking about and return valid JSON only.
 
1
  import json
2
  import logging
3
+ import time
4
 
5
  import dspy
6
  import httpx
 
214
 
215
 
216
  def _llama_chat_complete(messages: list[dict]) -> str:
217
+ """Call llama-server /v1/chat/completions, waiting through cold starts."""
218
+ payload = {
219
+ "messages": messages,
220
+ "n_predict": LLAMA_MAX_TOKENS,
221
+ "temperature": LLAMA_TEMPERATURE,
222
+ "chat_template_kwargs": {"enable_thinking": False},
223
+ }
224
+ deadline = time.monotonic() + 120.0
225
+ backoff = 2.0
226
+ last_err: Exception | None = None
227
+ while time.monotonic() < deadline:
228
+ try:
229
+ resp = httpx.post(
230
+ f"{LLAMA_SERVER_URL}/v1/chat/completions",
231
+ json=payload,
232
+ timeout=90,
233
+ )
234
+ if resp.status_code == 503:
235
+ time.sleep(backoff)
236
+ backoff = min(backoff * 1.5, 8.0)
237
+ continue
238
+ if resp.status_code != 200:
239
+ logger.error("llama-server %s: %s", resp.status_code, resp.text[:500])
240
+ resp.raise_for_status()
241
+ return resp.json()["choices"][0]["message"]["content"]
242
+ except (httpx.ConnectError, httpx.ReadError, httpx.TimeoutException) as e:
243
+ last_err = e
244
+ time.sleep(backoff)
245
+ backoff = min(backoff * 1.5, 8.0)
246
+ raise RuntimeError(f"llama-server did not become ready within 120s: {last_err}")
247
 
248
 
249
  _PLACES_SYSTEM_PROMPT = """You are a geographic entity extractor. Extract the place names the user is asking about and return valid JSON only.
src/gazet/sql.py CHANGED
@@ -22,34 +22,62 @@ from .config import DIVISIONS_AREA_PATH, MAX_SQL_ITERATIONS, NATURAL_EARTH_PATH,
22
  from .lm import generate_sql, write_sql
23
 
24
 
25
- def _needs_geometry_normalization(result_df: pd.DataFrame) -> bool:
26
- """Return True when `geometry` is a binary blob likely from DuckDB GEOMETRY."""
27
- if "geometry" not in result_df.columns or result_df.empty:
28
- return False
29
- sample = result_df["geometry"].dropna().head(5)
30
- if sample.empty:
31
- return False
32
- return sample.apply(lambda v: isinstance(v, (bytes, bytearray, memoryview))).all()
 
 
 
 
 
33
 
34
 
35
  def _normalize_geometry_to_geojson(
36
  result_df: pd.DataFrame,
37
  ) -> pd.DataFrame:
38
- """If result has binary geometry, convert it to GeoJSON text in Python."""
39
- if not _needs_geometry_normalization(result_df):
 
 
 
 
40
  return result_df
41
 
42
- def _blob_to_geojson_text(val: Any) -> Optional[str]:
43
- if not isinstance(val, (bytes, bytearray, memoryview)):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
  return val
45
- try:
46
- geom = wkb.loads(bytes(val))
47
- return json.dumps(mapping(geom))
48
- except Exception:
 
49
  return None
 
50
 
51
  normalized_df = result_df.copy()
52
- normalized_df["geometry"] = normalized_df["geometry"].apply(_blob_to_geojson_text)
 
53
  return normalized_df
54
 
55
 
 
22
  from .lm import generate_sql, write_sql
23
 
24
 
25
+ SIMPLIFY_TOLERANCE = 0.001 # ~100m; adequate for web map display
26
+ COORD_PRECISION = 5 # ~1.1m; sufficient for web maps, shrinks payload 30-50%
27
+
28
+
29
+ def _round_coords(obj: Any, precision: int) -> Any:
30
+ """Recursively round numeric coordinates in a GeoJSON geometry dict."""
31
+ if isinstance(obj, float):
32
+ return round(obj, precision)
33
+ if isinstance(obj, list):
34
+ return [_round_coords(x, precision) for x in obj]
35
+ if isinstance(obj, dict):
36
+ return {k: _round_coords(v, precision) for k, v in obj.items()}
37
+ return obj
38
 
39
 
40
  def _normalize_geometry_to_geojson(
41
  result_df: pd.DataFrame,
42
  ) -> pd.DataFrame:
43
+ """Simplify geometries and convert to compact GeoJSON text.
44
+
45
+ Accepts either binary WKB blobs or GeoJSON strings in the `geometry` column.
46
+ Runs ST_SimplifyPreserveTopology then rounds coordinates to reduce payload.
47
+ """
48
+ if "geometry" not in result_df.columns or result_df.empty:
49
  return result_df
50
 
51
+ sample = result_df["geometry"].dropna().head(5)
52
+ if sample.empty:
53
+ return result_df
54
+
55
+ con = duckdb.connect()
56
+ con.execute("INSTALL spatial")
57
+ con.execute("LOAD spatial")
58
+
59
+ def _simplify(val: Any) -> Optional[str]:
60
+ if val is None:
61
+ return None
62
+ if isinstance(val, (bytes, bytearray, memoryview)):
63
+ geom_expr = "ST_GeomFromWKB(?::BLOB)"
64
+ arg: Any = bytes(val)
65
+ elif isinstance(val, str) and val.lstrip().startswith('{"'):
66
+ geom_expr = "ST_GeomFromGeoJSON(?)"
67
+ arg = val
68
+ else:
69
  return val
70
+ row = con.execute(
71
+ f"SELECT ST_AsGeoJSON(ST_SimplifyPreserveTopology({geom_expr}, ?))",
72
+ [arg, SIMPLIFY_TOLERANCE],
73
+ ).fetchone()
74
+ if not row or not row[0]:
75
  return None
76
+ return json.dumps(_round_coords(json.loads(row[0]), COORD_PRECISION))
77
 
78
  normalized_df = result_df.copy()
79
+ normalized_df["geometry"] = normalized_df["geometry"].apply(_simplify)
80
+ con.close()
81
  return normalized_df
82
 
83