Tonic commited on
Commit
eb91462
·
verified ·
1 Parent(s): 01a81a8

ci: deploy game_server (a9efbe759359)

Browse files
README.package.md CHANGED
@@ -33,6 +33,12 @@ Optional: copy **`.env.example`** to **`server/.env`** so local runs pick up **`
33
  | `FEATURE_GUESSES_RECORD` / `NUTONIC_FEATURE_GUESSES_RECORD` | No | Default **`false`**. When **`true`**, enables **`POST /api/v1/maps/{map_id}/guesses/record`** and sets `features.guesses_record` on **`GET /api/v1/config`**. Use **`true`** in local dev when exercising telemetry (`rules/05`, `docs/GAME-ENGINE.md` §12.3). |
34
  | `NUTONIC_GUESS_TELEMETRY_DATABASE_URL` | No | Default `sqlite:///data/nutonic_guess_telemetry.db`. SQLite file for optional guess rows (created under `data/` like the leaderboard DB). |
35
  | `NUTONIC_RANKED_DATABASE_URL` | No | Default `sqlite:///data/nutonic_ranked.db`. SQLite for ranked round rows (**IMP-090**). |
 
 
 
 
 
 
36
  | `NUTONIC_RANKED_STALE_OPEN_ROUND_MAX_AGE_SECONDS` | No | Default **604800** (7d). Abandoned **`open`** ranked rounds older than this are deleted on each **`POST /api/v1/ranked/rounds/start`** (housekeeping). |
37
  | `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH` | No | Default **`false`**: **`GET /api/v1/cache/manifest`** omits `locations` and `ai_guesses` (public spoiler hygiene). Set **`true`** for local dev / tests that need full fixture slices. |
38
  | `NUTONIC_MANIFEST_FULL_PATH` / `MANIFEST_FULL_PATH` | No | When set to an existing **`manifest.full.json`** (same schema as **`assemble_manifest.py`** output), replaces the builtin demo **`PUBLISHED_MAPS` / `MANIFEST_LOCATIONS` / `MANIFEST_AI_GUESSES`** at process start so **`GET /api/v1/maps`**, **`GET /api/v1/cache/manifest`**, and **`POST /api/v1/ranked/rounds/start`** use the full GeoGuessr-derived catalog. Optional **`satellite_caption_sidecar`** on each location is preserved and echoed on **`RankedClueOut`** when present. |
@@ -64,6 +70,7 @@ The game server is deployed by **`.github/workflows/huggingface-deploy.yml`**, n
64
  | Runtime profile | `tools/hf_deploy/profiles/game_server.yaml`: `cpu-basic`, `FEATURE_RANKED=false`, `FEATURE_COMMUNITY_LB_POST=false`, `FEATURE_PRO_JOBS=false`, `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH=false`. |
65
  | Hub auth | `HF_TOKEN_NUTONIC`, then `HF_TOKEN`, then `HF_API_WRITE`. |
66
  | Runtime secrets | `NUTONIC_JWT_SECRET` is pushed to the Space as `JWT_SECRET`; `NUTONIC_LEADERBOARD_DATABASE_URL` and `NUTONIC_RANKED_DATABASE_URL` are pushed under the same names when set. |
 
67
  | Post-deploy smoke | `python tools/live_inference_smoke.py --preset game-deploy --timeout 30 --strict --json-report-path artifacts/hf-smoke-game.json`. |
68
 
69
  The deployed Space defaults are intentionally conservative: ranked, community writes, PRO jobs, and manifest truth are off until the Space profile or Space settings opt in. Enabling live PRO on the game server also requires worker URLs and the same `NUTONIC_INFERENCE_HMAC_SECRET` used by any HMAC-protected worker.
 
33
  | `FEATURE_GUESSES_RECORD` / `NUTONIC_FEATURE_GUESSES_RECORD` | No | Default **`false`**. When **`true`**, enables **`POST /api/v1/maps/{map_id}/guesses/record`** and sets `features.guesses_record` on **`GET /api/v1/config`**. Use **`true`** in local dev when exercising telemetry (`rules/05`, `docs/GAME-ENGINE.md` §12.3). |
34
  | `NUTONIC_GUESS_TELEMETRY_DATABASE_URL` | No | Default `sqlite:///data/nutonic_guess_telemetry.db`. SQLite file for optional guess rows (created under `data/` like the leaderboard DB). |
35
  | `NUTONIC_RANKED_DATABASE_URL` | No | Default `sqlite:///data/nutonic_ranked.db`. SQLite for ranked round rows (**IMP-090**). |
36
+ | `NUTONIC_HF_PERSISTENCE_ENABLED` / `HF_PERSISTENCE_ENABLED` | No | Default **`false`**. When **`true`**, SQLite stores bootstrap from and sync writes to a Hugging Face Dataset repo. |
37
+ | `NUTONIC_HF_PERSISTENCE_REQUIRED` / `HF_PERSISTENCE_REQUIRED` | No | Default **`false`**. When **`true`** with HF persistence enabled, startup fails if repo id/token is missing and runtime fails on sync errors (fail-fast mode). |
38
+ | `NUTONIC_HF_PERSISTENCE_REPO_ID` / `HF_PERSISTENCE_REPO_ID` | No | Dataset repo id (`owner/name`) used for persisted SQLite files. |
39
+ | `NUTONIC_HF_PERSISTENCE_SUBDIR` / `HF_PERSISTENCE_SUBDIR` | No | Default `server-persistence`. Subdirectory inside the dataset repo for DB files. |
40
+ | `NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE` / `HF_PERSISTENCE_STARTUP_PULL_MODE` | No | Default `if_missing`. `if_missing` only pulls when local DB is absent; `always` always pulls from dataset at startup (remote wins). |
41
+ | `HF_API_WRITE` / `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` | No | Hub token used by HF persistence sync client for dataset read/write calls. Prefer a dedicated write token in production. |
42
  | `NUTONIC_RANKED_STALE_OPEN_ROUND_MAX_AGE_SECONDS` | No | Default **604800** (7d). Abandoned **`open`** ranked rounds older than this are deleted on each **`POST /api/v1/ranked/rounds/start`** (housekeeping). |
43
  | `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH` | No | Default **`false`**: **`GET /api/v1/cache/manifest`** omits `locations` and `ai_guesses` (public spoiler hygiene). Set **`true`** for local dev / tests that need full fixture slices. |
44
  | `NUTONIC_MANIFEST_FULL_PATH` / `MANIFEST_FULL_PATH` | No | When set to an existing **`manifest.full.json`** (same schema as **`assemble_manifest.py`** output), replaces the builtin demo **`PUBLISHED_MAPS` / `MANIFEST_LOCATIONS` / `MANIFEST_AI_GUESSES`** at process start so **`GET /api/v1/maps`**, **`GET /api/v1/cache/manifest`**, and **`POST /api/v1/ranked/rounds/start`** use the full GeoGuessr-derived catalog. Optional **`satellite_caption_sidecar`** on each location is preserved and echoed on **`RankedClueOut`** when present. |
 
70
  | Runtime profile | `tools/hf_deploy/profiles/game_server.yaml`: `cpu-basic`, `FEATURE_RANKED=false`, `FEATURE_COMMUNITY_LB_POST=false`, `FEATURE_PRO_JOBS=false`, `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH=false`. |
71
  | Hub auth | `HF_TOKEN_NUTONIC`, then `HF_TOKEN`, then `HF_API_WRITE`. |
72
  | Runtime secrets | `NUTONIC_JWT_SECRET` is pushed to the Space as `JWT_SECRET`; `NUTONIC_LEADERBOARD_DATABASE_URL` and `NUTONIC_RANKED_DATABASE_URL` are pushed under the same names when set. |
73
+ | HF dataset persistence wiring | `NUTONIC_HF_PERSISTENCE_ENABLED`, `NUTONIC_HF_PERSISTENCE_REQUIRED`, `NUTONIC_HF_PERSISTENCE_REPO_ID`, `NUTONIC_HF_PERSISTENCE_SUBDIR`, and `NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE` are forwarded as runtime variables; `NUTONIC_HF_PERSISTENCE_TOKEN` is pushed as `HF_API_WRITE` secret for dataset writes. |
74
  | Post-deploy smoke | `python tools/live_inference_smoke.py --preset game-deploy --timeout 30 --strict --json-report-path artifacts/hf-smoke-game.json`. |
75
 
76
  The deployed Space defaults are intentionally conservative: ranked, community writes, PRO jobs, and manifest truth are off until the Space profile or Space settings opt in. Enabling live PRO on the game server also requires worker URLs and the same `NUTONIC_INFERENCE_HMAC_SECRET` used by any HMAC-protected worker.
pyproject.toml CHANGED
@@ -16,6 +16,7 @@ dependencies = [
16
  "httpx>=0.27.0",
17
  "PyJWT>=2.9.0",
18
  "sqlalchemy>=2.0.36",
 
19
  ]
20
 
21
  [project.optional-dependencies]
 
16
  "httpx>=0.27.0",
17
  "PyJWT>=2.9.0",
18
  "sqlalchemy>=2.0.36",
19
+ "huggingface_hub>=0.35.0",
20
  ]
21
 
22
  [project.optional-dependencies]
src/nutonic_server/deps.py CHANGED
@@ -7,9 +7,11 @@ from fastapi import Depends, HTTPException, status
7
  from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
8
 
9
  from nutonic_server.jwt_tokens import decode_bearer_token
 
10
  from nutonic_server.pro_jobs_runner import ProJobRunner
11
  from nutonic_server.pro_jobs_store import ProJobStore, create_pro_job_store
12
  from nutonic_server.settings import Settings, load_settings
 
13
 
14
  _bearer = HTTPBearer(auto_error=False)
15
  _pro_job_stores: dict[str, ProJobStore] = {}
@@ -31,7 +33,14 @@ def get_pro_job_store_for_settings(settings: Settings) -> ProJobStore:
31
  raise HTTPException(status_code=500, detail="Unsupported PRO job backend")
32
  url = settings.pro_job_database_url.strip()
33
  if url not in _pro_job_stores:
34
- _pro_job_stores[url] = create_pro_job_store(url)
 
 
 
 
 
 
 
35
  return _pro_job_stores[url]
36
 
37
 
 
7
  from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
8
 
9
  from nutonic_server.jwt_tokens import decode_bearer_token
10
+ from nutonic_server.hf_persistence import HfSqliteSync
11
  from nutonic_server.pro_jobs_runner import ProJobRunner
12
  from nutonic_server.pro_jobs_store import ProJobStore, create_pro_job_store
13
  from nutonic_server.settings import Settings, load_settings
14
+ from nutonic_server.leaderboard_store import sqlite_file_path_from_url
15
 
16
  _bearer = HTTPBearer(auto_error=False)
17
  _pro_job_stores: dict[str, ProJobStore] = {}
 
33
  raise HTTPException(status_code=500, detail="Unsupported PRO job backend")
34
  url = settings.pro_job_database_url.strip()
35
  if url not in _pro_job_stores:
36
+ sync_hook = None
37
+ db_path = sqlite_file_path_from_url(url)
38
+ if db_path is not None:
39
+ hf = HfSqliteSync.from_settings(settings)
40
+ if hf is not None:
41
+ hf.bootstrap_sqlite_file(local_path=db_path, logical_name="pro_jobs")
42
+ sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="pro_jobs")
43
+ _pro_job_stores[url] = create_pro_job_store(url, on_write=sync_hook)
44
  return _pro_job_stores[url]
45
 
46
 
src/nutonic_server/guess_telemetry_store.py CHANGED
@@ -4,7 +4,7 @@ from __future__ import annotations
4
 
5
  from dataclasses import dataclass
6
  from threading import Lock
7
- from typing import TYPE_CHECKING
8
 
9
  from sqlalchemy import Column, Float, Integer, MetaData, String, Table, insert, select
10
 
@@ -49,9 +49,10 @@ class GuessTelemetryIn:
49
 
50
 
51
  class GuessTelemetryStore:
52
- def __init__(self, engine: Engine) -> None:
53
  self._engine = engine
54
  self._lock = Lock()
 
55
 
56
  def initialize_schema(self) -> None:
57
  metadata.create_all(self._engine)
@@ -98,8 +99,14 @@ class GuessTelemetryStore:
98
  row_id=rid,
99
  )
100
  )
 
101
  return int(rid)
102
 
 
 
 
 
 
103
 
104
  def create_guess_telemetry_engine(url: str):
105
  from nutonic_server.leaderboard_store import create_leaderboard_engine
@@ -108,13 +115,22 @@ def create_guess_telemetry_engine(url: str):
108
 
109
 
110
  def create_guess_telemetry_store(url: str) -> GuessTelemetryStore | None:
111
- from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file
 
 
112
 
113
  u = url.strip()
114
  if not u or u.lower() == "disabled":
115
  return None
116
  _ensure_parent_dir_for_sqlite_file(u)
 
 
 
 
 
 
 
117
  eng = create_guess_telemetry_engine(u)
118
- st = GuessTelemetryStore(eng)
119
  st.initialize_schema()
120
  return st
 
4
 
5
  from dataclasses import dataclass
6
  from threading import Lock
7
+ from typing import TYPE_CHECKING, Callable
8
 
9
  from sqlalchemy import Column, Float, Integer, MetaData, String, Table, insert, select
10
 
 
49
 
50
 
51
  class GuessTelemetryStore:
52
+ def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
53
  self._engine = engine
54
  self._lock = Lock()
55
+ self._on_write = on_write
56
 
57
  def initialize_schema(self) -> None:
58
  metadata.create_all(self._engine)
 
99
  row_id=rid,
100
  )
101
  )
102
+ self._sync_after_write()
103
  return int(rid)
104
 
105
+ def _sync_after_write(self) -> None:
106
+ if self._on_write is None:
107
+ return
108
+ self._on_write()
109
+
110
 
111
  def create_guess_telemetry_engine(url: str):
112
  from nutonic_server.leaderboard_store import create_leaderboard_engine
 
115
 
116
 
117
  def create_guess_telemetry_store(url: str) -> GuessTelemetryStore | None:
118
+ from nutonic_server.hf_persistence import HfSqliteSync
119
+ from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file, sqlite_file_path_from_url
120
+ from nutonic_server.settings import load_settings
121
 
122
  u = url.strip()
123
  if not u or u.lower() == "disabled":
124
  return None
125
  _ensure_parent_dir_for_sqlite_file(u)
126
+ sync_hook = None
127
+ db_path = sqlite_file_path_from_url(u)
128
+ if db_path is not None:
129
+ hf = HfSqliteSync.from_settings(load_settings())
130
+ if hf is not None:
131
+ hf.bootstrap_sqlite_file(local_path=db_path, logical_name="guess_telemetry")
132
+ sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="guess_telemetry")
133
  eng = create_guess_telemetry_engine(u)
134
+ st = GuessTelemetryStore(eng, on_write=sync_hook)
135
  st.initialize_schema()
136
  return st
src/nutonic_server/hf_persistence.py ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import os
4
+ from dataclasses import dataclass
5
+ from pathlib import Path
6
+ from threading import Lock
7
+
8
+
9
+ @dataclass(frozen=True)
10
+ class HfSqliteSyncConfig:
11
+ enabled: bool
12
+ repo_id: str
13
+ dataset_subdir: str
14
+ startup_pull_mode: str = "if_missing"
15
+ required: bool = False
16
+ token: str | None = None
17
+
18
+
19
+ class HfSqliteSync:
20
+ """Sync local SQLite files to a Hugging Face Dataset repo."""
21
+
22
+ def __init__(self, config: HfSqliteSyncConfig) -> None:
23
+ self._config = config
24
+ self._lock = Lock()
25
+ self._api = None
26
+
27
+ @classmethod
28
+ def from_settings(cls, settings) -> HfSqliteSync | None:
29
+ enabled = bool(getattr(settings, "hf_persistence_enabled", False))
30
+ required = bool(getattr(settings, "hf_persistence_required", False))
31
+ # Lazy-check optional dependency and env wiring only when explicitly enabled.
32
+ if not enabled:
33
+ return None
34
+ repo_id = str(getattr(settings, "hf_persistence_repo_id", "") or "").strip()
35
+ if not repo_id:
36
+ if required:
37
+ raise RuntimeError(
38
+ "HF persistence is required but NUTONIC_HF_PERSISTENCE_REPO_ID/HF_PERSISTENCE_REPO_ID is missing."
39
+ )
40
+ return None
41
+ token = (
42
+ os.environ.get("NUTONIC_HF_PERSISTENCE_TOKEN")
43
+ or os.environ.get("HF_API_WRITE")
44
+ or os.environ.get("HF_TOKEN")
45
+ or os.environ.get("HUGGING_FACE_HUB_TOKEN")
46
+ or None
47
+ )
48
+ if required and not token:
49
+ raise RuntimeError(
50
+ "HF persistence is required but no HF write token is set "
51
+ "(NUTONIC_HF_PERSISTENCE_TOKEN/HF_API_WRITE/HF_TOKEN/HUGGING_FACE_HUB_TOKEN)."
52
+ )
53
+ subdir = str(getattr(settings, "hf_persistence_dataset_subdir", "server-persistence") or "server-persistence").strip(
54
+ "/"
55
+ )
56
+ startup_pull_mode = str(getattr(settings, "hf_persistence_startup_pull_mode", "if_missing") or "if_missing").strip().lower()
57
+ if startup_pull_mode not in ("if_missing", "always"):
58
+ raise RuntimeError(
59
+ "Invalid HF persistence startup pull mode. "
60
+ "Use NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE=if_missing|always."
61
+ )
62
+ return cls(
63
+ HfSqliteSyncConfig(
64
+ enabled=True,
65
+ repo_id=repo_id,
66
+ dataset_subdir=subdir,
67
+ startup_pull_mode=startup_pull_mode,
68
+ required=required,
69
+ token=token,
70
+ )
71
+ )
72
+
73
+ def bootstrap_sqlite_file(self, *, local_path: Path, logical_name: str) -> None:
74
+ """Pull remote DB file at startup based on configured pull mode."""
75
+ if self._config.startup_pull_mode == "if_missing" and local_path.exists():
76
+ return
77
+ try:
78
+ from huggingface_hub import hf_hub_download
79
+ except Exception:
80
+ if self._config.required:
81
+ raise RuntimeError("HF persistence required but huggingface_hub is unavailable.")
82
+ return
83
+ local_path.parent.mkdir(parents=True, exist_ok=True)
84
+ remote_path = self._remote_path(logical_name)
85
+ try:
86
+ hf_hub_download(
87
+ repo_id=self._config.repo_id,
88
+ repo_type="dataset",
89
+ filename=remote_path,
90
+ local_dir=str(local_path.parent),
91
+ local_dir_use_symlinks=False,
92
+ token=self._config.token,
93
+ )
94
+ downloaded = local_path.parent / remote_path
95
+ if downloaded.exists() and downloaded != local_path:
96
+ downloaded.replace(local_path)
97
+ except Exception:
98
+ if self._config.required:
99
+ raise
100
+ return
101
+
102
+ def make_write_sync_hook(self, *, local_path: Path, logical_name: str):
103
+ def _sync() -> None:
104
+ if not local_path.exists():
105
+ return
106
+ try:
107
+ self._upload_file(local_path=local_path, logical_name=logical_name)
108
+ except Exception:
109
+ if self._config.required:
110
+ raise
111
+ return
112
+
113
+ return _sync
114
+
115
+ def _upload_file(self, *, local_path: Path, logical_name: str) -> None:
116
+ with self._lock:
117
+ api = self._get_api()
118
+ api.create_repo(
119
+ repo_id=self._config.repo_id,
120
+ repo_type="dataset",
121
+ private=True,
122
+ exist_ok=True,
123
+ )
124
+ api.upload_file(
125
+ path_or_fileobj=str(local_path),
126
+ path_in_repo=self._remote_path(logical_name),
127
+ repo_id=self._config.repo_id,
128
+ repo_type="dataset",
129
+ )
130
+
131
+ def _remote_path(self, logical_name: str) -> str:
132
+ safe = "".join(ch if ch.isalnum() or ch in ("-", "_", ".") else "_" for ch in logical_name).strip("._")
133
+ safe = safe or "database"
134
+ return f"{self._config.dataset_subdir}/{safe}.sqlite3"
135
+
136
+ def _get_api(self):
137
+ if self._api is not None:
138
+ return self._api
139
+ from huggingface_hub import HfApi
140
+
141
+ self._api = HfApi(token=self._config.token)
142
+ return self._api
src/nutonic_server/leaderboard_store.py CHANGED
@@ -3,7 +3,7 @@ from __future__ import annotations
3
  from dataclasses import dataclass, field
4
  from pathlib import Path
5
  from threading import Lock
6
- from typing import TYPE_CHECKING, Protocol, runtime_checkable
7
 
8
  from sqlalchemy import (
9
  Column,
@@ -163,9 +163,10 @@ def _row_from_db(
163
  class SqliteLeaderboardStore:
164
  """IMP-060: durable community leaderboard rows + POST idempotency."""
165
 
166
- def __init__(self, engine: Engine) -> None:
167
  self._engine = engine
168
  self._lock = Lock()
 
169
 
170
  def initialize_schema(self) -> None:
171
  metadata.create_all(self._engine)
@@ -192,6 +193,7 @@ class SqliteLeaderboardStore:
192
  distance_km=r.distance_km,
193
  )
194
  )
 
195
 
196
  def list_rows(self, map_id: str) -> list[LeaderboardRow]:
197
  with self._lock:
@@ -262,8 +264,14 @@ class SqliteLeaderboardStore:
262
  row_id=row_id,
263
  )
264
  )
 
265
  return row
266
 
 
 
 
 
 
267
 
268
  def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
269
  """Factory: SQLite by default; `memory` (case-insensitive) keeps in-process S0 store."""
@@ -271,7 +279,26 @@ def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
271
  if url.lower() == "memory":
272
  return InMemoryLeaderboardStore()
273
  _ensure_parent_dir_for_sqlite_file(url)
 
 
 
 
 
 
 
 
 
274
  engine = create_leaderboard_engine(url)
275
- store = SqliteLeaderboardStore(engine)
276
  store.initialize_schema()
277
  return store
 
 
 
 
 
 
 
 
 
 
 
3
  from dataclasses import dataclass, field
4
  from pathlib import Path
5
  from threading import Lock
6
+ from typing import TYPE_CHECKING, Callable, Protocol, runtime_checkable
7
 
8
  from sqlalchemy import (
9
  Column,
 
163
  class SqliteLeaderboardStore:
164
  """IMP-060: durable community leaderboard rows + POST idempotency."""
165
 
166
+ def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
167
  self._engine = engine
168
  self._lock = Lock()
169
+ self._on_write = on_write
170
 
171
  def initialize_schema(self) -> None:
172
  metadata.create_all(self._engine)
 
193
  distance_km=r.distance_km,
194
  )
195
  )
196
+ self._sync_after_write()
197
 
198
  def list_rows(self, map_id: str) -> list[LeaderboardRow]:
199
  with self._lock:
 
264
  row_id=row_id,
265
  )
266
  )
267
+ self._sync_after_write()
268
  return row
269
 
270
+ def _sync_after_write(self) -> None:
271
+ if self._on_write is None:
272
+ return
273
+ self._on_write()
274
+
275
 
276
  def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
277
  """Factory: SQLite by default; `memory` (case-insensitive) keeps in-process S0 store."""
 
279
  if url.lower() == "memory":
280
  return InMemoryLeaderboardStore()
281
  _ensure_parent_dir_for_sqlite_file(url)
282
+ sync_hook = None
283
+ db_path = sqlite_file_path_from_url(url)
284
+ if db_path is not None:
285
+ from nutonic_server.hf_persistence import HfSqliteSync
286
+
287
+ hf = HfSqliteSync.from_settings(settings)
288
+ if hf is not None:
289
+ hf.bootstrap_sqlite_file(local_path=db_path, logical_name="leaderboard")
290
+ sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="leaderboard")
291
  engine = create_leaderboard_engine(url)
292
+ store = SqliteLeaderboardStore(engine, on_write=sync_hook)
293
  store.initialize_schema()
294
  return store
295
+
296
+
297
+ def sqlite_file_path_from_url(url: str) -> Path | None:
298
+ u = make_url(url)
299
+ if u.drivername not in ("sqlite", "sqlite+pysqlite"):
300
+ return None
301
+ db = u.database
302
+ if not db or db == ":memory:":
303
+ return None
304
+ return Path(db)
src/nutonic_server/pro_jobs_store.py CHANGED
@@ -6,7 +6,7 @@ from dataclasses import dataclass
6
  from datetime import UTC, datetime, timedelta
7
  from pathlib import Path
8
  from threading import Lock
9
- from typing import Any, Literal, Protocol
10
 
11
  from sqlalchemy import Boolean, Column, Integer, MetaData, String, Table, create_engine, delete, event, insert, select, text, update
12
  from sqlalchemy.engine import Engine, make_url
@@ -107,9 +107,10 @@ class ProJobStore(Protocol):
107
 
108
 
109
  class SqliteProJobStore:
110
- def __init__(self, engine: Engine) -> None:
111
  self._engine = engine
112
  self._lock = Lock()
 
113
 
114
  def initialize_schema(self) -> None:
115
  metadata.create_all(self._engine)
@@ -139,6 +140,7 @@ class SqliteProJobStore:
139
  for name, ddl in required_columns.items():
140
  if name not in existing:
141
  conn.execute(text(f"ALTER TABLE pro_jobs ADD COLUMN {name} {ddl}"))
 
142
 
143
  def create_job(self, *, session_id: str, analysis_profile: str, request_params: dict[str, Any]) -> ProJobRecord:
144
  now = _utc_now()
@@ -163,6 +165,7 @@ class SqliteProJobStore:
163
  progress_pct=record.progress_pct,
164
  )
165
  )
 
166
  return record
167
 
168
  def get_job(self, job_id: str, *, session_id: str | None = None) -> ProJobRecord | None:
@@ -227,6 +230,7 @@ class SqliteProJobStore:
227
  )
228
  if not res.rowcount:
229
  return None
 
230
  return self.get_job(job_id)
231
 
232
  def update_progress(self, job_id: str, *, progress_pct: int) -> None:
@@ -237,6 +241,7 @@ class SqliteProJobStore:
237
  .where(pro_jobs.c.job_id == job_id)
238
  .values(progress_pct=max(0, min(int(progress_pct), 100)))
239
  )
 
240
 
241
  def complete(
242
  self,
@@ -266,6 +271,7 @@ class SqliteProJobStore:
266
  )
267
  if not res.rowcount:
268
  return None
 
269
  return self.get_job(job_id)
270
 
271
  def fail(self, job_id: str, *, error_class: str, error_detail: str | None = None) -> ProJobRecord | None:
@@ -283,6 +289,7 @@ class SqliteProJobStore:
283
  )
284
  if not res.rowcount:
285
  return None
 
286
  return self.get_job(job_id)
287
 
288
  def request_cancel(self, job_id: str, *, session_id: str) -> str:
@@ -303,6 +310,7 @@ class SqliteProJobStore:
303
  .where(pro_jobs.c.job_id == job_id)
304
  .values(status="cancelled", finished_at=_utc_now(), progress_pct=0, error_class="cancelled")
305
  )
 
306
  return "cancelled"
307
  if current == "running":
308
  conn.execute(
@@ -310,6 +318,7 @@ class SqliteProJobStore:
310
  .where(pro_jobs.c.job_id == job_id)
311
  .values(cancel_requested=True)
312
  )
 
313
  return "cancelling"
314
  return current
315
 
@@ -330,12 +339,19 @@ class SqliteProJobStore:
330
  pro_jobs.c.finished_at < cutoff_s,
331
  )
332
  )
 
 
333
  if artifact_root:
334
  root = Path(artifact_root)
335
  for (job_id,) in rows:
336
  _delete_job_artifacts(root / str(job_id))
337
  return len(rows)
338
 
 
 
 
 
 
339
 
340
  def create_pro_job_engine(url: str) -> Engine:
341
  u = make_url(url)
@@ -360,9 +376,9 @@ def create_pro_job_engine(url: str) -> Engine:
360
  return engine
361
 
362
 
363
- def create_pro_job_store(url: str) -> SqliteProJobStore:
364
  _ensure_parent_dir_for_sqlite_file(url)
365
- store = SqliteProJobStore(create_pro_job_engine(url))
366
  store.initialize_schema()
367
  return store
368
 
 
6
  from datetime import UTC, datetime, timedelta
7
  from pathlib import Path
8
  from threading import Lock
9
+ from typing import Any, Callable, Literal, Protocol
10
 
11
  from sqlalchemy import Boolean, Column, Integer, MetaData, String, Table, create_engine, delete, event, insert, select, text, update
12
  from sqlalchemy.engine import Engine, make_url
 
107
 
108
 
109
  class SqliteProJobStore:
110
+ def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
111
  self._engine = engine
112
  self._lock = Lock()
113
+ self._on_write = on_write
114
 
115
  def initialize_schema(self) -> None:
116
  metadata.create_all(self._engine)
 
140
  for name, ddl in required_columns.items():
141
  if name not in existing:
142
  conn.execute(text(f"ALTER TABLE pro_jobs ADD COLUMN {name} {ddl}"))
143
+ self._sync_after_write()
144
 
145
  def create_job(self, *, session_id: str, analysis_profile: str, request_params: dict[str, Any]) -> ProJobRecord:
146
  now = _utc_now()
 
165
  progress_pct=record.progress_pct,
166
  )
167
  )
168
+ self._sync_after_write()
169
  return record
170
 
171
  def get_job(self, job_id: str, *, session_id: str | None = None) -> ProJobRecord | None:
 
230
  )
231
  if not res.rowcount:
232
  return None
233
+ self._sync_after_write()
234
  return self.get_job(job_id)
235
 
236
  def update_progress(self, job_id: str, *, progress_pct: int) -> None:
 
241
  .where(pro_jobs.c.job_id == job_id)
242
  .values(progress_pct=max(0, min(int(progress_pct), 100)))
243
  )
244
+ self._sync_after_write()
245
 
246
  def complete(
247
  self,
 
271
  )
272
  if not res.rowcount:
273
  return None
274
+ self._sync_after_write()
275
  return self.get_job(job_id)
276
 
277
  def fail(self, job_id: str, *, error_class: str, error_detail: str | None = None) -> ProJobRecord | None:
 
289
  )
290
  if not res.rowcount:
291
  return None
292
+ self._sync_after_write()
293
  return self.get_job(job_id)
294
 
295
  def request_cancel(self, job_id: str, *, session_id: str) -> str:
 
310
  .where(pro_jobs.c.job_id == job_id)
311
  .values(status="cancelled", finished_at=_utc_now(), progress_pct=0, error_class="cancelled")
312
  )
313
+ self._sync_after_write()
314
  return "cancelled"
315
  if current == "running":
316
  conn.execute(
 
318
  .where(pro_jobs.c.job_id == job_id)
319
  .values(cancel_requested=True)
320
  )
321
+ self._sync_after_write()
322
  return "cancelling"
323
  return current
324
 
 
339
  pro_jobs.c.finished_at < cutoff_s,
340
  )
341
  )
342
+ if rows:
343
+ self._sync_after_write()
344
  if artifact_root:
345
  root = Path(artifact_root)
346
  for (job_id,) in rows:
347
  _delete_job_artifacts(root / str(job_id))
348
  return len(rows)
349
 
350
+ def _sync_after_write(self) -> None:
351
+ if self._on_write is None:
352
+ return
353
+ self._on_write()
354
+
355
 
356
  def create_pro_job_engine(url: str) -> Engine:
357
  u = make_url(url)
 
376
  return engine
377
 
378
 
379
+ def create_pro_job_store(url: str, *, on_write: Callable[[], None] | None = None) -> SqliteProJobStore:
380
  _ensure_parent_dir_for_sqlite_file(url)
381
+ store = SqliteProJobStore(create_pro_job_engine(url), on_write=on_write)
382
  store.initialize_schema()
383
  return store
384
 
src/nutonic_server/ranked_store.py CHANGED
@@ -6,7 +6,7 @@ import uuid
6
  from dataclasses import dataclass
7
  from datetime import UTC, datetime
8
  from threading import Lock
9
- from typing import TYPE_CHECKING, Literal
10
 
11
  from sqlalchemy import Column, Float, Integer, MetaData, String, Table, create_engine, delete, desc, insert, select, text, update
12
  from sqlalchemy.dialects.sqlite import insert as sqlite_insert
@@ -74,9 +74,10 @@ class RankedVerifiedLbRow:
74
 
75
 
76
  class SqliteRankedStore:
77
- def __init__(self, engine: Engine) -> None:
78
  self._engine = engine
79
  self._lock = Lock()
 
80
 
81
  def initialize_schema(self) -> None:
82
  metadata.create_all(self._engine)
@@ -90,6 +91,7 @@ class SqliteRankedStore:
90
  names = {str(r[1]) for r in rows}
91
  if "opened_at_epoch" not in names:
92
  conn.execute(text("ALTER TABLE ranked_rounds ADD COLUMN opened_at_epoch INTEGER NOT NULL DEFAULT 0"))
 
93
 
94
  def prune_stale_open_rounds(self, *, now_epoch: int, max_age_seconds: int) -> int:
95
  """Delete ``open`` rounds whose ``opened_at_epoch`` is older than ``now_epoch - max_age_seconds``."""
@@ -104,6 +106,8 @@ class SqliteRankedStore:
104
  ranked_rounds.c.opened_at_epoch < cutoff,
105
  )
106
  )
 
 
107
  return int(res.rowcount or 0)
108
 
109
  def create_round(
@@ -131,6 +135,7 @@ class SqliteRankedStore:
131
  opened_at_epoch=opened,
132
  )
133
  )
 
134
  return rid
135
 
136
  def get_round(self, round_id: str) -> RankedRoundRow | None:
@@ -160,6 +165,7 @@ class SqliteRankedStore:
160
  .where(ranked_rounds.c.round_id == round_id)
161
  .values(status="submitted")
162
  )
 
163
 
164
  def forfeit_round(self, round_id: str, session_id: str) -> RankedForfeitResult:
165
  """Mark ``open`` round as ``forfeited`` when ``session_id`` matches (IMP-091)."""
@@ -183,6 +189,7 @@ class SqliteRankedStore:
183
  .where(ranked_rounds.c.round_id == round_id)
184
  .values(status="forfeited")
185
  )
 
186
  return "ok"
187
 
188
  def get_submit_if_exists(self, round_id: str, idempotency_key: str) -> tuple[float, int] | None:
@@ -225,6 +232,7 @@ class SqliteRankedStore:
225
  score_points=score_points,
226
  )
227
  )
 
228
  return distance_km, score_points, True
229
 
230
  def append_verified_leaderboard(
@@ -247,6 +255,7 @@ class SqliteRankedStore:
247
  distance_km=distance_km,
248
  )
249
  conn.execute(stmt.on_conflict_do_nothing(index_elements=["round_id"]))
 
250
 
251
  def list_ranked_leaderboard(self, map_id: str) -> list[RankedVerifiedLbRow]:
252
  """Verified ranked rows for ``map_id``, highest score first."""
@@ -275,6 +284,11 @@ class SqliteRankedStore:
275
  )
276
  return out
277
 
 
 
 
 
 
278
 
279
  def create_ranked_engine(url: str) -> Engine:
280
  from nutonic_server.leaderboard_store import create_leaderboard_engine
@@ -283,10 +297,19 @@ def create_ranked_engine(url: str) -> Engine:
283
 
284
 
285
  def create_ranked_store(url: str) -> SqliteRankedStore:
286
- from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file
 
 
287
 
288
  _ensure_parent_dir_for_sqlite_file(url)
 
 
 
 
 
 
 
289
  eng = create_ranked_engine(url)
290
- st = SqliteRankedStore(eng)
291
  st.initialize_schema()
292
  return st
 
6
  from dataclasses import dataclass
7
  from datetime import UTC, datetime
8
  from threading import Lock
9
+ from typing import TYPE_CHECKING, Callable, Literal
10
 
11
  from sqlalchemy import Column, Float, Integer, MetaData, String, Table, create_engine, delete, desc, insert, select, text, update
12
  from sqlalchemy.dialects.sqlite import insert as sqlite_insert
 
74
 
75
 
76
  class SqliteRankedStore:
77
+ def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
78
  self._engine = engine
79
  self._lock = Lock()
80
+ self._on_write = on_write
81
 
82
  def initialize_schema(self) -> None:
83
  metadata.create_all(self._engine)
 
91
  names = {str(r[1]) for r in rows}
92
  if "opened_at_epoch" not in names:
93
  conn.execute(text("ALTER TABLE ranked_rounds ADD COLUMN opened_at_epoch INTEGER NOT NULL DEFAULT 0"))
94
+ self._sync_after_write()
95
 
96
  def prune_stale_open_rounds(self, *, now_epoch: int, max_age_seconds: int) -> int:
97
  """Delete ``open`` rounds whose ``opened_at_epoch`` is older than ``now_epoch - max_age_seconds``."""
 
106
  ranked_rounds.c.opened_at_epoch < cutoff,
107
  )
108
  )
109
+ if (res.rowcount or 0) > 0:
110
+ self._sync_after_write()
111
  return int(res.rowcount or 0)
112
 
113
  def create_round(
 
135
  opened_at_epoch=opened,
136
  )
137
  )
138
+ self._sync_after_write()
139
  return rid
140
 
141
  def get_round(self, round_id: str) -> RankedRoundRow | None:
 
165
  .where(ranked_rounds.c.round_id == round_id)
166
  .values(status="submitted")
167
  )
168
+ self._sync_after_write()
169
 
170
  def forfeit_round(self, round_id: str, session_id: str) -> RankedForfeitResult:
171
  """Mark ``open`` round as ``forfeited`` when ``session_id`` matches (IMP-091)."""
 
189
  .where(ranked_rounds.c.round_id == round_id)
190
  .values(status="forfeited")
191
  )
192
+ self._sync_after_write()
193
  return "ok"
194
 
195
  def get_submit_if_exists(self, round_id: str, idempotency_key: str) -> tuple[float, int] | None:
 
232
  score_points=score_points,
233
  )
234
  )
235
+ self._sync_after_write()
236
  return distance_km, score_points, True
237
 
238
  def append_verified_leaderboard(
 
255
  distance_km=distance_km,
256
  )
257
  conn.execute(stmt.on_conflict_do_nothing(index_elements=["round_id"]))
258
+ self._sync_after_write()
259
 
260
  def list_ranked_leaderboard(self, map_id: str) -> list[RankedVerifiedLbRow]:
261
  """Verified ranked rows for ``map_id``, highest score first."""
 
284
  )
285
  return out
286
 
287
+ def _sync_after_write(self) -> None:
288
+ if self._on_write is None:
289
+ return
290
+ self._on_write()
291
+
292
 
293
  def create_ranked_engine(url: str) -> Engine:
294
  from nutonic_server.leaderboard_store import create_leaderboard_engine
 
297
 
298
 
299
  def create_ranked_store(url: str) -> SqliteRankedStore:
300
+ from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file, sqlite_file_path_from_url
301
+ from nutonic_server.hf_persistence import HfSqliteSync
302
+ from nutonic_server.settings import load_settings
303
 
304
  _ensure_parent_dir_for_sqlite_file(url)
305
+ sync_hook = None
306
+ db_path = sqlite_file_path_from_url(url)
307
+ if db_path is not None:
308
+ hf = HfSqliteSync.from_settings(load_settings())
309
+ if hf is not None:
310
+ hf.bootstrap_sqlite_file(local_path=db_path, logical_name="ranked")
311
+ sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="ranked")
312
  eng = create_ranked_engine(url)
313
+ st = SqliteRankedStore(eng, on_write=sync_hook)
314
  st.initialize_schema()
315
  return st
src/nutonic_server/settings.py CHANGED
@@ -265,6 +265,42 @@ class Settings(BaseSettings):
265
  ),
266
  )
267
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
268
  @field_validator("cors_origins", mode="before")
269
  @classmethod
270
  def strip_origins(cls, v: object) -> str:
 
265
  ),
266
  )
267
 
268
+ hf_persistence_enabled: bool = Field(
269
+ default=False,
270
+ validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_ENABLED", "HF_PERSISTENCE_ENABLED"),
271
+ description="When true, mirror SQLite server DB files to a Hugging Face Dataset repo.",
272
+ )
273
+
274
+ hf_persistence_required: bool = Field(
275
+ default=False,
276
+ validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_REQUIRED", "HF_PERSISTENCE_REQUIRED"),
277
+ description=(
278
+ "When true with HF persistence enabled, fail fast if the dataset repo/token is missing "
279
+ "or sync operations fail."
280
+ ),
281
+ )
282
+
283
+ hf_persistence_repo_id: str = Field(
284
+ default="",
285
+ validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_REPO_ID", "HF_PERSISTENCE_REPO_ID"),
286
+ description="Dataset repo id (owner/name) used for SQLite persistence mirroring.",
287
+ )
288
+
289
+ hf_persistence_dataset_subdir: str = Field(
290
+ default="server-persistence",
291
+ validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_SUBDIR", "HF_PERSISTENCE_SUBDIR"),
292
+ description="Subdirectory inside the Dataset repo where DB files are stored.",
293
+ )
294
+
295
+ hf_persistence_startup_pull_mode: str = Field(
296
+ default="if_missing",
297
+ validation_alias=AliasChoices(
298
+ "NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE",
299
+ "HF_PERSISTENCE_STARTUP_PULL_MODE",
300
+ ),
301
+ description="Startup pull policy for local SQLite files: if_missing (default) or always.",
302
+ )
303
+
304
  @field_validator("cors_origins", mode="before")
305
  @classmethod
306
  def strip_origins(cls, v: object) -> str: