Spaces:
Sleeping
Sleeping
ci: deploy game_server (a9efbe759359)
Browse files- README.package.md +7 -0
- pyproject.toml +1 -0
- src/nutonic_server/deps.py +10 -1
- src/nutonic_server/guess_telemetry_store.py +20 -4
- src/nutonic_server/hf_persistence.py +142 -0
- src/nutonic_server/leaderboard_store.py +30 -3
- src/nutonic_server/pro_jobs_store.py +20 -4
- src/nutonic_server/ranked_store.py +27 -4
- src/nutonic_server/settings.py +36 -0
README.package.md
CHANGED
|
@@ -33,6 +33,12 @@ Optional: copy **`.env.example`** to **`server/.env`** so local runs pick up **`
|
|
| 33 |
| `FEATURE_GUESSES_RECORD` / `NUTONIC_FEATURE_GUESSES_RECORD` | No | Default **`false`**. When **`true`**, enables **`POST /api/v1/maps/{map_id}/guesses/record`** and sets `features.guesses_record` on **`GET /api/v1/config`**. Use **`true`** in local dev when exercising telemetry (`rules/05`, `docs/GAME-ENGINE.md` §12.3). |
|
| 34 |
| `NUTONIC_GUESS_TELEMETRY_DATABASE_URL` | No | Default `sqlite:///data/nutonic_guess_telemetry.db`. SQLite file for optional guess rows (created under `data/` like the leaderboard DB). |
|
| 35 |
| `NUTONIC_RANKED_DATABASE_URL` | No | Default `sqlite:///data/nutonic_ranked.db`. SQLite for ranked round rows (**IMP-090**). |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
| `NUTONIC_RANKED_STALE_OPEN_ROUND_MAX_AGE_SECONDS` | No | Default **604800** (7d). Abandoned **`open`** ranked rounds older than this are deleted on each **`POST /api/v1/ranked/rounds/start`** (housekeeping). |
|
| 37 |
| `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH` | No | Default **`false`**: **`GET /api/v1/cache/manifest`** omits `locations` and `ai_guesses` (public spoiler hygiene). Set **`true`** for local dev / tests that need full fixture slices. |
|
| 38 |
| `NUTONIC_MANIFEST_FULL_PATH` / `MANIFEST_FULL_PATH` | No | When set to an existing **`manifest.full.json`** (same schema as **`assemble_manifest.py`** output), replaces the builtin demo **`PUBLISHED_MAPS` / `MANIFEST_LOCATIONS` / `MANIFEST_AI_GUESSES`** at process start so **`GET /api/v1/maps`**, **`GET /api/v1/cache/manifest`**, and **`POST /api/v1/ranked/rounds/start`** use the full GeoGuessr-derived catalog. Optional **`satellite_caption_sidecar`** on each location is preserved and echoed on **`RankedClueOut`** when present. |
|
|
@@ -64,6 +70,7 @@ The game server is deployed by **`.github/workflows/huggingface-deploy.yml`**, n
|
|
| 64 |
| Runtime profile | `tools/hf_deploy/profiles/game_server.yaml`: `cpu-basic`, `FEATURE_RANKED=false`, `FEATURE_COMMUNITY_LB_POST=false`, `FEATURE_PRO_JOBS=false`, `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH=false`. |
|
| 65 |
| Hub auth | `HF_TOKEN_NUTONIC`, then `HF_TOKEN`, then `HF_API_WRITE`. |
|
| 66 |
| Runtime secrets | `NUTONIC_JWT_SECRET` is pushed to the Space as `JWT_SECRET`; `NUTONIC_LEADERBOARD_DATABASE_URL` and `NUTONIC_RANKED_DATABASE_URL` are pushed under the same names when set. |
|
|
|
|
| 67 |
| Post-deploy smoke | `python tools/live_inference_smoke.py --preset game-deploy --timeout 30 --strict --json-report-path artifacts/hf-smoke-game.json`. |
|
| 68 |
|
| 69 |
The deployed Space defaults are intentionally conservative: ranked, community writes, PRO jobs, and manifest truth are off until the Space profile or Space settings opt in. Enabling live PRO on the game server also requires worker URLs and the same `NUTONIC_INFERENCE_HMAC_SECRET` used by any HMAC-protected worker.
|
|
|
|
| 33 |
| `FEATURE_GUESSES_RECORD` / `NUTONIC_FEATURE_GUESSES_RECORD` | No | Default **`false`**. When **`true`**, enables **`POST /api/v1/maps/{map_id}/guesses/record`** and sets `features.guesses_record` on **`GET /api/v1/config`**. Use **`true`** in local dev when exercising telemetry (`rules/05`, `docs/GAME-ENGINE.md` §12.3). |
|
| 34 |
| `NUTONIC_GUESS_TELEMETRY_DATABASE_URL` | No | Default `sqlite:///data/nutonic_guess_telemetry.db`. SQLite file for optional guess rows (created under `data/` like the leaderboard DB). |
|
| 35 |
| `NUTONIC_RANKED_DATABASE_URL` | No | Default `sqlite:///data/nutonic_ranked.db`. SQLite for ranked round rows (**IMP-090**). |
|
| 36 |
+
| `NUTONIC_HF_PERSISTENCE_ENABLED` / `HF_PERSISTENCE_ENABLED` | No | Default **`false`**. When **`true`**, SQLite stores bootstrap from and sync writes to a Hugging Face Dataset repo. |
|
| 37 |
+
| `NUTONIC_HF_PERSISTENCE_REQUIRED` / `HF_PERSISTENCE_REQUIRED` | No | Default **`false`**. When **`true`** with HF persistence enabled, startup fails if repo id/token is missing and runtime fails on sync errors (fail-fast mode). |
|
| 38 |
+
| `NUTONIC_HF_PERSISTENCE_REPO_ID` / `HF_PERSISTENCE_REPO_ID` | No | Dataset repo id (`owner/name`) used for persisted SQLite files. |
|
| 39 |
+
| `NUTONIC_HF_PERSISTENCE_SUBDIR` / `HF_PERSISTENCE_SUBDIR` | No | Default `server-persistence`. Subdirectory inside the dataset repo for DB files. |
|
| 40 |
+
| `NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE` / `HF_PERSISTENCE_STARTUP_PULL_MODE` | No | Default `if_missing`. `if_missing` only pulls when local DB is absent; `always` always pulls from dataset at startup (remote wins). |
|
| 41 |
+
| `HF_API_WRITE` / `HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN` | No | Hub token used by HF persistence sync client for dataset read/write calls. Prefer a dedicated write token in production. |
|
| 42 |
| `NUTONIC_RANKED_STALE_OPEN_ROUND_MAX_AGE_SECONDS` | No | Default **604800** (7d). Abandoned **`open`** ranked rounds older than this are deleted on each **`POST /api/v1/ranked/rounds/start`** (housekeeping). |
|
| 43 |
| `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH` | No | Default **`false`**: **`GET /api/v1/cache/manifest`** omits `locations` and `ai_guesses` (public spoiler hygiene). Set **`true`** for local dev / tests that need full fixture slices. |
|
| 44 |
| `NUTONIC_MANIFEST_FULL_PATH` / `MANIFEST_FULL_PATH` | No | When set to an existing **`manifest.full.json`** (same schema as **`assemble_manifest.py`** output), replaces the builtin demo **`PUBLISHED_MAPS` / `MANIFEST_LOCATIONS` / `MANIFEST_AI_GUESSES`** at process start so **`GET /api/v1/maps`**, **`GET /api/v1/cache/manifest`**, and **`POST /api/v1/ranked/rounds/start`** use the full GeoGuessr-derived catalog. Optional **`satellite_caption_sidecar`** on each location is preserved and echoed on **`RankedClueOut`** when present. |
|
|
|
|
| 70 |
| Runtime profile | `tools/hf_deploy/profiles/game_server.yaml`: `cpu-basic`, `FEATURE_RANKED=false`, `FEATURE_COMMUNITY_LB_POST=false`, `FEATURE_PRO_JOBS=false`, `NUTONIC_EXPOSE_MANIFEST_ROUND_TRUTH=false`. |
|
| 71 |
| Hub auth | `HF_TOKEN_NUTONIC`, then `HF_TOKEN`, then `HF_API_WRITE`. |
|
| 72 |
| Runtime secrets | `NUTONIC_JWT_SECRET` is pushed to the Space as `JWT_SECRET`; `NUTONIC_LEADERBOARD_DATABASE_URL` and `NUTONIC_RANKED_DATABASE_URL` are pushed under the same names when set. |
|
| 73 |
+
| HF dataset persistence wiring | `NUTONIC_HF_PERSISTENCE_ENABLED`, `NUTONIC_HF_PERSISTENCE_REQUIRED`, `NUTONIC_HF_PERSISTENCE_REPO_ID`, `NUTONIC_HF_PERSISTENCE_SUBDIR`, and `NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE` are forwarded as runtime variables; `NUTONIC_HF_PERSISTENCE_TOKEN` is pushed as `HF_API_WRITE` secret for dataset writes. |
|
| 74 |
| Post-deploy smoke | `python tools/live_inference_smoke.py --preset game-deploy --timeout 30 --strict --json-report-path artifacts/hf-smoke-game.json`. |
|
| 75 |
|
| 76 |
The deployed Space defaults are intentionally conservative: ranked, community writes, PRO jobs, and manifest truth are off until the Space profile or Space settings opt in. Enabling live PRO on the game server also requires worker URLs and the same `NUTONIC_INFERENCE_HMAC_SECRET` used by any HMAC-protected worker.
|
pyproject.toml
CHANGED
|
@@ -16,6 +16,7 @@ dependencies = [
|
|
| 16 |
"httpx>=0.27.0",
|
| 17 |
"PyJWT>=2.9.0",
|
| 18 |
"sqlalchemy>=2.0.36",
|
|
|
|
| 19 |
]
|
| 20 |
|
| 21 |
[project.optional-dependencies]
|
|
|
|
| 16 |
"httpx>=0.27.0",
|
| 17 |
"PyJWT>=2.9.0",
|
| 18 |
"sqlalchemy>=2.0.36",
|
| 19 |
+
"huggingface_hub>=0.35.0",
|
| 20 |
]
|
| 21 |
|
| 22 |
[project.optional-dependencies]
|
src/nutonic_server/deps.py
CHANGED
|
@@ -7,9 +7,11 @@ from fastapi import Depends, HTTPException, status
|
|
| 7 |
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
| 8 |
|
| 9 |
from nutonic_server.jwt_tokens import decode_bearer_token
|
|
|
|
| 10 |
from nutonic_server.pro_jobs_runner import ProJobRunner
|
| 11 |
from nutonic_server.pro_jobs_store import ProJobStore, create_pro_job_store
|
| 12 |
from nutonic_server.settings import Settings, load_settings
|
|
|
|
| 13 |
|
| 14 |
_bearer = HTTPBearer(auto_error=False)
|
| 15 |
_pro_job_stores: dict[str, ProJobStore] = {}
|
|
@@ -31,7 +33,14 @@ def get_pro_job_store_for_settings(settings: Settings) -> ProJobStore:
|
|
| 31 |
raise HTTPException(status_code=500, detail="Unsupported PRO job backend")
|
| 32 |
url = settings.pro_job_database_url.strip()
|
| 33 |
if url not in _pro_job_stores:
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
return _pro_job_stores[url]
|
| 36 |
|
| 37 |
|
|
|
|
| 7 |
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
| 8 |
|
| 9 |
from nutonic_server.jwt_tokens import decode_bearer_token
|
| 10 |
+
from nutonic_server.hf_persistence import HfSqliteSync
|
| 11 |
from nutonic_server.pro_jobs_runner import ProJobRunner
|
| 12 |
from nutonic_server.pro_jobs_store import ProJobStore, create_pro_job_store
|
| 13 |
from nutonic_server.settings import Settings, load_settings
|
| 14 |
+
from nutonic_server.leaderboard_store import sqlite_file_path_from_url
|
| 15 |
|
| 16 |
_bearer = HTTPBearer(auto_error=False)
|
| 17 |
_pro_job_stores: dict[str, ProJobStore] = {}
|
|
|
|
| 33 |
raise HTTPException(status_code=500, detail="Unsupported PRO job backend")
|
| 34 |
url = settings.pro_job_database_url.strip()
|
| 35 |
if url not in _pro_job_stores:
|
| 36 |
+
sync_hook = None
|
| 37 |
+
db_path = sqlite_file_path_from_url(url)
|
| 38 |
+
if db_path is not None:
|
| 39 |
+
hf = HfSqliteSync.from_settings(settings)
|
| 40 |
+
if hf is not None:
|
| 41 |
+
hf.bootstrap_sqlite_file(local_path=db_path, logical_name="pro_jobs")
|
| 42 |
+
sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="pro_jobs")
|
| 43 |
+
_pro_job_stores[url] = create_pro_job_store(url, on_write=sync_hook)
|
| 44 |
return _pro_job_stores[url]
|
| 45 |
|
| 46 |
|
src/nutonic_server/guess_telemetry_store.py
CHANGED
|
@@ -4,7 +4,7 @@ from __future__ import annotations
|
|
| 4 |
|
| 5 |
from dataclasses import dataclass
|
| 6 |
from threading import Lock
|
| 7 |
-
from typing import TYPE_CHECKING
|
| 8 |
|
| 9 |
from sqlalchemy import Column, Float, Integer, MetaData, String, Table, insert, select
|
| 10 |
|
|
@@ -49,9 +49,10 @@ class GuessTelemetryIn:
|
|
| 49 |
|
| 50 |
|
| 51 |
class GuessTelemetryStore:
|
| 52 |
-
def __init__(self, engine: Engine) -> None:
|
| 53 |
self._engine = engine
|
| 54 |
self._lock = Lock()
|
|
|
|
| 55 |
|
| 56 |
def initialize_schema(self) -> None:
|
| 57 |
metadata.create_all(self._engine)
|
|
@@ -98,8 +99,14 @@ class GuessTelemetryStore:
|
|
| 98 |
row_id=rid,
|
| 99 |
)
|
| 100 |
)
|
|
|
|
| 101 |
return int(rid)
|
| 102 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
def create_guess_telemetry_engine(url: str):
|
| 105 |
from nutonic_server.leaderboard_store import create_leaderboard_engine
|
|
@@ -108,13 +115,22 @@ def create_guess_telemetry_engine(url: str):
|
|
| 108 |
|
| 109 |
|
| 110 |
def create_guess_telemetry_store(url: str) -> GuessTelemetryStore | None:
|
| 111 |
-
from nutonic_server.
|
|
|
|
|
|
|
| 112 |
|
| 113 |
u = url.strip()
|
| 114 |
if not u or u.lower() == "disabled":
|
| 115 |
return None
|
| 116 |
_ensure_parent_dir_for_sqlite_file(u)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
eng = create_guess_telemetry_engine(u)
|
| 118 |
-
st = GuessTelemetryStore(eng)
|
| 119 |
st.initialize_schema()
|
| 120 |
return st
|
|
|
|
| 4 |
|
| 5 |
from dataclasses import dataclass
|
| 6 |
from threading import Lock
|
| 7 |
+
from typing import TYPE_CHECKING, Callable
|
| 8 |
|
| 9 |
from sqlalchemy import Column, Float, Integer, MetaData, String, Table, insert, select
|
| 10 |
|
|
|
|
| 49 |
|
| 50 |
|
| 51 |
class GuessTelemetryStore:
|
| 52 |
+
def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
|
| 53 |
self._engine = engine
|
| 54 |
self._lock = Lock()
|
| 55 |
+
self._on_write = on_write
|
| 56 |
|
| 57 |
def initialize_schema(self) -> None:
|
| 58 |
metadata.create_all(self._engine)
|
|
|
|
| 99 |
row_id=rid,
|
| 100 |
)
|
| 101 |
)
|
| 102 |
+
self._sync_after_write()
|
| 103 |
return int(rid)
|
| 104 |
|
| 105 |
+
def _sync_after_write(self) -> None:
|
| 106 |
+
if self._on_write is None:
|
| 107 |
+
return
|
| 108 |
+
self._on_write()
|
| 109 |
+
|
| 110 |
|
| 111 |
def create_guess_telemetry_engine(url: str):
|
| 112 |
from nutonic_server.leaderboard_store import create_leaderboard_engine
|
|
|
|
| 115 |
|
| 116 |
|
| 117 |
def create_guess_telemetry_store(url: str) -> GuessTelemetryStore | None:
|
| 118 |
+
from nutonic_server.hf_persistence import HfSqliteSync
|
| 119 |
+
from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file, sqlite_file_path_from_url
|
| 120 |
+
from nutonic_server.settings import load_settings
|
| 121 |
|
| 122 |
u = url.strip()
|
| 123 |
if not u or u.lower() == "disabled":
|
| 124 |
return None
|
| 125 |
_ensure_parent_dir_for_sqlite_file(u)
|
| 126 |
+
sync_hook = None
|
| 127 |
+
db_path = sqlite_file_path_from_url(u)
|
| 128 |
+
if db_path is not None:
|
| 129 |
+
hf = HfSqliteSync.from_settings(load_settings())
|
| 130 |
+
if hf is not None:
|
| 131 |
+
hf.bootstrap_sqlite_file(local_path=db_path, logical_name="guess_telemetry")
|
| 132 |
+
sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="guess_telemetry")
|
| 133 |
eng = create_guess_telemetry_engine(u)
|
| 134 |
+
st = GuessTelemetryStore(eng, on_write=sync_hook)
|
| 135 |
st.initialize_schema()
|
| 136 |
return st
|
src/nutonic_server/hf_persistence.py
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import os
|
| 4 |
+
from dataclasses import dataclass
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
from threading import Lock
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
@dataclass(frozen=True)
|
| 10 |
+
class HfSqliteSyncConfig:
|
| 11 |
+
enabled: bool
|
| 12 |
+
repo_id: str
|
| 13 |
+
dataset_subdir: str
|
| 14 |
+
startup_pull_mode: str = "if_missing"
|
| 15 |
+
required: bool = False
|
| 16 |
+
token: str | None = None
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
class HfSqliteSync:
|
| 20 |
+
"""Sync local SQLite files to a Hugging Face Dataset repo."""
|
| 21 |
+
|
| 22 |
+
def __init__(self, config: HfSqliteSyncConfig) -> None:
|
| 23 |
+
self._config = config
|
| 24 |
+
self._lock = Lock()
|
| 25 |
+
self._api = None
|
| 26 |
+
|
| 27 |
+
@classmethod
|
| 28 |
+
def from_settings(cls, settings) -> HfSqliteSync | None:
|
| 29 |
+
enabled = bool(getattr(settings, "hf_persistence_enabled", False))
|
| 30 |
+
required = bool(getattr(settings, "hf_persistence_required", False))
|
| 31 |
+
# Lazy-check optional dependency and env wiring only when explicitly enabled.
|
| 32 |
+
if not enabled:
|
| 33 |
+
return None
|
| 34 |
+
repo_id = str(getattr(settings, "hf_persistence_repo_id", "") or "").strip()
|
| 35 |
+
if not repo_id:
|
| 36 |
+
if required:
|
| 37 |
+
raise RuntimeError(
|
| 38 |
+
"HF persistence is required but NUTONIC_HF_PERSISTENCE_REPO_ID/HF_PERSISTENCE_REPO_ID is missing."
|
| 39 |
+
)
|
| 40 |
+
return None
|
| 41 |
+
token = (
|
| 42 |
+
os.environ.get("NUTONIC_HF_PERSISTENCE_TOKEN")
|
| 43 |
+
or os.environ.get("HF_API_WRITE")
|
| 44 |
+
or os.environ.get("HF_TOKEN")
|
| 45 |
+
or os.environ.get("HUGGING_FACE_HUB_TOKEN")
|
| 46 |
+
or None
|
| 47 |
+
)
|
| 48 |
+
if required and not token:
|
| 49 |
+
raise RuntimeError(
|
| 50 |
+
"HF persistence is required but no HF write token is set "
|
| 51 |
+
"(NUTONIC_HF_PERSISTENCE_TOKEN/HF_API_WRITE/HF_TOKEN/HUGGING_FACE_HUB_TOKEN)."
|
| 52 |
+
)
|
| 53 |
+
subdir = str(getattr(settings, "hf_persistence_dataset_subdir", "server-persistence") or "server-persistence").strip(
|
| 54 |
+
"/"
|
| 55 |
+
)
|
| 56 |
+
startup_pull_mode = str(getattr(settings, "hf_persistence_startup_pull_mode", "if_missing") or "if_missing").strip().lower()
|
| 57 |
+
if startup_pull_mode not in ("if_missing", "always"):
|
| 58 |
+
raise RuntimeError(
|
| 59 |
+
"Invalid HF persistence startup pull mode. "
|
| 60 |
+
"Use NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE=if_missing|always."
|
| 61 |
+
)
|
| 62 |
+
return cls(
|
| 63 |
+
HfSqliteSyncConfig(
|
| 64 |
+
enabled=True,
|
| 65 |
+
repo_id=repo_id,
|
| 66 |
+
dataset_subdir=subdir,
|
| 67 |
+
startup_pull_mode=startup_pull_mode,
|
| 68 |
+
required=required,
|
| 69 |
+
token=token,
|
| 70 |
+
)
|
| 71 |
+
)
|
| 72 |
+
|
| 73 |
+
def bootstrap_sqlite_file(self, *, local_path: Path, logical_name: str) -> None:
|
| 74 |
+
"""Pull remote DB file at startup based on configured pull mode."""
|
| 75 |
+
if self._config.startup_pull_mode == "if_missing" and local_path.exists():
|
| 76 |
+
return
|
| 77 |
+
try:
|
| 78 |
+
from huggingface_hub import hf_hub_download
|
| 79 |
+
except Exception:
|
| 80 |
+
if self._config.required:
|
| 81 |
+
raise RuntimeError("HF persistence required but huggingface_hub is unavailable.")
|
| 82 |
+
return
|
| 83 |
+
local_path.parent.mkdir(parents=True, exist_ok=True)
|
| 84 |
+
remote_path = self._remote_path(logical_name)
|
| 85 |
+
try:
|
| 86 |
+
hf_hub_download(
|
| 87 |
+
repo_id=self._config.repo_id,
|
| 88 |
+
repo_type="dataset",
|
| 89 |
+
filename=remote_path,
|
| 90 |
+
local_dir=str(local_path.parent),
|
| 91 |
+
local_dir_use_symlinks=False,
|
| 92 |
+
token=self._config.token,
|
| 93 |
+
)
|
| 94 |
+
downloaded = local_path.parent / remote_path
|
| 95 |
+
if downloaded.exists() and downloaded != local_path:
|
| 96 |
+
downloaded.replace(local_path)
|
| 97 |
+
except Exception:
|
| 98 |
+
if self._config.required:
|
| 99 |
+
raise
|
| 100 |
+
return
|
| 101 |
+
|
| 102 |
+
def make_write_sync_hook(self, *, local_path: Path, logical_name: str):
|
| 103 |
+
def _sync() -> None:
|
| 104 |
+
if not local_path.exists():
|
| 105 |
+
return
|
| 106 |
+
try:
|
| 107 |
+
self._upload_file(local_path=local_path, logical_name=logical_name)
|
| 108 |
+
except Exception:
|
| 109 |
+
if self._config.required:
|
| 110 |
+
raise
|
| 111 |
+
return
|
| 112 |
+
|
| 113 |
+
return _sync
|
| 114 |
+
|
| 115 |
+
def _upload_file(self, *, local_path: Path, logical_name: str) -> None:
|
| 116 |
+
with self._lock:
|
| 117 |
+
api = self._get_api()
|
| 118 |
+
api.create_repo(
|
| 119 |
+
repo_id=self._config.repo_id,
|
| 120 |
+
repo_type="dataset",
|
| 121 |
+
private=True,
|
| 122 |
+
exist_ok=True,
|
| 123 |
+
)
|
| 124 |
+
api.upload_file(
|
| 125 |
+
path_or_fileobj=str(local_path),
|
| 126 |
+
path_in_repo=self._remote_path(logical_name),
|
| 127 |
+
repo_id=self._config.repo_id,
|
| 128 |
+
repo_type="dataset",
|
| 129 |
+
)
|
| 130 |
+
|
| 131 |
+
def _remote_path(self, logical_name: str) -> str:
|
| 132 |
+
safe = "".join(ch if ch.isalnum() or ch in ("-", "_", ".") else "_" for ch in logical_name).strip("._")
|
| 133 |
+
safe = safe or "database"
|
| 134 |
+
return f"{self._config.dataset_subdir}/{safe}.sqlite3"
|
| 135 |
+
|
| 136 |
+
def _get_api(self):
|
| 137 |
+
if self._api is not None:
|
| 138 |
+
return self._api
|
| 139 |
+
from huggingface_hub import HfApi
|
| 140 |
+
|
| 141 |
+
self._api = HfApi(token=self._config.token)
|
| 142 |
+
return self._api
|
src/nutonic_server/leaderboard_store.py
CHANGED
|
@@ -3,7 +3,7 @@ from __future__ import annotations
|
|
| 3 |
from dataclasses import dataclass, field
|
| 4 |
from pathlib import Path
|
| 5 |
from threading import Lock
|
| 6 |
-
from typing import TYPE_CHECKING, Protocol, runtime_checkable
|
| 7 |
|
| 8 |
from sqlalchemy import (
|
| 9 |
Column,
|
|
@@ -163,9 +163,10 @@ def _row_from_db(
|
|
| 163 |
class SqliteLeaderboardStore:
|
| 164 |
"""IMP-060: durable community leaderboard rows + POST idempotency."""
|
| 165 |
|
| 166 |
-
def __init__(self, engine: Engine) -> None:
|
| 167 |
self._engine = engine
|
| 168 |
self._lock = Lock()
|
|
|
|
| 169 |
|
| 170 |
def initialize_schema(self) -> None:
|
| 171 |
metadata.create_all(self._engine)
|
|
@@ -192,6 +193,7 @@ class SqliteLeaderboardStore:
|
|
| 192 |
distance_km=r.distance_km,
|
| 193 |
)
|
| 194 |
)
|
|
|
|
| 195 |
|
| 196 |
def list_rows(self, map_id: str) -> list[LeaderboardRow]:
|
| 197 |
with self._lock:
|
|
@@ -262,8 +264,14 @@ class SqliteLeaderboardStore:
|
|
| 262 |
row_id=row_id,
|
| 263 |
)
|
| 264 |
)
|
|
|
|
| 265 |
return row
|
| 266 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
|
| 268 |
def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
|
| 269 |
"""Factory: SQLite by default; `memory` (case-insensitive) keeps in-process S0 store."""
|
|
@@ -271,7 +279,26 @@ def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
|
|
| 271 |
if url.lower() == "memory":
|
| 272 |
return InMemoryLeaderboardStore()
|
| 273 |
_ensure_parent_dir_for_sqlite_file(url)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 274 |
engine = create_leaderboard_engine(url)
|
| 275 |
-
store = SqliteLeaderboardStore(engine)
|
| 276 |
store.initialize_schema()
|
| 277 |
return store
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
from dataclasses import dataclass, field
|
| 4 |
from pathlib import Path
|
| 5 |
from threading import Lock
|
| 6 |
+
from typing import TYPE_CHECKING, Callable, Protocol, runtime_checkable
|
| 7 |
|
| 8 |
from sqlalchemy import (
|
| 9 |
Column,
|
|
|
|
| 163 |
class SqliteLeaderboardStore:
|
| 164 |
"""IMP-060: durable community leaderboard rows + POST idempotency."""
|
| 165 |
|
| 166 |
+
def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
|
| 167 |
self._engine = engine
|
| 168 |
self._lock = Lock()
|
| 169 |
+
self._on_write = on_write
|
| 170 |
|
| 171 |
def initialize_schema(self) -> None:
|
| 172 |
metadata.create_all(self._engine)
|
|
|
|
| 193 |
distance_km=r.distance_km,
|
| 194 |
)
|
| 195 |
)
|
| 196 |
+
self._sync_after_write()
|
| 197 |
|
| 198 |
def list_rows(self, map_id: str) -> list[LeaderboardRow]:
|
| 199 |
with self._lock:
|
|
|
|
| 264 |
row_id=row_id,
|
| 265 |
)
|
| 266 |
)
|
| 267 |
+
self._sync_after_write()
|
| 268 |
return row
|
| 269 |
|
| 270 |
+
def _sync_after_write(self) -> None:
|
| 271 |
+
if self._on_write is None:
|
| 272 |
+
return
|
| 273 |
+
self._on_write()
|
| 274 |
+
|
| 275 |
|
| 276 |
def create_leaderboard_store(settings: Settings) -> LeaderboardStore:
|
| 277 |
"""Factory: SQLite by default; `memory` (case-insensitive) keeps in-process S0 store."""
|
|
|
|
| 279 |
if url.lower() == "memory":
|
| 280 |
return InMemoryLeaderboardStore()
|
| 281 |
_ensure_parent_dir_for_sqlite_file(url)
|
| 282 |
+
sync_hook = None
|
| 283 |
+
db_path = sqlite_file_path_from_url(url)
|
| 284 |
+
if db_path is not None:
|
| 285 |
+
from nutonic_server.hf_persistence import HfSqliteSync
|
| 286 |
+
|
| 287 |
+
hf = HfSqliteSync.from_settings(settings)
|
| 288 |
+
if hf is not None:
|
| 289 |
+
hf.bootstrap_sqlite_file(local_path=db_path, logical_name="leaderboard")
|
| 290 |
+
sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="leaderboard")
|
| 291 |
engine = create_leaderboard_engine(url)
|
| 292 |
+
store = SqliteLeaderboardStore(engine, on_write=sync_hook)
|
| 293 |
store.initialize_schema()
|
| 294 |
return store
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
def sqlite_file_path_from_url(url: str) -> Path | None:
|
| 298 |
+
u = make_url(url)
|
| 299 |
+
if u.drivername not in ("sqlite", "sqlite+pysqlite"):
|
| 300 |
+
return None
|
| 301 |
+
db = u.database
|
| 302 |
+
if not db or db == ":memory:":
|
| 303 |
+
return None
|
| 304 |
+
return Path(db)
|
src/nutonic_server/pro_jobs_store.py
CHANGED
|
@@ -6,7 +6,7 @@ from dataclasses import dataclass
|
|
| 6 |
from datetime import UTC, datetime, timedelta
|
| 7 |
from pathlib import Path
|
| 8 |
from threading import Lock
|
| 9 |
-
from typing import Any, Literal, Protocol
|
| 10 |
|
| 11 |
from sqlalchemy import Boolean, Column, Integer, MetaData, String, Table, create_engine, delete, event, insert, select, text, update
|
| 12 |
from sqlalchemy.engine import Engine, make_url
|
|
@@ -107,9 +107,10 @@ class ProJobStore(Protocol):
|
|
| 107 |
|
| 108 |
|
| 109 |
class SqliteProJobStore:
|
| 110 |
-
def __init__(self, engine: Engine) -> None:
|
| 111 |
self._engine = engine
|
| 112 |
self._lock = Lock()
|
|
|
|
| 113 |
|
| 114 |
def initialize_schema(self) -> None:
|
| 115 |
metadata.create_all(self._engine)
|
|
@@ -139,6 +140,7 @@ class SqliteProJobStore:
|
|
| 139 |
for name, ddl in required_columns.items():
|
| 140 |
if name not in existing:
|
| 141 |
conn.execute(text(f"ALTER TABLE pro_jobs ADD COLUMN {name} {ddl}"))
|
|
|
|
| 142 |
|
| 143 |
def create_job(self, *, session_id: str, analysis_profile: str, request_params: dict[str, Any]) -> ProJobRecord:
|
| 144 |
now = _utc_now()
|
|
@@ -163,6 +165,7 @@ class SqliteProJobStore:
|
|
| 163 |
progress_pct=record.progress_pct,
|
| 164 |
)
|
| 165 |
)
|
|
|
|
| 166 |
return record
|
| 167 |
|
| 168 |
def get_job(self, job_id: str, *, session_id: str | None = None) -> ProJobRecord | None:
|
|
@@ -227,6 +230,7 @@ class SqliteProJobStore:
|
|
| 227 |
)
|
| 228 |
if not res.rowcount:
|
| 229 |
return None
|
|
|
|
| 230 |
return self.get_job(job_id)
|
| 231 |
|
| 232 |
def update_progress(self, job_id: str, *, progress_pct: int) -> None:
|
|
@@ -237,6 +241,7 @@ class SqliteProJobStore:
|
|
| 237 |
.where(pro_jobs.c.job_id == job_id)
|
| 238 |
.values(progress_pct=max(0, min(int(progress_pct), 100)))
|
| 239 |
)
|
|
|
|
| 240 |
|
| 241 |
def complete(
|
| 242 |
self,
|
|
@@ -266,6 +271,7 @@ class SqliteProJobStore:
|
|
| 266 |
)
|
| 267 |
if not res.rowcount:
|
| 268 |
return None
|
|
|
|
| 269 |
return self.get_job(job_id)
|
| 270 |
|
| 271 |
def fail(self, job_id: str, *, error_class: str, error_detail: str | None = None) -> ProJobRecord | None:
|
|
@@ -283,6 +289,7 @@ class SqliteProJobStore:
|
|
| 283 |
)
|
| 284 |
if not res.rowcount:
|
| 285 |
return None
|
|
|
|
| 286 |
return self.get_job(job_id)
|
| 287 |
|
| 288 |
def request_cancel(self, job_id: str, *, session_id: str) -> str:
|
|
@@ -303,6 +310,7 @@ class SqliteProJobStore:
|
|
| 303 |
.where(pro_jobs.c.job_id == job_id)
|
| 304 |
.values(status="cancelled", finished_at=_utc_now(), progress_pct=0, error_class="cancelled")
|
| 305 |
)
|
|
|
|
| 306 |
return "cancelled"
|
| 307 |
if current == "running":
|
| 308 |
conn.execute(
|
|
@@ -310,6 +318,7 @@ class SqliteProJobStore:
|
|
| 310 |
.where(pro_jobs.c.job_id == job_id)
|
| 311 |
.values(cancel_requested=True)
|
| 312 |
)
|
|
|
|
| 313 |
return "cancelling"
|
| 314 |
return current
|
| 315 |
|
|
@@ -330,12 +339,19 @@ class SqliteProJobStore:
|
|
| 330 |
pro_jobs.c.finished_at < cutoff_s,
|
| 331 |
)
|
| 332 |
)
|
|
|
|
|
|
|
| 333 |
if artifact_root:
|
| 334 |
root = Path(artifact_root)
|
| 335 |
for (job_id,) in rows:
|
| 336 |
_delete_job_artifacts(root / str(job_id))
|
| 337 |
return len(rows)
|
| 338 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 339 |
|
| 340 |
def create_pro_job_engine(url: str) -> Engine:
|
| 341 |
u = make_url(url)
|
|
@@ -360,9 +376,9 @@ def create_pro_job_engine(url: str) -> Engine:
|
|
| 360 |
return engine
|
| 361 |
|
| 362 |
|
| 363 |
-
def create_pro_job_store(url: str) -> SqliteProJobStore:
|
| 364 |
_ensure_parent_dir_for_sqlite_file(url)
|
| 365 |
-
store = SqliteProJobStore(create_pro_job_engine(url))
|
| 366 |
store.initialize_schema()
|
| 367 |
return store
|
| 368 |
|
|
|
|
| 6 |
from datetime import UTC, datetime, timedelta
|
| 7 |
from pathlib import Path
|
| 8 |
from threading import Lock
|
| 9 |
+
from typing import Any, Callable, Literal, Protocol
|
| 10 |
|
| 11 |
from sqlalchemy import Boolean, Column, Integer, MetaData, String, Table, create_engine, delete, event, insert, select, text, update
|
| 12 |
from sqlalchemy.engine import Engine, make_url
|
|
|
|
| 107 |
|
| 108 |
|
| 109 |
class SqliteProJobStore:
|
| 110 |
+
def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
|
| 111 |
self._engine = engine
|
| 112 |
self._lock = Lock()
|
| 113 |
+
self._on_write = on_write
|
| 114 |
|
| 115 |
def initialize_schema(self) -> None:
|
| 116 |
metadata.create_all(self._engine)
|
|
|
|
| 140 |
for name, ddl in required_columns.items():
|
| 141 |
if name not in existing:
|
| 142 |
conn.execute(text(f"ALTER TABLE pro_jobs ADD COLUMN {name} {ddl}"))
|
| 143 |
+
self._sync_after_write()
|
| 144 |
|
| 145 |
def create_job(self, *, session_id: str, analysis_profile: str, request_params: dict[str, Any]) -> ProJobRecord:
|
| 146 |
now = _utc_now()
|
|
|
|
| 165 |
progress_pct=record.progress_pct,
|
| 166 |
)
|
| 167 |
)
|
| 168 |
+
self._sync_after_write()
|
| 169 |
return record
|
| 170 |
|
| 171 |
def get_job(self, job_id: str, *, session_id: str | None = None) -> ProJobRecord | None:
|
|
|
|
| 230 |
)
|
| 231 |
if not res.rowcount:
|
| 232 |
return None
|
| 233 |
+
self._sync_after_write()
|
| 234 |
return self.get_job(job_id)
|
| 235 |
|
| 236 |
def update_progress(self, job_id: str, *, progress_pct: int) -> None:
|
|
|
|
| 241 |
.where(pro_jobs.c.job_id == job_id)
|
| 242 |
.values(progress_pct=max(0, min(int(progress_pct), 100)))
|
| 243 |
)
|
| 244 |
+
self._sync_after_write()
|
| 245 |
|
| 246 |
def complete(
|
| 247 |
self,
|
|
|
|
| 271 |
)
|
| 272 |
if not res.rowcount:
|
| 273 |
return None
|
| 274 |
+
self._sync_after_write()
|
| 275 |
return self.get_job(job_id)
|
| 276 |
|
| 277 |
def fail(self, job_id: str, *, error_class: str, error_detail: str | None = None) -> ProJobRecord | None:
|
|
|
|
| 289 |
)
|
| 290 |
if not res.rowcount:
|
| 291 |
return None
|
| 292 |
+
self._sync_after_write()
|
| 293 |
return self.get_job(job_id)
|
| 294 |
|
| 295 |
def request_cancel(self, job_id: str, *, session_id: str) -> str:
|
|
|
|
| 310 |
.where(pro_jobs.c.job_id == job_id)
|
| 311 |
.values(status="cancelled", finished_at=_utc_now(), progress_pct=0, error_class="cancelled")
|
| 312 |
)
|
| 313 |
+
self._sync_after_write()
|
| 314 |
return "cancelled"
|
| 315 |
if current == "running":
|
| 316 |
conn.execute(
|
|
|
|
| 318 |
.where(pro_jobs.c.job_id == job_id)
|
| 319 |
.values(cancel_requested=True)
|
| 320 |
)
|
| 321 |
+
self._sync_after_write()
|
| 322 |
return "cancelling"
|
| 323 |
return current
|
| 324 |
|
|
|
|
| 339 |
pro_jobs.c.finished_at < cutoff_s,
|
| 340 |
)
|
| 341 |
)
|
| 342 |
+
if rows:
|
| 343 |
+
self._sync_after_write()
|
| 344 |
if artifact_root:
|
| 345 |
root = Path(artifact_root)
|
| 346 |
for (job_id,) in rows:
|
| 347 |
_delete_job_artifacts(root / str(job_id))
|
| 348 |
return len(rows)
|
| 349 |
|
| 350 |
+
def _sync_after_write(self) -> None:
|
| 351 |
+
if self._on_write is None:
|
| 352 |
+
return
|
| 353 |
+
self._on_write()
|
| 354 |
+
|
| 355 |
|
| 356 |
def create_pro_job_engine(url: str) -> Engine:
|
| 357 |
u = make_url(url)
|
|
|
|
| 376 |
return engine
|
| 377 |
|
| 378 |
|
| 379 |
+
def create_pro_job_store(url: str, *, on_write: Callable[[], None] | None = None) -> SqliteProJobStore:
|
| 380 |
_ensure_parent_dir_for_sqlite_file(url)
|
| 381 |
+
store = SqliteProJobStore(create_pro_job_engine(url), on_write=on_write)
|
| 382 |
store.initialize_schema()
|
| 383 |
return store
|
| 384 |
|
src/nutonic_server/ranked_store.py
CHANGED
|
@@ -6,7 +6,7 @@ import uuid
|
|
| 6 |
from dataclasses import dataclass
|
| 7 |
from datetime import UTC, datetime
|
| 8 |
from threading import Lock
|
| 9 |
-
from typing import TYPE_CHECKING, Literal
|
| 10 |
|
| 11 |
from sqlalchemy import Column, Float, Integer, MetaData, String, Table, create_engine, delete, desc, insert, select, text, update
|
| 12 |
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
|
@@ -74,9 +74,10 @@ class RankedVerifiedLbRow:
|
|
| 74 |
|
| 75 |
|
| 76 |
class SqliteRankedStore:
|
| 77 |
-
def __init__(self, engine: Engine) -> None:
|
| 78 |
self._engine = engine
|
| 79 |
self._lock = Lock()
|
|
|
|
| 80 |
|
| 81 |
def initialize_schema(self) -> None:
|
| 82 |
metadata.create_all(self._engine)
|
|
@@ -90,6 +91,7 @@ class SqliteRankedStore:
|
|
| 90 |
names = {str(r[1]) for r in rows}
|
| 91 |
if "opened_at_epoch" not in names:
|
| 92 |
conn.execute(text("ALTER TABLE ranked_rounds ADD COLUMN opened_at_epoch INTEGER NOT NULL DEFAULT 0"))
|
|
|
|
| 93 |
|
| 94 |
def prune_stale_open_rounds(self, *, now_epoch: int, max_age_seconds: int) -> int:
|
| 95 |
"""Delete ``open`` rounds whose ``opened_at_epoch`` is older than ``now_epoch - max_age_seconds``."""
|
|
@@ -104,6 +106,8 @@ class SqliteRankedStore:
|
|
| 104 |
ranked_rounds.c.opened_at_epoch < cutoff,
|
| 105 |
)
|
| 106 |
)
|
|
|
|
|
|
|
| 107 |
return int(res.rowcount or 0)
|
| 108 |
|
| 109 |
def create_round(
|
|
@@ -131,6 +135,7 @@ class SqliteRankedStore:
|
|
| 131 |
opened_at_epoch=opened,
|
| 132 |
)
|
| 133 |
)
|
|
|
|
| 134 |
return rid
|
| 135 |
|
| 136 |
def get_round(self, round_id: str) -> RankedRoundRow | None:
|
|
@@ -160,6 +165,7 @@ class SqliteRankedStore:
|
|
| 160 |
.where(ranked_rounds.c.round_id == round_id)
|
| 161 |
.values(status="submitted")
|
| 162 |
)
|
|
|
|
| 163 |
|
| 164 |
def forfeit_round(self, round_id: str, session_id: str) -> RankedForfeitResult:
|
| 165 |
"""Mark ``open`` round as ``forfeited`` when ``session_id`` matches (IMP-091)."""
|
|
@@ -183,6 +189,7 @@ class SqliteRankedStore:
|
|
| 183 |
.where(ranked_rounds.c.round_id == round_id)
|
| 184 |
.values(status="forfeited")
|
| 185 |
)
|
|
|
|
| 186 |
return "ok"
|
| 187 |
|
| 188 |
def get_submit_if_exists(self, round_id: str, idempotency_key: str) -> tuple[float, int] | None:
|
|
@@ -225,6 +232,7 @@ class SqliteRankedStore:
|
|
| 225 |
score_points=score_points,
|
| 226 |
)
|
| 227 |
)
|
|
|
|
| 228 |
return distance_km, score_points, True
|
| 229 |
|
| 230 |
def append_verified_leaderboard(
|
|
@@ -247,6 +255,7 @@ class SqliteRankedStore:
|
|
| 247 |
distance_km=distance_km,
|
| 248 |
)
|
| 249 |
conn.execute(stmt.on_conflict_do_nothing(index_elements=["round_id"]))
|
|
|
|
| 250 |
|
| 251 |
def list_ranked_leaderboard(self, map_id: str) -> list[RankedVerifiedLbRow]:
|
| 252 |
"""Verified ranked rows for ``map_id``, highest score first."""
|
|
@@ -275,6 +284,11 @@ class SqliteRankedStore:
|
|
| 275 |
)
|
| 276 |
return out
|
| 277 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 278 |
|
| 279 |
def create_ranked_engine(url: str) -> Engine:
|
| 280 |
from nutonic_server.leaderboard_store import create_leaderboard_engine
|
|
@@ -283,10 +297,19 @@ def create_ranked_engine(url: str) -> Engine:
|
|
| 283 |
|
| 284 |
|
| 285 |
def create_ranked_store(url: str) -> SqliteRankedStore:
|
| 286 |
-
from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file
|
|
|
|
|
|
|
| 287 |
|
| 288 |
_ensure_parent_dir_for_sqlite_file(url)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 289 |
eng = create_ranked_engine(url)
|
| 290 |
-
st = SqliteRankedStore(eng)
|
| 291 |
st.initialize_schema()
|
| 292 |
return st
|
|
|
|
| 6 |
from dataclasses import dataclass
|
| 7 |
from datetime import UTC, datetime
|
| 8 |
from threading import Lock
|
| 9 |
+
from typing import TYPE_CHECKING, Callable, Literal
|
| 10 |
|
| 11 |
from sqlalchemy import Column, Float, Integer, MetaData, String, Table, create_engine, delete, desc, insert, select, text, update
|
| 12 |
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
|
|
|
|
| 74 |
|
| 75 |
|
| 76 |
class SqliteRankedStore:
|
| 77 |
+
def __init__(self, engine: Engine, *, on_write: Callable[[], None] | None = None) -> None:
|
| 78 |
self._engine = engine
|
| 79 |
self._lock = Lock()
|
| 80 |
+
self._on_write = on_write
|
| 81 |
|
| 82 |
def initialize_schema(self) -> None:
|
| 83 |
metadata.create_all(self._engine)
|
|
|
|
| 91 |
names = {str(r[1]) for r in rows}
|
| 92 |
if "opened_at_epoch" not in names:
|
| 93 |
conn.execute(text("ALTER TABLE ranked_rounds ADD COLUMN opened_at_epoch INTEGER NOT NULL DEFAULT 0"))
|
| 94 |
+
self._sync_after_write()
|
| 95 |
|
| 96 |
def prune_stale_open_rounds(self, *, now_epoch: int, max_age_seconds: int) -> int:
|
| 97 |
"""Delete ``open`` rounds whose ``opened_at_epoch`` is older than ``now_epoch - max_age_seconds``."""
|
|
|
|
| 106 |
ranked_rounds.c.opened_at_epoch < cutoff,
|
| 107 |
)
|
| 108 |
)
|
| 109 |
+
if (res.rowcount or 0) > 0:
|
| 110 |
+
self._sync_after_write()
|
| 111 |
return int(res.rowcount or 0)
|
| 112 |
|
| 113 |
def create_round(
|
|
|
|
| 135 |
opened_at_epoch=opened,
|
| 136 |
)
|
| 137 |
)
|
| 138 |
+
self._sync_after_write()
|
| 139 |
return rid
|
| 140 |
|
| 141 |
def get_round(self, round_id: str) -> RankedRoundRow | None:
|
|
|
|
| 165 |
.where(ranked_rounds.c.round_id == round_id)
|
| 166 |
.values(status="submitted")
|
| 167 |
)
|
| 168 |
+
self._sync_after_write()
|
| 169 |
|
| 170 |
def forfeit_round(self, round_id: str, session_id: str) -> RankedForfeitResult:
|
| 171 |
"""Mark ``open`` round as ``forfeited`` when ``session_id`` matches (IMP-091)."""
|
|
|
|
| 189 |
.where(ranked_rounds.c.round_id == round_id)
|
| 190 |
.values(status="forfeited")
|
| 191 |
)
|
| 192 |
+
self._sync_after_write()
|
| 193 |
return "ok"
|
| 194 |
|
| 195 |
def get_submit_if_exists(self, round_id: str, idempotency_key: str) -> tuple[float, int] | None:
|
|
|
|
| 232 |
score_points=score_points,
|
| 233 |
)
|
| 234 |
)
|
| 235 |
+
self._sync_after_write()
|
| 236 |
return distance_km, score_points, True
|
| 237 |
|
| 238 |
def append_verified_leaderboard(
|
|
|
|
| 255 |
distance_km=distance_km,
|
| 256 |
)
|
| 257 |
conn.execute(stmt.on_conflict_do_nothing(index_elements=["round_id"]))
|
| 258 |
+
self._sync_after_write()
|
| 259 |
|
| 260 |
def list_ranked_leaderboard(self, map_id: str) -> list[RankedVerifiedLbRow]:
|
| 261 |
"""Verified ranked rows for ``map_id``, highest score first."""
|
|
|
|
| 284 |
)
|
| 285 |
return out
|
| 286 |
|
| 287 |
+
def _sync_after_write(self) -> None:
|
| 288 |
+
if self._on_write is None:
|
| 289 |
+
return
|
| 290 |
+
self._on_write()
|
| 291 |
+
|
| 292 |
|
| 293 |
def create_ranked_engine(url: str) -> Engine:
|
| 294 |
from nutonic_server.leaderboard_store import create_leaderboard_engine
|
|
|
|
| 297 |
|
| 298 |
|
| 299 |
def create_ranked_store(url: str) -> SqliteRankedStore:
|
| 300 |
+
from nutonic_server.leaderboard_store import _ensure_parent_dir_for_sqlite_file, sqlite_file_path_from_url
|
| 301 |
+
from nutonic_server.hf_persistence import HfSqliteSync
|
| 302 |
+
from nutonic_server.settings import load_settings
|
| 303 |
|
| 304 |
_ensure_parent_dir_for_sqlite_file(url)
|
| 305 |
+
sync_hook = None
|
| 306 |
+
db_path = sqlite_file_path_from_url(url)
|
| 307 |
+
if db_path is not None:
|
| 308 |
+
hf = HfSqliteSync.from_settings(load_settings())
|
| 309 |
+
if hf is not None:
|
| 310 |
+
hf.bootstrap_sqlite_file(local_path=db_path, logical_name="ranked")
|
| 311 |
+
sync_hook = hf.make_write_sync_hook(local_path=db_path, logical_name="ranked")
|
| 312 |
eng = create_ranked_engine(url)
|
| 313 |
+
st = SqliteRankedStore(eng, on_write=sync_hook)
|
| 314 |
st.initialize_schema()
|
| 315 |
return st
|
src/nutonic_server/settings.py
CHANGED
|
@@ -265,6 +265,42 @@ class Settings(BaseSettings):
|
|
| 265 |
),
|
| 266 |
)
|
| 267 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 268 |
@field_validator("cors_origins", mode="before")
|
| 269 |
@classmethod
|
| 270 |
def strip_origins(cls, v: object) -> str:
|
|
|
|
| 265 |
),
|
| 266 |
)
|
| 267 |
|
| 268 |
+
hf_persistence_enabled: bool = Field(
|
| 269 |
+
default=False,
|
| 270 |
+
validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_ENABLED", "HF_PERSISTENCE_ENABLED"),
|
| 271 |
+
description="When true, mirror SQLite server DB files to a Hugging Face Dataset repo.",
|
| 272 |
+
)
|
| 273 |
+
|
| 274 |
+
hf_persistence_required: bool = Field(
|
| 275 |
+
default=False,
|
| 276 |
+
validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_REQUIRED", "HF_PERSISTENCE_REQUIRED"),
|
| 277 |
+
description=(
|
| 278 |
+
"When true with HF persistence enabled, fail fast if the dataset repo/token is missing "
|
| 279 |
+
"or sync operations fail."
|
| 280 |
+
),
|
| 281 |
+
)
|
| 282 |
+
|
| 283 |
+
hf_persistence_repo_id: str = Field(
|
| 284 |
+
default="",
|
| 285 |
+
validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_REPO_ID", "HF_PERSISTENCE_REPO_ID"),
|
| 286 |
+
description="Dataset repo id (owner/name) used for SQLite persistence mirroring.",
|
| 287 |
+
)
|
| 288 |
+
|
| 289 |
+
hf_persistence_dataset_subdir: str = Field(
|
| 290 |
+
default="server-persistence",
|
| 291 |
+
validation_alias=AliasChoices("NUTONIC_HF_PERSISTENCE_SUBDIR", "HF_PERSISTENCE_SUBDIR"),
|
| 292 |
+
description="Subdirectory inside the Dataset repo where DB files are stored.",
|
| 293 |
+
)
|
| 294 |
+
|
| 295 |
+
hf_persistence_startup_pull_mode: str = Field(
|
| 296 |
+
default="if_missing",
|
| 297 |
+
validation_alias=AliasChoices(
|
| 298 |
+
"NUTONIC_HF_PERSISTENCE_STARTUP_PULL_MODE",
|
| 299 |
+
"HF_PERSISTENCE_STARTUP_PULL_MODE",
|
| 300 |
+
),
|
| 301 |
+
description="Startup pull policy for local SQLite files: if_missing (default) or always.",
|
| 302 |
+
)
|
| 303 |
+
|
| 304 |
@field_validator("cors_origins", mode="before")
|
| 305 |
@classmethod
|
| 306 |
def strip_origins(cls, v: object) -> str:
|