Spaces:
Sleeping
Sleeping
| title: Kink Discovery | |
| emoji: 🔥 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| pinned: false | |
| # Hub merges these onto Docker responses; without this, COEP defaults (e.g. require-corp) break CDN catalog images. | |
| custom_headers: | |
| cross-origin-embedder-policy: credentialless | |
| cross-origin-opener-policy: same-origin | |
| cross-origin-resource-policy: same-origin | |
| # Kink Discovery (Hugging Face Space) | |
| ## Standards alignment (Phase 0-A) | |
| This tree uses **staged alignment** with the **parser** repo policy, not a full copy-paste gate on every path yet. | |
| - **Deploy and HF bootstrap** still rely on the process environment variables documented under [Settings (recommended)](#settings-recommended) (Hugging Face Spaces and B2 expect them). | |
| - **Recommender source (graph / OTS / hybrid)** uses **typed settings**: optional `recsys/settings.json` next to the SQLite store (see [`deploy/recsys/settings.example.json`](deploy/recsys/settings.example.json)), optional `KINK_RECSYS_*` env overrides when not injecting `Backend(..., recsys_settings=...)`, or constructor injection in code/tests. Details: [`docs/recsys_ots.md`](docs/recsys_ots.md). | |
| - **Makefile**: `make ruff-kink` runs Ruff ([`ruff.toml`](ruff.toml)) on staged paths (`backend/models.py`, `scripts/recsys_benchmark.py`, `tests/test_recsys_ots.py`). `make semgrep-kink` runs Semgrep from repo-root [`semgrep.yml`](semgrep.yml) (same rules as parser) on Python sources, with **`SEMGREP_KINK_EXCLUDE_RULES`** defaults so broad scans stay usable while debt is paid down (override or clear that variable to tighten). `make policy-kink` runs Ruff plus **`semgrep-recsys-strict`** ([`backend/models.py`](backend/models.py) only, **0** Semgrep findings). `make semgrep-recsys-core` scans recommender modules with the same excludes as `semgrep-kink`. `make dead-kink` runs `dead` when this tree is a **git** checkout (see [`.dead-allowlist`](.dead-allowlist)). `PARSER_ROOT` is only needed for **`scripts/install_cursor_from_parser.sh`** when refreshing from a non-default parser checkout. | |
| - **Cursor hooks and rules**: run **`bash scripts/install_cursor_from_parser.sh`** once (or after parser updates) to copy `.cursor/hooks/`, `.cursor/rules/`, root **`semgrep.yml`**, `scripts/codex_*.py`, and regenerate **`.cursor/hooks.json`**. See [`.cursor/README.md`](.cursor/README.md). The stop hook prefers this repo’s **`semgrep.yml`** when present. | |
| This repo runs as a **Docker Space** with the **same FastAPI backend** as local development. The **Dockerfile** defaults to **`KINK_HF_REQUIRE_FULL_CATALOG=0`** (bundled seed in the image for reliable cold start). For a **full** catalog on the Space, set **`KINK_HF_REQUIRE_FULL_CATALOG=1`** plus **`KINK_CATALOG_URL`** (public HTTPS), or **Backblaze B2** via Space secrets **`B2_KEY_ID` / `B2_APPLICATION_KEY` / `B2_BUCKET` / `B2_REGION`** plus **`KINK_B2_OBJECT_KEY`**, or **`KINK_HF_DATASET_REPO`** / **`KINK_HF_DATASET_FILENAME`** on **Hugging Face Hub** (see `deploy/hf/README.md`). Precedence: **`KINK_CATALOG_URL`**, then B2, then Hub. | |
| ## Settings (recommended) | |
| | Variable | Purpose | | |
| |----------|---------| | |
| | `KINK_STORE_PATH` | SQLite file path. Use a **mounted Storage Bucket** path (e.g. `/data/store.db`) so **user accounts persist** across restarts. Default in the image is `/tmp/kink_store.db` (ephemeral unless you use persistent storage). | | |
| | `KINK_HF_REQUIRE_FULL_CATALOG` | **`0` in the image** (bundled seed). Set **`1`** on the Space when you supply a full catalog (URL, B2, or Hub). | | |
| | `KINK_CATALOG_URL` | Optional public HTTPS URL to `store_slim.db` (wins over B2 and Hub when set). | | |
| | `KINK_CATALOG_URL_BEARER` | Optional bearer for private **HTTP** catalog URLs. | | |
| | `B2_KEY_ID`, `B2_APPLICATION_KEY`, `B2_BUCKET`, `B2_REGION` | B2 S3 credentials (Space secrets; **parser**-compatible). With `KINK_B2_OBJECT_KEY`, downloads the catalog without a public URL. | | |
| | `KINK_B2_OBJECT_KEY` | Object path in the bucket (default `kink/catalog/store_slim.db`). | | |
| | `KINK_HF_DATASET_REPO` | Hub dataset id (with `KINK_HF_REQUIRE_FULL_CATALOG=1`, or with `KINK_HF_USE_HUB_DATASET=1` when full catalog is off). | | |
| | `KINK_HF_DATASET_FILENAME` | Remote filename in the dataset (e.g. `store_slim.db`). | | |
| | `KINK_HF_USE_HUB_DATASET` | **`1`** = Hub download even when `KINK_HF_REQUIRE_FULL_CATALOG=0`. If unset, a leftover `KINK_HF_DATASET_REPO` variable is ignored and the **bundled** seed is used. | | |
| | `HF_TOKEN` | Required if the Hub dataset is **private** (not needed for public HTTPS URLs). | | |
| | `KINK_CORS_ORIGINS` | Only if you split frontend/API origins. Same-origin Space UI usually needs no change. | | |
| | `KINK_RECSYS_SOURCE` | Optional. When set and the app is **not** constructed with `recsys_settings=`, overrides `recsys/settings.json` for `source` (`graph` / `ots` / `hybrid`). | | |
| | `KINK_RECSYS_CANDIDATES_PATH` | Optional. Absolute path to OTS `candidates.json` when it is not beside the SQLite file. | | |
| | `KINK_FAIL_ON_EPHEMERAL_STORE` | Optional. When truthy, refuse to start if `KINK_STORE_PATH` points at ephemeral storage such as `/tmp`. | | |
| ### One-time: publish the full catalog (Hub **or** B2) | |
| 1. `python scripts/build_slim_store.py` → `data/store_slim.db` | |
| 2. **Either** `export HF_TOKEN=hf_...` and `python scripts/upload_catalog_to_hub.py` **or** (same `[b2]` block as **parser** `secrets.toml`) `python scripts/upload_catalog_to_b2.py` — optional `KINK_B2_CATALOG_BUCKET` / `--bucket` for a dedicated bucket. It prints `KINK_CATALOG_URL=…` for **public** Spaces; for **private** B2, add `B2_*` secrets + `KINK_B2_OBJECT_KEY` instead. Then `python scripts/verify_b2_catalog_http.py`. | |
| 3. For Hub only: `python scripts/verify_hub_dataset.py` must print `OK`. | |
| Bootstrap uses `hf_hub_download(..., repo_type="dataset")` so the file is fetched from a **dataset** repo, not a model repo. | |
| First cold start after deploy may take a long time while the multi‑GB file downloads. | |
| ### Discover shows “Nothing new to show right now.” | |
| That message means the client received recommendations but **every returned kink was already on your play list** (or the API returned an empty list). With a **full** `store_slim.db`, the backend still has many unrated plays after dozens of saves (see `tests/test_discover_recommendations_invariant.py`). If you see exhaustion after only ~40–50 ratings, check **`GET /health`**: `stats` should report a **large** kink count. A **tiny** or wrong SQLite file (ephemeral `/tmp` path, failed bootstrap, or seed-only DB) shrinks the unrated pool until it is genuinely empty—fix **`KINK_STORE_PATH`** and catalog bootstrap (Hub / B2 / URL) rather than the recommender. | |
| `/health` now also exposes `store_path`, `storage_mode`, `store_path_persistent`, and OTS candidate-cache metadata (`recsys_candidate_cache_exists`, `recsys_candidate_cache_age_s`, `recsys_candidate_user_count`) so deploy verification can distinguish persistence mistakes from recommender depletion. | |
| ### Deploy proof (catalog, kink detail, recommendations) | |
| After a push, prove the live Space (replace the URL): | |
| ```bash | |
| HF_VERIFY_BASE_URL=https://your-space.hf.space HF_VERIFY_MIN_KINKS=1000 \ | |
| HF_VERIFY_SKIP_COEP=1 python scripts/verify_hf_stack.py --smoke | |
| ``` | |
| `--smoke` checks `GET /health` (catalog counts), **served** `index → app.js → api.js` for HTML/non-JSON error shaping (so a bad recs response cannot dump a full HF HTML page into the UI), `GET /health/catalog-sample` plus `GET /kinks/{id}` (one catalog row, not a full list), creates a user, saves one play, and asserts `GET /users/{id}/recommendations` returns items. When `stats.assets >= 1`, it **GETs** the bundled demo tile ``/media/hf_seed/demo_kink_1.jpg`` (required), optionally ``/media/fetlife_fetishes/1/146853406.jpg`` when that seed file is on the edge, then **best-effort** GETs one live catalog ``asset_url`` (``https://…`` or ``/media/…``) and treats **404** on that row as “bytes not mirrored for this id” while the ``hf_seed`` check still proves ``/media`` works. See ``deploy/hf/seed/cached_assets/`` + ``deploy/hf/entrypoint.sh``. Set `HF_VERIFY_SKIP_FRONTEND=1` to skip the static bundle probe; set `HF_VERIFY_SKIP_PICTURE_PROBE=1` to skip all image probes. Run without `--smoke` for the full partner/group exercise (that path includes the same frontend and image probes). | |
| On `*.hf.space`, the static bundle probe **retries** (same idea as COEP) until the new container serves updated `api.js` / `discover.js` after a push. Tunables: `HF_VERIFY_FRONTEND_RETRIES`, `HF_VERIFY_FRONTEND_INTERVAL_S`, `HF_VERIFY_CATALOG_SAMPLE_TIMEOUT_S` (wait if `/health/catalog-sample` returns `starting: true`), `HF_VERIFY_HTTP_TIMEOUT_S` (per-request read timeout for large `GET /kinks/{id}` on cold edges), `HF_VERIFY_IMAGE_TIMEOUT_S` (image `asset_url` fetch timeout). | |
| ### Browser 401 on `/users/.../roles` or `.../partner-groups` | |
| Those routes require the **`x-private-token`** that belongs to **that** `user_id`. **401** means the token is missing, wrong for this user, or the profile was wiped (ephemeral `/tmp` after a restart). Log in again with the user id and private token from onboarding, or create a new profile. | |
| ## Current product flow | |
| - Fresh users see a **12-card starter gate** before full discovery unlocks. | |
| - Positive reactions in Discover ask for **direction** after the reaction instead of before it. | |
| - Partner invites use a shareable **`?link=<user_id>`** link; manual invite codes remain as fallback. | |
| - `Together` is organized around **Shared / Worth Exploring / Their List**. | |
| ## Local dev reset (wipe users, optional port kill) | |
| [`scripts/dev_reset_local.py`](scripts/dev_reset_local.py) deletes **all** user profiles and related rows (`partnergroup*`, `partnerlink*`, `scenariopreference`, `playpreference`, etc.) from a SQLite store. It does **not** remove kinks or catalog data. | |
| - Preview: `python scripts/dev_reset_local.py --db data/store.db --dry-run` | |
| - Apply: `python scripts/dev_reset_local.py --db data/store.db --i-am-sure` (also tries to free TCP **8011** by default; use `--skip-port-kill` when you only want the DB wipe, e.g. in CI) | |
| - Custom ports: `--port 8012 --port 8013` | |
| After changing merge detection in [`backend/kink_merge.py`](backend/kink_merge.py), re-run merge tooling on a **copy** of production data, review JSON, apply merges, then refresh the catalog cache on that store: | |
| ```bash | |
| python scripts/propose_kink_merge_pairs.py --db path/to/store.db --out /tmp/merge_proposals.json | |
| python scripts/apply_kink_merge_pair.py --db path/to/store.db --from DUP_ID --to CANON_ID # repeat per reviewed pair | |
| # then warm/refresh Backend on that path (or restart the API) so `detail_by_id` picks up row removals | |
| ``` | |
| For **title-fingerprint** duplicate review (aligned with [`frontend/name-fingerprint.js`](frontend/name-fingerprint.js)): | |
| ```bash | |
| python scripts/find_name_duplicate_clusters.py --db data/store_slim.db --min-size 2 --limit 50 | |
| ``` | |
| Export **both** cluster keys to JSONL for labeling / benchmarks: | |
| ```bash | |
| python scripts/export_normalization_clusters.py --db data/store_slim.db --out /tmp/norm_clusters.jsonl --mode both --min-size 2 | |
| ``` | |
| Golden expectations for CI live in [`tests/fixtures/normalization_benchmark_v1.json`](tests/fixtures/normalization_benchmark_v1.json) ([`tests/test_normalization_benchmark.py`](tests/test_normalization_benchmark.py)); the file embeds **20** seed-42 random case-only clusters plus two curated cases. Regenerate with [`scripts/generate_normalization_benchmark_fixture.py`](scripts/generate_normalization_benchmark_fixture.py) when you intentionally change merge behavior and want a fresh frozen sample block. | |
| **Pair sourcing for benchmarks:** merge tooling buckets whole clusters; labeled **pairs** should be drawn with a tiered strategy so you get both obvious merges and hard negatives. See [`backend/benchmark_pair_sources.py`](backend/benchmark_pair_sources.py) (docstring: merge-signature buckets, fingerprint collisions, similarity edges). Export JSONL for review: | |
| ```bash | |
| python scripts/source_normalization_benchmark_pairs.py --db data/store_slim.db --out /tmp/bench_pairs.jsonl --sample-random 800 --seed 42 | |
| ``` | |
| ## Verify locally | |
| ```bash | |
| docker build -t kink-cli-hf:latest . | |
| python scripts/verify_hf_stack.py --docker | |
| ``` | |
| ## Product audits | |
| Use Playwright as the canonical browser audit path for the current product shell: | |
| ```bash | |
| python scripts/e2e_playwright.py --base-url http://127.0.0.1:8012 | |
| python scripts/e2e_playwright.py --scenario starter,settings --base-url http://127.0.0.1:8012 | |
| python scripts/couple_product_audit.py --base-url http://127.0.0.1:8012 --json-out /tmp/kink_couple_audit.json | |
| ``` | |
| `scripts/e2e_smoke.py` is now legacy/manual Selenium coverage, not the primary maintained browser gate. | |
| To review the current starter deck without mutating user data: | |
| ```bash | |
| python scripts/audit_starter_deck.py --store data/store_slim.db --limit 24 | |
| ``` | |
| ## Live deployment | |
| - **Hub:** [huggingface.co/spaces/Perplexed7675/kink-discovery](https://huggingface.co/spaces/Perplexed7675/kink-discovery) | |
| - **App URL:** [perplexed7675-kink-discovery.hf.space](https://perplexed7675-kink-discovery.hf.space) (same-origin API + static UI) | |
| - **Account persistence:** pick **one** option so user profiles survive restarts: | |
| - **Paid:** enable HF Persistent Storage on the Space and set `KINK_STORE_PATH=/data/store_slim.db` + `KINK_FAIL_ON_EPHEMERAL_STORE=1`. Zero engineering, perfect durability. | |
| - **Free (Hub-snapshot):** create a private HF Hub dataset (e.g. `Perplexed7675/kink-userstate`), then set Space variables `KINK_USER_SNAPSHOT_REPO=Perplexed7675/kink-userstate` and `HF_TOKEN=<write-token>`. The container then pulls the latest user-state snapshot on cold boot and pushes back every 60s when the user tables change (catalog stays on `/tmp`). Tunables: `KINK_USER_SNAPSHOT_FILENAME` (default `user_state.db`), `KINK_USER_SNAPSHOT_INTERVAL_S` (default 60). Trade-off: up to ~60s of writes can be lost on a hard crash; the Space must stay single-replica (cpu-basic is). | |
| Re-verify anytime: | |
| ```bash | |
| HF_VERIFY_BASE_URL=https://perplexed7675-kink-discovery.hf.space python scripts/verify_hf_stack.py | |
| ``` | |
| After **frontend** changes, bump the `?v=` query on `frontend/index.html` styles + `app.js` so browsers load the new bundle, then redeploy (command below). | |
| ## Deploy to Hugging Face (working Space) | |
| 1. Create a [Hub access token](https://huggingface.co/settings/tokens) with write access (or reuse `[huggingface].token` from the `parser` repo’s `secrets.toml` — same file `scripts/publish_hf_space.py` reads when `HF_TOKEN` is unset; path: `KINK_PARSER_SECRETS_TOML` or `~/PycharmProjects/parser/secrets.toml`). | |
| 2. From the repo root: | |
| ```bash | |
| # Either export credentials, or rely on parser secrets.toml + [huggingface].username → <username>/kink-discovery | |
| export HF_TOKEN=hf_... | |
| export HF_SPACE_REPO=yourname/your-space-name # e.g. Perplexed7675/kink-discovery | |
| python scripts/publish_hf_space.py --verify | |
| ``` | |
| **Release order:** run tests (`make test`), then—**only if** you changed the catalog object or B2/Hub secrets—[`scripts/sync_hf_space_b2_catalog.py`](scripts/sync_hf_space_b2_catalog.py) and/or [`scripts/deploy_hf_bootstrap_catalog.py`](scripts/deploy_hf_bootstrap_catalog.py); then `publish_hf_space.py --verify`. Full checklist: [`deploy/hf/README.md`](deploy/hf/README.md#release-checklist). | |
| This uploads the project (Docker SDK), waits until the Space is **RUNNING**, resolves the direct app URL (`*.hf.space`), and runs the same checks as `verify_hf_stack.py` against the **live** deployment (including COEP on `/health`). | |
| To only upload and print the verify command: | |
| ```bash | |
| python scripts/publish_hf_space.py | |
| ``` | |
| To prove a Space that is already deployed: | |
| ```bash | |
| HF_VERIFY_BASE_URL=https://yourname-yourspace.hf.space python scripts/verify_hf_stack.py | |
| ``` | |
| See `deploy/hf/README.md` for persistence details. | |