README.md

metadata

title: Kink Discovery
emoji: 🔥
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
custom_headers:
  cross-origin-embedder-policy: credentialless
  cross-origin-opener-policy: same-origin
  cross-origin-resource-policy: same-origin

Kink Discovery (Hugging Face Space)

Standards alignment (Phase 0-A)

This tree uses staged alignment with the parser repo policy, not a full copy-paste gate on every path yet.

• Deploy and HF bootstrap still rely on the process environment variables documented under Settings (recommended) (Hugging Face Spaces and B2 expect them).
• Recommender source (graph / OTS / hybrid) uses typed settings: optional recsys/settings.json next to the SQLite store (see deploy/recsys/settings.example.json), optional KINK_RECSYS_* env overrides when not injecting Backend(..., recsys_settings=...), or constructor injection in code/tests. Details: docs/recsys_ots.md.
• Makefile: make ruff-kink runs Ruff (ruff.toml) on staged paths (backend/models.py, scripts/recsys_benchmark.py, tests/test_recsys_ots.py). make semgrep-kink runs Semgrep from repo-root semgrep.yml (same rules as parser) on Python sources, with SEMGREP_KINK_EXCLUDE_RULES defaults so broad scans stay usable while debt is paid down (override or clear that variable to tighten). make policy-kink runs Ruff plus semgrep-recsys-strict (backend/models.py only, 0 Semgrep findings). make semgrep-recsys-core scans recommender modules with the same excludes as semgrep-kink. make dead-kink runs dead when this tree is a git checkout (see .dead-allowlist). PARSER_ROOT is only needed for scripts/install_cursor_from_parser.sh when refreshing from a non-default parser checkout.
• Cursor hooks and rules: run bash scripts/install_cursor_from_parser.sh once (or after parser updates) to copy .cursor/hooks/, .cursor/rules/, root semgrep.yml, scripts/codex_*.py, and regenerate .cursor/hooks.json. See .cursor/README.md. The stop hook prefers this repo’s semgrep.yml when present.

This repo runs as a Docker Space with the same FastAPI backend as local development. The Dockerfile defaults to KINK_HF_REQUIRE_FULL_CATALOG=0 (bundled seed in the image for reliable cold start). For a full catalog on the Space, set KINK_HF_REQUIRE_FULL_CATALOG=1 plus KINK_CATALOG_URL (public HTTPS), or Backblaze B2 via Space secrets B2_KEY_ID / B2_APPLICATION_KEY / B2_BUCKET / B2_REGION plus KINK_B2_OBJECT_KEY, or KINK_HF_DATASET_REPO / KINK_HF_DATASET_FILENAME on Hugging Face Hub (see deploy/hf/README.md). Precedence: KINK_CATALOG_URL, then B2, then Hub.

Settings (recommended)

Variable	Purpose
KINK_STORE_PATH	SQLite file path. Use a mounted Storage Bucket path (e.g. /data/store.db) so user accounts persist across restarts. Default in the image is /tmp/kink_store.db (ephemeral unless you use persistent storage).
KINK_HF_REQUIRE_FULL_CATALOG	0 in the image (bundled seed). Set 1 on the Space when you supply a full catalog (URL, B2, or Hub).
KINK_CATALOG_URL	Optional public HTTPS URL to store_slim.db (wins over B2 and Hub when set).
KINK_CATALOG_URL_BEARER	Optional bearer for private HTTP catalog URLs.
B2_KEY_ID, B2_APPLICATION_KEY, B2_BUCKET, B2_REGION	B2 S3 credentials (Space secrets; parser-compatible). With KINK_B2_OBJECT_KEY, downloads the catalog without a public URL.
KINK_B2_OBJECT_KEY	Object path in the bucket (default kink/catalog/store_slim.db).
KINK_HF_DATASET_REPO	Hub dataset id (with KINK_HF_REQUIRE_FULL_CATALOG=1, or with KINK_HF_USE_HUB_DATASET=1 when full catalog is off).
KINK_HF_DATASET_FILENAME	Remote filename in the dataset (e.g. store_slim.db).
KINK_HF_USE_HUB_DATASET	1 = Hub download even when KINK_HF_REQUIRE_FULL_CATALOG=0. If unset, a leftover KINK_HF_DATASET_REPO variable is ignored and the bundled seed is used.
HF_TOKEN	Required if the Hub dataset is private (not needed for public HTTPS URLs).
KINK_CORS_ORIGINS	Only if you split frontend/API origins. Same-origin Space UI usually needs no change.
KINK_RECSYS_SOURCE	Optional. When set and the app is not constructed with recsys_settings=, overrides recsys/settings.json for source (graph / ots / hybrid).
KINK_RECSYS_CANDIDATES_PATH	Optional. Absolute path to OTS candidates.json when it is not beside the SQLite file.
KINK_FAIL_ON_EPHEMERAL_STORE	Optional. When truthy, refuse to start if KINK_STORE_PATH points at ephemeral storage such as /tmp.

One-time: publish the full catalog (Hub or B2)

1. python scripts/build_slim_store.py → data/store_slim.db
2. Either export HF_TOKEN=hf_... and python scripts/upload_catalog_to_hub.py or (same [b2] block as parser secrets.toml) python scripts/upload_catalog_to_b2.py — optional KINK_B2_CATALOG_BUCKET / --bucket for a dedicated bucket. It prints KINK_CATALOG_URL=… for public Spaces; for private B2, add B2_* secrets + KINK_B2_OBJECT_KEY instead. Then python scripts/verify_b2_catalog_http.py.
3. For Hub only: python scripts/verify_hub_dataset.py must print OK.

Bootstrap uses hf_hub_download(..., repo_type="dataset") so the file is fetched from a dataset repo, not a model repo.

First cold start after deploy may take a long time while the multi‑GB file downloads.

Discover shows “Nothing new to show right now.”

That message means the client received recommendations but every returned kink was already on your play list (or the API returned an empty list). With a full store_slim.db, the backend still has many unrated plays after dozens of saves (see tests/test_discover_recommendations_invariant.py). If you see exhaustion after only ~40–50 ratings, check GET /health: stats should report a large kink count. A tiny or wrong SQLite file (ephemeral /tmp path, failed bootstrap, or seed-only DB) shrinks the unrated pool until it is genuinely empty—fix KINK_STORE_PATH and catalog bootstrap (Hub / B2 / URL) rather than the recommender.

/health now also exposes store_path, storage_mode, store_path_persistent, and OTS candidate-cache metadata (recsys_candidate_cache_exists, recsys_candidate_cache_age_s, recsys_candidate_user_count) so deploy verification can distinguish persistence mistakes from recommender depletion.

Deploy proof (catalog, kink detail, recommendations)

After a push, prove the live Space (replace the URL):

HF_VERIFY_BASE_URL=https://your-space.hf.space HF_VERIFY_MIN_KINKS=1000 \
  HF_VERIFY_SKIP_COEP=1 python scripts/verify_hf_stack.py --smoke

--smoke checks GET /health (catalog counts), served index → app.js → api.js for HTML/non-JSON error shaping (so a bad recs response cannot dump a full HF HTML page into the UI), GET /health/catalog-sample plus GET /kinks/{id} (one catalog row, not a full list), creates a user, saves one play, and asserts GET /users/{id}/recommendations returns items. When stats.assets >= 1, it GETs the bundled demo tile /media/hf_seed/demo_kink_1.jpg (required), optionally /media/fetlife_fetishes/1/146853406.jpg when that seed file is on the edge, then best-effort GETs one live catalog asset_url (https://… or /media/…) and treats 404 on that row as “bytes not mirrored for this id” while the hf_seed check still proves /media works. See deploy/hf/seed/cached_assets/ + deploy/hf/entrypoint.sh. Set HF_VERIFY_SKIP_FRONTEND=1 to skip the static bundle probe; set HF_VERIFY_SKIP_PICTURE_PROBE=1 to skip all image probes. Run without --smoke for the full partner/group exercise (that path includes the same frontend and image probes).

On *.hf.space, the static bundle probe retries (same idea as COEP) until the new container serves updated api.js / discover.js after a push. Tunables: HF_VERIFY_FRONTEND_RETRIES, HF_VERIFY_FRONTEND_INTERVAL_S, HF_VERIFY_CATALOG_SAMPLE_TIMEOUT_S (wait if /health/catalog-sample returns starting: true), HF_VERIFY_HTTP_TIMEOUT_S (per-request read timeout for large GET /kinks/{id} on cold edges), HF_VERIFY_IMAGE_TIMEOUT_S (image asset_url fetch timeout).

Browser 401 on /users/.../roles or .../partner-groups

Those routes require the x-private-token that belongs to that user_id. 401 means the token is missing, wrong for this user, or the profile was wiped (ephemeral /tmp after a restart). Log in again with the user id and private token from onboarding, or create a new profile.

Current product flow

• Fresh users see a 12-card starter gate before full discovery unlocks.
• Positive reactions in Discover ask for direction after the reaction instead of before it.
• Partner invites use a shareable ?link=<user_id> link; manual invite codes remain as fallback.
• Together is organized around Shared / Worth Exploring / Their List.

Local dev reset (wipe users, optional port kill)

scripts/dev_reset_local.py deletes all user profiles and related rows (partnergroup*, partnerlink*, scenariopreference, playpreference, etc.) from a SQLite store. It does not remove kinks or catalog data.

• Preview: python scripts/dev_reset_local.py --db data/store.db --dry-run
• Apply: python scripts/dev_reset_local.py --db data/store.db --i-am-sure (also tries to free TCP 8011 by default; use --skip-port-kill when you only want the DB wipe, e.g. in CI)
• Custom ports: --port 8012 --port 8013

After changing merge detection in backend/kink_merge.py, re-run merge tooling on a copy of production data, review JSON, apply merges, then refresh the catalog cache on that store:

python scripts/propose_kink_merge_pairs.py --db path/to/store.db --out /tmp/merge_proposals.json
python scripts/apply_kink_merge_pair.py --db path/to/store.db --from DUP_ID --to CANON_ID   # repeat per reviewed pair
# then warm/refresh Backend on that path (or restart the API) so `detail_by_id` picks up row removals

For title-fingerprint duplicate review (aligned with frontend/name-fingerprint.js):

python scripts/find_name_duplicate_clusters.py --db data/store_slim.db --min-size 2 --limit 50

Export both cluster keys to JSONL for labeling / benchmarks:

python scripts/export_normalization_clusters.py --db data/store_slim.db --out /tmp/norm_clusters.jsonl --mode both --min-size 2

Golden expectations for CI live in tests/fixtures/normalization_benchmark_v1.json (tests/test_normalization_benchmark.py); the file embeds 20 seed-42 random case-only clusters plus two curated cases. Regenerate with scripts/generate_normalization_benchmark_fixture.py when you intentionally change merge behavior and want a fresh frozen sample block.

Pair sourcing for benchmarks: merge tooling buckets whole clusters; labeled pairs should be drawn with a tiered strategy so you get both obvious merges and hard negatives. See backend/benchmark_pair_sources.py (docstring: merge-signature buckets, fingerprint collisions, similarity edges). Export JSONL for review:

python scripts/source_normalization_benchmark_pairs.py --db data/store_slim.db --out /tmp/bench_pairs.jsonl --sample-random 800 --seed 42

Verify locally

docker build -t kink-cli-hf:latest .
python scripts/verify_hf_stack.py --docker

Product audits

Use Playwright as the canonical browser audit path for the current product shell:

python scripts/e2e_playwright.py --base-url http://127.0.0.1:8012
python scripts/e2e_playwright.py --scenario starter,settings --base-url http://127.0.0.1:8012
python scripts/couple_product_audit.py --base-url http://127.0.0.1:8012 --json-out /tmp/kink_couple_audit.json

scripts/e2e_smoke.py is now legacy/manual Selenium coverage, not the primary maintained browser gate.

To review the current starter deck without mutating user data:

python scripts/audit_starter_deck.py --store data/store_slim.db --limit 24

Live deployment

• Hub: huggingface.co/spaces/Perplexed7675/kink-discovery
• App URL: perplexed7675-kink-discovery.hf.space (same-origin API + static UI)
• Account persistence: pick one option so user profiles survive restarts:
  • Paid: enable HF Persistent Storage on the Space and set KINK_STORE_PATH=/data/store_slim.db + KINK_FAIL_ON_EPHEMERAL_STORE=1. Zero engineering, perfect durability.
  • Free (Hub-snapshot): create a private HF Hub dataset (e.g. Perplexed7675/kink-userstate), then set Space variables KINK_USER_SNAPSHOT_REPO=Perplexed7675/kink-userstate and HF_TOKEN=<write-token>. The container then pulls the latest user-state snapshot on cold boot and pushes back every 60s when the user tables change (catalog stays on /tmp). Tunables: KINK_USER_SNAPSHOT_FILENAME (default user_state.db), KINK_USER_SNAPSHOT_INTERVAL_S (default 60). Trade-off: up to ~60s of writes can be lost on a hard crash; the Space must stay single-replica (cpu-basic is).

Re-verify anytime:

HF_VERIFY_BASE_URL=https://perplexed7675-kink-discovery.hf.space python scripts/verify_hf_stack.py

After frontend changes, bump the ?v= query on frontend/index.html styles + app.js so browsers load the new bundle, then redeploy (command below).

Deploy to Hugging Face (working Space)

1. Create a Hub access token with write access (or reuse [huggingface].token from the parser repo’s secrets.toml — same file scripts/publish_hf_space.py reads when HF_TOKEN is unset; path: KINK_PARSER_SECRETS_TOML or ~/PycharmProjects/parser/secrets.toml).
2. From the repo root:

# Either export credentials, or rely on parser secrets.toml + [huggingface].username → <username>/kink-discovery
export HF_TOKEN=hf_...
export HF_SPACE_REPO=yourname/your-space-name   # e.g. Perplexed7675/kink-discovery
python scripts/publish_hf_space.py --verify

Release order: run tests (make test), then—only if you changed the catalog object or B2/Hub secrets—scripts/sync_hf_space_b2_catalog.py and/or scripts/deploy_hf_bootstrap_catalog.py; then publish_hf_space.py --verify. Full checklist: deploy/hf/README.md.

This uploads the project (Docker SDK), waits until the Space is RUNNING, resolves the direct app URL (*.hf.space), and runs the same checks as verify_hf_stack.py against the live deployment (including COEP on /health).

To only upload and print the verify command:

python scripts/publish_hf_space.py

To prove a Space that is already deployed:

HF_VERIFY_BASE_URL=https://yourname-yourspace.hf.space python scripts/verify_hf_stack.py

See deploy/hf/README.md for persistence details.