| --- |
| title: Ingestics SearXNG |
| emoji: 🔎 |
| colorFrom: indigo |
| colorTo: blue |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| license: agpl-3.0 |
| short_description: Private SearXNG backend for Ingestics deep research |
| --- |
| |
| # Ingestics SearXNG (Private Research Backend) |
|
|
| This Space hosts a hardened [SearXNG](https://docs.searxng.org/) instance used |
| **only** by the Ingestics web app as a deep-research backbone. It is not a |
| public search frontend; rate limiting and quality filtering happen on the |
| Cloudflare Worker side that consumes it. |
|
|
| ## What it does |
|
|
| - Aggregates results primarily from tolerant engines (DuckDuckGo, Brave, |
| Qwant, Bing, Yahoo, Mojeek) plus Wikipedia/Wikidata/PubMed/Crossref/ |
| Semantic Scholar. Google remains low-weight backup only. |
| - Exposes the SearXNG **JSON API** (`/search?q=...&format=json`) for the |
| Ingestics Cloudflare Pages worker to consume. |
| - Nginx sits in front of SearXNG with micro-caching, connection caps, and |
| request throttles. SearXNG's own limiter can be enabled when Valkey exists. |
| - Engine timeouts are tightened (4–8 s) so one slow upstream cannot pin a |
| worker thread. |
| - Music / video / social media / torrent engines are disabled because |
| they're noise for a health-info agent and tend to attract bot |
| challenges that would burn our shared HF egress IP. |
|
|
| ## Hardware |
|
|
| - HF Spaces **CPU Basic** (2 vCPU, 16 GB RAM, 50 GB ephemeral disk). |
| - Granian is tuned for I/O-heavy workloads: 8 workers × 40 threads behind |
| Nginx. The Worker-side cache/coalescing and Nginx micro-cache prevent |
| duplicate bursts from becoming upstream engine floods. |
|
|
| ## Bot / IP-ban resilience |
|
|
| Five mitigations layer on top of each other: |
|
|
| 1. **Rotating-proxy ready** — set `SEARXNG_PROXY_HTTP` and/or |
| `SEARXNG_PROXY_HTTPS` as HF secrets. The boot script injects them into |
| `outgoing.proxies`, so a premium provider can rotate IPs behind one |
| endpoint without committing credentials. |
| 2. **Nginx micro-cache** — identical JSON searches cache for 3 minutes, |
| use stale-on-error, and lock concurrent cache misses. |
| 3. **Ingress throttles** — Nginx caps per-IP request rate/connections and |
| adds a separate client-marker budget for Worker traffic. |
| 4. **Engine load distribution** — tolerant engines have higher weights; |
| Google is low-weight backup and noisy/video/torrent engines are disabled. |
| 5. **Worker-side dedup + cache** — the Cloudflare Worker coalesces identical |
| in-flight queries and 10-minute caches completed research. |
|
|
| ## Security |
|
|
| - Set HF secrets `SEARXNG_PROXY_HTTP` / `SEARXNG_PROXY_HTTPS` for the rotating |
| proxy provider endpoint. |
| - Set the HF secret `SEARXNG_API_KEY` if you want to require an API key |
| on every call. The Worker pool will send it as `X-API-KEY`. When |
| unset, the Space is effectively open — but its URL isn't publicised |
| and HF lists it as private to the operator. |
| - `SEARXNG_SECRET` is auto-generated at every cold start unless pinned |
| as a HF secret. Cookies are not used by JSON consumers anyway. |
| - `public_instance: false` keeps SearXNG's link_token middleware off so |
| our consumer parses raw URLs. |
| |
| ## Endpoints |
| |
| - `GET /` — the SearXNG search HTML page (useful for human debugging). |
| - `GET /healthz` — upstream health endpoint. |
| - `GET /search?q=...&format=json` — primary JSON API. |
| - `GET /config` — engine list + version (debug). |
| |
