README.md

metadata

title: Ingestics SearXNG
emoji: 🔎
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: agpl-3.0
short_description: Private SearXNG backend for Ingestics deep research

Ingestics SearXNG (Private Research Backend)

This Space hosts a hardened SearXNG instance used only by the Ingestics web app as a deep-research backbone. It is not a public search frontend; rate limiting and quality filtering happen on the Cloudflare Worker side that consumes it.

What it does

• Aggregates results primarily from tolerant engines (DuckDuckGo, Brave, Qwant, Bing, Yahoo, Mojeek) plus Wikipedia/Wikidata/PubMed/Crossref/ Semantic Scholar. Google remains low-weight backup only.
• Exposes the SearXNG JSON API (/search?q=...&format=json) for the Ingestics Cloudflare Pages worker to consume.
• Nginx sits in front of SearXNG with micro-caching, connection caps, and request throttles. SearXNG's own limiter can be enabled when Valkey exists.
• Engine timeouts are tightened (4–8 s) so one slow upstream cannot pin a worker thread.
• Music / video / social media / torrent engines are disabled because they're noise for a health-info agent and tend to attract bot challenges that would burn our shared HF egress IP.

Hardware

• HF Spaces CPU Basic (2 vCPU, 16 GB RAM, 50 GB ephemeral disk).
• Granian is tuned for I/O-heavy workloads: 8 workers × 40 threads behind Nginx. The Worker-side cache/coalescing and Nginx micro-cache prevent duplicate bursts from becoming upstream engine floods.

Bot / IP-ban resilience

Five mitigations layer on top of each other:

1. Rotating-proxy ready — set SEARXNG_PROXY_HTTP and/or SEARXNG_PROXY_HTTPS as HF secrets. The boot script injects them into outgoing.proxies, so a premium provider can rotate IPs behind one endpoint without committing credentials.
2. Nginx micro-cache — identical JSON searches cache for 3 minutes, use stale-on-error, and lock concurrent cache misses.
3. Ingress throttles — Nginx caps per-IP request rate/connections and adds a separate client-marker budget for Worker traffic.
4. Engine load distribution — tolerant engines have higher weights; Google is low-weight backup and noisy/video/torrent engines are disabled.
5. Worker-side dedup + cache — the Cloudflare Worker coalesces identical in-flight queries and 10-minute caches completed research.

Security

• Set HF secrets SEARXNG_PROXY_HTTP / SEARXNG_PROXY_HTTPS for the rotating proxy provider endpoint.
• Set the HF secret SEARXNG_API_KEY if you want to require an API key on every call. The Worker pool will send it as X-API-KEY. When unset, the Space is effectively open — but its URL isn't publicised and HF lists it as private to the operator.
• SEARXNG_SECRET is auto-generated at every cold start unless pinned as a HF secret. Cookies are not used by JSON consumers anyway.
• public_instance: false keeps SearXNG's link_token middleware off so our consumer parses raw URLs.

Endpoints

• GET / — the SearXNG search HTML page (useful for human debugging).
• GET /healthz — upstream health endpoint.
• GET /search?q=...&format=json — primary JSON API.
• GET /config — engine list + version (debug).