---
title: Ingestics SearXNG
emoji: 🔎
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: agpl-3.0
short_description: Private SearXNG backend for Ingestics deep research
---

# Ingestics SearXNG (Private Research Backend)

This Space hosts a hardened [SearXNG](https://docs.searxng.org/) instance used
**only** by the Ingestics web app as a deep-research backbone. It is not a
public search frontend; rate limiting and quality filtering happen on the
Cloudflare Worker side that consumes it.

## What it does

- Aggregates results primarily from tolerant engines (DuckDuckGo, Brave,
  Qwant, Bing, Yahoo, Mojeek) plus Wikipedia/Wikidata/PubMed/Crossref/
  Semantic Scholar. Google remains low-weight backup only.
- Exposes the SearXNG **JSON API** (`/search?q=...&format=json`) for the
  Ingestics Cloudflare Pages worker to consume.
- Nginx sits in front of SearXNG with micro-caching, connection caps, and
  request throttles. SearXNG's own limiter can be enabled when Valkey exists.
- Engine timeouts are tightened (4–8 s) so one slow upstream cannot pin a
  worker thread.
- Music / video / social media / torrent engines are disabled because
  they're noise for a health-info agent and tend to attract bot
  challenges that would burn our shared HF egress IP.

## Hardware

- HF Spaces **CPU Basic** (2 vCPU, 16 GB RAM, 50 GB ephemeral disk).
- Granian is tuned for I/O-heavy workloads: 8 workers × 40 threads behind
  Nginx. The Worker-side cache/coalescing and Nginx micro-cache prevent
  duplicate bursts from becoming upstream engine floods.

## Bot / IP-ban resilience

Five mitigations layer on top of each other:

1. **Rotating-proxy ready** — set `SEARXNG_PROXY_HTTP` and/or
   `SEARXNG_PROXY_HTTPS` as HF secrets. The boot script injects them into
   `outgoing.proxies`, so a premium provider can rotate IPs behind one
   endpoint without committing credentials.
2. **Nginx micro-cache** — identical JSON searches cache for 3 minutes,
   use stale-on-error, and lock concurrent cache misses.
3. **Ingress throttles** — Nginx caps per-IP request rate/connections and
   adds a separate client-marker budget for Worker traffic.
4. **Engine load distribution** — tolerant engines have higher weights;
   Google is low-weight backup and noisy/video/torrent engines are disabled.
5. **Worker-side dedup + cache** — the Cloudflare Worker coalesces identical
   in-flight queries and 10-minute caches completed research.

## Security

- Set HF secrets `SEARXNG_PROXY_HTTP` / `SEARXNG_PROXY_HTTPS` for the rotating
  proxy provider endpoint.
- Set the HF secret `SEARXNG_API_KEY` if you want to require an API key
  on every call. The Worker pool will send it as `X-API-KEY`. When
  unset, the Space is effectively open — but its URL isn't publicised
  and HF lists it as private to the operator.
- `SEARXNG_SECRET` is auto-generated at every cold start unless pinned
  as a HF secret. Cookies are not used by JSON consumers anyway.
- `public_instance: false` keeps SearXNG's link_token middleware off so
  our consumer parses raw URLs.

## Endpoints

- `GET /` — the SearXNG search HTML page (useful for human debugging).
- `GET /healthz` — upstream health endpoint.
- `GET /search?q=...&format=json` — primary JSON API.
- `GET /config` — engine list + version (debug).