Fetching metadata from the HF Docker repository... Today
-
Scale to 1M+ docs: SQLite core, FAISS auto-tiering, hybrid retrieval, PII protection, ingestion worker queue
- Storage: single SQLite (WAL) core (documents/chunks/chunk_vectors/tombstones); BLOB embeddings as source of truth
- Retrieval: hybrid dense FAISS + bm25s sparse with Reciprocal Rank Fusion; optional cross-encoder rerank; auto tiering flat -> IVF/SQ8 with compaction via tombstones
- Ingestion: streaming batched embed->index pipeline, O(N) chunker fix, persistent SQLite-backed worker queue with job status (survives restarts)
- PII protection: India/USA/global detectors with checksum validators (Aadhaar Verhoeff, card Luhn), reversible placeholder masking at every LLM boundary (mappings memory-only), custom keyword/regex rules persisted, log masking, optional NER (presidio)
- UI: new PII Protection and Ingestion Queue Gradio tabs; MCP tools scan_pii, pii_custom_rule_add/remove/list, ingest_directory_queue, ingest_queued, job_status, list_ingest_jobs
- Ops: SCALING.md (architecture, capacity math, env knobs, ops playbook), migrate + benchmark scripts, 13 offline tests (pipeline, RRF, IVF…
