Spaces:
Running on Zero
Running on Zero
Commit History
2 days ago
-
Fix voice QA + podcast pipeline (llama_index workflow API, PII boundary, honest failures)
953fef2 -
feat: persistence across Space restarts (snapshot backup + /data autodetect)
- services/backup_service.py: tar.gz snapshot of the whole data tree (WAL-checkpointed SQLite + FAISS + BM25 + uploads) pushed to a private HF dataset repo; restore on boot only into an empty dir so newer local data is never clobbered; traversal-guarded extraction; oversize files (podcast audio, PERSIST_BACKUP_MAX_FILE_MB) excluded. - config.py: auto-redirect DATA_DIR/DB_PATH to the /data mount when HF Persistent Storage is attached; PERSIST_BACKUP_* knobs. - bootstrap: restore() before stores open in __init__ (never fails boot) and debounced flush_if_due() in the maintenance loop whenever library.db changed - vectors/deletes/uploads survive restarts within ~2 min of any change. - tests/test_backup_service.py: round-trip + traversal-refusal + disabled-without-secrets + max-MB trim (offline, stubbed network). - SCALING.md: persistence section + honest '100 docs in 1 min on GPU?' stage table (GPU speeds encode; parse/OCR stays CPU-bound).23c41aa -
feat: multi-file upload to queue; honest 1M/1min throughput notes; crash recovery
- Upload widget accepts multiple files; each is archived + enqueued as its own background job and the request returns instantly (verified 3 at once locally). Message lists job ids per file. - SCALING.md: 'Ingestion speed reality' section with the actual math - 1M docs/1min would need ~500k-1M chunk-embeddings/s (hundreds of GPUs or precomputed snapshots); table comparing CPU/API/GPU/cluster paths and the wired levers. - bootstrap: register the queue worker task on the service so worker_alive reports truthfully (was always 'stopped'). - ingestion_queue_service: crash-recovery sweep requeues stale 'running' jobs at worker boot (claim_next only picked 'pending' before), with idempotency test (18/18).4021ac2
4 days ago
-
Scale to 1M+ docs: SQLite core, FAISS auto-tiering, hybrid retrieval, PII protection, ingestion worker queue
- Storage: single SQLite (WAL) core (documents/chunks/chunk_vectors/tombstones); BLOB embeddings as source of truth - Retrieval: hybrid dense FAISS + bm25s sparse with Reciprocal Rank Fusion; optional cross-encoder rerank; auto tiering flat -> IVF/SQ8 with compaction via tombstones - Ingestion: streaming batched embed->index pipeline, O(N) chunker fix, persistent SQLite-backed worker queue with job status (survives restarts) - PII protection: India/USA/global detectors with checksum validators (Aadhaar Verhoeff, card Luhn), reversible placeholder masking at every LLM boundary (mappings memory-only), custom keyword/regex rules persisted, log masking, optional NER (presidio) - UI: new PII Protection and Ingestion Queue Gradio tabs; MCP tools scan_pii, pii_custom_rule_add/remove/list, ingest_directory_queue, ingest_queued, job_status, list_ingest_jobs - Ops: SCALING.md (architecture, capacity math, env knobs, ops playbook), migrate + benchmark scripts, 13 offline tests (pipeline, RRF, IVF…106c7b9