CHANGELOG.md

Changelog

All notable changes to this project are documented here. The format is based on Keep a Changelog and the project adheres to Semantic Versioning.

[1.1.0]

Added

• Layered architecture: api/, core/, analytics/, safety/, ui/.
• Settings module (pydantic-settings) — single source of truth for env vars.
• App factory create_app() with FastAPI lifespan for clean startup/shutdown.
• LLM registry with TTL cache + hash key — reuses MarkItDown(LLM) clients per (api_key, base_url, model) triple, with a connection-pooled httpx.Client per OpenAI client.
• Streaming uploads to disk in a worker thread; no double-write to tmp.
• HTML cleanup powered by selectolax (strips <style>, <script>, hidden / aria-hidden / display:none nodes before MarkItDown sees them).
• SSRF guard: every URL is resolved and rejected if the host is private, loopback, link-local, multicast or unspecified.
• Anti-crash safety: ConversionGate (bounded concurrency + queue), PerIPLimiter (per-IP fairness), ResourceGuard (RAM/disk preflight), CircuitBreaker, MemoryWatchdog, DiskCleanup.
• Endpoints: GET /v1/healthz, GET /v1/readyz, GET /v1/stats.
• Request-ID middleware with ContextVar propagation into JSON logs.
• Security headers middleware (X-Content-Type-Options, X-Frame-Options, Referrer-Policy).
• Body-size limit middleware enforcing DOCSIFER_MAX_UPLOAD_BYTES.
• ORJSON default response class + GZip middleware (>1KB).
• Refined Gradio UI: Soft theme, two-column layout, status pill, metric summary (chars / KB / tokens), auto-loaded stats tab.
• Multi-stage slim Dockerfile (Python 3.11) with HEALTHCHECK, BuildKit cache mounts, gunicorn + uvicorn workers with request recycling, HF Spaces-compatible non-root user.
• GitHub Actions CI workflow: ruff + pytest + Docker build smoke (only on push to main).
• Hugging Face Space auto-sync workflow gated on CI success (workflow_run trigger).
• pre-commit config (ruff, mypy, basic hygiene hooks).
• Makefile shortcuts for install / lint / format / type / test / cov / run.
• Comprehensive test suite: unit tests for html_cleaner, url_guard, tokenizer, llm_registry, periods, safety, analytics_service; integration tests for /v1/healthz, /v1/readyz, /v1/convert, /v1/stats, SSRF rejection.
• Comprehensive README.md, ARCHITECTURE.md, CONTRIBUTING.md.

Changed

• requirements.txt and pyproject.toml synchronised; dependencies pinned.
• Analytics persistence is opt-in: DOCSIFER_REDIS_URL empty by default → in-memory store (HF Spaces works out of the box).
• Analytics period keys use ISO calendar (%G-W%V) and survive year boundaries; flush is snapshot-then-swap and pipelined to Upstash.
• Default response class is ORJSONResponse (3–10× faster JSON encoding).
• Logging routed through dictConfig with optional JSON output.
• datetime.utcnow() replaced by datetime.now(timezone.utc) everywhere.
• CORS: allow_credentials is automatically disabled when origins contain * (otherwise non-compliant per the CORS spec).

Fixed

• A1: Gradio UI was always forwarding OpenAI config even when no API key was provided — now only forwarded when a key is present.
• A2: HTTP cookies were dropped because MarkItDown was instantiated without a requests.Session; cookies now reach the upstream request.
• A3: Path-traversal via crafted upload filenames — sanitized with Path(name).name.
• A4: SSRF via URL conversion — see new core/url_guard.py.
• A5/A6: Analytics race condition on concurrent flushes; pending counters are now restored on flush failure.
• A7: Deprecated FastAPI on_event replaced with lifespan.
• A8: Wrong ISO week formatting at year boundaries.
• A9: Invalid CORS combination allow_credentials=True with *.
• A10: Gradio UI was looping back through HTTP to itself — now calls the in-process service directly.
• A11/A13/A14: HTML conversion no longer double-writes the body to disk; cleaned HTML is streamed via BytesIO.
• F4: 500 errors echoed str(exception) to clients — now generic with request_id in the response.
• F5: Server-side extension allowlist (not just UI-side).
• F6: Internal exception messages no longer leak.
• C6: Body streaming is now async-safe (handled in a worker thread).
• LLM client cache no longer leaks: TTL eviction + bounded size.

Removed

• Legacy flat modules docsifer/router.py, docsifer/service.py, docsifer/analytics.py.
• Eager / module-level side effects on import — replaced by lifespan.

[1.0.0]

• Initial public version.