# ARCHITECTURE — marker ## Tech stack - **Language / runtime:** Python 3.11 - **Framework:** FastAPI + Uvicorn (single-process, GPU-bound work doesn't benefit from multiple workers). - **Database:** none. Stateless. - **Key libraries:** - `marker-pdf` — the actual PDF→markdown engine (note: package name is `marker-pdf`, import is `marker`). - `fastapi`, `uvicorn`, `python-multipart` — HTTP layer + multipart upload. - `torch` (CUDA build) — pulled in transitively by marker-pdf, must match the Space's CUDA version. - **Frontend:** none. JSON API only. - **Build / package manager:** `uv` for dependency resolution; `pip` inside the Docker image (smaller layer than installing `uv` into the runtime). - **Hosting:** Hugging Face Space (`hirokamitai/trenchlesspm`, **public**), Docker SDK, A10 small GPU, HF Bucket `hirokamitai/trenchlesspm-cache` mounted at `/data` for the model cache. ## Components - **`app/main.py`** — FastAPI app. Public: `GET /` (endpoint blurb), `GET /health` (liveness + gpu/cpu). Bearer-gated: - `POST /parse` — NDJSON streaming. Marker runs on `asyncio.to_thread` so the route can yield NDJSON heartbeats every 5 s while the parse is in flight (HF's edge proxy closes idle connections mid-parse otherwise). Kept for back-compat and short/CLI use. - `POST /parse/jobs` — submit. Writes the upload + a `state.json` to the job directory (see `app/jobs.py`), enqueues the job_id, and returns 202 with the job_id in <1 s. Recommended for any non-trivial parse. - `GET /parse/jobs/{id}` — read job state from disk. No coupling to the in-memory queue, so polling clients can hit any replica safely. - **`app/jobs.py`** — persistent job queue. Single-worker (A10 only fits one parse at a time), backed by `/data/jobs//state.json` on the HF Bucket. Lifespan-managed: on startup, re-enqueues any `pending` or stale-`parsing` jobs left from a prior container; runs a GC loop that prunes job dirs older than 24 h. Also runs a **keepalive loop** that self-pings `/health` every 4 min while any job is in-flight, so HF's autosleep timer (5 min idle on inbound HTTP) doesn't trip mid-parse if no client is polling. - **`app/parser.py`** — wraps marker-pdf. Loads the marker model dict once at import time (module-level singleton) so subsequent requests reuse weights on GPU. Exposes `parse(pdf_path, mode) -> tuple[str, int]` returning `(markdown, page_count)`. `mode="quality"` flips marker's LLM-enhancement flag on and routes through `marker.services.claude.ClaudeService` (defaults to `claude-sonnet-4-6`). - **`Dockerfile`** — CUDA-enabled base image, installs system deps marker needs, `pip install` of marker-pdf + FastAPI stack, sets `HF_HOME=/data/hf` and `TORCH_HOME=/data/torch` so model weights persist on the bucket. CMD runs Uvicorn with `--timeout-keep-alive 1800` so long parses don't drop the connection. Exposes port 7860 (HF Space convention). - **`README.md`** (Space root) — HF Space frontmatter (`sdk: docker`, `app_port: 7860`, `sleep_time: 300`, `suggested_hardware: a10g-small`, `suggested_storage: small`), plus a short usage blurb. ## Data flow ``` 1. Caller POSTs PDF as multipart/form-data to /parse with header `Authorization: Bearer ` and form fields: file: the PDF parse_mode: "fast" (default) | "quality" 2. FastAPI handler validates the bearer token + form fields and pre-checks that quality mode has ANTHROPIC_API_KEY. Hard 4xx for these — they're deterministic config issues, not per-request runtime failures. 3. Upload streams in 1 MiB chunks to a NamedTemporaryFile so 100+ MB PDFs don't OOM the A10's host RAM. 4. Handler returns 200 with Content-Type: application/x-ndjson and starts yielding events from an async generator: - parser.parse(tmp_path, mode) is launched on a worker thread via asyncio.to_thread (marker is blocking, would otherwise stall the loop). - The generator wakes every 5s; if the thread is still running, it yields {"type":"progress","stage":"parsing","elapsed_ms":N} so the connection has bytes flowing and HF's edge proxy doesn't idle-close. - When the thread completes, one terminal event is yielded: {"type":"result","markdown":"...","mode":"...","page_count":N, "duration_ms":N} or {"type":"error","error":"..."} if marker raised. 5. quality mode routes the LLM-enhancement pass through Claude (Sonnet 4.6 by default) via marker's `ClaudeService`. See _project/ENVIRONMENT.md. page_count comes from marker's rendered.metadata, falling back to pypdf. 6. Space autosleeps after 5 min of no requests. Next request triggers a cold start; model weights load from /data (HF Bucket) instead of re-downloading. ``` The streaming shape is the *only* `/parse` response shape — there is no synchronous JSON form. Clients must consume line-by-line; the **last line** is the terminal event, everything before it is a heartbeat. ## Key directories | Path | Purpose | |---|---| | `app/` | FastAPI app code (`main.py`, `parser.py`, `jobs.py`). | | `/data/` (runtime, on the Space) | Mount point for HF Bucket `hirokamitai/trenchlesspm-cache`. Holds every cache marker writes: `hf/` (HF Hub), `torch/` (Torch hub), `cache/` (XDG fallback — covers `datalab` text-detection + OCR-error-detection models), and `datalab/` (explicit datalab override). Also `/data/jobs//` for the job queue's state + inputs (auto-GC'd at 24 h). Not in this repo. | ## External touchpoints - **Hugging Face Hub** — marker pulls its layout / OCR models from the Hub on first cold start; cached to `/data/hf` (HF Bucket mount) thereafter. No auth needed for public models. - **Anthropic API (Claude)** — only called when `parse_mode=quality`. Marker uses its built-in `ClaudeService`; we pin `claude-sonnet-4-6`. API key stored as a Space secret. See [DEPENDENCIES.md](DEPENDENCIES.md). ## Known sharp edges - **Cold start is slow.** First request after sleep waits on container boot + model load from `/data` (~30–60s). This is by design for cost; don't try to paper over it with a 24/7 keepalive. (The job worker's *conditional* keepalive — only while a parse is in flight — is fine and necessary.) - **Model cache MUST land on `/data`.** If the bucket isn't mounted, every cold start re-downloads ~5 GB of weights into ephemeral container disk (lost on the next sleep). The Dockerfile sets `HF_HOME=/data/hf` and `TORCH_HOME=/data/torch` so marker writes through the bucket. - **Public Space, bearer auth.** The Space URL is reachable without HF auth. Real access control is the bearer-token check in `app/main.py` — `/parse` 401s without `Authorization: Bearer `. `/` and `/health` are intentionally open and leak nothing. - **Package name vs import name.** `pip install marker-pdf`, but `import marker`. Easy to get wrong. - **Single-process, GPU-bound.** Don't run multiple Uvicorn workers — they'd each try to load the marker models onto the same GPU and OOM. Concurrency comes from the Space scaling, not in-process workers. - **Synchronous from the GPU's perspective; streaming over the wire.** A long parse still holds the HTTP connection open for the duration, but the response body is NDJSON heartbeats every 5 s, so HF's edge proxy doesn't idle-close. Callers still need a generous client-side timeout (a few minutes for big contracts in quality mode), and they MUST consume the body line-by-line — `await response.json()` will not work. - **A10 small VRAM (24 GB) is enough for marker** including LLM-enhancement *post-processing*, but the LLM call itself is external — don't try to run a local LLM on the same GPU.