_project/ARCHITECTURE.md

# ARCHITECTURE — marker

<!-- UPDATE WHEN: tech stack changes, a component is added/removed, data flow changes, or a major directory is renamed -->

## Tech stack

- **Language / runtime:** Python 3.11
- **Framework:** FastAPI + Uvicorn (single-process, GPU-bound work doesn't benefit
  from multiple workers).
- **Database:** none. Stateless.
- **Key libraries:**
  - `marker-pdf` — the actual PDF→markdown engine (note: package name is
    `marker-pdf`, import is `marker`).
  - `fastapi`, `uvicorn`, `python-multipart` — HTTP layer + multipart upload.
  - `torch` (CUDA build) — pulled in transitively by marker-pdf, must match the
    Space's CUDA version.
- **Frontend:** none. JSON API only.
- **Build / package manager:** `uv` for dependency resolution; `pip` inside the
  Docker image (smaller layer than installing `uv` into the runtime).
- **Hosting:** Hugging Face Space (`hirokamitai/trenchlesspm`, **public**),
  Docker SDK, A10 small GPU, HF Bucket `hirokamitai/trenchlesspm-cache`
  mounted at `/data` for the model cache.

## Components

- **`app/main.py`** — FastAPI app. Public: `GET /` (endpoint blurb), `GET
  /health` (liveness + gpu/cpu). Bearer-gated:
  - `POST /parse` — NDJSON streaming. Marker runs on `asyncio.to_thread`
    so the route can yield NDJSON heartbeats every 5 s while the parse is
    in flight (HF's edge proxy closes idle connections mid-parse otherwise).
    Kept for back-compat and short/CLI use.
  - `POST /parse/jobs` — submit. Writes the upload + a `state.json` to the
    job directory (see `app/jobs.py`), enqueues the job_id, and returns
    202 with the job_id in <1 s. Recommended for any non-trivial parse.
  - `GET /parse/jobs/{id}` — read job state from disk. No coupling to the
    in-memory queue, so polling clients can hit any replica safely.
- **`app/jobs.py`** — persistent job queue. Single-worker (A10 only fits
  one parse at a time), backed by `/data/jobs/<id>/state.json` on the HF
  Bucket. Lifespan-managed: on startup, re-enqueues any `pending` or
  stale-`parsing` jobs left from a prior container; runs a GC loop that
  prunes job dirs older than 24 h. Also runs a **keepalive loop** that
  self-pings `/health` every 4 min while any job is in-flight, so HF's
  autosleep timer (5 min idle on inbound HTTP) doesn't trip mid-parse if
  no client is polling.
- **`app/parser.py`** — wraps marker-pdf. Loads the marker model dict once at
  import time (module-level singleton) so subsequent requests reuse weights on
  GPU. Exposes `parse(pdf_path, mode) -> tuple[str, int]` returning
  `(markdown, page_count)`. `mode="quality"` flips marker's LLM-enhancement
  flag on and routes through `marker.services.claude.ClaudeService`
  (defaults to `claude-sonnet-4-6`).
- **`Dockerfile`** — CUDA-enabled base image, installs system deps marker
  needs, `pip install` of marker-pdf + FastAPI stack, sets `HF_HOME=/data/hf`
  and `TORCH_HOME=/data/torch` so model weights persist on the bucket. CMD
  runs Uvicorn with `--timeout-keep-alive 1800` so long parses don't drop
  the connection. Exposes port 7860 (HF Space convention).
- **`README.md`** (Space root) — HF Space frontmatter (`sdk: docker`,
  `app_port: 7860`, `sleep_time: 300`, `suggested_hardware: a10g-small`,
  `suggested_storage: small`), plus a short usage blurb.

## Data flow

```
1. Caller POSTs PDF as multipart/form-data to /parse
   with header `Authorization: Bearer <API_TOKEN>` and form fields:
     file:       the PDF
     parse_mode: "fast" (default) | "quality"

2. FastAPI handler validates the bearer token + form fields and pre-checks
   that quality mode has ANTHROPIC_API_KEY. Hard 4xx for these — they're
   deterministic config issues, not per-request runtime failures.

3. Upload streams in 1 MiB chunks to a NamedTemporaryFile so 100+ MB PDFs
   don't OOM the A10's host RAM.

4. Handler returns 200 with Content-Type: application/x-ndjson and starts
   yielding events from an async generator:
     - parser.parse(tmp_path, mode) is launched on a worker thread via
       asyncio.to_thread (marker is blocking, would otherwise stall the loop).
     - The generator wakes every 5s; if the thread is still running, it
       yields {"type":"progress","stage":"parsing","elapsed_ms":N} so the
       connection has bytes flowing and HF's edge proxy doesn't idle-close.
     - When the thread completes, one terminal event is yielded:
         {"type":"result","markdown":"...","mode":"...","page_count":N,
          "duration_ms":N}
       or {"type":"error","error":"..."} if marker raised.

5. quality mode routes the LLM-enhancement pass through Claude (Sonnet 4.6
   by default) via marker's `ClaudeService`. See _project/ENVIRONMENT.md.
   page_count comes from marker's rendered.metadata, falling back to pypdf.

6. Space autosleeps after 5 min of no requests. Next request triggers a cold
   start; model weights load from /data (HF Bucket) instead of re-downloading.
```

The streaming shape is the *only* `/parse` response shape — there is no
synchronous JSON form. Clients must consume line-by-line; the **last line**
is the terminal event, everything before it is a heartbeat.

## Key directories

| Path | Purpose |
|---|---|
| `app/` | FastAPI app code (`main.py`, `parser.py`, `jobs.py`). |
| `/data/` (runtime, on the Space) | Mount point for HF Bucket `hirokamitai/trenchlesspm-cache`. Holds every cache marker writes: `hf/` (HF Hub), `torch/` (Torch hub), `cache/` (XDG fallback — covers `datalab` text-detection + OCR-error-detection models), and `datalab/` (explicit datalab override). Also `/data/jobs/<job_id>/` for the job queue's state + inputs (auto-GC'd at 24 h). Not in this repo. |

## External touchpoints

- **Hugging Face Hub** — marker pulls its layout / OCR models from the Hub on
  first cold start; cached to `/data/hf` (HF Bucket mount) thereafter. No auth
  needed for public models.
- **Anthropic API (Claude)** — only called when `parse_mode=quality`. Marker
  uses its built-in `ClaudeService`; we pin `claude-sonnet-4-6`. API key
  stored as a Space secret. See [DEPENDENCIES.md](DEPENDENCIES.md).

## Known sharp edges

- **Cold start is slow.** First request after sleep waits on container boot +
  model load from `/data` (~30–60s). This is by design for cost; don't try
  to paper over it with a 24/7 keepalive. (The job worker's *conditional*
  keepalive — only while a parse is in flight — is fine and necessary.)
- **Model cache MUST land on `/data`.** If the bucket isn't mounted, every
  cold start re-downloads ~5 GB of weights into ephemeral container disk
  (lost on the next sleep). The Dockerfile sets `HF_HOME=/data/hf` and
  `TORCH_HOME=/data/torch` so marker writes through the bucket.
- **Public Space, bearer auth.** The Space URL is reachable without HF auth.
  Real access control is the bearer-token check in `app/main.py` — `/parse`
  401s without `Authorization: Bearer <API_TOKEN>`. `/` and `/health` are
  intentionally open and leak nothing.
- **Package name vs import name.** `pip install marker-pdf`, but `import marker`.
  Easy to get wrong.
- **Single-process, GPU-bound.** Don't run multiple Uvicorn workers — they'd each
  try to load the marker models onto the same GPU and OOM. Concurrency comes
  from the Space scaling, not in-process workers.
- **Synchronous from the GPU's perspective; streaming over the wire.** A
  long parse still holds the HTTP connection open for the duration, but
  the response body is NDJSON heartbeats every 5 s, so HF's edge proxy
  doesn't idle-close. Callers still need a generous client-side timeout
  (a few minutes for big contracts in quality mode), and they MUST consume
  the body line-by-line — `await response.json()` will not work.
- **A10 small VRAM (24 GB) is enough for marker** including LLM-enhancement
  *post-processing*, but the LLM call itself is external — don't try to run a
  local LLM on the same GPU.