_project/ARCHITECTURE.md

ARCHITECTURE — marker

Tech stack

• Language / runtime: Python 3.11
• Framework: FastAPI + Uvicorn (single-process, GPU-bound work doesn't benefit from multiple workers).
• Database: none. Stateless.
• Key libraries:
  • marker-pdf — the actual PDF→markdown engine (note: package name is marker-pdf, import is marker).
  • fastapi, uvicorn, python-multipart — HTTP layer + multipart upload.
  • torch (CUDA build) — pulled in transitively by marker-pdf, must match the Space's CUDA version.
• Frontend: none. JSON API only.
• Build / package manager: uv for dependency resolution; pip inside the Docker image (smaller layer than installing uv into the runtime).
• Hosting: Hugging Face Space (hirokamitai/trenchlesspm, public), Docker SDK, A10 small GPU, HF Bucket hirokamitai/trenchlesspm-cache mounted at /data for the model cache.

Components

• app/main.py — FastAPI app. Public: GET / (endpoint blurb), GET /health (liveness + gpu/cpu). Bearer-gated:
  • POST /parse — NDJSON streaming. Marker runs on asyncio.to_thread so the route can yield NDJSON heartbeats every 5 s while the parse is in flight (HF's edge proxy closes idle connections mid-parse otherwise). Kept for back-compat and short/CLI use.
  • POST /parse/jobs — submit. Writes the upload + a state.json to the job directory (see app/jobs.py), enqueues the job_id, and returns 202 with the job_id in <1 s. Recommended for any non-trivial parse.
  • GET /parse/jobs/{id} — read job state from disk. No coupling to the in-memory queue, so polling clients can hit any replica safely.
• app/jobs.py — persistent job queue. Single-worker (A10 only fits one parse at a time), backed by /data/jobs/<id>/state.json on the HF Bucket. Lifespan-managed: on startup, re-enqueues any pending or stale-parsing jobs left from a prior container; runs a GC loop that prunes job dirs older than 24 h. Also runs a keepalive loop that self-pings /health every 4 min while any job is in-flight, so HF's autosleep timer (5 min idle on inbound HTTP) doesn't trip mid-parse if no client is polling.
• app/parser.py — wraps marker-pdf. Loads the marker model dict once at import time (module-level singleton) so subsequent requests reuse weights on GPU. Exposes parse(pdf_path, mode) -> tuple[str, int] returning (markdown, page_count). mode="quality" flips marker's LLM-enhancement flag on and routes through marker.services.claude.ClaudeService (defaults to claude-sonnet-4-6).
• Dockerfile — CUDA-enabled base image, installs system deps marker needs, pip install of marker-pdf + FastAPI stack, sets HF_HOME=/data/hf and TORCH_HOME=/data/torch so model weights persist on the bucket. CMD runs Uvicorn with --timeout-keep-alive 1800 so long parses don't drop the connection. Exposes port 7860 (HF Space convention).
• README.md (Space root) — HF Space frontmatter (sdk: docker, app_port: 7860, sleep_time: 300, suggested_hardware: a10g-small, suggested_storage: small), plus a short usage blurb.

Data flow

1. Caller POSTs PDF as multipart/form-data to /parse
   with header `Authorization: Bearer <API_TOKEN>` and form fields:
     file:       the PDF
     parse_mode: "fast" (default) | "quality"

2. FastAPI handler validates the bearer token + form fields and pre-checks
   that quality mode has ANTHROPIC_API_KEY. Hard 4xx for these — they're
   deterministic config issues, not per-request runtime failures.

3. Upload streams in 1 MiB chunks to a NamedTemporaryFile so 100+ MB PDFs
   don't OOM the A10's host RAM.

4. Handler returns 200 with Content-Type: application/x-ndjson and starts
   yielding events from an async generator:
     - parser.parse(tmp_path, mode) is launched on a worker thread via
       asyncio.to_thread (marker is blocking, would otherwise stall the loop).
     - The generator wakes every 5s; if the thread is still running, it
       yields {"type":"progress","stage":"parsing","elapsed_ms":N} so the
       connection has bytes flowing and HF's edge proxy doesn't idle-close.
     - When the thread completes, one terminal event is yielded:
         {"type":"result","markdown":"...","mode":"...","page_count":N,
          "duration_ms":N}
       or {"type":"error","error":"..."} if marker raised.

5. quality mode routes the LLM-enhancement pass through Claude (Sonnet 4.6
   by default) via marker's `ClaudeService`. See _project/ENVIRONMENT.md.
   page_count comes from marker's rendered.metadata, falling back to pypdf.

6. Space autosleeps after 5 min of no requests. Next request triggers a cold
   start; model weights load from /data (HF Bucket) instead of re-downloading.

The streaming shape is the only /parse response shape — there is no synchronous JSON form. Clients must consume line-by-line; the last line is the terminal event, everything before it is a heartbeat.

Key directories

Path	Purpose
app/	FastAPI app code (main.py, parser.py, jobs.py).
/data/ (runtime, on the Space)	Mount point for HF Bucket hirokamitai/trenchlesspm-cache. Holds every cache marker writes: hf/ (HF Hub), torch/ (Torch hub), cache/ (XDG fallback — covers datalab text-detection + OCR-error-detection models), and datalab/ (explicit datalab override). Also /data/jobs/<job_id>/ for the job queue's state + inputs (auto-GC'd at 24 h). Not in this repo.

External touchpoints

• Hugging Face Hub — marker pulls its layout / OCR models from the Hub on first cold start; cached to /data/hf (HF Bucket mount) thereafter. No auth needed for public models.
• Anthropic API (Claude) — only called when parse_mode=quality. Marker uses its built-in ClaudeService; we pin claude-sonnet-4-6. API key stored as a Space secret. See DEPENDENCIES.md.

Known sharp edges

• Cold start is slow. First request after sleep waits on container boot + model load from /data (~30–60s). This is by design for cost; don't try to paper over it with a 24/7 keepalive. (The job worker's conditional keepalive — only while a parse is in flight — is fine and necessary.)
• Model cache MUST land on /data. If the bucket isn't mounted, every cold start re-downloads ~5 GB of weights into ephemeral container disk (lost on the next sleep). The Dockerfile sets HF_HOME=/data/hf and TORCH_HOME=/data/torch so marker writes through the bucket.
• Public Space, bearer auth. The Space URL is reachable without HF auth. Real access control is the bearer-token check in app/main.py — /parse 401s without Authorization: Bearer <API_TOKEN>. / and /health are intentionally open and leak nothing.
• Package name vs import name. pip install marker-pdf, but import marker. Easy to get wrong.
• Single-process, GPU-bound. Don't run multiple Uvicorn workers — they'd each try to load the marker models onto the same GPU and OOM. Concurrency comes from the Space scaling, not in-process workers.
• Synchronous from the GPU's perspective; streaming over the wire. A long parse still holds the HTTP connection open for the duration, but the response body is NDJSON heartbeats every 5 s, so HF's edge proxy doesn't idle-close. Callers still need a generous client-side timeout (a few minutes for big contracts in quality mode), and they MUST consume the body line-by-line — await response.json() will not work.
• A10 small VRAM (24 GB) is enough for marker including LLM-enhancement post-processing, but the LLM call itself is external — don't try to run a local LLM on the same GPU.