--- title: zeroshotGPU sdk: gradio app_file: app.py python_version: 3.11 suggested_hardware: l4x1 short_description: Agentic zero-shot document parser with GT metrics + repair. --- # Zero-Shot GPU Document Parser A self-hosted parsing control plane that profiles documents, routes pages to parser experts, normalizes outputs, verifies quality with GT-comparison metrics, repairs weak regions through a bounded verify/repair loop (with optional GPU escalation), and emits auditable parsed-document artifacts plus strategy-aware chunks. Implements the project described in [`zero_shot_gpu_document_parser_project_spec.md`](zero_shot_gpu_document_parser_project_spec.md). The codebase is intentionally dependency-light by default. Text and Markdown work with the standard library; PyMuPDF, Docling, Marker, MinerU, olmOCR, PaddleOCR, and Unstructured plug in via optional extras. Live GPU repair (Qwen2.5-VL-3B) and the embedding retriever (jina-embeddings-v3) are gated behind explicit config flags so a fresh clone never silently downloads multi-gigabyte weights. --- ## Install For the local MVP (text + PyMuPDF + Docling): ```bash python -m pip install -e ".[pdf,yaml,docling,dev]" ``` Optional extras: | Extra | Adds | Required for | |---------------|--------------------------------------------------|-----------------------------------------------| | `embedding` | `sentence-transformers`, `transformers` | `benchmarks.retriever.backend=embedding` | | `gpu_repair` | `transformers` | `repair.execute_gpu_escalations=true` | | `spaces` | mirrors `requirements.txt` for HF Spaces parity | running `app.py` locally as a Space simulant | External parser CLIs (Marker, MinerU, olmOCR, PaddleOCR) install separately; configure each via `parsers..command`, `output_args`, and `extra_args` in your YAML config. Secrets: ```bash cp .env.example .env # Set HF_TOKEN if you'll use gated models (jina-embeddings-v3, private repos). ``` `.env` is gitignored. The CLI and `app.py` load it on startup; pre-set environment variables (e.g. Space-side secrets) always win. --- ## Quick start ### Parse one document or a folder ```bash python -m zsgdp.cli parse --input ./docs/sample.md --output ./out/sample python -m zsgdp.cli parse-folder --input ./docs --output ./parsed --workers 4 python -m zsgdp.cli parse --input ./docs/report.pdf --output ./out/report --config configs/docling.yaml ``` Each parse writes a full artifact bundle. `parsed_document.json` is the canonical record; `chunks.jsonl` is the retrieval-ready output; `quality_report.json` carries every metric the verifier computed. ### Run a benchmark ```bash # Custom corpus, no GT — runs every metric that doesn't need labels: python -m zsgdp.cli benchmark --input ./docs --output ./bench # Labelled datasets — adds layout F1 / table structure / formula CER: python -m zsgdp.cli benchmark --input ./omnidocbench --dataset omnidocbench --output ./bench/omni python -m zsgdp.cli benchmark --input ./doclaynet --dataset doclaynet --output ./bench/doclay ``` ### Compare parsers (ablation) ```bash python -m zsgdp.cli benchmark-ablate \ --input ./docs --output ./bench/ablation \ --parser docling --parser pymupdf --parser text ``` Runs the benchmark once per parser plus a merged arm; emits `ablation_comparison.csv`. ### Compare across datasets ```bash python -m zsgdp.cli combine-benchmarks \ --input ./bench/omni --label omnidocbench \ --input ./bench/doclay --label doclaynet \ --output ./bench/cross ``` Emits `dataset_summary.csv` and `parser_matrix.csv` (parser × dataset). ### Before pushing to a Space — preflight ```bash make preflight # unit + regression + space-check + parsers (~10s) make preflight-full # ...plus an end-to-end benchmark smoke ``` A green preflight is the local signal that the branch is ready for the Space. Pre-commit and pre-push hooks (see [CONTRIBUTING.md](CONTRIBUTING.md)) make this automatic on every `git push`. ### On the Space — smoke validation Once deployed, exercise the deferred GPU/model paths: ```bash make space-smoke # runs whichever of 5 smokes have their deps python -m scripts.run_space_smoke --strict --output ./space_smoke.json ``` See [docs/space_smoke.md](docs/space_smoke.md) for the manual fallback procedure (real PDF uploads, full Marker parses) and per-smoke acceptance criteria. --- ## Opt-ins ### Embedding retriever Default retriever is lexical TF-IDF (zero deps). To use a real embedder: ```yaml # configs/myrun.yaml benchmarks: retriever: backend: embedding model_id: jinaai/jina-embeddings-v3 # or any sentence-transformers model task: retrieval.passage ``` ```bash python -m pip install -e ".[embedding]" python -m zsgdp.cli benchmark --input ./docs --output ./bench --config configs/myrun.yaml ``` The first call lazy-loads the model; subsequent calls reuse it in-process. Set `HF_TOKEN` in `.env` for gated models. ### Live GPU repair The repair controller plans GPU tasks for verification failures (invalid tables, OCR coverage gaps, reading-order issues, missing figure captions). By default these are dry-run only. To execute: ```yaml # configs/myrun.yaml repair: gpu_escalation: true execute_gpu_escalations: true # invokes the configured backend gpu: backend: transformers # or "vllm" for OpenAI-compat models: table: model_id: Qwen/Qwen2.5-VL-3B-Instruct ``` Each executed task writes its output back into the merged document with a `gpu_repair_task_id` provenance field. --- ## Outputs Every parse writes: - `parsed_document.json` — canonical record (carries `schema_version`). - `document.md` — human-readable Markdown reconstruction. - `elements.jsonl` / `tables.jsonl` / `figures.jsonl` / `chunks.jsonl` — JSONL streams. - `chunking_plan.json` — strategy ladder + per-strategy metadata. - `parser_metrics.json` — per-parser candidate-level stats. - `quality_report.json` — every verifier metric (text coverage, reading order, table validity, parser disagreement, repair resolution/regression rates, GT-comparison metrics when applicable). - `routing_report.json` — page → parser routing decisions. - `profile.json` — document profiler output. - `gpu_runtime.json` — detected GPU/device state at parse time. - `gpu_tasks.jsonl` (when model-backed work is planned) and `gpu_task_report.json` (preflight validation). - `conflict_report.json` (when multiple parsers ran). - `artifact_manifest.json` with SHA-256 checksums and the parsed-document schema version. - `assets/pages/*.png`, `assets/tables/*.png`, `assets/figures/*.png` — rendered PDF page and region crops. Benchmark runs additionally write: - `results.json` — full structured summary including aggregate means. - `leaderboard.csv` and `per_parser_gt_leaderboard.csv` — parser leaderboards (without and with GT comparison). - `per_parser_metrics.csv` — per-document, per-parser GT-comparison breakdown. - `layout_runs.csv`, `table_structure_runs.csv`, `formula_runs.csv`, `retrieval_runs.csv`, `repair_runs.csv` — per-document detail per metric family. - `parser_runs.csv`, `chunk_runs.csv`, `structure_runs.csv`, `chunk_quality.csv`, `throughput_runs.csv`, `ablations.json` — additional detail. `benchmark-ablate` adds `ablation_comparison.csv`. `combine-benchmarks` adds `dataset_summary.csv`, `parser_matrix.csv`, and `cross_dataset_comparison.json`. --- ## Architecture map | Module | Responsibility | |-------------------------|-------------------------------------------------------------------------| | `zsgdp/profiling/` | Cheap per-page features (scanned-score, table density, columns, etc.) | | `zsgdp/routing/` | Deterministic page → parser-expert decisions with budget | | `zsgdp/parsers/` | Adapters; one canonical schema regardless of source | | `zsgdp/normalize/` | Convert each parser's output into the schema | | `zsgdp/merge/` | Align candidates, dedupe, detect conflicts | | `zsgdp/verify/` | Coverage / reading order / table / figure / formula / chunk readiness, plus GT-comparison: layout F1, table structure, formula CER, retrieval recall, parser disagreement, repair success | | `zsgdp/repair/` | Deterministic header/table fixes plus GPU escalation through `gpu/worker.py` | | `zsgdp/chunking/` | Agentic planner + structure / parent-child / table / figure / page chunkers, with semantic / late / vision / proposition deterministic stubs | | `zsgdp/gpu/` | Task planning, batching, dry-run worker, transformers + vLLM clients | | `zsgdp/benchmarks/` | Dataset loaders, metric runners, ablation, cross-dataset, retrieval | | `zsgdp/cli.py` | All entry points | | `app.py` | Gradio Space UI | The full spec is in [`zero_shot_gpu_document_parser_project_spec.md`](zero_shot_gpu_document_parser_project_spec.md). §10 (schema) and §17 (chunking ladder) are the most useful sections to skim before touching those modules. --- ## Production benchmark numbers Spec §29 success criteria for reference: - **MVP:** full agentic loop improves table QA by ≥20% over best single parser; agentic chunking improves citation accuracy by ≥10% over recursive baseline. - **Production-style (HR / financial reports / etc.):** retrieval recall@5 ≥ 90%, citation accuracy ≥ 90%, table QA exactness ≥ 85%, manual review rate ≤ 10%, parser blocking failure rate ≤ 5%. ### Smoke validation — `arjun10g/zeroshotGPU` ZeroGPU Space, `2026-05-07` Ran via `/gradio_api/call/run_smokes_in_space` against commit `de03f34`. | Smoke | Status | Detail | |---------------|--------|-----------------------------------------------------------------------------------| | `lexical` | pass | `mean_quality_score=0.972`, `mean_retrieval_recall_at_1=1.0`, n=1 doc | | `ablation` | pass | 3 arms (text / pymupdf / merged), comparison CSV emitted | | `embedding` | pass | `mean_retrieval_recall_at_5=1.0`, `mean_retrieval_recall_at_1=1.0`, 17.06s on A10g | | `gpu_repair` | pass | wiring verified: `dry_run_task_count=1`, `repair_iterations=1` | | `marker` | skip | `marker_single` not installed (expected; opt-in via `requirements.txt`) | ### Benchmark — regression fixtures, `2026-05-07` Ran via `/gradio_api/call/run_benchmark_in_space` against commit `abadf36`. | Metric | Value | Notes | |---------------------------------|--------|--------------------------------------------------------| | `document_count` | 1 | `markdown_basic.input.md` only | | `mean_quality_score` | 0.964 | passes the §29 quality bar trivially on this fixture | | `mean_layout_f1` | 0.000 | no GT for `custom_folder` dataset | | `mean_table_structure_score` | 0.000 | no GT | | `mean_formula_cer` | 0.000 | no GT | | `mean_retrieval_recall_at_1` | 0.000 | first-rank miss; truth at rank 3 (single-doc corpus) | | `mean_retrieval_recall_at_5` | 1.000 | found within top-5 | | `mean_retrieval_mrr` | 0.333 | implies truth rank ≈ 3 | | `mean_parser_disagreement_rate` | 0.000 | only one parser (`text`) ran on `.md` | | `mean_repair_resolution_rate` | 1.000 | every blocking issue pre-repair was resolved | | `mean_repair_regression_rate` | 0.000 | repair introduced no new blocking issues | **Reading the numbers:** this fixture-only benchmark validates the metric pipeline end-to-end on production hardware. The `mean_layout_f1`/`table_structure_score`/`formula_cer` zeros are because `custom_folder` has no GT — those metrics activate when running against `omnidocbench` or `doclaynet`. To benchmark a real corpus, drop documents into a Space-side directory and call `run_benchmark_in_space` (or run `zsgdp benchmark --input ./corpus --output ./bench` from a Pro-tier Dev Mode terminal). ### Live GPU repair invocation — verified, deferred end-to-end `Live GPU repair` pipeline mode loads `configs/live_gpu_repair.yaml`, runs the iterative verify/repair loop, detects `cuda` runtime, and dispatches GPU tasks when needed. On committed test fixtures, the deterministic markdown-table normalizer resolves invalid-table issues without needing the GPU — `repair_resolution_rate=1.0` after one iteration, zero GPU tasks executed. Genuinely firing the live `Qwen2.5-VL-3B` invocation requires either a PDF with bbox-aware table extraction that pymupdf can't deterministically normalize, or a config that explicitly skips `repair.table_repair` so the GPU path is the only resolution route. Both are tractable follow-ups when a real malformed PDF is available; the wiring on production is confirmed up to that point. --- ## Deployment Targeted: Hugging Face Spaces, hardware `l4x1`, GPU/model target `zeroshotGPU`. Pre-deploy gate: 1. `make preflight` (local). 2. `make preflight-full` (local with end-to-end benchmark smoke). 3. Duplicate the Space, set `HF_TOKEN` and any other secrets in **Variables and secrets**. 4. Push. 5. `make space-smoke` from the Space's JupyterLab terminal. 6. Inspect [docs/space_smoke.md](docs/space_smoke.md) Smoke 3 (live GPU repair) manually if the runner-level wiring smoke passed but you want full model-invocation validation. 7. Run `python -m zsgdp.cli benchmark` against your representative corpus and update the table above. The Space defaults to `configs/docling.yaml` (Docling + PyMuPDF co-enabled so the parser disagreement rate has signal). Override via `ZSGDP_CONFIG_PATH` in Space variables for custom configs. --- ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, hooks, test layout, fixture format, parser/metric/schema-bump procedures, and the PR checklist. For changes touching the on-disk schema, bump `zsgdp.schema.SCHEMA_VERSION` and add an entry under `### Schema` in [CHANGELOG.md](CHANGELOG.md). The artifact manifest surfaces the version under `parsed_document_schema_version` so downstream consumers can gate.