Spaces:
Running on Zero
Running on Zero
| title: zeroshotGPU | |
| sdk: gradio | |
| app_file: app.py | |
| python_version: 3.11 | |
| suggested_hardware: l4x1 | |
| short_description: Agentic zero-shot document parser with GT metrics + repair. | |
| # Zero-Shot GPU Document Parser | |
| A self-hosted parsing control plane that profiles documents, routes pages to | |
| parser experts, normalizes outputs, verifies quality with GT-comparison | |
| metrics, repairs weak regions through a bounded verify/repair loop (with | |
| optional GPU escalation), and emits auditable parsed-document artifacts plus | |
| strategy-aware chunks. Implements the project described in | |
| [`zero_shot_gpu_document_parser_project_spec.md`](zero_shot_gpu_document_parser_project_spec.md). | |
| The codebase is intentionally dependency-light by default. Text and Markdown | |
| work with the standard library; PyMuPDF, Docling, Marker, MinerU, olmOCR, | |
| PaddleOCR, and Unstructured plug in via optional extras. Live GPU repair | |
| (Qwen2.5-VL-3B) and the embedding retriever (jina-embeddings-v3) are gated | |
| behind explicit config flags so a fresh clone never silently downloads | |
| multi-gigabyte weights. | |
| --- | |
| ## Install | |
| For the local MVP (text + PyMuPDF + Docling): | |
| ```bash | |
| python -m pip install -e ".[pdf,yaml,docling,dev]" | |
| ``` | |
| Optional extras: | |
| | Extra | Adds | Required for | | |
| |---------------|--------------------------------------------------|-----------------------------------------------| | |
| | `embedding` | `sentence-transformers`, `transformers` | `benchmarks.retriever.backend=embedding` | | |
| | `gpu_repair` | `transformers` | `repair.execute_gpu_escalations=true` | | |
| | `spaces` | mirrors `requirements.txt` for HF Spaces parity | running `app.py` locally as a Space simulant | | |
| External parser CLIs (Marker, MinerU, olmOCR, PaddleOCR) install separately; | |
| configure each via `parsers.<name>.command`, `output_args`, and `extra_args` | |
| in your YAML config. | |
| Secrets: | |
| ```bash | |
| cp .env.example .env | |
| # Set HF_TOKEN if you'll use gated models (jina-embeddings-v3, private repos). | |
| ``` | |
| `.env` is gitignored. The CLI and `app.py` load it on startup; pre-set | |
| environment variables (e.g. Space-side secrets) always win. | |
| --- | |
| ## Quick start | |
| ### Parse one document or a folder | |
| ```bash | |
| python -m zsgdp.cli parse --input ./docs/sample.md --output ./out/sample | |
| python -m zsgdp.cli parse-folder --input ./docs --output ./parsed --workers 4 | |
| python -m zsgdp.cli parse --input ./docs/report.pdf --output ./out/report --config configs/docling.yaml | |
| ``` | |
| Each parse writes a full artifact bundle. `parsed_document.json` is the | |
| canonical record; `chunks.jsonl` is the retrieval-ready output; | |
| `quality_report.json` carries every metric the verifier computed. | |
| ### Run a benchmark | |
| ```bash | |
| # Custom corpus, no GT — runs every metric that doesn't need labels: | |
| python -m zsgdp.cli benchmark --input ./docs --output ./bench | |
| # Labelled datasets — adds layout F1 / table structure / formula CER: | |
| python -m zsgdp.cli benchmark --input ./omnidocbench --dataset omnidocbench --output ./bench/omni | |
| python -m zsgdp.cli benchmark --input ./doclaynet --dataset doclaynet --output ./bench/doclay | |
| ``` | |
| ### Compare parsers (ablation) | |
| ```bash | |
| python -m zsgdp.cli benchmark-ablate \ | |
| --input ./docs --output ./bench/ablation \ | |
| --parser docling --parser pymupdf --parser text | |
| ``` | |
| Runs the benchmark once per parser plus a merged arm; emits | |
| `ablation_comparison.csv`. | |
| ### Compare across datasets | |
| ```bash | |
| python -m zsgdp.cli combine-benchmarks \ | |
| --input ./bench/omni --label omnidocbench \ | |
| --input ./bench/doclay --label doclaynet \ | |
| --output ./bench/cross | |
| ``` | |
| Emits `dataset_summary.csv` and `parser_matrix.csv` (parser × dataset). | |
| ### Before pushing to a Space — preflight | |
| ```bash | |
| make preflight # unit + regression + space-check + parsers (~10s) | |
| make preflight-full # ...plus an end-to-end benchmark smoke | |
| ``` | |
| A green preflight is the local signal that the branch is ready for the | |
| Space. Pre-commit and pre-push hooks (see [CONTRIBUTING.md](CONTRIBUTING.md)) | |
| make this automatic on every `git push`. | |
| ### On the Space — smoke validation | |
| Once deployed, exercise the deferred GPU/model paths: | |
| ```bash | |
| make space-smoke # runs whichever of 5 smokes have their deps | |
| python -m scripts.run_space_smoke --strict --output ./space_smoke.json | |
| ``` | |
| See [docs/space_smoke.md](docs/space_smoke.md) for the manual fallback | |
| procedure (real PDF uploads, full Marker parses) and per-smoke | |
| acceptance criteria. | |
| --- | |
| ## Opt-ins | |
| ### Embedding retriever | |
| Default retriever is lexical TF-IDF (zero deps). To use a real embedder: | |
| ```yaml | |
| # configs/myrun.yaml | |
| benchmarks: | |
| retriever: | |
| backend: embedding | |
| model_id: jinaai/jina-embeddings-v3 # or any sentence-transformers model | |
| task: retrieval.passage | |
| ``` | |
| ```bash | |
| python -m pip install -e ".[embedding]" | |
| python -m zsgdp.cli benchmark --input ./docs --output ./bench --config configs/myrun.yaml | |
| ``` | |
| The first call lazy-loads the model; subsequent calls reuse it in-process. | |
| Set `HF_TOKEN` in `.env` for gated models. | |
| ### Live GPU repair | |
| The repair controller plans GPU tasks for verification failures (invalid | |
| tables, OCR coverage gaps, reading-order issues, missing figure captions). | |
| By default these are dry-run only. To execute: | |
| ```yaml | |
| # configs/myrun.yaml | |
| repair: | |
| gpu_escalation: true | |
| execute_gpu_escalations: true # invokes the configured backend | |
| gpu: | |
| backend: transformers # or "vllm" for OpenAI-compat | |
| models: | |
| table: | |
| model_id: Qwen/Qwen2.5-VL-3B-Instruct | |
| ``` | |
| Each executed task writes its output back into the merged document with a | |
| `gpu_repair_task_id` provenance field. | |
| --- | |
| ## Outputs | |
| Every parse writes: | |
| - `parsed_document.json` — canonical record (carries `schema_version`). | |
| - `document.md` — human-readable Markdown reconstruction. | |
| - `elements.jsonl` / `tables.jsonl` / `figures.jsonl` / `chunks.jsonl` — JSONL streams. | |
| - `chunking_plan.json` — strategy ladder + per-strategy metadata. | |
| - `parser_metrics.json` — per-parser candidate-level stats. | |
| - `quality_report.json` — every verifier metric (text coverage, reading order, table validity, parser disagreement, repair resolution/regression rates, GT-comparison metrics when applicable). | |
| - `routing_report.json` — page → parser routing decisions. | |
| - `profile.json` — document profiler output. | |
| - `gpu_runtime.json` — detected GPU/device state at parse time. | |
| - `gpu_tasks.jsonl` (when model-backed work is planned) and `gpu_task_report.json` (preflight validation). | |
| - `conflict_report.json` (when multiple parsers ran). | |
| - `artifact_manifest.json` with SHA-256 checksums and the parsed-document schema version. | |
| - `assets/pages/*.png`, `assets/tables/*.png`, `assets/figures/*.png` — rendered PDF page and region crops. | |
| Benchmark runs additionally write: | |
| - `results.json` — full structured summary including aggregate means. | |
| - `leaderboard.csv` and `per_parser_gt_leaderboard.csv` — parser leaderboards (without and with GT comparison). | |
| - `per_parser_metrics.csv` — per-document, per-parser GT-comparison breakdown. | |
| - `layout_runs.csv`, `table_structure_runs.csv`, `formula_runs.csv`, `retrieval_runs.csv`, `repair_runs.csv` — per-document detail per metric family. | |
| - `parser_runs.csv`, `chunk_runs.csv`, `structure_runs.csv`, `chunk_quality.csv`, `throughput_runs.csv`, `ablations.json` — additional detail. | |
| `benchmark-ablate` adds `ablation_comparison.csv`. `combine-benchmarks` | |
| adds `dataset_summary.csv`, `parser_matrix.csv`, and | |
| `cross_dataset_comparison.json`. | |
| --- | |
| ## Architecture map | |
| | Module | Responsibility | | |
| |-------------------------|-------------------------------------------------------------------------| | |
| | `zsgdp/profiling/` | Cheap per-page features (scanned-score, table density, columns, etc.) | | |
| | `zsgdp/routing/` | Deterministic page → parser-expert decisions with budget | | |
| | `zsgdp/parsers/` | Adapters; one canonical schema regardless of source | | |
| | `zsgdp/normalize/` | Convert each parser's output into the schema | | |
| | `zsgdp/merge/` | Align candidates, dedupe, detect conflicts | | |
| | `zsgdp/verify/` | Coverage / reading order / table / figure / formula / chunk readiness, plus GT-comparison: layout F1, table structure, formula CER, retrieval recall, parser disagreement, repair success | | |
| | `zsgdp/repair/` | Deterministic header/table fixes plus GPU escalation through `gpu/worker.py` | | |
| | `zsgdp/chunking/` | Agentic planner + structure / parent-child / table / figure / page chunkers, with semantic / late / vision / proposition deterministic stubs | | |
| | `zsgdp/gpu/` | Task planning, batching, dry-run worker, transformers + vLLM clients | | |
| | `zsgdp/benchmarks/` | Dataset loaders, metric runners, ablation, cross-dataset, retrieval | | |
| | `zsgdp/cli.py` | All entry points | | |
| | `app.py` | Gradio Space UI | | |
| The full spec is in [`zero_shot_gpu_document_parser_project_spec.md`](zero_shot_gpu_document_parser_project_spec.md). §10 (schema) and §17 (chunking ladder) are the most useful sections to skim before touching those modules. | |
| --- | |
| ## Production benchmark numbers | |
| Spec §29 success criteria for reference: | |
| - **MVP:** full agentic loop improves table QA by ≥20% over best single parser; agentic chunking improves citation accuracy by ≥10% over recursive baseline. | |
| - **Production-style (HR / financial reports / etc.):** retrieval recall@5 ≥ 90%, citation accuracy ≥ 90%, table QA exactness ≥ 85%, manual review rate ≤ 10%, parser blocking failure rate ≤ 5%. | |
| ### Smoke validation — `arjun10g/zeroshotGPU` ZeroGPU Space, `2026-05-07` | |
| Ran via `/gradio_api/call/run_smokes_in_space` against commit `de03f34`. | |
| | Smoke | Status | Detail | | |
| |---------------|--------|-----------------------------------------------------------------------------------| | |
| | `lexical` | pass | `mean_quality_score=0.972`, `mean_retrieval_recall_at_1=1.0`, n=1 doc | | |
| | `ablation` | pass | 3 arms (text / pymupdf / merged), comparison CSV emitted | | |
| | `embedding` | pass | `mean_retrieval_recall_at_5=1.0`, `mean_retrieval_recall_at_1=1.0`, 17.06s on A10g | | |
| | `gpu_repair` | pass | wiring verified: `dry_run_task_count=1`, `repair_iterations=1` | | |
| | `marker` | skip | `marker_single` not installed (expected; opt-in via `requirements.txt`) | | |
| ### Benchmark — regression fixtures, `2026-05-07` | |
| Ran via `/gradio_api/call/run_benchmark_in_space` against commit `abadf36`. | |
| | Metric | Value | Notes | | |
| |---------------------------------|--------|--------------------------------------------------------| | |
| | `document_count` | 1 | `markdown_basic.input.md` only | | |
| | `mean_quality_score` | 0.964 | passes the §29 quality bar trivially on this fixture | | |
| | `mean_layout_f1` | 0.000 | no GT for `custom_folder` dataset | | |
| | `mean_table_structure_score` | 0.000 | no GT | | |
| | `mean_formula_cer` | 0.000 | no GT | | |
| | `mean_retrieval_recall_at_1` | 0.000 | first-rank miss; truth at rank 3 (single-doc corpus) | | |
| | `mean_retrieval_recall_at_5` | 1.000 | found within top-5 | | |
| | `mean_retrieval_mrr` | 0.333 | implies truth rank ≈ 3 | | |
| | `mean_parser_disagreement_rate` | 0.000 | only one parser (`text`) ran on `.md` | | |
| | `mean_repair_resolution_rate` | 1.000 | every blocking issue pre-repair was resolved | | |
| | `mean_repair_regression_rate` | 0.000 | repair introduced no new blocking issues | | |
| **Reading the numbers:** this fixture-only benchmark validates the metric pipeline end-to-end on production hardware. The `mean_layout_f1`/`table_structure_score`/`formula_cer` zeros are because `custom_folder` has no GT — those metrics activate when running against `omnidocbench` or `doclaynet`. To benchmark a real corpus, drop documents into a Space-side directory and call `run_benchmark_in_space` (or run `zsgdp benchmark --input ./corpus --output ./bench` from a Pro-tier Dev Mode terminal). | |
| ### Live GPU repair invocation — verified, deferred end-to-end | |
| `Live GPU repair` pipeline mode loads `configs/live_gpu_repair.yaml`, runs the iterative verify/repair loop, detects `cuda` runtime, and dispatches GPU tasks when needed. On committed test fixtures, the deterministic markdown-table normalizer resolves invalid-table issues without needing the GPU — `repair_resolution_rate=1.0` after one iteration, zero GPU tasks executed. | |
| Genuinely firing the live `Qwen2.5-VL-3B` invocation requires either a PDF with bbox-aware table extraction that pymupdf can't deterministically normalize, or a config that explicitly skips `repair.table_repair` so the GPU path is the only resolution route. Both are tractable follow-ups when a real malformed PDF is available; the wiring on production is confirmed up to that point. | |
| --- | |
| ## Deployment | |
| Targeted: Hugging Face Spaces, hardware `l4x1`, GPU/model target | |
| `zeroshotGPU`. | |
| Pre-deploy gate: | |
| 1. `make preflight` (local). | |
| 2. `make preflight-full` (local with end-to-end benchmark smoke). | |
| 3. Duplicate the Space, set `HF_TOKEN` and any other secrets in **Variables and secrets**. | |
| 4. Push. | |
| 5. `make space-smoke` from the Space's JupyterLab terminal. | |
| 6. Inspect [docs/space_smoke.md](docs/space_smoke.md) Smoke 3 (live GPU repair) manually if the runner-level wiring smoke passed but you want full model-invocation validation. | |
| 7. Run `python -m zsgdp.cli benchmark` against your representative corpus and update the table above. | |
| The Space defaults to `configs/docling.yaml` (Docling + PyMuPDF | |
| co-enabled so the parser disagreement rate has signal). Override via | |
| `ZSGDP_CONFIG_PATH` in Space variables for custom configs. | |
| --- | |
| ## Contributing | |
| See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, hooks, test layout, | |
| fixture format, parser/metric/schema-bump procedures, and the PR checklist. | |
| For changes touching the on-disk schema, bump `zsgdp.schema.SCHEMA_VERSION` | |
| and add an entry under `### Schema` in [CHANGELOG.md](CHANGELOG.md). The | |
| artifact manifest surfaces the version under | |
| `parsed_document_schema_version` so downstream consumers can gate. | |