Add PDF text extraction reliability review

PDF_TEXT_EXTRACTION_RELIABILITY_REVIEW.md

@@ -0,0 +1,533 @@
+# PDF text extraction reliability review
+
+Workpackage: `WP15 — PDF text extraction reliability review only`  
+Repository: `solidprivacy-nl/scrub`  
+Status: completed review/specification only
+
+This document evaluates whether Scrub should support PDF input in the reinsert workflow.
+
+No code, UI, tests, dependencies, export behavior, Scrub Key behavior, AI calls, OCR or cloud processing are changed by this review.
+
+---
+
+## Executive recommendation
+
+Scrub should **not** implement full PDF reinsert now.
+
+Scrub may later consider **text-based PDF input to restored TXT output only**, but only as a helper-first, test-first feature with strong warnings and audit output.
+
+Recommended product position for the next phase:
+
+```text
+B/C — text-extraction-only PDF input, producing restored TXT preview/download.
+```
+
+More explicitly:
+
+```text
+PDF input -> local text extraction -> deterministic Scrub Key reinsert -> restored TXT output + audit.
+```
+
+Not recommended now:
+
+```text
+PDF input -> restored PDF output
+PDF input -> restored DOCX output
+OCR for scanned PDFs
+Automatic layout reconstruction
+```
+
+DOCX remains the preferred document-level reinsert path.
+
+---
+
+## 1. PDF types and reliability
+
+### Text-based PDFs
+
+Text-based PDFs contain selectable text. These are the only PDFs that are realistic candidates for a future limited extraction path.
+
+Even then, reliability is not guaranteed. The visual order on the page can differ from the extracted text order. Placeholders may be split, duplicated, hidden, hyphenated or encoded differently from their visible form.
+
+### Scanned or image-only PDFs
+
+Scanned PDFs are images. They do not contain usable text unless an OCR layer exists or OCR is performed.
+
+Scrub should treat scanned/image-only PDFs as unsupported for reinsert for now. Adding OCR would create a separate accuracy, dependency, privacy and user-expectation problem.
+
+### OCR-dependent PDFs
+
+Some PDFs contain a text layer produced by OCR. The user sees an image, while extraction reads OCR text behind it.
+
+This is risky because OCR may introduce wrong characters, missing brackets, confused numbers, broken line order or incomplete text. A placeholder such as `[PERSOON_1]` may become `[PERS0ON_1]`, `[PERSOON l]`, or be split across text fragments.
+
+### PDFs generated from Word
+
+Word-generated PDFs may preserve a usable text layer, but layout elements such as headers, footers, page numbers, footnotes, tables and columns may extract in an unexpected order.
+
+A Word-generated PDF is often less reliable for reinsert than the original DOCX. For reliable document-level reinsert, Scrub should prefer DOCX whenever possible.
+
+### PDFs generated by AI tools
+
+AI tools may generate PDFs through unknown conversion pipelines. Their PDFs may contain odd text layers, duplicated text, hidden content, visual text that differs from extracted text, or paragraph/order changes.
+
+This makes AI-generated PDF output a weak foundation for high-trust legal restoration.
+
+### PDFs with columns, tables, footnotes, headers, footers or page numbers
+
+These layouts are high risk for extraction. Text may be returned by coordinate order, stream order or internal object order, not legal reading order.
+
+Risks include:
+
+- two columns interleaving line by line;
+- table rows becoming a long unreadable sentence;
+- footnotes appearing in the middle of body text;
+- headers and footers appearing repeatedly;
+- page numbers entering the restored text;
+- captions and exhibit labels moving away from their context.
+
+### PDFs with hidden or duplicated text layers
+
+Some PDFs contain duplicated text for accessibility, search, OCR or rendering reasons. Extraction may return both visible and hidden text.
+
+This can produce duplicate placeholders and duplicate restored values, which may mislead users.
+
+### PDFs where reading order differs from visual order
+
+PDF is a presentation format, not a reliable semantic document format. The internal text order can differ from what a human reads on screen.
+
+That matters for legal documents because paragraph order, headings, footnotes, tables and references often carry legal meaning.
+
+---
+
+## 2. Extraction risks
+
+### Incomplete text extraction
+
+Some text may not be extracted at all. This can happen with scanned pages, images, vector text, unsupported fonts, form fields, annotations, headers, footers or unusual PDF structures.
+
+If a placeholder is not extracted, Scrub cannot restore it.
+
+### Broken reading order
+
+The extracted text may be in the wrong order. This is especially likely with multi-column layouts, tables, footnotes and positioned text blocks.
+
+Broken order can make the restored TXT output legally confusing or misleading.
+
+### Missing headers and footers
+
+PDF extraction tools may include, exclude or reorder headers and footers. This creates uncertainty around document titles, page identifiers, case references and repeated legal context.
+
+### Duplicated text
+
+Text may be duplicated because of hidden layers, OCR layers, accessibility layers or repeated rendering objects.
+
+If a placeholder appears twice in extracted text but once visually, the audit may show extra replacements and the restored output may appear over-complete.
+
+### Split placeholders
+
+A placeholder can be split across PDF text objects, for example:
+
+```text
+[PERS
+OON_1]
+```
+
+or:
+
+```text
+[PERSOON_
+1]
+```
+
+The current deterministic placeholder replacement expects intact placeholder tokens. Split placeholders may not be restored.
+
+### Changed placeholder characters
+
+PDF extraction may alter brackets, underscores, hyphens, digits or letters. Examples include smart punctuation, substituted glyphs, OCR confusion and encoding changes.
+
+If `[ZAAKNUMMER_1]` is extracted as `[ZAAKNUMMER-l]` or `[ZAAKNUMMER 1]`, Scrub should report an unknown or missing placeholder rather than guessing.
+
+### Ligatures or encoding issues
+
+PDFs can contain ligatures or custom font encodings. Extracted text may differ from visible text. That may damage names, legal terms or placeholders.
+
+### Hyphenation and line wrapping
+
+PDF text extraction may preserve hard line breaks and hyphenation. A word split at line end may be extracted as two parts. This can affect readability and placeholder detection.
+
+### Tables becoming unreadable
+
+Tables in PDFs often lose cell boundaries during extraction. A table row can become a sequence of unrelated values. A restored TXT output should never be presented as a faithful table reconstruction.
+
+### Loss of legal formatting and context
+
+PDF extraction loses or weakens legal formatting such as headings, numbering, indentation, footnotes, quote blocks, signatures, exhibits and page layout.
+
+This can change how a legal reader interprets the result.
+
+### False completeness
+
+The biggest user risk is that a user assumes the restored output is complete because Scrub produced a file.
+
+For PDF text extraction, Scrub must communicate that output is a convenience preview/download, not a guaranteed complete restored legal document.
+
+---
+
+## 3. Privacy and legal risk
+
+Restored output may contain original confidential or personal values again. That means the output must be treated as sensitive.
+
+PDF extraction can miss placeholders. It can also produce unknown placeholder-like tokens or duplicates. The audit must clearly show:
+
+- placeholders from the key that were not found;
+- unknown placeholders found in extracted text;
+- duplicate placeholders in the key;
+- validation issues;
+- whether extraction was text-based only;
+- whether OCR was used, which should be `False` for the recommended next phase.
+
+The user must manually check the full restored result before sharing, filing, sending or using it in legal work.
+
+Privacy boundaries:
+
+- no cloud OCR;
+- no external PDF conversion service;
+- no AI calls;
+- no external document processing;
+- no real personal data in tests;
+- deterministic local processing only;
+- Scrub Key remains local and protected.
+
+---
+
+## 4. Product strategy
+
+### Option A — Unsupported for reinsert
+
+This is the safest product option.
+
+Pros:
+
+- avoids misleading users;
+- avoids unreliable extraction;
+- reinforces DOCX/TXT as trusted paths.
+
+Cons:
+
+- users with PDF-only AI output must manually copy text or convert outside Scrub;
+- creates workflow friction.
+
+Assessment:
+
+```text
+Safe baseline, acceptable until a helper-only extraction path is proven.
+```
+
+### Option B — Text-extraction-only, output as TXT
+
+This means Scrub accepts a text-based PDF, extracts text locally, reinserts values, and returns restored TXT.
+
+Pros:
+
+- realistic limited scope;
+- avoids layout reconstruction claims;
+- can reuse existing deterministic text reinsert logic;
+- audit can show limitations clearly.
+
+Cons:
+
+- extraction can still be incomplete or reordered;
+- users may still over-trust the output if warnings are weak.
+
+Assessment:
+
+```text
+Best candidate for a future helper-only spike.
+```
+
+### Option C — PDF input -> restored TXT output
+
+This is the practical product form of option B.
+
+Recommended future behavior:
+
+```text
+Upload text-based PDF
+Extract text locally
+Run Scrub Key reinsert
+Show restored TXT preview
+Allow restored TXT download
+Show audit and limitations
+```
+
+Assessment:
+
+```text
+Recommended next implementation direction, if PDF support is pursued.
+```
+
+### Option D — PDF input -> restored DOCX output
+
+Not recommended now.
+
+PDF-to-DOCX reconstruction is not a simple reinsert operation. It implies layout interpretation, table reconstruction, paragraph reconstruction and potential semantic changes.
+
+Assessment:
+
+```text
+Out of scope.
+```
+
+### Option E — PDF input -> restored PDF output
+
+Not recommended now.
+
+A restored PDF would imply visual fidelity. Achieving this safely would require layout reconstruction, font handling, coordinate mapping, metadata handling and possibly OCR. This is far beyond the safe reinsert helper model.
+
+Assessment:
+
+```text
+Out of scope.
+```
+
+### Product recommendation
+
+Recommended status:
+
+```text
+Do not support full PDF reinsert now.
+Keep PDF unsupported for document-level reinsert until text extraction is proven.
+Consider only text-based PDF input -> restored TXT output as a future helper-only package.
+Keep DOCX as the preferred document-level reinsert workflow.
+```
+
+---
+
+## 5. Technical architecture
+
+### Existing dependencies
+
+The inspected `requirements.txt` does not include a dedicated PDF extraction dependency such as `pypdf`, `pdfplumber`, `PyMuPDF`, `pdfminer.six`, `pytesseract` or OCR tooling.
+
+Therefore a future PDF extraction helper would likely require either:
+
+- a new dependency decision; or
+- use of an already transitive dependency, which should be avoided unless explicit and tested.
+
+This review does not add dependencies.
+
+### Existing helper architecture
+
+The current deterministic reinsert core is text-based:
+
+```python
+reinsert_from_scrub_key(text, scrub_key)
+```
+
+The document wrapper foundation already exists for TXT and DOCX:
+
+```python
+reinsert_text_document(text, scrub_key)
+reinsert_txt_bytes(content, scrub_key, encoding="utf-8")
+reinsert_docx_bytes(content, scrub_key)
+```
+
+A future PDF helper should not create a new replacement engine. It should extract text and then feed the extracted text into the existing deterministic text reinsert path.
+
+### Possible future helper shape
+
+A future helper-only workpackage could define something like:
+
+```python
+extract_text_from_pdf_bytes(content) -> dict
+reinsert_pdf_text_bytes(content, scrub_key) -> dict
+```
+
+The result should include:
+
+- extracted text;
+- restored text;
+- restored TXT bytes;
+- extraction warnings;
+- unsupported PDF reason when applicable;
+- placeholder audit fields from the existing reinsert helper;
+- local-only/no-AI/no-cloud fields;
+- `ocr_processing=False`.
+
+### Determinism and locality
+
+A text-based PDF extraction helper can be deterministic and local if it uses a local parser and no OCR/cloud calls.
+
+However, deterministic does not mean reliable. The same parser may consistently extract the wrong order. The audit and UI must describe this clearly.
+
+### Why PDF output reconstruction is much harder
+
+PDF output reconstruction is hard because PDF is layout-oriented. Text may be stored in small positioned fragments, not semantic paragraphs.
+
+Reconstructing a restored PDF safely would require:
+
+- preserving or recreating coordinates;
+- font and glyph handling;
+- line wrapping decisions;
+- table and column reconstruction;
+- annotation and form handling;
+- metadata decisions;
+- page-break behavior;
+- checks for hidden text layers;
+- possibly OCR.
+
+That is outside the current safe helper model.
+
+### Why scanned PDFs remain out of scope
+
+Scanned PDFs require OCR. OCR introduces:
+
+- recognition errors;
+- missed placeholders;
+- altered characters;
+- additional dependencies;
+- more compute cost;
+- new privacy questions if not strictly local;
+- stronger user warnings.
+
+OCR should remain out of scope unless separately approved in a dedicated workpackage.
+
+---
+
+## 6. UX recommendation
+
+If PDF text extraction is later added, the UI should communicate limitations before the user clicks restore.
+
+Required UX language should cover:
+
+```text
+PDF-tekstextractie is niet gegarandeerd volledig.
+Opmaak wordt niet behouden.
+De uitvoer is herstelde TXT, geen herstelde PDF.
+Controleer het volledige resultaat handmatig.
+```
+
+The UI should also show:
+
+- not-found placeholders;
+- unknown placeholders;
+- duplicate placeholders;
+- extraction warnings;
+- whether OCR was used, expected to be `nee`;
+- local-only/no-AI/no-cloud;
+- warning that restored output contains confidential or personal values again.
+
+Recommended UI position if implemented later:
+
+```text
+Originele waarden terugzetten
+-> PDF tekst-extractie, experimenteel / beperkt
+-> Herstelde TXT-preview
+-> Download herstelde TXT (.txt)
+-> Controleverslag PDF-tekstextractie
+```
+
+The UI should not offer `Download herstelde PDF`.
+
+The UI should not imply that formatting, page layout, tables, headers, footers or legal document fidelity are preserved.
+
+---
+
+## 7. Test strategy for future work
+
+No tests are implemented by this review. Future tests should be added before any PDF extraction helper is introduced.
+
+Recommended future synthetic tests:
+
+1. Text-based PDF with intact placeholders.
+2. Text-based PDF with two columns.
+3. Text-based PDF with table-like layout.
+4. Text-based PDF with header/footer placeholders.
+5. Text-based PDF with split placeholder.
+6. Text-based PDF with duplicated hidden text layer if feasible.
+7. Text-based PDF with unknown placeholder.
+8. Text-based PDF where key mappings are not found.
+9. Scanned/image-only PDF should be rejected or clearly marked unsupported.
+10. OCR-dependent PDF should not trigger OCR automatically.
+11. Audit includes local-only/no-AI/no-cloud fields.
+12. Audit includes OCR not used.
+13. Restored output is TXT, not PDF.
+14. Existing TXT/DOCX reinsert behavior remains unchanged.
+15. Existing scrubbed export/download behavior remains unchanged.
+
+Future test fixtures must use synthetic data only.
+
+---
+
+## Required recommendation answers
+
+### 1. Should Scrub implement full PDF reinsert now?
+
+No.
+
+Full PDF reinsert should not be implemented now because it would imply layout and completeness guarantees that Scrub cannot safely provide at this stage.
+
+### 2. Should Scrub implement PDF text extraction only?
+
+Not immediately in this review package.
+
+It may be considered as a future helper-only workpackage for text-based PDFs, producing restored TXT output with strong warnings and audit.
+
+### 3. Should restored PDF output remain out of scope?
+
+Yes.
+
+Restored PDF output should remain out of scope.
+
+### 4. Should OCR remain out of scope?
+
+Yes.
+
+OCR should remain out of scope unless separately approved after a dedicated reliability, privacy and dependency review.
+
+### 5. What is the safest next implementation workpackage, if any?
+
+Recommended next implementation workpackage:
+
+```text
+WP16 — Text-based PDF extraction helper spike, restored TXT output only
+```
+
+Suggested scope for WP16:
+
+- choose and justify a local PDF text extraction dependency;
+- add a pure helper only;
+- support text-based PDFs only;
+- reject or clearly mark scanned/image-only PDFs as unsupported;
+- feed extracted text into existing deterministic reinsert logic;
+- output restored TXT only;
+- add synthetic tests;
+- do not add UI yet;
+- do not add OCR;
+- do not output PDF.
+
+### 6. What should explicitly not be built yet?
+
+Do not build yet:
+
+- full restored PDF output;
+- PDF-to-DOCX reconstruction;
+- OCR;
+- cloud PDF conversion;
+- AI-based extraction;
+- UI upload controls for PDF reinsert;
+- automatic PDF rehydration;
+- promises of layout preservation;
+- batch PDF processing;
+- real-data PDF test cases.
+
+---
+
+## Final decision
+
+PDF should stay out of the main reinsert workflow until a limited text-extraction helper proves safe enough.
+
+The next safe step is not UI. It is a pure helper spike for text-based PDF extraction to restored TXT output, with strict tests, local-only processing and clear audit.
+
+DOCX remains the preferred document-level reinsert path.