Hugging Face's logo

Spaces:
Text-to-Document-Generation
/
Docgenie-API
Paused

App Files Community
Docgenie-API / TESTING_PLAN.md
Ahadhassan-2003
deploy: update HF Space
dc4e6da
preview code
|
raw
history blame contribute delete
62.2 kB

Comprehensive Testing Plan & Test Cases

DocGenie Synthetic Document Generation API

Document Version: 1.0
Date: March 4, 2026
Project: DocGenie - AI-Powered Synthetic Document Dataset Generator

Table of Contents

  1. Testing Overview
  2. Functional Testing
  3. Non-Functional Testing
  4. Test Environment Setup
  5. Testing Tools & Frameworks
  6. Test Execution Plan
  7. Success Criteria & Metrics
  8. Risk Assessment

Testing Overview

Purpose

This document outlines the comprehensive testing strategy for DocGenie API, ensuring quality, reliability, and performance of the synthetic document generation system across all 19 pipeline stages.

Scope

  • API endpoints testing (/generate, /generate/pdf, /generate/async)
  • 19-stage pipeline validation
  • External service integrations (Claude API, RunPod handwriting service)
  • Database operations (Supabase)
  • Background job processing (Redis Queue)
  • Error handling and recovery mechanisms

Testing Approach

  • Test-Driven Development (TDD): Write tests before implementation where applicable
  • Continuous Integration: Automated test execution on every commit
  • Coverage Target: Minimum 80% code coverage for critical paths
  • Risk-Based Testing: Prioritize high-risk components (LLM integration, handwriting service)

Functional Testing

A.1 Unit Testing

Unit tests verify individual functions and methods in isolation. Target: 85% code coverage.

A.1.1 Seed Image Processing (Stage 01)

Module: api/utils.py::download_seed_images()

Test Case ID Test Name Input Expected Output Priority
UT-SEED-001 Download valid image URL Valid HTTPS URL (JPEG) Base64-encoded image string High
UT-SEED-002 Download PNG format Valid PNG URL Base64-encoded PNG High
UT-SEED-003 Handle 503 timeout error URL returning 503 Retry 3 times, eventual success Critical
UT-SEED-004 Handle 502 bad gateway URL returning 502 Retry with exponential backoff High
UT-SEED-005 Handle 404 not found Invalid URL Raise HTTPException(400) High
UT-SEED-006 Handle connection timeout Slow/unresponsive server Retry then raise exception Medium
UT-SEED-007 Validate image format Non-image URL (HTML) Raise validation error Medium
UT-SEED-008 Handle oversized images >10MB image Process or reject gracefully Low
UT-SEED-009 Test retry backoff timing Mock 503 responses Delays: 2s, 4s, 8s Medium
UT-SEED-010 Test max retries exhausted Persistent 503 errors Raise exception after 3 attempts High

Test Implementation:

# test_seed_download.py
import pytest
from api.utils import download_seed_images
from unittest.mock import patch, Mock

@pytest.mark.asyncio
async def test_download_valid_image():
    url = "https://example.com/test.jpg"
    with patch('httpx.AsyncClient') as mock_client:
        mock_response = Mock()
        mock_response.content = b'\xff\xd8\xff\xe0'  # JPEG header
        mock_client.return_value.__aenter__.return_value.get.return_value = mock_response
        
        result = await download_seed_images([url])
        assert len(result) == 1
        assert isinstance(result[0], str)  # base64 string

@pytest.mark.asyncio
async def test_download_503_retry():
    url = "https://example.com/test.jpg"
    with patch('httpx.AsyncClient') as mock_client:
        # First two calls: 503, third call: success
        responses = [
            Mock(status_code=503, raise_for_status=Mock(side_effect=httpx.HTTPStatusError("503", request=Mock(), response=Mock()))),
            Mock(status_code=503, raise_for_status=Mock(side_effect=httpx.HTTPStatusError("503", request=Mock(), response=Mock()))),
            Mock(content=b'\xff\xd8\xff\xe0', raise_for_status=Mock())
        ]
        mock_client.return_value.__aenter__.return_value.get.side_effect = responses
        
        result = await download_seed_images([url])
        assert len(result) == 1
        assert mock_client.return_value.__aenter__.return_value.get.call_count == 3

A.1.2 HTML Processing (Stage 03)

Module: api/utils.py::extract_html_documents_from_response()

Test Case ID Test Name Input Expected Output Priority
UT-HTML-001 Extract single HTML LLM response with 1 HTML List with 1 HTML document High
UT-HTML-002 Extract multiple HTMLs Response with 3 HTMLs List with 3 documents High
UT-HTML-003 Extract ground truth HTML with <script id="GT"> GT JSON extracted, script removed Critical
UT-HTML-004 Handle malformed HTML Invalid HTML tags Parse with BeautifulSoup recovery Medium
UT-HTML-005 Handle missing DOCTYPE HTML without DOCTYPE Add DOCTYPE or flag error Low
UT-HTML-006 Validate CSS presence HTML without <style> Raise validation error High
UT-HTML-007 Extract handwriting markers HTML with class="handwritten" Identify 5 handwriting elements High
UT-HTML-008 Extract visual elements HTML with data-placeholder Identify 3 visual elements High
UT-HTML-009 Handle empty response Empty string from LLM Return empty list Medium
UT-HTML-010 Prettify minified HTML Single-line HTML Multi-line formatted HTML Low

A.1.3 PDF Rendering (Stage 04)

Module: api/utils.py::render_html_to_pdf()

Test Case ID Test Name Input Expected Output Priority
UT-PDF-001 Render A4 document HTML with A4 page size PDF 210×297mm High
UT-PDF-002 Render Letter size HTML with Letter page PDF 215.9×279.4mm Medium
UT-PDF-003 Extract geometries HTML with handwriting Geometries JSON with rects Critical
UT-PDF-004 Handle custom fonts HTML with @font-face PDF with embedded fonts Low
UT-PDF-005 Preserve CSS styling HTML with colors/borders PDF matches visual style Medium
UT-PDF-006 Handle images in HTML HTML with tags Images embedded in PDF Low
UT-PDF-007 Extract text coordinates HTML with paragraphs Accurate bbox coordinates High
UT-PDF-008 Handle landscape orientation HTML with landscape CSS PDF in landscape mode Low
UT-PDF-009 Validate page dimensions Various page sizes Dimensions match CSS @page High
UT-PDF-010 Handle Playwright errors Browser crash scenario Retry or graceful failure Medium

A.1.4 Bbox Extraction (Stage 05)

Module: api/utils.py::extract_bboxes_from_rendered_pdf()

Test Case ID Test Name Input Expected Output Priority
UT-BBOX-001 Extract word bboxes Standard PDF List of word-level bboxes Critical
UT-BBOX-002 Extract char bboxes Same PDF List of char-level bboxes High
UT-BBOX-003 Handle multi-line text PDF with paragraphs Correct block/line grouping High
UT-BBOX-004 Filter whitespace PDF with spaces/tabs No whitespace-only bboxes Medium
UT-BBOX-005 Handle special characters PDF with ©, ®, ™ Characters properly extracted Medium
UT-BBOX-006 Handle non-Latin scripts PDF with Chinese/Arabic Correct unicode extraction Low
UT-BBOX-007 Validate coordinates Extracted bboxes All coords within page bounds High
UT-BBOX-008 Handle empty PDF PDF with no text Return empty list Low
UT-BBOX-009 Handle rotated text PDF with rotation Bboxes account for rotation Low
UT-BBOX-010 Parse bbox strings "0_0_0 Hello 10 20 50 30" OCRBox object with correct fields High

A.1.5 Handwriting Region Extraction (Stage 07)

Module: api/utils.py::process_stage3_complete() - handwriting section

Test Case ID Test Name Input Expected Output Priority
UT-HW-001 Filter by handwriting_ratio 10 regions, ratio=0.3 ~3 regions selected Critical
UT-HW-002 Parse author IDs class="handwritten author1" author_id="author1" High
UT-HW-003 Match to word bboxes Geometry + bboxes Correct bbox mapping Critical
UT-HW-004 Handle signature class class="handwritten signature" is_signature=True Medium
UT-HW-005 DPI coordinate conversion Browser coords (96 DPI) PDF coords (72 DPI) with 0.75 scale High
UT-HW-006 Handle overlapping regions 2 regions, same text Prevent duplicate bbox usage Medium
UT-HW-007 Validate rect boundaries Geometries with rect Check bboxes within rect threshold High
UT-HW-008 Test seed reproducibility Same seed, same input Identical region selection High
UT-HW-009 Handle zero ratio ratio=0.0 No regions selected Medium
UT-HW-010 Handle full ratio ratio=1.0 All regions selected Medium

A.1.6 Handwriting Service Integration

Module: api/utils.py::call_handwriting_service_batch()

Test Case ID Test Name Input Expected Output Priority
UT-HWSVC-001 Batch request format 10 texts with metadata Correct RunPod JSON format Critical
UT-HWSVC-002 Handle sync response Immediate completion Parse output.images[] High
UT-HWSVC-003 Handle IN_PROGRESS Delayed completion Poll status endpoint Critical
UT-HWSVC-004 Status polling timeout Job exceeds 30 polls Raise timeout exception High
UT-HWSVC-005 Handle FAILED status RunPod job failure Raise exception with error High
UT-HWSVC-006 Parse image results Batch response Map hw_id to image_base64 Critical
UT-HWSVC-007 Calculate dynamic timeout 50 texts Timeout = 50×20+30 = 1030s Medium
UT-HWSVC-008 Handle network errors Connection timeout Retry up to max_retries High
UT-HWSVC-009 Validate authorization Missing API key Request includes Bearer token Medium
UT-HWSVC-010 Test exponential backoff Status polling Delays: 5s, 6s, 7s... up to 10s Low

A.1.7 Visual Element Generation (Stage 10)

Module: api/utils.py::generate_visual_element_images()

Test Case ID Test Name Input Expected Output Priority
UT-VE-001 Select logo prefab type="logo" Random logo from prefabs/ High
UT-VE-002 Select photo prefab type="photo" Random photo image High
UT-VE-003 Generate barcode type="barcode" EAN-13 barcode image Medium
UT-VE-004 Generate QR code type="qr_code", content="URL" QR code image Medium
UT-VE-005 Test seed reproducibility Same seed, same type Identical prefab selection High
UT-VE-006 Handle missing prefabs type with no files Fallback or error Medium
UT-VE-007 Load SVG prefabs SVG logo file Convert to PNG Low
UT-VE-008 Filter by requested types types=["logo","signature"] Only matching types generated High
UT-VE-009 Normalize type synonyms "chart" → "figure" Consistent type mapping Medium
UT-VE-010 Return base64 encoding All image types Valid base64 strings High

A.1.8 PDF Modification (Stages 12-13)

Module: api/utils.py::process_stage3_complete() - insertion sections

Test Case ID Test Name Input Expected Output Priority
UT-PDFMOD-001 Whiteout text regions 5 word bboxes White rectangles drawn High
UT-PDFMOD-002 Insert handwriting image Image + bbox Image at correct position Critical
UT-PDFMOD-003 Apply random offsets Word bbox Position offset within limits Medium
UT-PDFMOD-004 Resize with aspect ratio Wide/tall images Scaled to fit bbox High
UT-PDFMOD-005 Insert visual element Logo + rect Centered in bbox High
UT-PDFMOD-006 Handle rotation Element with rotation=45 Rotated image insertion Low
UT-PDFMOD-007 Save intermediate PDF After handwriting _with_handwriting.pdf created Medium
UT-PDFMOD-008 Save final PDF After visual elements _final.pdf created High
UT-PDFMOD-009 Scale factor application 3x upscale High-res image quality Medium
UT-PDFMOD-010 Handle insertion errors Invalid image data Log error, continue Medium

A.1.9 OCR Processing (Stage 15)

Module: api/utils.py::run_paddle_ocr()

Test Case ID Test Name Input Expected Output Priority
UT-OCR-001 OCR English text English document image Accurate word recognition Critical
UT-OCR-002 OCR with handwriting Mixed typed/handwritten Both text types detected High
UT-OCR-003 Extract word bboxes Document image List of word-level bboxes Critical
UT-OCR-004 Calculate confidence OCR results Confidence score per word High
UT-OCR-005 Handle low quality Blurry/noisy image Reasonable accuracy (>70%) Medium
UT-OCR-006 Handle rotated text 90° rotated document Correct orientation detection Low
UT-OCR-007 Multi-language support Document with German text lang="de" parameter works Medium
UT-OCR-008 Handle empty image Blank white image Empty results list Low
UT-OCR-009 DPI configuration Various DPI settings Consistent accuracy Medium
UT-OCR-010 Return image dimensions Any image width, height in pixels High

A.1.10 Bbox Normalization (Stage 16)

Module: api/utils.py::normalize_bboxes()

Test Case ID Test Name Input Expected Output Priority
UT-NORM-001 Normalize to [0,1] Pixel bboxes, image dims Normalized coordinates Critical
UT-NORM-002 Handle out-of-bounds x1 > image_width Clipped to [0, 1] High
UT-NORM-003 Preserve text data Bboxes with text field Text preserved in output High
UT-NORM-004 Create segment bboxes Word-level bboxes Aggregated segment bboxes Medium
UT-NORM-005 Handle zero dimensions Image with width=0 Raise validation error Low
UT-NORM-006 Round to precision Float coordinates 6 decimal places Low
UT-NORM-007 Maintain bbox order Ordered input list Same order in output Medium
UT-NORM-008 Handle negative coords bbox with x0=-5 Clipped to 0 Medium
UT-NORM-009 Validate bbox format Various input formats Consistent output schema High
UT-NORM-010 Handle empty list No bboxes Return empty list Low

A.1.11 Dataset Export (Stage 19)

Module: api/utils.py::export_to_msgpack()

Test Case ID Test Name Input Expected Output Priority
UT-EXPORT-001 Create msgpack file Complete document data Valid .msgpack file Critical
UT-EXPORT-002 Encode image bytes PNG image Binary image in msgpack High
UT-EXPORT-003 Store normalized bboxes Normalized coordinates Bboxes in [0,1] range High
UT-EXPORT-004 Store ground truth GT JSON GT dict in msgpack High
UT-EXPORT-005 Store metadata Document metadata Metadata dict in msgpack Medium
UT-EXPORT-006 Validate msgpack format Generated file Readable by msgpack.load() Critical
UT-EXPORT-007 Handle large files 10MB+ image Compression applied Low
UT-EXPORT-008 Store words list OCR words Ordered word list High
UT-EXPORT-009 Handle missing fields Partial data Fill with null/defaults Medium
UT-EXPORT-010 Return file path Export operation Absolute path to .msgpack Medium

A.1.12 Validation Functions

Module: api/utils.py::validate_*()

Test Case ID Test Name Input Expected Output Priority
UT-VAL-001 Validate HTML structure Valid HTML5 (True, None) High
UT-VAL-002 Detect missing DOCTYPE HTML without DOCTYPE (False, "Missing DOCTYPE") Medium
UT-VAL-003 Detect missing CSS HTML without