preprocessing/README.md

ShastraDocs Preprocessing Package

An advanced document preprocessing pipeline for RAG (Retrieval-Augmented Generation) systems. This modular package handles document ingestion, text extraction, chunking, embedding generation, and vector storage for multiple document formats.

🚀 Features

Document Format Support

• PDF: Advanced text extraction with table handling and CID font support (Malayalam, complex scripts)
• DOCX: Complete Word document processing with tables and text boxes
• PPTX: PowerPoint extraction with OCR for images using OCR Space API
• XLSX: Excel spreadsheet processing with image OCR support
• Images: PNG, JPEG, JPG with table detection and OCR
• Plain Text: TXT and CSV file support
• URLs: Direct URL processing and Google Docs conversion

Advanced Processing Capabilities

• Smart Text Chunking: Sentence-boundary aware chunking with configurable overlap
• Embedding Generation: Sentence transformer-based embeddings with batch processing
• Vector Storage: Qdrant integration for efficient similarity search
• Table Extraction: Automated table detection and formatting
• OCR Integration: OCR Space API for image text extraction
• Metadata Management: Comprehensive document metadata tracking
• Parallel Processing: Multi-threaded document processing
• Caching: Intelligent caching to avoid reprocessing

📁 Package Structure

preprocessing/
├── __init__.py                    # Package initialization
├── preprocessing.py               # Main entry point and CLI
└── preprocessing_modules/
    ├── __init__.py
    ├── modular_preprocessor.py    # Main orchestrator class
    ├── file_downloader.py         # Universal file downloading
    ├── pdf_extractor.py           # PDF text extraction
    ├── docx_extractor.py          # DOCX processing
    ├── pptx_extractor.py          # PowerPoint processing
    ├── xlsx_extractor.py          # Excel processing
    ├── image_extractor.py         # Image and table extraction
    ├── text_chunker.py            # Smart text chunking
    ├── embedding_manager.py       # Embedding generation
    ├── vector_storage.py          # Qdrant vector database
    └── metadata_manager.py        # Document metadata management

🛠️ Installation

Dependencies

Note: these packages are already included in requirements.txt of the project

# Core dependencies
pip install aiohttp asyncio numpy pandas pathlib
pip install sentence-transformers qdrant-client
pip install pdfplumber pymupdf python-docx python-pptx openpyxl
pip install opencv-python pytesseract pillow lxml

# For image processing
pip install opencv-python pytesseract pillow

# For document parsing
pip install pdfplumber pymupdf python-docx python-pptx openpyxl lxml

Environment Variables

Create a .env file with the following:

# Required for PowerPoint OCR
OCR_SPACE_API_KEY=your_ocr_space_api_key

# Optional: Custom paths
OUTPUT_DIR=./vector_db
EMBEDDING_MODEL=Bge-large-en #or any model
CHUNK_SIZE=1000
CHUNK_OVERLAP=200
BATCH_SIZE=32

🔧 Configuration

The package uses config/config.py for configuration:

# Embedding configuration
EMBEDDING_MODEL = "Bge-large-en"  # Sentence transformer model
BATCH_SIZE = 32                       # Embedding batch size

# Chunking configuration
CHUNK_SIZE = 1600                     # Characters per chunk
CHUNK_OVERLAP = 500                   # Overlap between chunks

# Storage configuration
OUTPUT_DIR = "./vector_db"            # Vector database directory

# OCR configuration (for PPTX images)
OCR_SPACE_API_KEY = "your_api_key"    # OCR Space API key

📖 Usage

Basic Usage

from preprocessing import ModularDocumentPreprocessor

# Initialize preprocessor
preprocessor = ModularDocumentPreprocessor()

# Process a single document
doc_id = await preprocessor.process_document("https://example.com/document.pdf")

# Process multiple documents
urls = [
    "https://example.com/doc1.pdf",
    "https://example.com/doc2.docx",
    "https://example.com/presentation.pptx"
]
results = await preprocessor.process_multiple_documents(urls)

# Check processing status
info = preprocessor.get_document_info("https://example.com/document.pdf")
print(f"Document processed: {info}")

Document Types and Return Values

# Different document types return different formats
result = await preprocessor.process_document(url)

# Regular documents (PDF, DOCX, TXT)
if isinstance(result, str):
    doc_id = result  # Normal processing, returns document ID

# Special cases
elif isinstance(result, list):
    content, doc_type = result[0], result[1]
    
    if doc_type == 'oneshot':
        # Small documents processed as single chunk
        # Use content directly with LLM
        
    elif doc_type == 'tabular':
        # Excel/CSV with structured data
        # Use content for data analysis
        
    elif doc_type == 'image':
        # Image file - content is file path
        # Process with image_extractor
        
    elif doc_type == 'unsupported':
        # File format not supported
        print(f"Unsupported format: {content}")

Advanced Usage

# Force reprocessing
doc_id = await preprocessor.process_document(url, force_reprocess=True)

# Custom timeout for large files
doc_id = await preprocessor.process_document(url, timeout=600)  # 10 minutes

# Get system information
system_info = preprocessor.get_system_info()
print(f"Embedding model: {system_info['embedding_model']}")

# Get collection statistics
stats = preprocessor.get_collection_stats()
print(f"Total documents: {stats['total_documents']}")
print(f"Total chunks: {stats['total_chunks']}")

# List all processed documents
docs = preprocessor.list_processed_documents()
for doc_id, info in docs.items():
    print(f"{doc_id}: {info['document_url']} ({info['chunk_count']} chunks)")

# Cleanup document
success = preprocessor.cleanup_document(url)

Image Processing

from preprocessing_modules.image_extractor import extract_image

# Extract text and tables from images
text_content = extract_image("path/to/image.png")
print(text_content)

# Output format:
# ### Non-Table Text:
# Regular text content from the image
# 
# ### Table 1 (Markdown):
# | Column 1 | Column 2 | Column 3 |
# |----------|----------|----------|
# | Data 1   | Data 2   | Data 3   |

🎯 Command Line Interface

# Process a single document
python -m preprocessing --url "https://example.com/document.pdf"

# Process multiple documents from file
python -m preprocessing --urls-file urls.txt

# Force reprocessing
python -m preprocessing --url "https://example.com/document.pdf" --force

# List processed documents
python -m preprocessing --list

# Show collection statistics
python -m preprocessing --stats

URLs File Format

https://example.com/doc1.pdf
https://example.com/doc2.docx
https://example.com/presentation.pptx
https://docs.google.com/document/d/abc123/edit?usp=sharing

🏗️ Architecture

Modular Design

The package follows a modular architecture with clear separation of concerns:

1. File Downloader: Handles downloading from various sources with retry logic
2. Text Extractors: Specialized extractors for each document format
3. Text Chunker: Smart chunking with sentence boundary detection
4. Embedding Manager: Generates embeddings using sentence transformers
5. Vector Storage: Manages Qdrant vector database operations
6. Metadata Manager: Tracks document processing metadata

Processing Pipeline

URL/File → Download → Extract Text → Chunk → Generate Embeddings → Store in Qdrant
                                     ↓
                               Save Metadata

Document Processing Flow

1. Download: Securely download document to temporary location
2. Format Detection: Identify document type and select appropriate extractor
3. Text Extraction: Extract text content with format-specific handling
4. Chunking: Split text into overlapping chunks with smart boundaries
5. Embedding: Generate embeddings using sentence transformers
6. Storage: Store embeddings and metadata in Qdrant vector database
7. Cleanup: Remove temporary files and update registries

📊 Supported Formats

Format	Extension	Features	Special Handling
PDF	.pdf	Text, tables, complex scripts	CID font mapping, parallel processing
Word	.docx	Text, tables, text boxes	XML parsing, gridSpan handling
PowerPoint	.pptx	Text, images, tables, notes	OCR Space API for images
Excel	.xlsx	Cells, images	OpenPyXL, OCR for embedded images
Images	.png, .jpg, .jpeg	Text, tables	OpenCV table detection, OCR
Text	.txt, .csv	Plain text	Direct processing
URLs	http/https	Web content	Google Docs conversion

🔍 Advanced Features

Table Processing

• Automatic table detection in PDFs and images
• GridSpan handling for complex table structures
• Markdown formatting for structured output
• Cell content extraction with proper spacing

CID Font Support

• Advanced handling of Malayalam and complex scripts
• Character mapping resolution
• Proper spacing and conjunct handling
• Fallback extraction methods

OCR Integration

• OCR Space API for PowerPoint images
• Tesseract OCR for Excel images
• Batch processing for efficiency
• Error handling and fallback options

Caching System

• Document-level caching to avoid reprocessing
• Chunk caching for repeated operations
• Temporary file management
• Automatic cleanup on exit

🛡️ Error Handling

The package includes comprehensive error handling:

• Network Issues: Retry logic with exponential backoff
• Corrupted Files: Fallback extraction methods
• Memory Issues: Batch processing and streaming
• Format Issues: Multiple parser fallbacks
• OCR Failures: Graceful degradation with error messages

📈 Performance

Optimization Features

• Parallel Processing: Multi-threaded document processing
• Batch Operations: Efficient embedding generation
• Streaming: Memory-efficient large file handling
• Caching: Avoid redundant processing
• Connection Pooling: Efficient HTTP operations

Benchmarks

• PDF Processing: ~2-5 pages/second (depends on complexity)
• Embedding Generation: ~100-500 chunks/second (depends on model)
• Vector Storage: ~1000+ vectors/second insertion rate

🔧 Troubleshooting

Common Issues

1. OCR Space API Errors

   # Ensure API key is set
   export OCR_SPACE_API_KEY="your_key_here"
   

2. Tesseract Not Found

   # Install tesseract
   apt-get install tesseract-ocr
   # or
   brew install tesseract
   

3. Memory Issues with Large Files

   # Reduce batch size in config
   BATCH_SIZE = 16
   

4. Vector Database Issues

   # Check permissions on OUTPUT_DIR
   # Ensure sufficient disk space
   

Debug Mode

import logging
logging.basicConfig(level=logging.DEBUG)

# Enable detailed logging for troubleshooting

📄 License

This package is part of the ShastraDocs project. See the main project license for details.

This preprocessing package is designed to handle the complex requirements of document processing in RAG systems, with a focus on reliability, performance, and format diversity.