Upload folder using huggingface_hub

.vscode/settings.json

@@ -0,0 +1,11 @@
+{
+    "files.watcherExclude": {
+        "**/.git/objects/**": true,
+        "**/.git/subtree-cache/**": true,
+        "**/.hg/store/**": true,
+        "**/.dart_tool": true,
+        "**/.git/**": true,
+        "**/node_modules/**": true,
+        "**/.vscode/**": true
+    }
+}

README.md

@@ -0,0 +1,305 @@
+# Automated Document Text Extraction Using Small Language Model (SLM)
+
+[![Python](https://img.shields.io/badge/python-v3.8+-blue.svg)](https://www.python.org/downloads/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-v2.0+-red.svg)](https://pytorch.org/)
+[![Transformers](https://img.shields.io/badge/Transformers-v4.30+-yellow.svg)](https://huggingface.co/transformers/)
+[![FastAPI](https://img.shields.io/badge/FastAPI-v0.100+-green.svg)](https://fastapi.tiangolo.com/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+> **Intelligent document processing system that extracts structured information from invoices, forms, and scanned documents using fine-tuned DistilBERT and transfer learning.**
+
+## Quick Start
+
+### 1. Installation
+
+```bash
+# Clone the repository
+git clone https://github.com/sanjanb/small-language-model.git
+cd small-language-model
+
+# Install dependencies
+pip install -r requirements.txt
+
+# Install Tesseract OCR (Windows)
+# Download from: https://github.com/UB-Mannheim/tesseract/wiki
+# Add to PATH or set TESSERACT_PATH environment variable
+
+# Install Tesseract OCR (Ubuntu/Debian)
+sudo apt install tesseract-ocr
+
+# Install Tesseract OCR (macOS)
+brew install tesseract
+```
+
+### 2. Quick Demo
+
+```bash
+# Run the interactive demo
+python demo.py
+
+# Option 1: Complete demo with training and inference
+# Option 2: Train model only
+# Option 3: Test specific text
+```
+
+### 3. Web Interface
+
+```bash
+# Start the web API server
+python api/app.py
+
+# Open your browser to http://localhost:8000
+# Upload documents or enter text for extraction
+```
+
+## Project Overview
+
+This system combines **OCR technology**, **text preprocessing**, and a **fine-tuned DistilBERT model** to automatically extract structured information from documents. It uses transfer learning to adapt a pretrained transformer for document-specific Named Entity Recognition (NER).
+
+### Key Capabilities
+
+- **Multi-format Support**: PDF, DOCX, PNG, JPG, TIFF, BMP
+- **Dual OCR Engine**: Tesseract + EasyOCR for maximum accuracy
+- **Smart Entity Extraction**: Names, dates, amounts, addresses, phones, emails
+- **Transfer Learning**: Fine-tuned DistilBERT for document-specific tasks
+- **Web API**: RESTful endpoints with interactive interface
+- **High Accuracy**: Regex validation + ML predictions
+
+## System Architecture
+
+```mermaid
+graph TD
+    A[Document Input] --> B[OCR Processing]
+    B --> C[Text Cleaning]
+    C --> D[Tokenization]
+    D --> E[DistilBERT NER Model]
+    E --> F[Entity Extraction]
+    F --> G[Post-processing]
+    G --> H[Structured JSON Output]
+
+    I[Training Data] --> J[Auto-labeling]
+    J --> K[Model Training]
+    K --> E
+```
+
+## Project Structure
+
+```
+small-language-model/
+├── src/                    # Core source code
+│   ├── data_preparation.py  # OCR & dataset creation
+│   ├── model.py             # DistilBERT NER model
+│   ├── training_pipeline.py # Training orchestration
+│   └── inference.py         # Document processing
+├── api/                    # Web API service
+│   └── app.py              # FastAPI application
+├── config/                 # Configuration files
+│   └── settings.py         # Project settings
+├── data/                   # Data directories
+│   ├── raw/                # Input documents
+│   └── processed/          # Processed datasets
+├── models/                 # Trained models
+├── results/               # Training results
+│   ├── plots/             # Training visualizations
+│   └── metrics/           # Evaluation metrics
+├── tests/                 # Unit tests
+├── demo.py               # Interactive demo
+├── requirements.txt      # Dependencies
+└── README.md            # This file
+```
+
+## Usage Examples
+
+### Python API
+
+```python
+from src.inference import DocumentInference
+
+# Load trained model
+inference = DocumentInference("models/document_ner_model")
+
+# Process a document
+result = inference.process_document("path/to/invoice.pdf")
+print(result['structured_data'])
+# Output: {'Name': 'John Doe', 'Date': '01/15/2025', 'Amount': '$1,500.00'}
+
+# Process text directly
+result = inference.process_text_directly(
+    "Invoice sent to Alice Smith on 03/20/2025 Amount: $2,300.50"
+)
+print(result['structured_data'])
+```
+
+### REST API
+
+```bash
+# Upload and process a file
+curl -X POST "http://localhost:8000/extract-from-file" \
+     -H "accept: application/json" \
+     -H "Content-Type: multipart/form-data" \
+     -F "file=@invoice.pdf"
+
+# Process text directly
+curl -X POST "http://localhost:8000/extract-from-text" \
+     -H "Content-Type: application/json" \
+     -d '{"text": "Invoice INV-001 for John Doe $1000"}'
+```
+
+### Web Interface
+
+![Document Text Extraction Web Interface](assets/Screenshot%202025-09-27%20184723.png)
+
+1. Go to `http://localhost:8000`
+2. Choose "Upload File" or "Enter Text" tab
+3. Upload document or paste text
+4. Click "Extract Information"
+5. View structured results
+
+## Configuration
+
+### Model Configuration
+
+```python
+from src.model import ModelConfig
+
+config = ModelConfig(
+    model_name="distilbert-base-uncased",
+    max_length=512,
+    batch_size=16,
+    learning_rate=2e-5,
+    num_epochs=3,
+    entity_labels=['O', 'B-NAME', 'I-NAME', 'B-DATE', 'I-DATE', ...]
+)
+```
+
+### Environment Variables
+
+```bash
+# Optional: Custom Tesseract path
+export TESSERACT_PATH="/usr/bin/tesseract"
+
+# Optional: CUDA for GPU acceleration
+export CUDA_VISIBLE_DEVICES=0
+```
+
+## Testing
+
+```bash
+# Run all tests
+python -m pytest tests/
+
+# Run specific test module
+python tests/test_extraction.py
+
+# Test with coverage
+python -m pytest tests/ --cov=src --cov-report=html
+```
+
+## Performance Metrics
+
+| Entity Type | Precision | Recall | F1-Score |
+| ----------- | --------- | ------ | -------- |
+| NAME        | 0.95      | 0.92   | 0.94     |
+| DATE        | 0.98      | 0.96   | 0.97     |
+| AMOUNT      | 0.93      | 0.91   | 0.92     |
+| INVOICE_NO  | 0.89      | 0.87   | 0.88     |
+| EMAIL       | 0.97      | 0.94   | 0.95     |
+| PHONE       | 0.91      | 0.89   | 0.90     |
+
+## Supported Entity Types
+
+- **NAME**: Person names (John Doe, Dr. Smith)
+- **DATE**: Dates in various formats (01/15/2025, March 15, 2025)
+- **AMOUNT**: Monetary amounts ($1,500.00, 1000 USD)
+- **INVOICE_NO**: Invoice numbers (INV-1001, BL-2045)
+- **ADDRESS**: Street addresses
+- **PHONE**: Phone numbers (555-123-4567, +1-555-123-4567)
+- **EMAIL**: Email addresses (user@domain.com)
+
+## Training Your Own Model
+
+### 1. Prepare Your Data
+
+```bash
+# Place your documents in data/raw/
+mkdir -p data/raw
+cp your_invoices/*.pdf data/raw/
+```
+
+### 2. Run Training Pipeline
+
+```python
+from src.training_pipeline import TrainingPipeline, create_custom_config
+
+# Create custom configuration
+config = create_custom_config()
+config.num_epochs = 5
+config.batch_size = 16
+
+# Run training
+pipeline = TrainingPipeline(config)
+model_path = pipeline.run_complete_pipeline("data/raw")
+```
+
+### 3. Evaluate Results
+
+Training automatically generates:
+
+- Loss curves: `results/plots/training_history.png`
+- Metrics: `results/metrics/evaluation_results.json`
+- Model checkpoints: `models/document_ner_model/`
+
+## Deployment
+
+### Docker Deployment
+
+```dockerfile
+FROM python:3.9-slim
+
+WORKDIR /app
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+# Install Tesseract
+RUN apt-get update && apt-get install -y tesseract-ocr
+
+COPY . .
+EXPOSE 8000
+
+CMD ["python", "api/app.py"]
+```
+
+### Cloud Deployment
+
+- **AWS**: Deploy using ECS or Lambda
+- **Google Cloud**: Use Cloud Run or Compute Engine
+- **Azure**: Deploy with Container Instances
+
+## Contributing
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
+3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
+4. Push to the branch (`git push origin feature/AmazingFeature`)
+5. Open a Pull Request
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- [Hugging Face Transformers](https://huggingface.co/transformers/) for the DistilBERT model
+- [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) for optical character recognition
+- [EasyOCR](https://github.com/JaidedAI/EasyOCR) for additional OCR capabilities
+- [FastAPI](https://fastapi.tiangolo.com/) for the web framework
+
+## Support
+
+- Email: your-email@domain.com
+- Issues: [GitHub Issues](https://github.com/your-username/small-language-model/issues)
+- Documentation: [Project Wiki](https://github.com/your-username/small-language-model/wiki)
+
+---
+
+**Star this repository if it helped you!**

api/app.py

@@ -0,0 +1,471 @@
+"""
+FastAPI web service for document text extraction.
+Provides REST API endpoints for uploading and processing documents.
+"""
+
+from fastapi import FastAPI, File, UploadFile, HTTPException, Form
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import HTMLResponse, JSONResponse
+from fastapi.staticfiles import StaticFiles
+import uvicorn
+import tempfile
+import os
+import json
+from pathlib import Path
+from typing import List, Optional, Dict, Any
+import shutil
+
+from src.inference import DocumentInference
+
+
+# Initialize FastAPI app
+app = FastAPI(
+    title="Document Text Extraction API",
+    description="Extract structured information from documents using Small Language Model (SLM)",
+    version="1.0.0"
+)
+
+# Add CORS middleware
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Global inference pipeline
+inference_pipeline: Optional[DocumentInference] = None
+
+def get_inference_pipeline() -> DocumentInference:
+    """Get or initialize the inference pipeline."""
+    global inference_pipeline
+    
+    if inference_pipeline is None:
+        model_path = "models/document_ner_model"
+        
+        if not Path(model_path).exists():
+            raise HTTPException(
+                status_code=503,
+                detail="Model not found. Please train the model first by running training_pipeline.py"
+            )
+        
+        try:
+            inference_pipeline = DocumentInference(model_path)
+        except Exception as e:
+            raise HTTPException(
+                status_code=503,
+                detail=f"Failed to load model: {str(e)}"
+            )
+    
+    return inference_pipeline
+
+
+@app.on_event("startup")
+async def startup_event():
+    """Initialize the model on startup."""
+    try:
+        get_inference_pipeline()
+        print("Model loaded successfully on startup")
+    except Exception as e:
+        print(f"Failed to load model on startup: {e}")
+        print("Model will be loaded on first request")
+
+
+@app.get("/", response_class=HTMLResponse)
+async def root():
+    """Serve the main HTML interface."""
+    html_content = """
+    <!DOCTYPE html>
+    <html>
+    <head>
+        <title>Document Text Extraction</title>
+        <style>
+            body {
+                font-family: Arial, sans-serif;
+                max-width: 800px;
+                margin: 0 auto;
+                padding: 20px;
+                background-color: #f5f5f5;
+            }
+            .container {
+                background: white;
+                padding: 30px;
+                border-radius: 10px;
+                box-shadow: 0 2px 10px rgba(0,0,0,0.1);
+            }
+            .header {
+                text-align: center;
+                color: #333;
+                margin-bottom: 30px;
+            }
+            .upload-area {
+                border: 2px dashed #ccc;
+                padding: 40px;
+                text-align: center;
+                margin: 20px 0;
+                border-radius: 8px;
+                background-color: #fafafa;
+            }
+            .upload-area:hover {
+                border-color: #007bff;
+                background-color: #f0f8ff;
+            }
+            .btn {
+                background-color: #007bff;
+                color: white;
+                padding: 10px 20px;
+                border: none;
+                border-radius: 5px;
+                cursor: pointer;
+                font-size: 16px;
+            }
+            .btn:hover {
+                background-color: #0056b3;
+            }
+            .result {
+                margin-top: 20px;
+                padding: 20px;
+                background-color: #f8f9fa;
+                border-radius: 5px;
+                border: 1px solid #dee2e6;
+            }
+            .json-output {
+                background-color: #f4f4f4;
+                padding: 15px;
+                border-radius: 5px;
+                font-family: monospace;
+                white-space: pre-wrap;
+                overflow-x: auto;
+                max-height: 400px;
+                overflow-y: auto;
+            }
+            .text-input {
+                width: 100%;
+                height: 100px;
+                padding: 10px;
+                border: 1px solid #ccc;
+                border-radius: 5px;
+                font-family: monospace;
+                resize: vertical;
+            }
+            .tab-container {
+                margin: 20px 0;
+            }
+            .tabs {
+                display: flex;
+                border-bottom: 1px solid #ccc;
+            }
+            .tab {
+                padding: 10px 20px;
+                cursor: pointer;
+                border-bottom: 2px solid transparent;
+                background-color: #f8f9fa;
+                margin-right: 5px;
+            }
+            .tab.active {
+                border-bottom-color: #007bff;
+                background-color: white;
+            }
+            .tab-content {
+                display: none;
+                padding: 20px 0;
+            }
+            .tab-content.active {
+                display: block;
+            }
+        </style>
+    </head>
+    <body>
+        <div class="container">
+            <div class="header">
+                <h1>Document Text Extraction</h1>
+                <p>Extract structured information from documents using AI</p>
+            </div>
+            
+            <div class="tab-container">
+                <div class="tabs">
+                    <div class="tab active" onclick="showTab('file')">Upload File</div>
+                    <div class="tab" onclick="showTab('text')">Enter Text</div>
+                </div>
+                
+                <div id="file-tab" class="tab-content active">
+                    <form id="uploadForm" enctype="multipart/form-data">
+                        <div class="upload-area">
+                            <p>Choose a document to extract information</p>
+                            <p><small>Supported: PDF, DOCX, Images (PNG, JPG, etc.)</small></p>
+                            <input type="file" id="fileInput" name="file" accept=".pdf,.docx,.png,.jpg,.jpeg,.tiff,.bmp" style="margin: 10px 0;">
+                            <br>
+                            <button type="submit" class="btn">Extract Information</button>
+                        </div>
+                    </form>
+                </div>
+                
+                <div id="text-tab" class="tab-content">
+                    <form id="textForm">
+                        <p>Enter text directly for information extraction:</p>
+                        <textarea id="textInput" class="text-input" placeholder="Enter document text here, e.g.:&#10;Invoice sent to John Doe on 01/15/2025&#10;Invoice No: INV-1001&#10;Amount: $1,500.00"></textarea>
+                        <br><br>
+                        <button type="submit" class="btn">Extract from Text</button>
+                    </form>
+                </div>
+            </div>
+            
+            <div id="result" class="result" style="display: none;">
+                <h3>Extraction Results</h3>
+                <div id="resultContent"></div>
+            </div>
+        </div>
+
+        <script>
+            function showTab(tabName) {
+                // Hide all tab contents
+                document.querySelectorAll('.tab-content').forEach(content => {
+                    content.classList.remove('active');
+                });
+                
+                // Remove active class from all tabs
+                document.querySelectorAll('.tab').forEach(tab => {
+                    tab.classList.remove('active');
+                });
+                
+                // Show selected tab content
+                document.getElementById(tabName + '-tab').classList.add('active');
+                
+                // Add active class to selected tab
+                event.target.classList.add('active');
+            }
+
+            // File upload form handler
+            document.getElementById('uploadForm').addEventListener('submit', async function(e) {
+                e.preventDefault();
+                
+                const fileInput = document.getElementById('fileInput');
+                if (!fileInput.files[0]) {
+                    alert('Please select a file');
+                    return;
+                }
+                
+                const formData = new FormData();
+                formData.append('file', fileInput.files[0]);
+                
+                try {
+                    showResult('Processing document, please wait...');
+                    
+                    const response = await fetch('/extract-from-file', {
+                        method: 'POST',
+                        body: formData
+                    });
+                    
+                    const result = await response.json();
+                    displayResult(result);
+                    
+                } catch (error) {
+                    showResult('Error: ' + error.message);
+                }
+            });
+            
+            // Text form handler
+            document.getElementById('textForm').addEventListener('submit', async function(e) {
+                e.preventDefault();
+                
+                const text = document.getElementById('textInput').value;
+                if (!text.trim()) {
+                    alert('Please enter some text');
+                    return;
+                }
+                
+                try {
+                    showResult('Processing text, please wait...');
+                    
+                    const response = await fetch('/extract-from-text', {
+                        method: 'POST',
+                        headers: {
+                            'Content-Type': 'application/json',
+                        },
+                        body: JSON.stringify({ text: text })
+                    });
+                    
+                    const result = await response.json();
+                    displayResult(result);
+                    
+                } catch (error) {
+                    showResult('Error: ' + error.message);
+                }
+            });
+            
+            function showResult(message) {
+                const resultDiv = document.getElementById('result');
+                const contentDiv = document.getElementById('resultContent');
+                contentDiv.innerHTML = message;
+                resultDiv.style.display = 'block';
+            }
+            
+            function displayResult(result) {
+                let html = '';
+                
+                if (result.error) {
+                    html = `<div style="color: red;">Error: ${result.error}</div>`;
+                } else {
+                    // Show structured data
+                    if (result.structured_data && Object.keys(result.structured_data).length > 0) {
+                        html += '<h4>Extracted Information:</h4>';
+                        html += '<table style="width: 100%; border-collapse: collapse; margin: 10px 0;">';
+                        html += '<tr style="background-color: #f8f9fa;"><th style="padding: 8px; border: 1px solid #dee2e6; text-align: left;">Field</th><th style="padding: 8px; border: 1px solid #dee2e6; text-align: left;">Value</th></tr>';
+                        
+                        for (const [key, value] of Object.entries(result.structured_data)) {
+                            html += `<tr><td style="padding: 8px; border: 1px solid #dee2e6; font-weight: bold;">${key}</td><td style="padding: 8px; border: 1px solid #dee2e6;">${value}</td></tr>`;
+                        }
+                        html += '</table>';
+                    } else {
+                        html += '<div style="color: orange;">No structured information found in the document.</div>';
+                    }
+                    
+                    // Show entities
+                    if (result.entities && result.entities.length > 0) {
+                        html += '<h4>Detected Entities:</h4>';
+                        html += '<div style="margin: 10px 0;">';
+                        result.entities.forEach(entity => {
+                            const confidence = Math.round(entity.confidence * 100);
+                            html += `<span style="display: inline-block; margin: 2px 4px; padding: 4px 8px; background-color: #e3f2fd; border: 1px solid #2196f3; border-radius: 15px; font-size: 12px;">
+                                ${entity.entity}: "${entity.text}" (${confidence}%)</span>`;
+                        });
+                        html += '</div>';
+                    }
+                    
+                    // Show raw JSON
+                    html += '<h4>Full Response:</h4>';
+                    html += `<div class="json-output">${JSON.stringify(result, null, 2)}</div>`;
+                }
+                
+                showResult(html);
+            }
+        </script>
+    </body>
+    </html>
+    """
+    return html_content
+
+
+@app.get("/health")
+async def health_check():
+    """Health check endpoint."""
+    try:
+        get_inference_pipeline()
+        return {"status": "healthy", "message": "Model loaded successfully"}
+    except Exception as e:
+        return {"status": "unhealthy", "message": str(e)}
+
+
+@app.post("/extract-from-file")
+async def extract_from_file(file: UploadFile = File(...)):
+    """Extract structured information from an uploaded file."""
+    if not file:
+        raise HTTPException(status_code=400, detail="No file provided")
+    
+    # Check file type
+    allowed_extensions = {'.pdf', '.docx', '.png', '.jpg', '.jpeg', '.tiff', '.bmp'}
+    file_extension = Path(file.filename).suffix.lower()
+    
+    if file_extension not in allowed_extensions:
+        raise HTTPException(
+            status_code=400,
+            detail=f"Unsupported file type: {file_extension}. Allowed: {', '.join(allowed_extensions)}"
+        )
+    
+    # Save uploaded file temporarily
+    with tempfile.NamedTemporaryFile(delete=False, suffix=file_extension) as temp_file:
+        shutil.copyfileobj(file.file, temp_file)
+        temp_file_path = temp_file.name
+    
+    try:
+        # Process the document
+        inference = get_inference_pipeline()
+        result = inference.process_document(temp_file_path)
+        
+        return JSONResponse(content=result)
+        
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+    
+    finally:
+        # Clean up temporary file
+        try:
+            os.unlink(temp_file_path)
+        except:
+            pass
+
+
+@app.post("/extract-from-text")
+async def extract_from_text(request: Dict[str, str]):
+    """Extract structured information from text."""
+    text = request.get("text", "").strip()
+    
+    if not text:
+        raise HTTPException(status_code=400, detail="No text provided")
+    
+    try:
+        # Process the text
+        inference = get_inference_pipeline()
+        result = inference.process_text_directly(text)
+        
+        return JSONResponse(content=result)
+        
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+@app.get("/supported-formats")
+async def get_supported_formats():
+    """Get list of supported file formats."""
+    return {
+        "supported_formats": [
+            {"extension": ".pdf", "description": "PDF documents"},
+            {"extension": ".docx", "description": "Microsoft Word documents"},
+            {"extension": ".png", "description": "PNG images"},
+            {"extension": ".jpg", "description": "JPEG images"},
+            {"extension": ".jpeg", "description": "JPEG images"},
+            {"extension": ".tiff", "description": "TIFF images"},
+            {"extension": ".bmp", "description": "BMP images"}
+        ],
+        "entity_types": [
+            "Name", "Date", "InvoiceNo", "Amount", "Address", "Phone", "Email"
+        ]
+    }
+
+
+@app.get("/model-info")
+async def get_model_info():
+    """Get information about the loaded model."""
+    try:
+        inference = get_inference_pipeline()
+        return {
+            "model_path": inference.model_path,
+            "model_name": inference.config.model_name,
+            "max_length": inference.config.max_length,
+            "entity_labels": inference.config.entity_labels,
+            "num_labels": inference.config.num_labels
+        }
+    except Exception as e:
+        raise HTTPException(status_code=503, detail=f"Model not loaded: {str(e)}")
+
+
+def main():
+    """Run the FastAPI server."""
+    print("Starting Document Text Extraction API Server...")
+    print("Server will be available at: http://localhost:8000")
+    print("Web interface: http://localhost:8000")
+    print("API docs: http://localhost:8000/docs")
+    print("Health check: http://localhost:8000/health")
+    
+    uvicorn.run(
+        "api.app:app",
+        host="0.0.0.0",
+        port=8000,
+        reload=True,
+        log_level="info"
+    )
+
+
+if __name__ == "__main__":
+    main()

config/settings.py

@@ -0,0 +1,64 @@
+"""
+Configuration settings for the document text extraction system.
+"""
+
+import os
+from pathlib import Path
+
+
+class Config:
+    """Global configuration settings."""
+    
+    # Project paths
+    PROJECT_ROOT = Path(__file__).parent.parent
+    DATA_DIR = PROJECT_ROOT / "data"
+    MODELS_DIR = PROJECT_ROOT / "models"
+    RESULTS_DIR = PROJECT_ROOT / "results"
+    
+    # Data paths
+    RAW_DATA_DIR = DATA_DIR / "raw"
+    PROCESSED_DATA_DIR = DATA_DIR / "processed"
+    
+    # Model settings
+    DEFAULT_MODEL_NAME = "distilbert-base-uncased"
+    DEFAULT_MODEL_PATH = MODELS_DIR / "document_ner_model"
+    
+    # Training settings
+    DEFAULT_BATCH_SIZE = 16
+    DEFAULT_LEARNING_RATE = 2e-5
+    DEFAULT_NUM_EPOCHS = 3
+    DEFAULT_MAX_LENGTH = 512
+    
+    # OCR settings
+    TESSERACT_PATH = os.getenv('TESSERACT_PATH', None)
+    
+    # API settings
+    API_HOST = "0.0.0.0"
+    API_PORT = 8000
+    
+    # Entity labels
+    ENTITY_LABELS = [
+        'O', 'B-NAME', 'I-NAME', 'B-DATE', 'I-DATE', 
+        'B-INVOICE_NO', 'I-INVOICE_NO', 'B-AMOUNT', 'I-AMOUNT',
+        'B-ADDRESS', 'I-ADDRESS', 'B-PHONE', 'I-PHONE',
+        'B-EMAIL', 'I-EMAIL'
+    ]
+    
+    # Supported file formats
+    SUPPORTED_FORMATS = ['.pdf', '.docx', '.png', '.jpg', '.jpeg', '.tiff', '.bmp']
+    
+    @classmethod
+    def create_directories(cls):
+        """Create necessary directories."""
+        directories = [
+            cls.DATA_DIR,
+            cls.RAW_DATA_DIR,
+            cls.PROCESSED_DATA_DIR,
+            cls.MODELS_DIR,
+            cls.RESULTS_DIR,
+            cls.RESULTS_DIR / "plots",
+            cls.RESULTS_DIR / "metrics"
+        ]
+        
+        for directory in directories:
+            directory.mkdir(parents=True, exist_ok=True)

demo.py

@@ -0,0 +1,227 @@
+"""
+Simple demo script for document text extraction.
+Demonstrates the complete workflow from training to inference.
+"""
+
+import os
+import sys
+from pathlib import Path
+import jso                    print(f"  {entity['entity']}: '{entity['text']}' ({confidence}%)")
+        else:
+            print(f"Error: {result['error']}")
+    
+    except Exception as e:
+        print(f"Failed to process text: {e}") Add src to path for imports
+sys.path.append(str(Path(__file__).parent))
+
+from src.data_preparation import DocumentProcessor, NERDatasetCreator
+from src.training_pipeline import TrainingPipeline, create_custom_config
+from src.inference import DocumentInference
+
+
+def run_quick_demo():
+    """Run a quick demonstration of the text extraction system."""
+    print("DOCUMENT TEXT EXTRACTION - QUICK DEMO")
+    print("=" * 60)
+    
+    # Sample documents for demonstration
+    demo_texts = [
+        {
+            "name": "Invoice Example 1",
+            "text": "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567"
+        },
+        {
+            "name": "Invoice Example 2",
+            "text": "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Email: sarah.johnson@email.com"
+        },
+        {
+            "name": "Receipt Example",
+            "text": "Receipt for Michael Brown 456 Oak Street Boston MA 02101 Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75"
+        }
+    ]
+    
+    print("\nSample Documents:")
+    for i, doc in enumerate(demo_texts, 1):
+        print(f"{i}. {doc['name']}: {doc['text'][:60]}...")
+    
+    # Check if model exists
+    model_path = "models/document_ner_model"
+    if not Path(model_path).exists():
+        print(f"\nModel not found at {model_path}")
+        print("Training a new model first...")
+        
+        # Train model
+        config = create_custom_config()
+        config.num_epochs = 2  # Quick training for demo
+        config.batch_size = 8
+        
+        pipeline = TrainingPipeline(config)
+        model_path = pipeline.run_complete_pipeline()
+        
+        print(f"Model trained and saved to {model_path}")
+    
+    # Load inference pipeline
+    print(f"\nLoading inference pipeline from {model_path}")
+    try:
+        inference = DocumentInference(model_path)
+        print("Inference pipeline loaded successfully")
+    except Exception as e:
+        print(f"Failed to load inference pipeline: {e}")
+        return
+    
+    # Process demo texts
+    print(f"\nProcessing {len(demo_texts)} demo documents...")
+    results = []
+    
+    for i, doc in enumerate(demo_texts, 1):
+        print(f"\nProcessing Document {i}: {doc['name']}")
+        print("-" * 50)
+        print(f"Text: {doc['text']}")
+        
+        # Extract information
+        result = inference.process_text_directly(doc['text'])
+        results.append({
+            'document_name': doc['name'],
+            'original_text': doc['text'],
+            'result': result
+        })
+        
+        # Display results
+        if 'error' not in result:
+            structured_data = result.get('structured_data', {})
+            entities = result.get('entities', [])
+            
+            print(f"\nExtraction Results:")
+            if structured_data:
+                print("Structured Data:")
+                for key, value in structured_data.items():
+                    print(f"   {key}: {value}")
+            else:
+                print("   No structured data extracted")
+            
+            if entities:
+                print(f"Found {len(entities)} entities:")
+                for entity in entities:
+                    confidence = int(entity['confidence'] * 100)
+                    print(f"   {entity['entity']}: '{entity['text']}' ({confidence}%)")
+        else:
+            print(f"Error: {result['error']}")
+    
+    # Save results
+    output_path = "results/demo_results.json"
+    Path(output_path).parent.mkdir(parents=True, exist_ok=True)
+    with open(output_path, 'w', encoding='utf-8') as f:
+        json.dump(results, f, indent=2, ensure_ascii=False)
+    
+    print(f"\nDemo results saved to: {output_path}")
+    
+    # Summary
+    successful_extractions = sum(1 for r in results if 'error' not in r['result'])
+    total_entities = sum(len(r['result'].get('entities', [])) for r in results if 'error' not in r['result'])
+    total_structured_fields = sum(len(r['result'].get('structured_data', {})) for r in results if 'error' not in r['result'])
+    
+    print(f"\nDemo Summary:")
+    print(f"   Successfully processed: {successful_extractions}/{len(demo_texts)} documents")
+    print(f"   Total entities found: {total_entities}")
+    print(f"   Total structured fields: {total_structured_fields}")
+    
+    print(f"\nDemo completed successfully!")
+    print(f"You can now:")
+    print(f"   - Run the web API: python api/app.py")
+    print(f"   - Process your own documents using inference.py")
+    print(f"   - Retrain with your data using training_pipeline.py")
+
+
+def train_model_only():
+    """Train the model without running inference demo."""
+    print("TRAINING MODEL ONLY")
+    print("=" * 40)
+    
+    config = create_custom_config()
+    pipeline = TrainingPipeline(config)
+    
+    model_path = pipeline.run_complete_pipeline()
+    
+    print(f"Model training completed!")
+    print(f"Model saved to: {model_path}")
+
+
+def test_specific_text():
+    """Test extraction on user-provided text."""
+    print("CUSTOM TEXT EXTRACTION")
+    print("=" * 40)
+    
+    # Check if model exists
+    model_path = "models/document_ner_model"
+    if not Path(model_path).exists():
+        print("No trained model found. Please run training first.")
+        return
+    
+    # Get text from user
+    print("Enter text to extract information from:")
+    print("(Example: Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00)")
+    text = input("Text: ").strip()
+    
+    if not text:
+        print("No text provided.")
+        return
+    
+    # Load inference and process
+    try:
+        inference = DocumentInference(model_path)
+        result = inference.process_text_directly(text)
+        
+        print(f"\nExtraction Results:")
+        if 'error' not in result:
+            structured_data = result.get('structured_data', {})
+            if structured_data:
+                print("Structured Information:")
+                for key, value in structured_data.items():
+                    print(f"  {key}: {value}")
+            else:
+                print("No structured information found.")
+            
+            entities = result.get('entities', [])
+            if entities:
+                print(f"\nEntities Found ({len(entities)}):")
+                for entity in entities:
+                    confidence = int(entity['confidence'] * 100)
+                    print(f"  {entity['entity']}: '{entity['text']}' ({confidence}%)")
+        else:
+            print(f"Error: {result['error']}")
+    
+    except Exception as e:
+        print(f"Failed to process text: {e}")
+
+
+def main():
+    """Main demo function with options."""
+    print("DOCUMENT TEXT EXTRACTION SYSTEM")
+    print("=" * 50)
+    print("Choose an option:")
+    print("1. Run complete demo (train + inference)")
+    print("2. Train model only")
+    print("3. Test specific text (requires trained model)")
+    print("4. Exit")
+    
+    while True:
+        choice = input("\nEnter your choice (1-4): ").strip()
+        
+        if choice == '1':
+            run_quick_demo()
+            break
+        elif choice == '2':
+            train_model_only()
+            break
+        elif choice == '3':
+            test_specific_text()
+            break
+        elif choice == '4':
+            print("👋 Goodbye!")
+            break
+        else:
+            print("Invalid choice. Please enter 1, 2, 3, or 4.")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,50 @@
+# Document Text Extraction using Small Language Model (SLM)
+# Core ML and NLP libraries
+torch>=2.0.0
+transformers>=4.30.0
+tokenizers>=0.13.0
+datasets>=2.14.0
+
+# OCR and image processing
+pytesseract>=0.3.10
+easyocr>=1.7.0
+opencv-python>=4.8.0
+Pillow>=10.0.0
+
+# PDF and document processing
+PyMuPDF>=1.23.0
+python-docx>=0.8.11
+
+# Data processing and analysis
+pandas>=2.0.0
+numpy>=1.24.0
+scikit-learn>=1.3.0
+
+# NER evaluation metrics
+seqeval>=1.2.2
+
+# Visualization
+matplotlib>=3.7.0
+seaborn>=0.12.0
+
+# Web API
+fastapi>=0.100.0
+uvicorn>=0.22.0
+python-multipart>=0.0.6
+
+# Utility libraries
+pathlib2>=2.3.7
+tqdm>=4.65.0
+python-dotenv>=1.0.0
+
+# Development and testing (optional)
+pytest>=7.4.0
+black>=23.0.0
+flake8>=6.0.0
+jupyter>=1.0.0
+ipykernel>=6.25.0
+
+# Optional: For GPU support (uncomment if you have CUDA)
+# torch>=2.0.0+cu118
+# torchvision>=0.15.0+cu118
+# torchaudio>=2.0.0+cu118

results/demo_extraction_results.json

@@ -0,0 +1,284 @@
+[
+  {
+    "document_name": "Invoice Example 1",
+    "original_text": "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567",
+    "entities": [
+      {
+        "entity": "NAME",
+        "text": "Invoice sent",
+        "start": 0,
+        "end": 12,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "to Robert",
+        "start": 13,
+        "end": 22,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "White on",
+        "start": 23,
+        "end": 31,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Invoice No",
+        "start": 43,
+        "end": 53,
+        "confidence": 0.8
+      },
+      {
+        "entity": "DATE",
+        "text": "15/09/2025",
+        "start": 32,
+        "end": 42,
+        "confidence": 0.85
+      },
+      {
+        "entity": "INVOICE_NO",
+        "text": "INV-1024",
+        "start": 43,
+        "end": 63,
+        "confidence": 0.9
+      },
+      {
+        "entity": "AMOUNT",
+        "text": "$1,250.00",
+        "start": 72,
+        "end": 81,
+        "confidence": 0.85
+      },
+      {
+        "entity": "PHONE",
+        "text": "(555) 123-4567",
+        "start": 89,
+        "end": 103,
+        "confidence": 0.9
+      }
+    ],
+    "structured_data": {
+      "Name": "Invoice Sent",
+      "Date": "15/09/2025",
+      "InvoiceNo": "INV-1024",
+      "Amount": "$1,250.00",
+      "Phone": "(555) 123-4567"
+    },
+    "processing_timestamp": "2025-09-27T18:26:31.996468",
+    "total_entities_found": 8,
+    "entity_types_found": [
+      "AMOUNT",
+      "NAME",
+      "DATE",
+      "INVOICE_NO",
+      "PHONE"
+    ]
+  },
+  {
+    "document_name": "Invoice Example 2",
+    "original_text": "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Email: sarah.johnson@email.com",
+    "entities": [
+      {
+        "entity": "NAME",
+        "text": "Sarah Johnson",
+        "start": 9,
+        "end": 26,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Bill for",
+        "start": 0,
+        "end": 8,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "dated March",
+        "start": 27,
+        "end": 38,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Invoice Number",
+        "start": 49,
+        "end": 63,
+        "confidence": 0.8
+      },
+      {
+        "entity": "DATE",
+        "text": "March 10, 2025",
+        "start": 33,
+        "end": 47,
+        "confidence": 0.85
+      },
+      {
+        "entity": "INVOICE_NO",
+        "text": "BL-2045",
+        "start": 49,
+        "end": 72,
+        "confidence": 0.9
+      },
+      {
+        "entity": "AMOUNT",
+        "text": "$2,300.50",
+        "start": 81,
+        "end": 90,
+        "confidence": 0.85
+      },
+      {
+        "entity": "EMAIL",
+        "text": "sarah.johnson@email.com",
+        "start": 98,
+        "end": 121,
+        "confidence": 0.95
+      }
+    ],
+    "structured_data": {
+      "Name": "Sarah Johnson",
+      "Date": "March 10, 2025",
+      "InvoiceNo": "BL-2045",
+      "Amount": "$2,300.50",
+      "Email": "sarah.johnson@email.com"
+    },
+    "processing_timestamp": "2025-09-27T18:26:31.997340",
+    "total_entities_found": 8,
+    "entity_types_found": [
+      "AMOUNT",
+      "NAME",
+      "EMAIL",
+      "DATE",
+      "INVOICE_NO"
+    ]
+  },
+  {
+    "document_name": "Receipt Example",
+    "original_text": "Receipt for Michael Brown Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75 Contact: +1-555-987-6543",
+    "entities": [
+      {
+        "entity": "NAME",
+        "text": "Receipt for",
+        "start": 0,
+        "end": 11,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Michael Brown",
+        "start": 12,
+        "end": 25,
+        "confidence": 0.8
+      },
+      {
+        "entity": "DATE",
+        "text": "2025-04-22",
+        "start": 50,
+        "end": 60,
+        "confidence": 0.85
+      },
+      {
+        "entity": "INVOICE_NO",
+        "text": "REC-3089",
+        "start": 35,
+        "end": 43,
+        "confidence": 0.9
+      },
+      {
+        "entity": "AMOUNT",
+        "text": "$890.75",
+        "start": 69,
+        "end": 76,
+        "confidence": 0.85
+      },
+      {
+        "entity": "PHONE",
+        "text": "+1-555-987-6543",
+        "start": 86,
+        "end": 101,
+        "confidence": 0.9
+      }
+    ],
+    "structured_data": {
+      "Name": "Receipt For",
+      "Date": "2025-04-22",
+      "InvoiceNo": "REC-3089",
+      "Amount": "$890.75",
+      "Phone": "+1 (555) 987-6543"
+    },
+    "processing_timestamp": "2025-09-27T18:26:31.998731",
+    "total_entities_found": 6,
+    "entity_types_found": [
+      "AMOUNT",
+      "NAME",
+      "DATE",
+      "INVOICE_NO",
+      "PHONE"
+    ]
+  },
+  {
+    "document_name": "Business Document",
+    "original_text": "Ms. Emma Wilson 456 Oak Street Payment due: January 15, 2025 Reference: INV-4567 Total: $1,750.25",
+    "entities": [
+      {
+        "entity": "NAME",
+        "text": "Emma Wilson",
+        "start": 0,
+        "end": 15,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Oak Street",
+        "start": 20,
+        "end": 30,
+        "confidence": 0.8
+      },
+      {
+        "entity": "NAME",
+        "text": "Payment due",
+        "start": 31,
+        "end": 42,
+        "confidence": 0.8
+      },
+      {
+        "entity": "DATE",
+        "text": "January 15, 2025",
+        "start": 44,
+        "end": 60,
+        "confidence": 0.85
+      },
+      {
+        "entity": "INVOICE_NO",
+        "text": "INV-4567",
+        "start": 72,
+        "end": 80,
+        "confidence": 0.9
+      },
+      {
+        "entity": "AMOUNT",
+        "text": "$1,750.25",
+        "start": 88,
+        "end": 97,
+        "confidence": 0.85
+      }
+    ],
+    "structured_data": {
+      "Name": "Emma Wilson",
+      "Date": "January 15, 2025",
+      "InvoiceNo": "INV-4567",
+      "Amount": "$1,750.25"
+    },
+    "processing_timestamp": "2025-09-27T18:26:32.000279",
+    "total_entities_found": 6,
+    "entity_types_found": [
+      "AMOUNT",
+      "INVOICE_NO",
+      "DATE",
+      "NAME"
+    ]
+  }
+]

setup.py

@@ -0,0 +1,274 @@
+#!/usr/bin/env python3
+"""
+Setup script for the Document Text Extraction system.
+Creates directories, checks dependencies, and initializes the project.
+"""
+
+import os
+import sys
+import subprocess
+from pathlib import Path
+import importlib.util
+
+
+def check_python_version():
+    """Check if Python version is compatible."""
+    if sys.version_info < (3, 8):
+        print("Python 3.8 or higher is required.")
+        print(f"Current version: {sys.version}")
+        return False
+    
+    print(f"Python {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}")
+    return True
+
+
+def create_directories():
+    """Create necessary project directories."""
+    directories = [
+        "data/raw",
+        "data/processed", 
+        "models",
+        "results/plots",
+        "results/metrics",
+        "logs"
+    ]
+    
+    print("\n📁 Creating project directories...")
+    for directory in directories:
+        Path(directory).mkdir(parents=True, exist_ok=True)
+        print(f"   {directory}")
+
+
+def check_dependencies():
+    """Check if required dependencies are installed."""
+    print("\n📦 Checking dependencies...")
+    
+    required_packages = [
+        ('torch', 'PyTorch'),
+        ('transformers', 'Transformers'),
+        ('PIL', 'Pillow'),
+        ('cv2', 'OpenCV'),
+        ('pandas', 'Pandas'),
+        ('numpy', 'NumPy'),
+        ('sklearn', 'Scikit-learn')
+    ]
+    
+    missing_packages = []
+    
+    for package, name in required_packages:
+        spec = importlib.util.find_spec(package)
+        if spec is None:
+            missing_packages.append(name)
+            print(f"   {name} not found")
+        else:
+            print(f"   {name}")
+    
+    return missing_packages
+
+
+def check_ocr_dependencies():
+    """Check OCR-related dependencies."""
+    print("\nChecking OCR dependencies...")
+    
+    # Check EasyOCR
+    try:
+        import easyocr
+        print("   EasyOCR")
+    except ImportError:
+        print("   EasyOCR not found")
+    
+    # Check Tesseract
+    try:
+        import pytesseract
+        print("   PyTesseract")
+        
+        # Try to run tesseract
+        try:
+            pytesseract.get_tesseract_version()
+            print("   Tesseract OCR engine")
+        except Exception:
+            print("   Tesseract OCR engine not found or not in PATH")
+            print("      Please install Tesseract OCR:")
+            print("      - Windows: https://github.com/UB-Mannheim/tesseract/wiki")
+            print("      - Ubuntu: sudo apt install tesseract-ocr")
+            print("      - macOS: brew install tesseract")
+            
+    except ImportError:
+        print("   PyTesseract not found")
+
+
+def install_dependencies():
+    """Install missing dependencies."""
+    print("\nInstalling dependencies from requirements.txt...")
+    
+    try:
+        result = subprocess.run([
+            sys.executable, "-m", "pip", "install", "-r", "requirements.txt"
+        ], capture_output=True, text=True, check=True)
+        
+        print("   Dependencies installed successfully")
+        return True
+        
+    except subprocess.CalledProcessError as e:
+        print(f"   Failed to install dependencies: {e}")
+        print(f"   Output: {e.stdout}")
+        print(f"   Error: {e.stderr}")
+        return False
+
+
+def check_gpu_support():
+    """Check if GPU support is available."""
+    print("\n🖥️  Checking GPU support...")
+    
+    try:
+        import torch
+        if torch.cuda.is_available():
+            gpu_count = torch.cuda.device_count()
+            gpu_name = torch.cuda.get_device_name(0)
+            print(f"   CUDA available - {gpu_count} GPU(s)")
+            print(f"   Primary GPU: {gpu_name}")
+        else:
+            print("   CUDA not available - will use CPU")
+    except ImportError:
+        print("   PyTorch not installed")
+
+
+def create_sample_documents():
+    """Create sample documents for testing."""
+    print("\nCreating sample test documents...")
+    
+    sample_texts = [
+        "Invoice sent to John Doe on 01/15/2025\nInvoice No: INV-1001\nAmount: $1,500.00\nPhone: (555) 123-4567",
+        "Bill for Dr. Sarah Johnson dated March 10, 2025.\nInvoice Number: BL-2045.\nTotal: $2,300.50\nEmail: sarah@email.com",
+        "Receipt for Michael Brown\n456 Oak Street, Boston MA 02101\nInvoice: REC-3089\nDate: 2025-04-22\nAmount: $890.75"
+    ]
+    
+    sample_dir = Path("data/raw/samples")
+    sample_dir.mkdir(parents=True, exist_ok=True)
+    
+    for i, text in enumerate(sample_texts, 1):
+        sample_file = sample_dir / f"sample_document_{i}.txt"
+        with open(sample_file, 'w', encoding='utf-8') as f:
+            f.write(text)
+        print(f"   {sample_file.name}")
+
+
+def run_initial_test():
+    """Run a basic test to verify setup."""
+    print("\nRunning initial setup test...")
+    
+    try:
+        # Test imports
+        from src.data_preparation import DocumentProcessor, NERDatasetCreator
+        from src.model import ModelConfig
+        print("   Core modules imported successfully")
+        
+        # Test document processor
+        processor = DocumentProcessor()
+        test_text = "Invoice sent to John Doe on 01/15/2025 Amount: $500.00"
+        cleaned_text = processor.clean_text(test_text)
+        print("   Document processor working")
+        
+        # Test dataset creator
+        dataset_creator = NERDatasetCreator(processor)
+        sample_dataset = dataset_creator.create_sample_dataset()
+        print(f"   Dataset creator working - {len(sample_dataset)} samples")
+        
+        # Test model config
+        config = ModelConfig()
+        print(f"   Model config created - {config.num_labels} labels")
+        
+        return True
+        
+    except Exception as e:
+        print(f"   Setup test failed: {e}")
+        return False
+
+
+def display_next_steps():
+    """Display next steps for the user."""
+    print("\n" + "=" * 30)
+    print("SETUP COMPLETED SUCCESSFULLY!")
+    print("=" * 30)
+    
+    print("\nNext Steps:")
+    print("1. Quick Demo:")
+    print("   python demo.py")
+    
+    print("\n2. Train Your Model:")
+    print("   # Add your documents to data/raw/")
+    print("   # Then run:")
+    print("   python src/training_pipeline.py")
+    
+    print("\n3. 🌐 Start Web API:")
+    print("   python api/app.py")
+    print("   # Then open: http://localhost:8000")
+    
+    print("\n4. Run Tests:")
+    print("   python tests/test_extraction.py")
+    
+    print("\n5. 📚 Documentation:")
+    print("   # View README.md for detailed usage")
+    print("   # API docs: http://localhost:8000/docs")
+    
+    print("\nPro Tips:")
+    print("   - Place your documents in data/raw/ for training")
+    print("   - Use GPU for faster training (if available)")
+    print("   - Adjust batch_size in config if you get memory errors")
+    print("   - Check logs/ directory for debugging information")
+
+
+def main():
+    """Main setup function."""
+    print("DOCUMENT TEXT EXTRACTION - SETUP SCRIPT")
+    print("=" * 60)
+    
+    # Check Python version
+    if not check_python_version():
+        return False
+    
+    # Create directories
+    create_directories()
+    
+    # Check and install dependencies
+    missing_packages = check_dependencies()
+    if missing_packages:
+        print(f"\nMissing packages: {', '.join(missing_packages)}")
+        install_deps = input("Install missing dependencies? (y/n): ").lower().strip()
+        
+        if install_deps == 'y':
+            if not install_dependencies():
+                print("Failed to install dependencies. Please install manually:")
+                print("   pip install -r requirements.txt")
+                return False
+        else:
+            print("Some features may not work without required dependencies.")
+    
+    # Check OCR dependencies
+    check_ocr_dependencies()
+    
+    # Check GPU support
+    check_gpu_support()
+    
+    # Create sample documents
+    create_sample_documents()
+    
+    # Run initial test
+    if not run_initial_test():
+        print("Setup test failed. Some features may not work correctly.")
+        print("   Check error messages above and ensure all dependencies are installed.")
+    
+    # Display next steps
+    display_next_steps()
+    
+    return True
+
+
+if __name__ == "__main__":
+    success = main()
+    
+    if success:
+        print(f"\nSetup completed! Ready to extract text from documents!")
+    else:
+        print(f"\nSetup encountered issues. Please check the messages above.")
+        sys.exit(1)

simple_api.py

@@ -0,0 +1,548 @@
+#!/usr/bin/env python3
+"""
+Simplified Document Text Extraction API
+Uses regex patterns instead of ML model for demonstration
+"""
+
+import json
+import re
+from datetime import datetime
+from typing import Dict, List, Any, Optional
+from pathlib import Path
+import sys
+import os
+
+# Add current directory to Python path
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+try:
+    from fastapi import FastAPI, HTTPException, File, UploadFile
+    from fastapi.responses import HTMLResponse, FileResponse
+    from fastapi.middleware.cors import CORSMiddleware
+    from pydantic import BaseModel
+    import uvicorn
+    HAS_FASTAPI = True
+except ImportError:
+    print("FastAPI not installed. Install with: pip install fastapi uvicorn python-multipart")
+    HAS_FASTAPI = False
+
+class SimpleDocumentProcessor:
+    """Simplified document processor using regex patterns"""
+    
+    def __init__(self):
+        # Define regex patterns for different entity types
+        self.patterns = {
+            'NAME': [
+                r'\b(?:Mr\.|Mrs\.|Ms\.|Dr\.|Prof\.)\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)',
+                r'\b([A-Z][a-z]+\s+[A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)\b',
+                r'(?:Invoice|Bill|Receipt)\s+(?:sent\s+)?(?:to\s+|for\s+)?([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)',
+            ],
+            'DATE': [
+                r'\b(\d{1,2}[\/\-]\d{1,2}[\/\-]\d{2,4})\b',
+                r'\b(\d{2,4}[\/\-]\d{1,2}[\/\-]\d{1,2})\b',
+                r'\b((?:January|February|March|April|May|June|July|August|September|October|November|December)\s+\d{1,2},?\s+\d{2,4})\b',
+                r'\b((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s+\d{1,2},?\s+\d{2,4})\b',
+            ],
+            'AMOUNT': [
+                r'\$\s*(\d{1,3}(?:,\d{3})*(?:\.\d{2})?)',
+                r'(?:Amount|Total|Sum):\s*\$?\s*(\d{1,3}(?:,\d{3})*(?:\.\d{2})?)',
+                r'(\d{1,3}(?:,\d{3})*(?:\.\d{2})?\s*(?:USD|dollars?))',
+            ],
+            'INVOICE_NO': [
+                r'(?:Invoice|Bill|Receipt)(?:\s+No\.?|#|Number):\s*([A-Z]{2,4}[-\s]?\d{3,6})',
+                r'(?:INV|BL|REC)[-\s]?(\d{3,6})',
+                r'Reference:\s*([A-Z]{2,4}[-\s]?\d{3,6})',
+            ],
+            'EMAIL': [
+                r'\b([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})\b',
+            ],
+            'PHONE': [
+                r'\b(\+?1[-.\s]?\(?[2-9]\d{2}\)?[-.\s]?\d{3}[-.\s]?\d{4})\b',
+                r'\b(\([2-9]\d{2}\)\s*[2-9]\d{2}[-.\s]?\d{4})\b',
+                r'\b([2-9]\d{2}[-.\s]?[2-9]\d{2}[-.\s]?\d{4})\b',
+            ],
+            'ADDRESS': [
+                r'\b(\d+\s+[A-Z][a-z]+\s+(?:Street|St|Avenue|Ave|Road|Rd|Lane|Ln|Drive|Dr|Boulevard|Blvd|Way))\b',
+            ]
+        }
+        
+        # Confidence scores for different entity types
+        self.confidence_scores = {
+            'NAME': 0.80,
+            'DATE': 0.85,
+            'AMOUNT': 0.85,
+            'INVOICE_NO': 0.90,
+            'EMAIL': 0.95,
+            'PHONE': 0.90,
+            'ADDRESS': 0.75
+        }
+
+    def extract_entities(self, text: str) -> List[Dict[str, Any]]:
+        """Extract entities from text using regex patterns"""
+        entities = []
+        
+        for entity_type, patterns in self.patterns.items():
+            for pattern in patterns:
+                matches = re.finditer(pattern, text, re.IGNORECASE)
+                for match in matches:
+                    entity = {
+                        'entity': entity_type,
+                        'text': match.group(1) if match.groups() else match.group(0),
+                        'start': match.start(),
+                        'end': match.end(),
+                        'confidence': self.confidence_scores[entity_type]
+                    }
+                    entities.append(entity)
+        
+        return entities
+
+    def create_structured_data(self, entities: List[Dict]) -> Dict[str, str]:
+        """Create structured data from extracted entities"""
+        structured = {}
+        
+        # Get the best entity for each type
+        entity_groups = {}
+        for entity in entities:
+            entity_type = entity['entity']
+            if entity_type not in entity_groups:
+                entity_groups[entity_type] = []
+            entity_groups[entity_type].append(entity)
+        
+        # Select best entity for each type
+        for entity_type, group in entity_groups.items():
+            if group:
+                # Sort by confidence and take the best one
+                best_entity = max(group, key=lambda x: x['confidence'])
+                
+                # Format field names
+                field_mapping = {
+                    'NAME': 'Name',
+                    'DATE': 'Date', 
+                    'AMOUNT': 'Amount',
+                    'INVOICE_NO': 'InvoiceNo',
+                    'EMAIL': 'Email',
+                    'PHONE': 'Phone',
+                    'ADDRESS': 'Address'
+                }
+                
+                field_name = field_mapping.get(entity_type, entity_type)
+                structured[field_name] = best_entity['text']
+        
+        return structured
+
+    def process_text(self, text: str) -> Dict[str, Any]:
+        """Process text and extract structured information"""
+        entities = self.extract_entities(text)
+        structured_data = self.create_structured_data(entities)
+        
+        # Get unique entity types
+        entity_types = list(set(entity['entity'] for entity in entities))
+        
+        return {
+            'status': 'success',
+            'data': {
+                'original_text': text,
+                'entities': entities,
+                'structured_data': structured_data,
+                'processing_timestamp': datetime.now().isoformat(),
+                'total_entities_found': len(entities),
+                'entity_types_found': sorted(entity_types)
+            }
+        }
+
+# Pydantic models for API
+if HAS_FASTAPI:
+    class TextRequest(BaseModel):
+        text: str
+
+def create_app():
+    """Create and configure FastAPI app"""
+    if not HAS_FASTAPI:
+        raise ImportError("FastAPI dependencies not installed")
+    
+    app = FastAPI(
+        title="Simple Document Text Extraction API",
+        description="Extract structured information from documents using regex patterns",
+        version="1.0.0"
+    )
+    
+    # Enable CORS
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=["*"],
+        allow_credentials=True,
+        allow_methods=["*"],
+        allow_headers=["*"],
+    )
+    
+    # Initialize processor
+    processor = SimpleDocumentProcessor()
+    
+    @app.get("/", response_class=HTMLResponse)
+    async def get_interface():
+        """Serve the web interface"""
+        return """
+        <!DOCTYPE html>
+        <html>
+        <head>
+            <title>Document Text Extraction Demo</title>
+            <style>
+                body { 
+                    font-family: Arial, sans-serif; 
+                    max-width: 1200px; 
+                    margin: 0 auto; 
+                    padding: 20px;
+                    background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+                    color: #333;
+                }
+                .container {
+                    background: white;
+                    padding: 30px;
+                    border-radius: 10px;
+                    box-shadow: 0 10px 30px rgba(0,0,0,0.2);
+                }
+                .header {
+                    text-align: center;
+                    margin-bottom: 30px;
+                }
+                .header h1 {
+                    color: #2c3e50;
+                    font-size: 2.5em;
+                    margin-bottom: 10px;
+                }
+                .header p {
+                    color: #7f8c8d;
+                    font-size: 1.2em;
+                }
+                .tabs {
+                    display: flex;
+                    margin-bottom: 20px;
+                }
+                .tab {
+                    flex: 1;
+                    text-align: center;
+                    padding: 15px;
+                    background: #ecf0f1;
+                    border: none;
+                    cursor: pointer;
+                    font-size: 16px;
+                    transition: background 0.3s;
+                }
+                .tab.active {
+                    background: #3498db;
+                    color: white;
+                }
+                .tab:hover {
+                    background: #3498db;
+                    color: white;
+                }
+                .tab-content {
+                    display: none;
+                    padding: 20px;
+                    border: 1px solid #ddd;
+                    border-radius: 5px;
+                }
+                .tab-content.active {
+                    display: block;
+                }
+                textarea {
+                    width: 100%;
+                    height: 150px;
+                    margin-bottom: 15px;
+                    padding: 10px;
+                    border: 1px solid #ddd;
+                    border-radius: 5px;
+                    font-size: 14px;
+                }
+                input[type="file"] {
+                    margin-bottom: 15px;
+                    padding: 10px;
+                }
+                button {
+                    background: #27ae60;
+                    color: white;
+                    padding: 12px 25px;
+                    border: none;
+                    border-radius: 5px;
+                    cursor: pointer;
+                    font-size: 16px;
+                    transition: background 0.3s;
+                }
+                button:hover {
+                    background: #2ecc71;
+                }
+                .results {
+                    margin-top: 20px;
+                    padding: 20px;
+                    background: #f8f9fa;
+                    border-radius: 5px;
+                    border-left: 4px solid #27ae60;
+                }
+                .entity {
+                    background: #e8f4fd;
+                    padding: 8px 12px;
+                    margin: 5px;
+                    border-radius: 20px;
+                    display: inline-block;
+                    font-size: 12px;
+                    border: 1px solid #3498db;
+                }
+                .entity.NAME { background: #ffeb3b; border-color: #ff9800; }
+                .entity.DATE { background: #4caf50; border-color: #2e7d32; color: white; }
+                .entity.AMOUNT { background: #f44336; border-color: #c62828; color: white; }
+                .entity.INVOICE_NO { background: #9c27b0; border-color: #6a1b9a; color: white; }
+                .entity.EMAIL { background: #00bcd4; border-color: #00838f; color: white; }
+                .entity.PHONE { background: #ff5722; border-color: #d84315; color: white; }
+                .entity.ADDRESS { background: #795548; border-color: #5d4037; color: white; }
+                .structured-data {
+                    background: #e8f5e8;
+                    padding: 15px;
+                    border-radius: 5px;
+                    margin-top: 15px;
+                }
+                .examples {
+                    background: #fff3cd;
+                    padding: 15px;
+                    border-radius: 5px;
+                    margin-top: 20px;
+                }
+                .example-btn {
+                    background: #6c757d;
+                    font-size: 12px;
+                    padding: 5px 10px;
+                    margin: 2px;
+                }
+                pre {
+                    background: #f8f9fa;
+                    padding: 15px;
+                    border-radius: 5px;
+                    overflow-x: auto;
+                    font-size: 12px;
+                    border: 1px solid #dee2e6;
+                }
+            </style>
+        </head>
+        <body>
+            <div class="container">
+                <div class="header">
+                    <h1> Document Text Extraction</h1>
+                    <p>Extract structured information from documents using AI patterns</p>
+                </div>
+                
+                <div class="tabs">
+                    <button class="tab active" onclick="showTab('text')">Enter Text</button>
+                    <button class="tab" onclick="showTab('file')">Upload File</button>
+                    <button class="tab" onclick="showTab('api')">API Docs</button>
+                </div>
+                
+                <div id="text-tab" class="tab-content active">
+                    <h3>Enter Text to Extract:</h3>
+                    <textarea id="textInput" placeholder="Paste your document text here...">Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567 Email: robert.white@email.com</textarea>
+                    <button onclick="extractFromText()">Extract Information</button>
+                    
+                    <div class="examples">
+                        <h4>Try These Examples:</h4>
+                        <button class="example-btn" onclick="useExample(0)">Invoice Example</button>
+                        <button class="example-btn" onclick="useExample(1)">Receipt Example</button>
+                        <button class="example-btn" onclick="useExample(2)">Business Document</button>
+                        <button class="example-btn" onclick="useExample(3)">Payment Notice</button>
+                    </div>
+                </div>
+                
+                <div id="file-tab" class="tab-content">
+                    <h3>Upload Document:</h3>
+                    <input type="file" id="fileInput" accept=".pdf,.docx,.txt,.jpg,.png,.tiff">
+                    <br>
+                    <button onclick="extractFromFile()">Upload & Extract</button>
+                    <p><em>Note: File upload processing is simplified in this demo</em></p>
+                </div>
+                
+                <div id="api-tab" class="tab-content">
+                    <h3>API Documentation</h3>
+                    <h4>Endpoints:</h4>
+                    <pre><strong>POST /extract-from-text</strong>
+Content-Type: application/json
+{
+  "text": "Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00"
+}</pre>
+                    
+                    <pre><strong>POST /extract-from-file</strong>
+Content-Type: multipart/form-data
+file: [uploaded file]</pre>
+                    
+                    <h4>Response Format:</h4>
+                    <pre>{
+  "status": "success",
+  "data": {
+    "original_text": "...",
+    "entities": [...],
+    "structured_data": {...},
+    "processing_timestamp": "2025-09-27T...",
+    "total_entities_found": 7,
+    "entity_types_found": ["NAME", "DATE", "AMOUNT", "INVOICE_NO"]
+  }
+}</pre>
+                </div>
+                
+                <div id="results"></div>
+            </div>
+            
+            <script>
+                const examples = [
+                    "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567 Email: robert.white@email.com",
+                    "Receipt for Michael Brown Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75 Contact: +1-555-987-6543",
+                    "Ms. Emma Wilson 456 Oak Street Payment due: January 15, 2025 Reference: INV-4567 Total: $1,750.25",
+                    "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Email: sarah.johnson@email.com"
+                ];
+                
+                function showTab(tabName) {
+                    // Hide all tabs
+                    document.querySelectorAll('.tab-content').forEach(content => {
+                        content.classList.remove('active');
+                    });
+                    document.querySelectorAll('.tab').forEach(tab => {
+                        tab.classList.remove('active');
+                    });
+                    
+                    // Show selected tab
+                    document.getElementById(tabName + '-tab').classList.add('active');
+                    event.target.classList.add('active');
+                }
+                
+                function useExample(index) {
+                    document.getElementById('textInput').value = examples[index];
+                }
+                
+                async function extractFromText() {
+                    const text = document.getElementById('textInput').value;
+                    if (!text.trim()) {
+                        alert('Please enter some text');
+                        return;
+                    }
+                    
+                    try {
+                        const response = await fetch('/extract-from-text', {
+                            method: 'POST',
+                            headers: {
+                                'Content-Type': 'application/json',
+                            },
+                            body: JSON.stringify({ text: text })
+                        });
+                        
+                        const result = await response.json();
+                        displayResults(result);
+                    } catch (error) {
+                        alert('Error: ' + error.message);
+                    }
+                }
+                
+                async function extractFromFile() {
+                    const fileInput = document.getElementById('fileInput');
+                    if (!fileInput.files[0]) {
+                        alert('Please select a file');
+                        return;
+                    }
+                    
+                    // For demo purposes, show that file upload would work
+                    alert('File upload processing would happen here. For now, using sample text extraction.');
+                    document.getElementById('textInput').value = examples[0];
+                    showTab('text');
+                    extractFromText();
+                }
+                
+                function displayResults(result) {
+                    const resultsDiv = document.getElementById('results');
+                    
+                    if (result.status !== 'success') {
+                        resultsDiv.innerHTML = '<div class="results"><h3>Error</h3><p>' + result.message + '</p></div>';
+                        return;
+                    }
+                    
+                    const data = result.data;
+                    let html = '<div class="results">';
+                    html += '<h3>Extraction Results</h3>';
+                    html += '<p><strong>Found:</strong> ' + data.total_entities_found + ' entities of ' + data.entity_types_found.length + ' types</p>';
+                    
+                    // Show entities
+                    html += '<h4>Detected Entities:</h4>';
+                    data.entities.forEach(entity => {
+                        html += '<span class="entity ' + entity.entity + '">' + entity.entity + ': ' + entity.text + ' (' + Math.round(entity.confidence * 100) + '%)</span> ';
+                    });
+                    
+                    // Show structured data
+                    if (Object.keys(data.structured_data).length > 0) {
+                        html += '<div class="structured-data">';
+                        html += '<h4>Structured Information:</h4>';
+                        html += '<ul>';
+                        for (const [key, value] of Object.entries(data.structured_data)) {
+                            html += '<li><strong>' + key + ':</strong> ' + value + '</li>';
+                        }
+                        html += '</ul>';
+                        html += '</div>';
+                    }
+                    
+                    // Show processing info
+                    html += '<p><small>🕒 Processed at: ' + new Date(data.processing_timestamp).toLocaleString() + '</small></p>';
+                    html += '</div>';
+                    
+                    resultsDiv.innerHTML = html;
+                }
+            </script>
+        </body>
+        </html>
+        """
+
+    @app.post("/extract-from-text")
+    async def extract_from_text(request: TextRequest):
+        """Extract entities from text"""
+        try:
+            result = processor.process_text(request.text)
+            return result
+        except Exception as e:
+            raise HTTPException(status_code=500, detail=str(e))
+
+    @app.post("/extract-from-file")
+    async def extract_from_file(file: UploadFile = File(...)):
+        """Extract entities from uploaded file"""
+        try:
+            # Read file content
+            content = await file.read()
+            
+            # For demo purposes, convert to text (simplified)
+            if file.filename.lower().endswith('.txt'):
+                text = content.decode('utf-8')
+            else:
+                # For other file types, use sample text in demo
+                text = "Demo processing for " + file.filename + ": Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00"
+            
+            result = processor.process_text(text)
+            return result
+            
+        except Exception as e:
+            raise HTTPException(status_code=500, detail=str(e))
+
+    @app.get("/health")
+    async def health_check():
+        """Health check endpoint"""
+        return {"status": "healthy", "timestamp": datetime.now().isoformat()}
+
+    return app
+
+def main():
+    """Main function to run the API server"""
+    if not HAS_FASTAPI:
+        print("FastAPI dependencies not installed.")
+        print("📦 Install with: pip install fastapi uvicorn python-multipart")
+        return
+    
+    print("Starting Simple Document Text Extraction API...")
+    print("Access the web interface at: http://localhost:7000")
+    print("API documentation at: http://localhost:7000/docs")
+    print("Health check at: http://localhost:7000/health")
+    print("\nServer starting...")
+    
+    app = create_app()
+    uvicorn.run(app, host="0.0.0.0", port=7000, log_level="info")
+
+if __name__ == "__main__":
+    main()

simple_demo.py

@@ -0,0 +1,565 @@
+"""
+Simplified demo of document text extraction without heavy ML dependencies.
+This demonstrates the core workflow and patterns without requiring PyTorch/Transformers.
+"""
+
+import json
+import re
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, List, Tuple, Any
+
+
+class SimpleDocumentProcessor:
+    """Simplified document processor for demo purposes."""
+    
+    def __init__(self):
+        """Initialize with regex patterns for entity extraction."""
+        self.entity_patterns = {
+            'NAME': [
+                r'\b(?:Mr\.|Mrs\.|Ms\.|Dr\.)\s+([A-Z][a-z]+ [A-Z][a-z]+)\b',
+                r'\b([A-Z][a-z]+ [A-Z][a-z]+)\b',
+            ],
+            'DATE': [
+                r'\b(\d{1,2}[/\-]\d{1,2}[/\-]\d{2,4})\b',
+                r'\b(\d{4}[/\-]\d{1,2}[/\-]\d{1,2})\b',
+                r'\b((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\s+\d{1,2},?\s+\d{2,4})\b'
+            ],
+            'INVOICE_NO': [
+                r'(?:Invoice\s+(?:No|Number|#):\s*)?([A-Z]{2,4}[-]?\d{3,6})',
+                r'(INV[-]?\d{3,6})',
+                r'(BL[-]?\d{3,6})',
+                r'(REC[-]?\d{3,6})',
+            ],
+            'AMOUNT': [
+                r'(\$\s*\d{1,3}(?:,\d{3})*(?:\.\d{2})?)',
+                r'(\d{1,3}(?:,\d{3})*(?:\.\d{2})?\s*(?:USD|EUR|GBP))',
+            ],
+            'PHONE': [
+                r'(\+?\d{1,3}[-.\s]?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4})',
+                r'(\(\d{3}\)\s*\d{3}-\d{4})',
+            ],
+            'EMAIL': [
+                r'\b([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,})\b',
+            ]
+        }
+    
+    def extract_entities(self, text: str) -> List[Dict[str, Any]]:
+        """Extract entities from text using regex patterns."""
+        entities = []
+        
+        for entity_type, patterns in self.entity_patterns.items():
+            for pattern in patterns:
+                matches = re.finditer(pattern, text, re.IGNORECASE)
+                for match in matches:
+                    entity_text = match.group(1) if match.groups() else match.group(0)
+                    entities.append({
+                        'entity': entity_type,
+                        'text': entity_text.strip(),
+                        'start': match.start(),
+                        'end': match.end(),
+                        'confidence': self.get_confidence_score(entity_type)
+                    })
+        
+        return entities
+    
+    def get_confidence_score(self, entity_type: str) -> float:
+        """Get confidence score for entity type."""
+        confidence_map = {
+            'NAME': 0.80,
+            'DATE': 0.85,
+            'AMOUNT': 0.85,
+            'INVOICE_NO': 0.90,
+            'EMAIL': 0.95,
+            'PHONE': 0.90,
+            'ADDRESS': 0.75
+        }
+        return confidence_map.get(entity_type, 0.70)
+    
+    def create_structured_data(self, entities: List[Dict[str, Any]]) -> Dict[str, str]:
+        """Create structured data from entities."""
+        structured = {}
+        
+        # Group entities by type
+        entity_groups = {}
+        for entity in entities:
+            entity_type = entity['entity']
+            if entity_type not in entity_groups:
+                entity_groups[entity_type] = []
+            entity_groups[entity_type].append(entity)
+        
+        # Select best entity for each type
+        for entity_type, group in entity_groups.items():
+            if group:
+                # Sort by confidence and length, take the best one
+                best_entity = max(group, key=lambda x: (x['confidence'], len(x['text'])))
+                
+                # Map to structured field names
+                field_mapping = {
+                    'NAME': 'Name',
+                    'DATE': 'Date', 
+                    'AMOUNT': 'Amount',
+                    'INVOICE_NO': 'InvoiceNo',
+                    'EMAIL': 'Email',
+                    'PHONE': 'Phone',
+                    'ADDRESS': 'Address'
+                }
+                
+                field_name = field_mapping.get(entity_type, entity_type)
+                structured[field_name] = best_entity['text']
+        
+        return structured
+    
+    def process_document(self, text: str) -> Dict[str, Any]:
+        """Process document text and extract information."""
+        entities = self.extract_entities(text)
+        structured_data = self.create_structured_data(entities)
+        
+        return {
+            'text': text,
+            'entities': entities,
+            'structured_data': structured_data,
+            'entity_count': len(entities),
+            'entity_types': list(set(e['entity'] for e in entities))
+        }
+
+
+def run_demo():
+    """Run the simplified document extraction demo."""
+    
+    print("SIMPLIFIED DOCUMENT TEXT EXTRACTION DEMO")
+    print("=" * 60)
+    print("This demo shows the core extraction logic using regex patterns")
+    print("(without the full ML pipeline for demonstration purposes)")
+    print()
+    
+    # Initialize processor
+    processor = SimpleDocumentProcessor()
+    
+    # Sample documents
+    sample_documents = [
+        {
+            "name": "Invoice Example 1",
+            "text": "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567 Email: robert.white@email.com"
+        },
+        {
+            "name": "Invoice Example 2", 
+            "text": "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Email: sarah.johnson@email.com"
+        },
+        {
+            "name": "Receipt Example",
+            "text": "Receipt for Michael Brown Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75 Contact: +1-555-987-6543"
+        },
+        {
+            "name": "Business Document",
+            "text": "Ms. Emma Wilson 456 Oak Street Payment due: January 15, 2025 Reference: INV-4567 Total: $1,750.25"
+        }
+    ]
+    
+    # Process each document
+    all_results = []
+    total_entities = 0
+    all_entity_types = set()
+    
+    for i, doc in enumerate(sample_documents, 1):
+        print(f"\nDocument {i}: {doc['name']}")
+        print("-" * 50)
+        print(f"Text: {doc['text']}")
+        print()
+        
+        # Process document
+        result = processor.process_document(doc['text'])
+        all_results.append(result)
+        
+        # Update totals
+        total_entities += result['entity_count']
+        all_entity_types.update(result['entity_types'])
+        
+        print(f"Extraction Results:")
+        print(f"   Found {result['entity_count']} entities")
+        print(f"   Entity types: {', '.join(result['entity_types'])}")
+        
+        # Show structured data if available
+        if result['structured_data']:
+            print(f"\nStructured Information:")
+            for key, value in result['structured_data'].items():
+                print(f"   {key}: {value}")
+        
+        # Show detailed entities
+        if result['entities']:
+            print(f"\nDetailed Entities:")
+            for entity in result['entities']:
+                print(f"   {entity['entity']}: '{entity['text']}' (confidence: {entity['confidence']*100:.0f}%)")
+    
+    # Save results
+    output_dir = Path("results")
+    output_dir.mkdir(exist_ok=True)
+    output_file = output_dir / "demo_extraction_results.json"
+    
+    # Prepare output data
+    output_data = {
+        'demo_info': {
+            'timestamp': datetime.now().isoformat(),
+            'documents_processed': len(sample_documents),
+            'total_entities_found': total_entities,
+            'unique_entity_types': sorted(list(all_entity_types))
+        },
+        'results': all_results
+    }
+    
+    # Save to file
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(output_data, f, indent=2, ensure_ascii=False)
+    
+    print(f"\nResults saved to: {output_file}")
+    
+    print(f"\nDemo Summary:")
+    print(f"   Documents processed: {len(sample_documents)}")
+    print(f"   Total entities found: {total_entities}")
+    print(f"   Total structured fields: {sum(len(r['structured_data']) for r in all_results)}")
+    print(f"   Unique entity types: {', '.join(sorted(all_entity_types))}")
+    
+    print(f"\nDemo completed successfully!")
+    
+    print(f"\nThis demonstrates the core extraction logic.")
+    print(f"   The full system would add:")
+    print(f"   - OCR for scanned documents")
+    print(f"   - ML model (DistilBERT) for better accuracy")
+    print(f"   - Web API for file uploads")
+    print(f"   - Training pipeline for custom domains")
+    
+    # Simulate API functionality
+    print(f"\nAPI FUNCTIONALITY SIMULATION")
+    print("=" * 40)
+    
+    sample_text = "Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00"
+    
+    print('API Request (POST /extract-from-text):')
+    print('  {')
+    print(f'  "text": "{sample_text}"')
+    print('}')
+    
+    print(f"\nAPI Response:")
+    api_result = processor.process_document(sample_text)
+    
+    api_response = {
+        "status": "success",
+        "data": {
+            "original_text": sample_text,
+            "entities": api_result['entities'],
+            "structured_data": api_result['structured_data'],
+            "processing_timestamp": datetime.now().isoformat(),
+            "total_entities_found": api_result['entity_count'],
+            "entity_types_found": api_result['entity_types']
+        }
+    }
+    
+    print(json.dumps(api_response, indent=2))
+    
+    print(f"\nTo run the full system:")
+    print(f"   1. Install ML dependencies: pip install torch transformers")
+    print(f"   2. Run training: python src/training_pipeline.py")
+    print(f"   3. Start API: python api/app.py")
+    print(f"   4. Open browser: http://localhost:8000")
+
+
+if __name__ == "__main__":
+    run_demo()
+    """Simplified document processor for demo purposes."""
+    
+    def __init__(self):
+        """Initialize with regex patterns for entity extraction."""
+        self.entity_patterns = {
+            'NAME': [
+                r'\b(?:Mr\.|Mrs\.|Ms\.|Dr\.)\s+([A-Z][a-z]+ [A-Z][a-z]+)\b',
+                r'\b([A-Z][a-z]+ [A-Z][a-z]+)\b',
+            ],
+            'DATE': [
+                r'\b(\d{1,2}[/\-]\d{1,2}[/\-]\d{2,4})\b',
+                r'\b(\d{4}[/\-]\d{1,2}[/\-]\d{1,2})\b',
+                r'\b((?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\s+\d{1,2},?\s+\d{2,4})\b'
+            ],
+            'INVOICE_NO': [
+                r'(?:Invoice\s+(?:No|Number|#):\s*)?([A-Z]{2,4}[-]?\d{3,6})',
+                r'(INV[-]?\d{3,6})',
+                r'(BL[-]?\d{3,6})',
+                r'(REC[-]?\d{3,6})',
+            ],
+            'AMOUNT': [
+                r'(\$\s*\d{1,3}(?:,\d{3})*(?:\.\d{2})?)',
+                r'(\d{1,3}(?:,\d{3})*(?:\.\d{2})?\s*(?:USD|EUR|GBP))',
+            ],
+            'PHONE': [
+                r'(\+?\d{1,3}[-.\s]?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4})',
+                r'(\(\d{3}\)\s*\d{3}-\d{4})',
+            ],
+            'EMAIL': [
+                r'\b([A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,})\b',
+            ]
+        }
+    
+    def extract_entities(self, text: str) -> List[Dict[str, Any]]:
+        """Extract entities from text using regex patterns."""
+        entities = []
+        
+        for entity_type, patterns in self.entity_patterns.items():
+            for pattern in patterns:
+                matches = re.finditer(pattern, text, re.IGNORECASE)
+                for match in matches:
+                    entity_text = match.group(1) if match.groups() else match.group(0)
+                    
+                    # Calculate position
+                    start_pos = match.start()
+                    end_pos = match.end()
+                    
+                    # Assign confidence based on pattern strength
+                    confidence = self._calculate_confidence(entity_type, entity_text, pattern)
+                    
+                    entity = {
+                        'entity': entity_type,
+                        'text': entity_text.strip(),
+                        'start': start_pos,
+                        'end': end_pos,
+                        'confidence': confidence
+                    }
+                    
+                    # Avoid duplicates
+                    if not self._is_duplicate(entity, entities):
+                        entities.append(entity)
+        
+        return entities
+    
+    def _calculate_confidence(self, entity_type: str, text: str, pattern: str) -> float:
+        """Calculate confidence score for extracted entity."""
+        base_confidence = 0.8
+        
+        # Boost confidence for specific patterns
+        if entity_type == 'EMAIL' and '@' in text:
+            base_confidence = 0.95
+        elif entity_type == 'PHONE' and len(re.sub(r'[^\d]', '', text)) >= 10:
+            base_confidence = 0.90
+        elif entity_type == 'AMOUNT' and '$' in text:
+            base_confidence = 0.85
+        elif entity_type == 'DATE':
+            base_confidence = 0.85
+        elif entity_type == 'INVOICE_NO' and any(prefix in text.upper() for prefix in ['INV', 'BL', 'REC']):
+            base_confidence = 0.90
+        
+        return min(base_confidence, 0.99)
+    
+    def _is_duplicate(self, new_entity: Dict, existing_entities: List[Dict]) -> bool:
+        """Check if entity is duplicate."""
+        for existing in existing_entities:
+            if (existing['entity'] == new_entity['entity'] and 
+                existing['text'].lower() == new_entity['text'].lower()):
+                return True
+        return False
+    
+    def postprocess_entities(self, entities: List[Dict], text: str) -> Dict[str, str]:
+        """Convert entities to structured data format."""
+        structured_data = {}
+        
+        # Group entities by type and pick the best one
+        entity_groups = {}
+        for entity in entities:
+            entity_type = entity['entity']
+            if entity_type not in entity_groups:
+                entity_groups[entity_type] = []
+            entity_groups[entity_type].append(entity)
+        
+        # Select best entity for each type
+        for entity_type, group in entity_groups.items():
+            best_entity = max(group, key=lambda x: x['confidence'])
+            
+            # Format the value
+            formatted_value = self._format_entity_value(best_entity['text'], entity_type)
+            
+            # Map to human-readable keys
+            readable_key = {
+                'NAME': 'Name',
+                'DATE': 'Date', 
+                'INVOICE_NO': 'InvoiceNo',
+                'AMOUNT': 'Amount',
+                'PHONE': 'Phone',
+                'EMAIL': 'Email'
+            }.get(entity_type, entity_type)
+            
+            structured_data[readable_key] = formatted_value
+        
+        return structured_data
+    
+    def _format_entity_value(self, text: str, entity_type: str) -> str:
+        """Format entity value based on type."""
+        text = text.strip()
+        
+        if entity_type == 'NAME':
+            return ' '.join(word.capitalize() for word in text.split())
+        elif entity_type == 'PHONE':
+            digits = re.sub(r'[^\d]', '', text)
+            if len(digits) == 10:
+                return f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
+            elif len(digits) == 11 and digits[0] == '1':
+                return f"+1 ({digits[1:4]}) {digits[4:7]}-{digits[7:]}"
+        elif entity_type == 'AMOUNT':
+            # Ensure proper formatting
+            if not text.startswith('$'):
+                return f"${text}"
+        
+        return text
+    
+    def process_text(self, text: str) -> Dict[str, Any]:
+        """Process text and return extraction results."""
+        # Extract entities
+        entities = self.extract_entities(text)
+        
+        # Create structured data
+        structured_data = self.postprocess_entities(entities, text)
+        
+        # Return complete result
+        return {
+            'original_text': text,
+            'entities': entities,
+            'structured_data': structured_data,
+            'processing_timestamp': datetime.now().isoformat(),
+            'total_entities_found': len(entities),
+            'entity_types_found': list(set(e['entity'] for e in entities))
+        }
+
+
+def run_demo():
+    """Run the document extraction demo."""
+    print("SIMPLIFIED DOCUMENT TEXT EXTRACTION DEMO")
+    print("=" * 60)
+    print("This demo shows the core extraction logic using regex patterns")
+    print("(without the full ML pipeline for demonstration purposes)")
+    print()
+    
+    # Initialize processor
+    processor = SimpleDocumentProcessor()
+    
+    # Sample documents
+    sample_docs = [
+        {
+            "name": "Invoice Example 1",
+            "text": "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250.00 Phone: (555) 123-4567"
+        },
+        {
+            "name": "Invoice Example 2", 
+            "text": "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Email: sarah.johnson@email.com"
+        },
+        {
+            "name": "Receipt Example",
+            "text": "Receipt for Michael Brown Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75 Contact: +1-555-987-6543"
+        },
+        {
+            "name": "Business Document",
+            "text": "Ms. Emma Wilson 456 Oak Street Payment due: January 15, 2025 Reference: INV-4567 Total: $1,750.25"
+        }
+    ]
+    
+    results = []
+    
+    for i, doc in enumerate(sample_docs, 1):
+        print(f"\nDocument {i}: {doc['name']}")
+        print("-" * 50)
+        print(f"Text: {doc['text']}")
+        
+        # Process the document
+        result = processor.process_text(doc['text'])
+        results.append({
+            'document_name': doc['name'],
+            **result
+        })
+        
+        # Display results
+        print(f"\nExtraction Results:")
+        print(f"   Found {result['total_entities_found']} entities")
+        print(f"   Entity types: {', '.join(result['entity_types_found'])}")
+        
+        # Show structured data
+        if result['structured_data']:
+            print(f"\nStructured Information:")
+            for key, value in result['structured_data'].items():
+                print(f"   {key}: {value}")
+        
+        # Show detailed entities
+        if result['entities']:
+            print(f"\nDetailed Entities:")
+            for entity in result['entities']:
+                confidence_pct = int(entity['confidence'] * 100)
+                print(f"   {entity['entity']}: '{entity['text']}' (confidence: {confidence_pct}%)")
+    
+    # Save results
+    output_dir = Path("results")
+    output_dir.mkdir(exist_ok=True)
+    
+    output_file = output_dir / "demo_extraction_results.json"
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(results, f, indent=2, ensure_ascii=False)
+    
+    print(f"\n💾 Results saved to: {output_file}")
+    
+    # Summary statistics
+    total_entities = sum(len(r['entities']) for r in results)
+    total_structured_fields = sum(len(r['structured_data']) for r in results)
+    unique_entity_types = set()
+    for r in results:
+        unique_entity_types.update(r['entity_types_found'])
+    
+    print(f"\nDemo Summary:")
+    print(f"   Documents processed: {len(results)}")
+    print(f"   Total entities found: {total_entities}")
+    print(f"   Total structured fields: {total_structured_fields}")
+    print(f"   Unique entity types: {', '.join(sorted(unique_entity_types))}")
+    
+    print(f"\nDemo completed successfully!")
+    print(f"\nThis demonstrates the core extraction logic.")
+    print(f"   The full system would add:")
+    print(f"   - OCR for scanned documents")
+    print(f"   - ML model (DistilBERT) for better accuracy")
+    print(f"   - Web API for file uploads")
+    print(f"   - Training pipeline for custom domains")
+    
+    return results
+
+
+def show_api_simulation():
+    """Simulate the API functionality."""
+    print(f"\n🌐 API FUNCTIONALITY SIMULATION")
+    print("=" * 40)
+    
+    processor = SimpleDocumentProcessor()
+    
+    # Simulate API request
+    sample_request = {
+        "text": "Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00"
+    }
+    
+    print(f"API Request (POST /extract-from-text):")
+    print(f"  {json.dumps(sample_request, indent=2)}")
+    
+    # Process
+    result = processor.process_text(sample_request["text"])
+    
+    # Simulate API response
+    api_response = {
+        "status": "success",
+        "data": result
+    }
+    
+    print(f"\nAPI Response:")
+    print(f"  {json.dumps(api_response, indent=2)}")
+
+
+if __name__ == "__main__":
+    # Run the main demo
+    results = run_demo()
+    
+    # Show API simulation
+    show_api_simulation()
+    
+    print(f"\nTo run the full system:")
+    print(f"   1. Install ML dependencies: pip install torch transformers")
+    print(f"   2. Run training: python src/training_pipeline.py")
+    print(f"   3. Start API: python api/app.py")
+    print(f"   4. Open browser: http://localhost:8000")

src/data_preparation.py

@@ -0,0 +1,339 @@
+"""
+Data preparation module for document text extraction.
+Handles OCR, text cleaning, and dataset creation for NER training.
+"""
+
+import os
+import json
+import re
+import pytesseract
+from PIL import Image
+import pandas as pd
+import cv2
+import numpy as np
+from typing import List, Dict, Tuple, Optional
+from pathlib import Path
+import fitz  # PyMuPDF for PDF processing
+from docx import Document
+import easyocr
+
+
+class DocumentProcessor:
+    """Handles document processing, OCR, and text extraction."""
+    
+    def __init__(self, tesseract_path: Optional[str] = None):
+        """Initialize document processor with OCR settings."""
+        if tesseract_path:
+            pytesseract.pytesseract.tesseract_cmd = tesseract_path
+        
+        # Initialize EasyOCR reader
+        self.ocr_reader = easyocr.Reader(['en'])
+        
+        # Entity patterns for initial labeling
+        self.entity_patterns = {
+            'NAME': [
+                r'\b[A-Z][a-z]+ [A-Z][a-z]+\b',  # First Last
+                r'(?:Mr\.|Mrs\.|Ms\.|Dr\.)\s+[A-Z][a-z]+ [A-Z][a-z]+',  # Title + Name
+            ],
+            'DATE': [
+                r'\b\d{1,2}[/\-]\d{1,2}[/\-]\d{2,4}\b',  # DD/MM/YYYY
+                r'\b\d{4}[/\-]\d{1,2}[/\-]\d{1,2}\b',    # YYYY/MM/DD
+                r'\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\s+\d{1,2},?\s+\d{2,4}\b'
+            ],
+            'INVOICE_NO': [
+                r'(?:Invoice\s+(?:No|Number|#):\s*)?([A-Z]{2,4}[-]?\d{3,6})',
+                r'(?:INV[-]?\d{3,6})',
+            ],
+            'AMOUNT': [
+                r'\$\s*\d{1,3}(?:,\d{3})*(?:\.\d{2})?',  # $1,000.00
+                r'\d{1,3}(?:,\d{3})*(?:\.\d{2})?\s*(?:USD|EUR|GBP)',  # 1000.00 USD
+            ],
+            'ADDRESS': [
+                r'\d+\s+[A-Za-z\s]+(?:Street|St|Avenue|Ave|Road|Rd|Drive|Dr|Lane|Ln).*',
+            ],
+            'PHONE': [
+                r'\+?\d{1,3}[-.\s]?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}',
+                r'\(\d{3}\)\s*\d{3}-\d{4}',
+            ],
+            'EMAIL': [
+                r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
+            ]
+        }
+    
+    def extract_text_from_pdf(self, pdf_path: str) -> str:
+        """Extract text from PDF file."""
+        try:
+            doc = fitz.open(pdf_path)
+            text = ""
+            for page_num in range(len(doc)):
+                page = doc.load_page(page_num)
+                text += page.get_text()
+            doc.close()
+            return text
+        except Exception as e:
+            print(f"Error extracting text from PDF {pdf_path}: {e}")
+            return ""
+    
+    def extract_text_from_docx(self, docx_path: str) -> str:
+        """Extract text from DOCX file."""
+        try:
+            doc = Document(docx_path)
+            text = ""
+            for paragraph in doc.paragraphs:
+                text += paragraph.text + "\n"
+            return text
+        except Exception as e:
+            print(f"Error extracting text from DOCX {docx_path}: {e}")
+            return ""
+    
+    def preprocess_image(self, image_path: str) -> np.ndarray:
+        """Preprocess image for better OCR results."""
+        img = cv2.imread(image_path)
+        if img is None:
+            raise ValueError(f"Could not load image: {image_path}")
+        
+        # Convert to grayscale
+        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+        
+        # Apply Gaussian blur to reduce noise
+        blurred = cv2.GaussianBlur(gray, (5, 5), 0)
+        
+        # Apply adaptive threshold
+        thresh = cv2.adaptiveThreshold(
+            blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2
+        )
+        
+        return thresh
+    
+    def extract_text_with_tesseract(self, image_path: str) -> str:
+        """Extract text using Tesseract OCR."""
+        try:
+            preprocessed_img = self.preprocess_image(image_path)
+            
+            # Configure Tesseract
+            custom_config = r'--oem 3 --psm 6'
+            text = pytesseract.image_to_string(preprocessed_img, config=custom_config)
+            
+            return text
+        except Exception as e:
+            print(f"Error with Tesseract OCR on {image_path}: {e}")
+            return ""
+    
+    def extract_text_with_easyocr(self, image_path: str) -> str:
+        """Extract text using EasyOCR."""
+        try:
+            results = self.ocr_reader.readtext(image_path)
+            text = " ".join([result[1] for result in results])
+            return text
+        except Exception as e:
+            print(f"Error with EasyOCR on {image_path}: {e}")
+            return ""
+    
+    def extract_text_from_image(self, image_path: str, use_easyocr: bool = True) -> str:
+        """Extract text from image using OCR."""
+        if use_easyocr:
+            text = self.extract_text_with_easyocr(image_path)
+            if not text.strip():  # Fallback to Tesseract
+                text = self.extract_text_with_tesseract(image_path)
+        else:
+            text = self.extract_text_with_tesseract(image_path)
+            if not text.strip():  # Fallback to EasyOCR
+                text = self.extract_text_with_easyocr(image_path)
+        
+        return text
+    
+    def clean_text(self, text: str) -> str:
+        """Clean and normalize extracted text."""
+        # Remove extra whitespace
+        text = re.sub(r'\s+', ' ', text)
+        
+        # Remove special characters but keep important punctuation
+        text = re.sub(r'[^\w\s\.\,\:\;\-\$\(\)\[\]\/]', '', text)
+        
+        # Normalize whitespace around punctuation
+        text = re.sub(r'\s*([,.;:])\s*', r'\1 ', text)
+        
+        return text.strip()
+    
+    def process_document(self, file_path: str) -> str:
+        """Process any document type and extract text."""
+        file_path = Path(file_path)
+        file_ext = file_path.suffix.lower()
+        
+        if file_ext == '.pdf':
+            text = self.extract_text_from_pdf(str(file_path))
+        elif file_ext == '.docx':
+            text = self.extract_text_from_docx(str(file_path))
+        elif file_ext in ['.png', '.jpg', '.jpeg', '.tiff', '.bmp']:
+            text = self.extract_text_from_image(str(file_path))
+        else:
+            raise ValueError(f"Unsupported file type: {file_ext}")
+        
+        return self.clean_text(text)
+
+
+class NERDatasetCreator:
+    """Creates NER training datasets from processed documents."""
+    
+    def __init__(self, document_processor: DocumentProcessor):
+        self.document_processor = document_processor
+        self.entity_labels = ['O', 'B-NAME', 'I-NAME', 'B-DATE', 'I-DATE', 
+                             'B-INVOICE_NO', 'I-INVOICE_NO', 'B-AMOUNT', 'I-AMOUNT',
+                             'B-ADDRESS', 'I-ADDRESS', 'B-PHONE', 'I-PHONE',
+                             'B-EMAIL', 'I-EMAIL']
+    
+    def auto_label_text(self, text: str) -> List[Tuple[str, str]]:
+        """Automatically label text using regex patterns."""
+        words = text.split()
+        labels = ['O'] * len(words)
+        
+        # Track word positions in original text
+        word_positions = []
+        start = 0
+        for word in words:
+            pos = text.find(word, start)
+            word_positions.append((pos, pos + len(word)))
+            start = pos + len(word)
+        
+        # Apply entity patterns
+        for entity_type, patterns in self.document_processor.entity_patterns.items():
+            for pattern in patterns:
+                matches = list(re.finditer(pattern, text, re.IGNORECASE))
+                for match in matches:
+                    match_start, match_end = match.span()
+                    
+                    # Find which words overlap with this match
+                    first_word_idx = None
+                    last_word_idx = None
+                    
+                    for i, (word_start, word_end) in enumerate(word_positions):
+                        if word_start >= match_start and word_end <= match_end:
+                            if first_word_idx is None:
+                                first_word_idx = i
+                            last_word_idx = i
+                        elif word_start < match_end and word_end > match_start:
+                            # Partial overlap
+                            if first_word_idx is None:
+                                first_word_idx = i
+                            last_word_idx = i
+                    
+                    # Apply BIO labeling
+                    if first_word_idx is not None:
+                        labels[first_word_idx] = f'B-{entity_type}'
+                        for i in range(first_word_idx + 1, last_word_idx + 1):
+                            labels[i] = f'I-{entity_type}'
+        
+        return list(zip(words, labels))
+    
+    def create_training_example(self, text: str) -> Dict:
+        """Create a training example from text."""
+        labeled_tokens = self.auto_label_text(text)
+        
+        tokens = [token for token, _ in labeled_tokens]
+        labels = [label for _, label in labeled_tokens]
+        
+        return {
+            'tokens': tokens,
+            'labels': labels,
+            'text': text
+        }
+    
+    def create_sample_dataset(self) -> List[Dict]:
+        """Create sample training data for demonstration."""
+        sample_texts = [
+            "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250",
+            "Bill for Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50",
+            "Payment due from Michael Brown on 01/12/2025. Reference: PAY-3067. Sum: $890.00",
+            "Receipt for Emma Wilson Invoice: REC-4089 Date: 2025-04-22 Amount: $1,750.25",
+            "Dr. James Smith 123 Main Street Boston MA 02101 Phone: (555) 123-4567 Email: james@email.com",
+            "Ms. Lisa Anderson 456 Oak Avenue New York NY 10001 Contact: +1-555-987-6543",
+            "Invoice INV-5678 issued to David Lee on February 5, 2025 for $3,400.00",
+            "Bill #BIL-9012 for Jennifer Garcia dated 2025-05-15. Total amount: $567.89"
+        ]
+        
+        dataset = []
+        for text in sample_texts:
+            example = self.create_training_example(text)
+            dataset.append(example)
+        
+        return dataset
+    
+    def process_documents_folder(self, folder_path: str) -> List[Dict]:
+        """Process all documents in a folder and create training dataset."""
+        folder_path = Path(folder_path)
+        dataset = []
+        
+        if not folder_path.exists():
+            print(f"Folder {folder_path} does not exist. Creating sample dataset instead.")
+            return self.create_sample_dataset()
+        
+        supported_extensions = ['.pdf', '.docx', '.png', '.jpg', '.jpeg', '.tiff', '.bmp']
+        
+        for file_path in folder_path.rglob('*'):
+            if file_path.suffix.lower() in supported_extensions:
+                try:
+                    print(f"Processing {file_path.name}...")
+                    text = self.document_processor.process_document(str(file_path))
+                    
+                    if text.strip():  # Only process non-empty texts
+                        example = self.create_training_example(text)
+                        example['source_file'] = str(file_path)
+                        dataset.append(example)
+                        print(f"Processed {file_path.name}")
+                    else:
+                        print(f"No text extracted from {file_path.name}")
+                        
+                except Exception as e:
+                    print(f"Error processing {file_path.name}: {e}")
+        
+        if not dataset:
+            print("No documents processed. Creating sample dataset.")
+            return self.create_sample_dataset()
+        
+        return dataset
+    
+    def save_dataset(self, dataset: List[Dict], output_path: str):
+        """Save dataset to JSON file."""
+        output_path = Path(output_path)
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        
+        with open(output_path, 'w', encoding='utf-8') as f:
+            json.dump(dataset, f, indent=2, ensure_ascii=False)
+        
+        print(f"Dataset saved to {output_path}")
+        print(f"Total examples: {len(dataset)}")
+        
+        # Print statistics
+        all_labels = []
+        for example in dataset:
+            all_labels.extend(example['labels'])
+        
+        label_counts = {}
+        for label in all_labels:
+            label_counts[label] = label_counts.get(label, 0) + 1
+        
+        print("\nLabel distribution:")
+        for label, count in sorted(label_counts.items()):
+            print(f"  {label}: {count}")
+
+
+def main():
+    """Main function to demonstrate data preparation."""
+    # Initialize components
+    processor = DocumentProcessor()
+    dataset_creator = NERDatasetCreator(processor)
+    
+    # Process documents (or create sample data)
+    raw_data_path = "data/raw"
+    dataset = dataset_creator.process_documents_folder(raw_data_path)
+    
+    # Save processed dataset
+    output_path = "data/processed/ner_dataset.json"
+    dataset_creator.save_dataset(dataset, output_path)
+    
+    print(f"\nData preparation completed!")
+    print(f"Processed {len(dataset)} documents")
+
+
+if __name__ == "__main__":
+    main()

src/inference.py

@@ -0,0 +1,437 @@
+"""
+Inference pipeline for document text extraction.
+Processes new documents and extracts structured information using trained SLM.
+"""
+
+import json
+import torch
+import re
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple, Any
+from datetime import datetime
+import numpy as np
+
+from src.data_preparation import DocumentProcessor
+from src.model import DocumentNERModel, NERTrainer, ModelConfig
+
+
+class DocumentInference:
+    """Inference pipeline for extracting structured data from documents."""
+    
+    def __init__(self, model_path: str):
+        """Initialize inference pipeline with trained model."""
+        self.model_path = model_path
+        self.config = self._load_config()
+        self.model = None
+        self.trainer = None
+        self.document_processor = DocumentProcessor()
+        
+        # Load the trained model
+        self._load_model()
+        
+        # Post-processing patterns for field validation and formatting
+        self.postprocess_patterns = {
+            'DATE': [
+                r'\b\d{1,2}[/\-]\d{1,2}[/\-]\d{2,4}\b',
+                r'\b\d{4}[/\-]\d{1,2}[/\-]\d{1,2}\b',
+                r'\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\s+\d{1,2},?\s+\d{2,4}\b'
+            ],
+            'AMOUNT': [
+                r'\$\s*\d{1,3}(?:,\d{3})*(?:\.\d{2})?',
+                r'\d{1,3}(?:,\d{3})*(?:\.\d{2})?\s*(?:USD|EUR|GBP)'
+            ],
+            'PHONE': [
+                r'\+?\d{1,3}[-.\s]?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}',
+                r'\(\d{3}\)\s*\d{3}-\d{4}'
+            ],
+            'EMAIL': [
+                r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
+            ]
+        }
+    
+    def _load_config(self) -> ModelConfig:
+        """Load training configuration."""
+        config_path = Path(self.model_path) / "training_config.json"
+        
+        if config_path.exists():
+            with open(config_path, 'r') as f:
+                config_dict = json.load(f)
+                config = ModelConfig(**config_dict)
+        else:
+            print("No training config found. Using default configuration.")
+            config = ModelConfig()
+        
+        return config
+    
+    def _load_model(self):
+        """Load the trained model and tokenizer."""
+        try:
+            # Create model and trainer
+            self.model = DocumentNERModel(self.config)
+            self.trainer = NERTrainer(self.model, self.config)
+            
+            # Load the trained weights
+            self.trainer.load_model(self.model_path)
+            
+            print(f"Model loaded successfully from {self.model_path}")
+            
+        except Exception as e:
+            raise Exception(f"Failed to load model from {self.model_path}: {e}")
+    
+    def predict_entities(self, text: str) -> List[Dict[str, Any]]:
+        """Predict entities from text using the trained model."""
+        # Tokenize the text
+        tokens = text.split()
+        
+        # Prepare input for the model
+        inputs = self.trainer.tokenizer(
+            tokens,
+            is_split_into_words=True,
+            padding='max_length',
+            truncation=True,
+            max_length=self.config.max_length,
+            return_tensors='pt'
+        )
+        
+        # Move to device
+        inputs = {k: v.to(self.trainer.device) for k, v in inputs.items()}
+        
+        # Get predictions
+        with torch.no_grad():
+            predictions, probabilities = self.model.predict(
+                inputs['input_ids'], 
+                inputs['attention_mask']
+            )
+        
+        # Convert predictions to labels
+        word_ids = inputs['input_ids'][0].cpu().numpy()
+        pred_labels = predictions[0].cpu().numpy()
+        probs = probabilities[0].cpu().numpy()
+        
+        # Align predictions with original tokens
+        word_ids_list = self.trainer.tokenizer.convert_ids_to_tokens(word_ids)
+        
+        # Extract entities
+        entities = self._extract_entities_from_predictions(
+            tokens, pred_labels, probs, word_ids_list
+        )
+        
+        return entities
+    
+    def _extract_entities_from_predictions(self, tokens: List[str], 
+                                         pred_labels: np.ndarray,
+                                         probs: np.ndarray,
+                                         word_ids_list: List[str]) -> List[Dict[str, Any]]:
+        """Extract entities from model predictions."""
+        entities = []
+        current_entity = None
+        
+        # Map tokenizer output back to original tokens
+        token_idx = 0
+        
+        for i, (token_id, label_id) in enumerate(zip(word_ids_list, pred_labels)):
+            if token_id in ['[CLS]', '[SEP]', '[PAD]']:
+                continue
+            
+            label = self.config.id2label.get(label_id, 'O')
+            confidence = float(np.max(probs[i]))
+            
+            if label.startswith('B-'):
+                # Start of new entity
+                if current_entity:
+                    entities.append(current_entity)
+                
+                entity_type = label[2:]  # Remove 'B-' prefix
+                current_entity = {
+                    'entity': entity_type,
+                    'text': token_id if not token_id.startswith('##') else token_id[2:],
+                    'start': token_idx,
+                    'end': token_idx + 1,
+                    'confidence': confidence
+                }
+            
+            elif label.startswith('I-') and current_entity:
+                # Continue current entity
+                entity_type = label[2:]  # Remove 'I-' prefix
+                if current_entity['entity'] == entity_type:
+                    if token_id.startswith('##'):
+                        current_entity['text'] += token_id[2:]
+                    else:
+                        current_entity['text'] += ' ' + token_id
+                    current_entity['end'] = token_idx + 1
+                    current_entity['confidence'] = min(current_entity['confidence'], confidence)
+            
+            else:
+                # 'O' label or end of entity
+                if current_entity:
+                    entities.append(current_entity)
+                    current_entity = None
+            
+            if not token_id.startswith('##'):
+                token_idx += 1
+        
+        # Add the last entity if it exists
+        if current_entity:
+            entities.append(current_entity)
+        
+        return entities
+    
+    def postprocess_entities(self, entities: List[Dict[str, Any]], 
+                           original_text: str) -> Dict[str, Any]:
+        """Post-process and structure extracted entities."""
+        structured_data = {}
+        
+        for entity in entities:
+            entity_type = entity['entity']
+            entity_text = entity['text']
+            confidence = entity['confidence']
+            
+            # Apply post-processing patterns for validation
+            if entity_type in self.postprocess_patterns:
+                is_valid = self._validate_entity(entity_text, entity_type)
+                if not is_valid:
+                    continue
+            
+            # Format the entity value
+            formatted_value = self._format_entity_value(entity_text, entity_type)
+            
+            # Store the best entity for each type (highest confidence)
+            if entity_type not in structured_data or confidence > structured_data[entity_type]['confidence']:
+                structured_data[entity_type] = {
+                    'value': formatted_value,
+                    'confidence': confidence,
+                    'original_text': entity_text
+                }
+        
+        # Convert to final format
+        final_data = {}
+        entity_mapping = {
+            'NAME': 'Name',
+            'DATE': 'Date',
+            'INVOICE_NO': 'InvoiceNo',
+            'AMOUNT': 'Amount',
+            'ADDRESS': 'Address',
+            'PHONE': 'Phone',
+            'EMAIL': 'Email'
+        }
+        
+        for entity_type, entity_data in structured_data.items():
+            human_readable_key = entity_mapping.get(entity_type, entity_type)
+            final_data[human_readable_key] = entity_data['value']
+        
+        return final_data
+    
+    def _validate_entity(self, text: str, entity_type: str) -> bool:
+        """Validate entity using regex patterns."""
+        patterns = self.postprocess_patterns.get(entity_type, [])
+        
+        for pattern in patterns:
+            if re.search(pattern, text, re.IGNORECASE):
+                return True
+        
+        return False
+    
+    def _format_entity_value(self, text: str, entity_type: str) -> str:
+        """Format entity value based on its type."""
+        text = text.strip()
+        
+        if entity_type == 'DATE':
+            # Normalize date format
+            date_patterns = [
+                (r'(\d{1,2})[/\-](\d{1,2})[/\-](\d{2,4})', r'\1/\2/\3'),
+                (r'(\d{4})[/\-](\d{1,2})[/\-](\d{1,2})', r'\3/\2/\1')
+            ]
+            
+            for pattern, replacement in date_patterns:
+                match = re.search(pattern, text)
+                if match:
+                    return re.sub(pattern, replacement, text)
+        
+        elif entity_type == 'AMOUNT':
+            # Normalize amount format
+            amount_match = re.search(r'[\$\d,\.]+', text)
+            if amount_match:
+                return amount_match.group()
+        
+        elif entity_type == 'PHONE':
+            # Normalize phone format
+            digits = re.sub(r'[^\d]', '', text)
+            if len(digits) == 10:
+                return f"({digits[:3]}) {digits[3:6]}-{digits[6:]}"
+            elif len(digits) == 11 and digits[0] == '1':
+                return f"+1 ({digits[1:4]}) {digits[4:7]}-{digits[7:]}"
+        
+        elif entity_type == 'NAME':
+            # Capitalize name properly
+            return ' '.join(word.capitalize() for word in text.split())
+        
+        return text
+    
+    def process_document(self, file_path: str) -> Dict[str, Any]:
+        """Process a document and extract structured information."""
+        print(f"Processing document: {file_path}")
+        
+        try:
+            # Extract text from document
+            text = self.document_processor.process_document(file_path)
+            
+            if not text.strip():
+                return {
+                    'error': 'No text could be extracted from the document',
+                    'file_path': file_path
+                }
+            
+            # Predict entities
+            entities = self.predict_entities(text)
+            
+            # Post-process and structure data
+            structured_data = self.postprocess_entities(entities, text)
+            
+            # Create result
+            result = {
+                'file_path': file_path,
+                'extracted_text': text[:500] + '...' if len(text) > 500 else text,
+                'entities': entities,
+                'structured_data': structured_data,
+                'processing_timestamp': datetime.now().isoformat(),
+                'model_path': self.model_path
+            }
+            
+            print(f"Successfully processed {file_path}")
+            print(f"   Found {len(entities)} entities")
+            print(f"   Structured fields: {list(structured_data.keys())}")
+            
+            return result
+            
+        except Exception as e:
+            error_result = {
+                'error': str(e),
+                'file_path': file_path,
+                'processing_timestamp': datetime.now().isoformat()
+            }
+            print(f"Error processing {file_path}: {e}")
+            return error_result
+    
+    def process_text_directly(self, text: str) -> Dict[str, Any]:
+        """Process text directly without file operations."""
+        print("Processing text directly...")
+        
+        try:
+            # Clean the text
+            cleaned_text = self.document_processor.clean_text(text)
+            
+            # Predict entities
+            entities = self.predict_entities(cleaned_text)
+            
+            # Post-process and structure data
+            structured_data = self.postprocess_entities(entities, cleaned_text)
+            
+            # Create result
+            result = {
+                'original_text': text,
+                'cleaned_text': cleaned_text,
+                'entities': entities,
+                'structured_data': structured_data,
+                'processing_timestamp': datetime.now().isoformat(),
+                'model_path': self.model_path
+            }
+            
+            print(f"Successfully processed text")
+            print(f"   Found {len(entities)} entities")
+            print(f"   Structured fields: {list(structured_data.keys())}")
+            
+            return result
+            
+        except Exception as e:
+            error_result = {
+                'error': str(e),
+                'original_text': text,
+                'processing_timestamp': datetime.now().isoformat()
+            }
+            print(f"Error processing text: {e}")
+            return error_result
+    
+    def batch_process_documents(self, file_paths: List[str]) -> List[Dict[str, Any]]:
+        """Process multiple documents in batch."""
+        print(f"Processing {len(file_paths)} documents...")
+        
+        results = []
+        for i, file_path in enumerate(file_paths):
+            print(f"\nProcessing {i+1}/{len(file_paths)}: {Path(file_path).name}")
+            result = self.process_document(file_path)
+            results.append(result)
+        
+        print(f"\nBatch processing completed!")
+        print(f"   Successfully processed: {sum(1 for r in results if 'error' not in r)}")
+        print(f"   Errors: {sum(1 for r in results if 'error' in r)}")
+        
+        return results
+    
+    def save_results(self, results: List[Dict[str, Any]], output_path: str):
+        """Save processing results to JSON file."""
+        output_path = Path(output_path)
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        
+        with open(output_path, 'w', encoding='utf-8') as f:
+            json.dump(results, f, indent=2, ensure_ascii=False)
+        
+        print(f"Results saved to: {output_path}")
+
+
+def create_demo_inference(model_path: str = "models/document_ner_model") -> DocumentInference:
+    """Create inference pipeline for demonstration."""
+    try:
+        inference = DocumentInference(model_path)
+        return inference
+    except Exception as e:
+        print(f"Failed to create inference pipeline: {e}")
+        print("Make sure you have trained the model first by running training_pipeline.py")
+        raise
+
+
+def demo_text_extraction():
+    """Demonstrate text extraction with sample texts."""
+    print("DOCUMENT TEXT EXTRACTION - INFERENCE DEMO")
+    print("=" * 60)
+    
+    # Sample texts for demonstration
+    sample_texts = [
+        "Invoice sent to Robert White on 15/09/2025 Invoice No: INV-1024 Amount: $1,250",
+        "Bill for Dr. Sarah Johnson dated March 10, 2025. Invoice Number: BL-2045. Total: $2,300.50 Phone: (555) 123-4567",
+        "Receipt for Michael Brown 456 Oak Street Boston MA Email: michael@email.com Invoice: REC-3089 Date: 2025-04-22 Amount: $890.75"
+    ]
+    
+    # Create inference pipeline
+    try:
+        inference = create_demo_inference()
+        
+        results = []
+        for i, text in enumerate(sample_texts):
+            print(f"\nProcessing Sample Text {i+1}:")
+            print("-" * 40)
+            print(f"Text: {text}")
+            
+            result = inference.process_text_directly(text)
+            results.append(result)
+            
+            if 'error' not in result:
+                print(f"Structured Output: {json.dumps(result['structured_data'], indent=2)}")
+            else:
+                print(f"Error: {result['error']}")
+        
+        # Save results
+        inference.save_results(results, "results/demo_extraction_results.json")
+        
+        print("\nDemo completed successfully!")
+        
+    except Exception as e:
+        print(f"Demo failed: {e}")
+
+
+def main():
+    """Main function for inference demonstration."""
+    demo_text_extraction()
+
+
+if __name__ == "__main__":
+    main()

src/model.py

@@ -0,0 +1,396 @@
+"""
+Small Language Model (SLM) architecture for document text extraction.
+Uses DistilBERT with transfer learning for Named Entity Recognition.
+"""
+
+import torch
+import torch.nn as nn
+from torch.utils.data import Dataset, DataLoader
+from transformers import (
+    DistilBertTokenizer, 
+    DistilBertForTokenClassification,
+    DistilBertConfig,
+    get_linear_schedule_with_warmup
+)
+from typing import List, Dict, Tuple, Optional
+import json
+import numpy as np
+from sklearn.model_selection import train_test_split
+from dataclasses import dataclass
+
+
+@dataclass
+class ModelConfig:
+    """Configuration for the SLM model."""
+    model_name: str = "distilbert-base-uncased"
+    max_length: int = 512
+    batch_size: int = 16
+    learning_rate: float = 2e-5
+    num_epochs: int = 3
+    warmup_steps: int = 500
+    weight_decay: float = 0.01
+    dropout_rate: float = 0.3
+    
+    # Entity labels
+    entity_labels: List[str] = None
+    
+    def __post_init__(self):
+        if self.entity_labels is None:
+            self.entity_labels = [
+                'O', 'B-NAME', 'I-NAME', 'B-DATE', 'I-DATE', 
+                'B-INVOICE_NO', 'I-INVOICE_NO', 'B-AMOUNT', 'I-AMOUNT',
+                'B-ADDRESS', 'I-ADDRESS', 'B-PHONE', 'I-PHONE',
+                'B-EMAIL', 'I-EMAIL'
+            ]
+    
+    @property
+    def num_labels(self) -> int:
+        return len(self.entity_labels)
+    
+    @property
+    def label2id(self) -> Dict[str, int]:
+        return {label: i for i, label in enumerate(self.entity_labels)}
+    
+    @property
+    def id2label(self) -> Dict[int, str]:
+        return {i: label for i, label in enumerate(self.entity_labels)}
+
+
+class NERDataset(Dataset):
+    """PyTorch Dataset for NER training."""
+    
+    def __init__(self, dataset: List[Dict], tokenizer: DistilBertTokenizer, 
+                 config: ModelConfig, mode: str = 'train'):
+        self.dataset = dataset
+        self.tokenizer = tokenizer
+        self.config = config
+        self.mode = mode
+        
+        # Prepare tokenized data
+        self.tokenized_data = self._tokenize_and_align_labels()
+    
+    def _tokenize_and_align_labels(self) -> List[Dict]:
+        """Tokenize text and align labels with subword tokens."""
+        tokenized_data = []
+        
+        for example in self.dataset:
+            tokens = example['tokens']
+            labels = example['labels']
+            
+            # Tokenize each word and track alignments
+            tokenized_inputs = self.tokenizer(
+                tokens,
+                is_split_into_words=True,
+                padding='max_length',
+                truncation=True,
+                max_length=self.config.max_length,
+                return_tensors='pt'
+            )
+            
+            # Align labels with subword tokens
+            word_ids = tokenized_inputs.word_ids()
+            aligned_labels = []
+            previous_word_idx = None
+            
+            for word_idx in word_ids:
+                if word_idx is None:
+                    # Special tokens get -100 (ignored in loss computation)
+                    aligned_labels.append(-100)
+                elif word_idx != previous_word_idx:
+                    # First subword of a word gets the original label
+                    if word_idx < len(labels):
+                        label = labels[word_idx]
+                        aligned_labels.append(self.config.label2id.get(label, 0))
+                    else:
+                        aligned_labels.append(-100)
+                else:
+                    # Subsequent subwords of the same word
+                    if word_idx < len(labels):
+                        label = labels[word_idx]
+                        if label.startswith('B-'):
+                            # Convert B- to I- for subword tokens
+                            i_label = label.replace('B-', 'I-')
+                            aligned_labels.append(self.config.label2id.get(i_label, 0))
+                        else:
+                            aligned_labels.append(self.config.label2id.get(label, 0))
+                    else:
+                        aligned_labels.append(-100)
+                
+                previous_word_idx = word_idx
+            
+            tokenized_data.append({
+                'input_ids': tokenized_inputs['input_ids'].squeeze(),
+                'attention_mask': tokenized_inputs['attention_mask'].squeeze(),
+                'labels': torch.tensor(aligned_labels, dtype=torch.long),
+                'original_tokens': tokens,
+                'original_labels': labels
+            })
+        
+        return tokenized_data
+    
+    def __len__(self) -> int:
+        return len(self.tokenized_data)
+    
+    def __getitem__(self, idx: int) -> Dict[str, torch.Tensor]:
+        return {
+            'input_ids': self.tokenized_data[idx]['input_ids'],
+            'attention_mask': self.tokenized_data[idx]['attention_mask'],
+            'labels': self.tokenized_data[idx]['labels']
+        }
+
+
+class DocumentNERModel(nn.Module):
+    """DistilBERT-based model for document NER."""
+    
+    def __init__(self, config: ModelConfig):
+        super().__init__()
+        self.config = config
+        
+        # Load pre-trained DistilBERT configuration
+        bert_config = DistilBertConfig.from_pretrained(
+            config.model_name,
+            num_labels=config.num_labels,
+            id2label=config.id2label,
+            label2id=config.label2id,
+            dropout=config.dropout_rate,
+            attention_dropout=config.dropout_rate
+        )
+        
+        # Initialize model with token classification head
+        self.model = DistilBertForTokenClassification.from_pretrained(
+            config.model_name,
+            config=bert_config
+        )
+        
+        # Additional dropout layer for regularization
+        self.dropout = nn.Dropout(config.dropout_rate)
+    
+    def forward(self, input_ids, attention_mask=None, labels=None):
+        """Forward pass through the model."""
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            labels=labels
+        )
+        
+        return outputs
+    
+    def predict(self, input_ids, attention_mask):
+        """Make predictions without computing loss."""
+        with torch.no_grad():
+            outputs = self.model(
+                input_ids=input_ids,
+                attention_mask=attention_mask
+            )
+            predictions = torch.argmax(outputs.logits, dim=-1)
+            probabilities = torch.softmax(outputs.logits, dim=-1)
+        
+        return predictions, probabilities
+
+
+class NERTrainer:
+    """Trainer class for the NER model."""
+    
+    def __init__(self, model: DocumentNERModel, config: ModelConfig):
+        self.model = model
+        self.config = config
+        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+        self.model.to(self.device)
+        
+        # Initialize tokenizer
+        self.tokenizer = DistilBertTokenizer.from_pretrained(config.model_name)
+    
+    def prepare_dataloaders(self, dataset: List[Dict], 
+                          test_size: float = 0.2) -> Tuple[DataLoader, DataLoader]:
+        """Prepare training and validation dataloaders."""
+        # Split dataset
+        train_data, val_data = train_test_split(
+            dataset, test_size=test_size, random_state=42
+        )
+        
+        # Create datasets
+        train_dataset = NERDataset(train_data, self.tokenizer, self.config, 'train')
+        val_dataset = NERDataset(val_data, self.tokenizer, self.config, 'val')
+        
+        # Create dataloaders
+        train_dataloader = DataLoader(
+            train_dataset, 
+            batch_size=self.config.batch_size, 
+            shuffle=True
+        )
+        val_dataloader = DataLoader(
+            val_dataset, 
+            batch_size=self.config.batch_size, 
+            shuffle=False
+        )
+        
+        return train_dataloader, val_dataloader
+    
+    def train(self, train_dataloader: DataLoader, 
+              val_dataloader: DataLoader) -> Dict[str, List[float]]:
+        """Train the NER model."""
+        # Initialize optimizer and scheduler
+        optimizer = torch.optim.AdamW(
+            self.model.parameters(),
+            lr=self.config.learning_rate,
+            weight_decay=self.config.weight_decay
+        )
+        
+        total_steps = len(train_dataloader) * self.config.num_epochs
+        scheduler = get_linear_schedule_with_warmup(
+            optimizer,
+            num_warmup_steps=self.config.warmup_steps,
+            num_training_steps=total_steps
+        )
+        
+        # Training history
+        history = {
+            'train_loss': [],
+            'val_loss': [],
+            'val_accuracy': []
+        }
+        
+        print(f"Training on device: {self.device}")
+        print(f"Total training steps: {total_steps}")
+        
+        for epoch in range(self.config.num_epochs):
+            print(f"\nEpoch {epoch + 1}/{self.config.num_epochs}")
+            print("-" * 50)
+            
+            # Training phase
+            train_loss = self._train_epoch(train_dataloader, optimizer, scheduler)
+            history['train_loss'].append(train_loss)
+            
+            # Validation phase
+            val_loss, val_accuracy = self._validate_epoch(val_dataloader)
+            history['val_loss'].append(val_loss)
+            history['val_accuracy'].append(val_accuracy)
+            
+            print(f"Train Loss: {train_loss:.4f}")
+            print(f"Val Loss: {val_loss:.4f}")
+            print(f"Val Accuracy: {val_accuracy:.4f}")
+        
+        return history
+    
+    def _train_epoch(self, dataloader: DataLoader, optimizer, scheduler) -> float:
+        """Train for one epoch."""
+        self.model.train()
+        total_loss = 0
+        
+        for batch_idx, batch in enumerate(dataloader):
+            # Move batch to device
+            batch = {k: v.to(self.device) for k, v in batch.items()}
+            
+            # Forward pass
+            outputs = self.model(**batch)
+            loss = outputs.loss
+            
+            # Backward pass
+            optimizer.zero_grad()
+            loss.backward()
+            
+            # Gradient clipping
+            torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
+            
+            optimizer.step()
+            scheduler.step()
+            
+            total_loss += loss.item()
+            
+            if batch_idx % 10 == 0:
+                print(f"Batch {batch_idx}/{len(dataloader)}, Loss: {loss.item():.4f}")
+        
+        return total_loss / len(dataloader)
+    
+    def _validate_epoch(self, dataloader: DataLoader) -> Tuple[float, float]:
+        """Validate for one epoch."""
+        self.model.eval()
+        total_loss = 0
+        total_correct = 0
+        total_tokens = 0
+        
+        with torch.no_grad():
+            for batch in dataloader:
+                batch = {k: v.to(self.device) for k, v in batch.items()}
+                
+                outputs = self.model(**batch)
+                loss = outputs.loss
+                
+                total_loss += loss.item()
+                
+                # Calculate accuracy (ignoring -100 labels)
+                predictions = torch.argmax(outputs.logits, dim=-1)
+                labels = batch['labels']
+                
+                # Mask for valid labels (not -100)
+                valid_mask = labels != -100
+                
+                correct = (predictions == labels) & valid_mask
+                total_correct += correct.sum().item()
+                total_tokens += valid_mask.sum().item()
+        
+        avg_loss = total_loss / len(dataloader)
+        accuracy = total_correct / total_tokens if total_tokens > 0 else 0
+        
+        return avg_loss, accuracy
+    
+    def save_model(self, save_path: str):
+        """Save the trained model and tokenizer."""
+        self.model.model.save_pretrained(save_path)
+        self.tokenizer.save_pretrained(save_path)
+        
+        # Save config
+        config_path = f"{save_path}/training_config.json"
+        with open(config_path, 'w') as f:
+            json.dump(vars(self.config), f, indent=2)
+        
+        print(f"Model saved to {save_path}")
+    
+    def load_model(self, model_path: str):
+        """Load a pre-trained model."""
+        self.model.model = DistilBertForTokenClassification.from_pretrained(model_path)
+        self.tokenizer = DistilBertTokenizer.from_pretrained(model_path)
+        self.model.to(self.device)
+        print(f"Model loaded from {model_path}")
+
+
+def create_model_and_trainer(config: Optional[ModelConfig] = None) -> Tuple[DocumentNERModel, NERTrainer]:
+    """Create model and trainer with configuration."""
+    if config is None:
+        config = ModelConfig()
+    
+    model = DocumentNERModel(config)
+    trainer = NERTrainer(model, config)
+    
+    return model, trainer
+
+
+def main():
+    """Demonstrate model creation and setup."""
+    # Create configuration
+    config = ModelConfig(
+        batch_size=8,  # Smaller batch size for demo
+        num_epochs=2,
+        learning_rate=3e-5
+    )
+    
+    print("Model Configuration:")
+    print(f"Model: {config.model_name}")
+    print(f"Max Length: {config.max_length}")
+    print(f"Batch Size: {config.batch_size}")
+    print(f"Learning Rate: {config.learning_rate}")
+    print(f"Number of Labels: {config.num_labels}")
+    print(f"Entity Labels: {config.entity_labels}")
+    
+    # Create model and trainer
+    model, trainer = create_model_and_trainer(config)
+    
+    print(f"\nModel created successfully!")
+    print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
+    print(f"Trainable parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}")
+    
+    return model, trainer
+
+
+if __name__ == "__main__":
+    main()

src/training_pipeline.py

@@ -0,0 +1,342 @@
+"""
+Complete training pipeline for document text extraction using SLM.
+Handles data loading, model training, evaluation, and saving.
+"""
+
+import os
+import json
+import torch
+from pathlib import Path
+from typing import Dict, List, Optional
+import matplotlib.pyplot as plt
+import seaborn as sns
+from sklearn.metrics import classification_report, confusion_matrix
+import numpy as np
+from seqeval.metrics import f1_score, precision_score, recall_score, classification_report as seq_classification_report
+
+from src.data_preparation import DocumentProcessor, NERDatasetCreator
+from src.model import DocumentNERModel, NERTrainer, ModelConfig, create_model_and_trainer
+
+
+class TrainingPipeline:
+    """Complete training pipeline for document NER."""
+    
+    def __init__(self, config: Optional[ModelConfig] = None):
+        """Initialize training pipeline."""
+        self.config = config or ModelConfig()
+        self.model = None
+        self.trainer = None
+        self.history = {}
+        
+        # Create necessary directories
+        self._create_directories()
+    
+    def _create_directories(self):
+        """Create necessary directories for training."""
+        directories = [
+            "data/raw",
+            "data/processed",
+            "models",
+            "results/plots",
+            "results/metrics"
+        ]
+        
+        for directory in directories:
+            Path(directory).mkdir(parents=True, exist_ok=True)
+    
+    def prepare_data(self, data_path: Optional[str] = None) -> List[Dict]:
+        """Prepare training data from documents or create sample data."""
+        print("=" * 60)
+        print("STEP 1: DATA PREPARATION")
+        print("=" * 60)
+        
+        # Initialize document processor and dataset creator
+        processor = DocumentProcessor()
+        dataset_creator = NERDatasetCreator(processor)
+        
+        # Process documents or create sample data
+        if data_path and Path(data_path).exists():
+            print(f"Processing documents from: {data_path}")
+            dataset = dataset_creator.process_documents_folder(data_path)
+        else:
+            print("No document path provided or path doesn't exist.")
+            print("Creating sample dataset for demonstration...")
+            dataset = dataset_creator.create_sample_dataset()
+        
+        # Save processed dataset
+        output_path = "data/processed/ner_dataset.json"
+        dataset_creator.save_dataset(dataset, output_path)
+        
+        print(f"Data preparation completed!")
+        print(f"Dataset saved to: {output_path}")
+        print(f"Total examples: {len(dataset)}")
+        
+        return dataset
+    
+    def initialize_model(self):
+        """Initialize model and trainer."""
+        print("\n" + "=" * 60)
+        print("STEP 2: MODEL INITIALIZATION")
+        print("=" * 60)
+        
+        self.model, self.trainer = create_model_and_trainer(self.config)
+        
+        print(f"Model initialized: {self.config.model_name}")
+        print(f"Model parameters: {sum(p.numel() for p in self.model.parameters()):,}")
+        print(f"Device: {self.trainer.device}")
+        print(f"Number of entity labels: {self.config.num_labels}")
+        
+        return self.model, self.trainer
+    
+    def train_model(self, dataset: List[Dict]) -> Dict[str, List[float]]:
+        """Train the NER model."""
+        print("\n" + "=" * 60)
+        print("STEP 3: MODEL TRAINING")
+        print("=" * 60)
+        
+        # Prepare dataloaders
+        print("Preparing training and validation data...")
+        train_dataloader, val_dataloader = self.trainer.prepare_dataloaders(dataset)
+        
+        print(f"Training samples: {len(train_dataloader.dataset)}")
+        print(f"Validation samples: {len(val_dataloader.dataset)}")
+        print(f"Training batches: {len(train_dataloader)}")
+        print(f"Validation batches: {len(val_dataloader)}")
+        
+        # Start training
+        print(f"\nStarting training for {self.config.num_epochs} epochs...")
+        self.history = self.trainer.train(train_dataloader, val_dataloader)
+        
+        print(f"Training completed!")
+        return self.history
+    
+    def evaluate_model(self, dataset: List[Dict]) -> Dict:
+        """Evaluate the trained model."""
+        print("\n" + "=" * 60)
+        print("STEP 4: MODEL EVALUATION")
+        print("=" * 60)
+        
+        # Prepare test data
+        _, test_dataloader = self.trainer.prepare_dataloaders(dataset, test_size=0.3)
+        
+        # Evaluate
+        evaluation_results = self._detailed_evaluation(test_dataloader)
+        
+        # Save evaluation results
+        results_path = "results/metrics/evaluation_results.json"
+        with open(results_path, 'w') as f:
+            json.dump(evaluation_results, f, indent=2)
+        
+        print(f"Evaluation completed!")
+        print(f"Results saved to: {results_path}")
+        
+        return evaluation_results
+    
+    def _detailed_evaluation(self, test_dataloader) -> Dict:
+        """Perform detailed evaluation of the model."""
+        self.model.eval()
+        
+        all_predictions = []
+        all_labels = []
+        all_tokens = []
+        
+        print("Running evaluation on test set...")
+        
+        with torch.no_grad():
+            for batch_idx, batch in enumerate(test_dataloader):
+                # Move to device
+                batch = {k: v.to(self.trainer.device) for k, v in batch.items()}
+                
+                # Get predictions
+                predictions, probabilities = self.model.predict(
+                    batch['input_ids'], 
+                    batch['attention_mask']
+                )
+                
+                # Convert to numpy
+                pred_np = predictions.cpu().numpy()
+                labels_np = batch['labels'].cpu().numpy()
+                
+                # Process each sequence in the batch
+                for i in range(pred_np.shape[0]):
+                    pred_seq = []
+                    label_seq = []
+                    
+                    for j in range(pred_np.shape[1]):
+                        if labels_np[i][j] != -100:  # Valid label
+                            pred_label = self.config.id2label[pred_np[i][j]]
+                            true_label = self.config.id2label[labels_np[i][j]]
+                            
+                            pred_seq.append(pred_label)
+                            label_seq.append(true_label)
+                    
+                    if pred_seq and label_seq:  # Non-empty sequences
+                        all_predictions.append(pred_seq)
+                        all_labels.append(label_seq)
+        
+        print(f"Processed {len(all_predictions)} sequences")
+        
+        # Calculate metrics using seqeval
+        f1 = f1_score(all_labels, all_predictions)
+        precision = precision_score(all_labels, all_predictions)
+        recall = recall_score(all_labels, all_predictions)
+        
+        # Detailed classification report
+        report = seq_classification_report(all_labels, all_predictions)
+        
+        evaluation_results = {
+            'f1_score': f1,
+            'precision': precision,
+            'recall': recall,
+            'classification_report': report,
+            'num_test_sequences': len(all_predictions)
+        }
+        
+        # Print results
+        print(f"\nEvaluation Results:")
+        print(f"F1 Score: {f1:.4f}")
+        print(f"Precision: {precision:.4f}")
+        print(f"Recall: {recall:.4f}")
+        print(f"\nDetailed Classification Report:")
+        print(report)
+        
+        return evaluation_results
+    
+    def plot_training_history(self):
+        """Plot training history."""
+        if not self.history:
+            print("No training history available.")
+            return
+        
+        print("\n" + "=" * 60)
+        print("STEP 5: PLOTTING TRAINING HISTORY")
+        print("=" * 60)
+        
+        # Create plots
+        fig, axes = plt.subplots(1, 2, figsize=(15, 5))
+        
+        # Loss plot
+        epochs = range(1, len(self.history['train_loss']) + 1)
+        axes[0].plot(epochs, self.history['train_loss'], 'b-', label='Training Loss')
+        axes[0].plot(epochs, self.history['val_loss'], 'r-', label='Validation Loss')
+        axes[0].set_title('Model Loss')
+        axes[0].set_xlabel('Epoch')
+        axes[0].set_ylabel('Loss')
+        axes[0].legend()
+        axes[0].grid(True)
+        
+        # Accuracy plot
+        axes[1].plot(epochs, self.history['val_accuracy'], 'g-', label='Validation Accuracy')
+        axes[1].set_title('Model Accuracy')
+        axes[1].set_xlabel('Epoch')
+        axes[1].set_ylabel('Accuracy')
+        axes[1].legend()
+        axes[1].grid(True)
+        
+        plt.tight_layout()
+        
+        # Save plot
+        plot_path = "results/plots/training_history.png"
+        plt.savefig(plot_path, dpi=300, bbox_inches='tight')
+        plt.close()
+        
+        print(f"Training history plot saved to: {plot_path}")
+    
+    def save_model(self, model_name: str = "document_ner_model"):
+        """Save the trained model."""
+        print("\n" + "=" * 60)
+        print("STEP 6: SAVING MODEL")
+        print("=" * 60)
+        
+        save_path = f"models/{model_name}"
+        self.trainer.save_model(save_path)
+        
+        # Save training history
+        history_path = f"{save_path}/training_history.json"
+        with open(history_path, 'w') as f:
+            json.dump(self.history, f, indent=2)
+        
+        print(f"Model saved to: {save_path}")
+        print(f"Training history saved to: {history_path}")
+        
+        return save_path
+    
+    def run_complete_pipeline(self, data_path: Optional[str] = None, 
+                            model_name: str = "document_ner_model") -> str:
+        """Run the complete training pipeline."""
+        print("STARTING COMPLETE TRAINING PIPELINE")
+        print("=" * 80)
+        
+        try:
+            # Step 1: Prepare data
+            dataset = self.prepare_data(data_path)
+            
+            # Step 2: Initialize model
+            self.initialize_model()
+            
+            # Step 3: Train model
+            self.train_model(dataset)
+            
+            # Step 4: Evaluate model
+            self.evaluate_model(dataset)
+            
+            # Step 5: Plot training history
+            self.plot_training_history()
+            
+            # Step 6: Save model
+            model_path = self.save_model(model_name)
+            
+            print("\n" + "=" * 20)
+            print("TRAINING PIPELINE COMPLETED SUCCESSFULLY!")
+            print("=" * 20)
+            print(f"Model saved to: {model_path}")
+            print(f"Training completed in {self.config.num_epochs} epochs")
+            print(f"Final validation accuracy: {self.history['val_accuracy'][-1]:.4f}")
+            
+            return model_path
+            
+        except Exception as e:
+            print(f"\nError in training pipeline: {e}")
+            raise
+
+
+def create_custom_config() -> ModelConfig:
+    """Create a custom configuration for training."""
+    config = ModelConfig(
+        model_name="distilbert-base-uncased",
+        max_length=256,  # Shorter sequences for faster training
+        batch_size=16,   # Adjust based on your GPU memory
+        learning_rate=2e-5,
+        num_epochs=3,
+        warmup_steps=500,
+        weight_decay=0.01,
+        dropout_rate=0.1
+    )
+    
+    return config
+
+
+def main():
+    """Main function to run the complete training pipeline."""
+    print("Document Text Extraction - Training Pipeline")
+    print("=" * 50)
+    
+    # Create custom configuration
+    config = create_custom_config()
+    
+    # Initialize training pipeline
+    pipeline = TrainingPipeline(config)
+    
+    # Run complete pipeline
+    # You can provide a path to your document folder here
+    # pipeline.run_complete_pipeline(data_path="data/raw")
+    
+    # For demonstration, we'll use sample data
+    model_path = pipeline.run_complete_pipeline()
+    
+    print(f"\nTraining completed! Model saved to: {model_path}")
+    print("You can now use this model for document text extraction!")
+
+
+if __name__ == "__main__":
+    main()

tests/test_extraction.py

@@ -0,0 +1,290 @@
+"""
+Test cases for the document text extraction system.
+"""
+
+import unittest
+import json
+from pathlib import Path
+import tempfile
+import os
+
+from src.data_preparation import DocumentProcessor, NERDatasetCreator
+from src.model import ModelConfig, create_model_and_trainer
+from src.inference import DocumentInference
+
+
+class TestDocumentProcessor(unittest.TestCase):
+    """Test cases for document processing."""
+    
+    def setUp(self):
+        """Set up test fixtures."""
+        self.processor = DocumentProcessor()
+    
+    def test_clean_text(self):
+        """Test text cleaning functionality."""
+        dirty_text = "  This   is    a    test  text!!!  "
+        clean_text = self.processor.clean_text(dirty_text)
+        self.assertEqual(clean_text, "This is a test text!")
+    
+    def test_entity_patterns(self):
+        """Test entity pattern matching."""
+        test_text = "Invoice sent to John Doe on 01/15/2025 Invoice No: INV-1001 Amount: $1,500.00"
+        
+        # Test that patterns exist
+        self.assertIn('NAME', self.processor.entity_patterns)
+        self.assertIn('DATE', self.processor.entity_patterns)
+        self.assertIn('INVOICE_NO', self.processor.entity_patterns)
+        self.assertIn('AMOUNT', self.processor.entity_patterns)
+
+
+class TestNERDatasetCreator(unittest.TestCase):
+    """Test cases for NER dataset creation."""
+    
+    def setUp(self):
+        """Set up test fixtures."""
+        self.processor = DocumentProcessor()
+        self.dataset_creator = NERDatasetCreator(self.processor)
+    
+    def test_auto_label_text(self):
+        """Test automatic text labeling."""
+        test_text = "Invoice sent to Robert White on 15/09/2025 Amount: $1,250"
+        labeled_tokens = self.dataset_creator.auto_label_text(test_text)
+        
+        # Check that we get tokens and labels
+        self.assertIsInstance(labeled_tokens, list)
+        self.assertGreater(len(labeled_tokens), 0)
+        
+        # Check that each item is a (token, label) tuple
+        for token, label in labeled_tokens:
+            self.assertIsInstance(token, str)
+            self.assertIsInstance(label, str)
+    
+    def test_create_training_example(self):
+        """Test training example creation."""
+        test_text = "Invoice INV-1001 for $500"
+        example = self.dataset_creator.create_training_example(test_text)
+        
+        # Check required fields
+        self.assertIn('tokens', example)
+        self.assertIn('labels', example)
+        self.assertIn('text', example)
+        
+        # Check that tokens and labels have the same length
+        self.assertEqual(len(example['tokens']), len(example['labels']))
+    
+    def test_create_sample_dataset(self):
+        """Test sample dataset creation."""
+        dataset = self.dataset_creator.create_sample_dataset()
+        
+        # Check that we get a non-empty dataset
+        self.assertIsInstance(dataset, list)
+        self.assertGreater(len(dataset), 0)
+        
+        # Check first example structure
+        first_example = dataset[0]
+        self.assertIn('tokens', first_example)
+        self.assertIn('labels', first_example)
+        self.assertIn('text', first_example)
+
+
+class TestModelConfig(unittest.TestCase):
+    """Test cases for model configuration."""
+    
+    def test_default_config(self):
+        """Test default configuration creation."""
+        config = ModelConfig()
+        
+        # Check default values
+        self.assertEqual(config.model_name, "distilbert-base-uncased")
+        self.assertEqual(config.max_length, 512)
+        self.assertEqual(config.batch_size, 16)
+        
+        # Check entity labels
+        self.assertIsInstance(config.entity_labels, list)
+        self.assertGreater(len(config.entity_labels), 0)
+        self.assertIn('O', config.entity_labels)
+        
+        # Check label mappings
+        self.assertIsInstance(config.label2id, dict)
+        self.assertIsInstance(config.id2label, dict)
+        self.assertEqual(len(config.label2id), len(config.entity_labels))
+    
+    def test_custom_config(self):
+        """Test custom configuration."""
+        custom_labels = ['O', 'B-TEST', 'I-TEST']
+        config = ModelConfig(
+            batch_size=32,
+            learning_rate=1e-5,
+            entity_labels=custom_labels
+        )
+        
+        self.assertEqual(config.batch_size, 32)
+        self.assertEqual(config.learning_rate, 1e-5)
+        self.assertEqual(config.entity_labels, custom_labels)
+        self.assertEqual(config.num_labels, 3)
+
+
+class TestModelCreation(unittest.TestCase):
+    """Test cases for model creation."""
+    
+    def test_create_model_and_trainer(self):
+        """Test model and trainer creation."""
+        config = ModelConfig(
+            batch_size=4,  # Small batch for testing
+            num_epochs=1,
+            entity_labels=['O', 'B-TEST', 'I-TEST']
+        )
+        
+        model, trainer = create_model_and_trainer(config)
+        
+        # Check that objects are created
+        self.assertIsNotNone(model)
+        self.assertIsNotNone(trainer)
+        
+        # Check configuration
+        self.assertEqual(trainer.config.batch_size, 4)
+        self.assertEqual(trainer.config.num_epochs, 1)
+
+
+class TestInference(unittest.TestCase):
+    """Test cases for inference pipeline."""
+    
+    @classmethod
+    def setUpClass(cls):
+        """Set up class-level fixtures."""
+        # Create a minimal trained model for testing
+        # This is a placeholder - in real testing, you'd use a pre-trained model
+        cls.model_path = "test_model"
+        cls.test_text = "Invoice sent to John Doe on 01/15/2025 Amount: $500.00"
+    
+    def test_entity_validation(self):
+        """Test entity validation patterns."""
+        # We can test the patterns without loading a full model
+        test_patterns = {
+            'DATE': ['01/15/2025', '2025-01-15', 'January 15, 2025'],
+            'AMOUNT': ['$500.00', '$1,250.50', '1000.00 USD'],
+            'EMAIL': ['test@email.com', 'user.name@domain.co.uk'],
+            'PHONE': ['(555) 123-4567', '+1-555-987-6543', '555-123-4567']
+        }
+        
+        # This test checks that our regex patterns work
+        import re
+        
+        date_pattern = r'\b\d{1,2}[/\-]\d{1,2}[/\-]\d{2,4}\b'
+        self.assertTrue(re.search(date_pattern, '01/15/2025'))
+        
+        amount_pattern = r'\$\s*\d{1,3}(?:,\d{3})*(?:\.\d{2})?'
+        self.assertTrue(re.search(amount_pattern, '$1,250.50'))
+        
+        email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
+        self.assertTrue(re.search(email_pattern, 'test@email.com'))
+
+
+class TestEndToEnd(unittest.TestCase):
+    """End-to-end integration tests."""
+    
+    def test_data_preparation_flow(self):
+        """Test the complete data preparation flow."""
+        # Create processor and dataset creator
+        processor = DocumentProcessor()
+        dataset_creator = NERDatasetCreator(processor)
+        
+        # Create sample dataset
+        dataset = dataset_creator.create_sample_dataset()
+        
+        # Verify dataset structure
+        self.assertIsInstance(dataset, list)
+        self.assertGreater(len(dataset), 0)
+        
+        for example in dataset:
+            self.assertIn('tokens', example)
+            self.assertIn('labels', example)
+            self.assertIn('text', example)
+            self.assertEqual(len(example['tokens']), len(example['labels']))
+    
+    def test_model_config_flow(self):
+        """Test model configuration and creation flow."""
+        # Create configuration
+        config = ModelConfig(batch_size=4, num_epochs=1)
+        
+        # Create model and trainer
+        model, trainer = create_model_and_trainer(config)
+        
+        # Verify objects exist and have correct configuration
+        self.assertIsNotNone(model)
+        self.assertIsNotNone(trainer)
+        self.assertEqual(trainer.config.batch_size, 4)
+        self.assertEqual(trainer.config.num_epochs, 1)
+    
+    def test_save_and_load_dataset(self):
+        """Test saving and loading dataset."""
+        # Create dataset
+        processor = DocumentProcessor()
+        dataset_creator = NERDatasetCreator(processor)
+        dataset = dataset_creator.create_sample_dataset()
+        
+        # Save to temporary file
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
+            temp_path = f.name
+            json.dump(dataset, f, indent=2)
+        
+        try:
+            # Load and verify
+            with open(temp_path, 'r') as f:
+                loaded_dataset = json.load(f)
+            
+            self.assertEqual(len(loaded_dataset), len(dataset))
+            self.assertEqual(loaded_dataset[0]['text'], dataset[0]['text'])
+            
+        finally:
+            # Clean up
+            os.unlink(temp_path)
+
+
+def run_tests():
+    """Run all tests."""
+    print("Running Document Text Extraction Tests")
+    print("=" * 50)
+    
+    # Create test suite
+    test_suite = unittest.TestSuite()
+    
+    # Add test classes
+    test_classes = [
+        TestDocumentProcessor,
+        TestNERDatasetCreator,
+        TestModelConfig,
+        TestModelCreation,
+        TestInference,
+        TestEndToEnd
+    ]
+    
+    for test_class in test_classes:
+        tests = unittest.TestLoader().loadTestsFromTestCase(test_class)
+        test_suite.addTests(tests)
+    
+    # Run tests
+    runner = unittest.TextTestRunner(verbosity=2)
+    result = runner.run(test_suite)
+    
+    # Print summary
+    if result.wasSuccessful():
+        print(f"\nAll tests passed! ({result.testsRun} tests)")
+    else:
+        print(f"\n{len(result.failures)} failures, {len(result.errors)} errors")
+        
+        if result.failures:
+            print("\nFailures:")
+            for test, failure in result.failures:
+                print(f"  {test}: {failure}")
+        
+        if result.errors:
+            print("\nErrors:")
+            for test, error in result.errors:
+                print(f"  {test}: {error}")
+    
+    return result.wasSuccessful()
+
+
+if __name__ == "__main__":
+    run_tests()