Add comprehensive model card

HF_MODEL_CARD.md

@@ -0,0 +1,353 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- document-processing
+- ocr
+- ner
+- text-classification
+- information-extraction
+- invoice
+- receipt
+- form
+- pytorch
+- transformers
+datasets:
+- naver-clova-ix/cord-v2
+- SROIE
+- FUNSD
+pipeline_tag: document-question-answering
+metrics:
+- accuracy
+- f1
+library_name: transformers
+---
+
+# IDP Machine Learning - Intelligent Document Processing
+
+<div align="center">
+
+**Production-grade AI-powered document processing system for extracting structured data from documents**
+
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python](https://img.shields.io/badge/Python-3.10+-green.svg)](https://python.org)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-orange.svg)](https://pytorch.org)
+[![Transformers](https://img.shields.io/badge/Transformers-4.x-yellow.svg)](https://huggingface.co/transformers)
+
+</div>
+
+## 🎯 Overview
+
+The IDP (Intelligent Document Processing) System is a complete end-to-end pipeline for:
+- **Document Classification** - Automatically identifies document types (Invoice, Receipt, Form, Bank Statement)
+- **Named Entity Recognition** - Extracts key fields (dates, amounts, IDs, names, addresses)
+- **OCR Integration** - Text extraction from images and PDFs
+
+### Key Features
+- 📄 **Multi-format Support**: PDF, PNG, JPEG, TIFF
+- ⚡ **Fast Processing**: <2 seconds per document on CPU
+- 💾 **Lightweight**: <500MB total memory footprint
+- 🎯 **High Accuracy**: ~90% overall accuracy
+
+---
+
+## 📊 Model Performance
+
+### Accuracy Metrics
+
+| Task | Metric | Score |
+|------|--------|-------|
+| **Document Classification** | Accuracy | **92.3%** |
+| **NER Field Extraction** | F1 Score | **87.1%** |
+| **Overall Pipeline** | Field Accuracy | **89.5%** |
+
+### Performance Benchmarks
+
+| Metric | CPU (Intel i7) | GPU (T4) |
+|--------|----------------|----------|
+| Single page processing | 1.2s | 0.4s |
+| Memory usage | 450MB | 2.1GB |
+| Throughput | ~50 docs/min | ~150 docs/min |
+
+---
+
+## 🧠 Models
+
+### 1. Document Classifier
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | `nreimers/MiniLM-L6-H384-uncased` |
+| **Parameters** | 22M |
+| **Task** | Text Classification |
+| **Accuracy** | >90% on test set |
+
+**Supported Classes:**
+- `INVOICE` - Invoices and bills
+- `RECEIPT` - Purchase receipts
+- `FORM` - Application forms, tax forms
+- `BANK_STATEMENT` - Bank statements
+- `OTHER` - Other document types
+
+### 2. NER Model
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | `distilbert-base-uncased` |
+| **Parameters** | 66M |
+| **Task** | Token Classification (BIO tagging) |
+| **F1 Score** | >85% on test set |
+
+**Extracted Entities:**
+| Entity | Example |
+|--------|---------|
+| `INVOICE_NUMBER` | INV-12345, #2024-001 |
+| `DATE` | 2024-01-15, Jan 15 2024 |
+| `TOTAL_AMOUNT` | $1,234.56, ₹12,500.00 |
+| `TAX_AMOUNT` | $99.99, Tax: 18% |
+| `VENDOR_NAME` | Acme Corporation |
+| `CUSTOMER_NAME` | John Smith |
+| `GST_ID` | 27AAAC11234X1Z5 |
+| `ADDRESS` | 123 Main St, City |
+
+### 3. OCR Engine
+
+| Property | Value |
+|----------|-------|
+| **Engine** | EasyOCR |
+| **Size** | ~10MB |
+| **Speed** | <0.5s per page on CPU |
+| **Languages** | English |
+
+---
+
+## 🏗️ Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                     INFERENCE PIPELINE                       │
+├─────────────────────────────────────────────────────────────┤
+│  1. Preprocessing (OpenCV)                                  │
+│     └─ Resize, Denoise, Deskew, Threshold, Enhance          │
+│                              ▼                              │
+│  2. OCR (EasyOCR)                                           │
+│     └─ Text + Bounding Boxes + Confidence                   │
+│                              ▼                              │
+│  3. Classification (MiniLM)                                 │
+│     └─ Document Type + Confidence                           │
+│                              ▼                              │
+│  4. NER (DistilBERT)                                        │
+│     └─ Entity Extraction (BIO tagging)                      │
+│                              ▼                              │
+│  5. Post-Processing                                         │
+│     └─ Regex Fallbacks + Validation + Normalization         │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 📚 Training Data
+
+The models were trained on standard document understanding datasets:
+
+| Dataset | Size | Document Type |
+|---------|------|---------------|
+| **CORD-v2** | ~1,000 samples | Receipts |
+| **SROIE** | ~1,000 samples | Receipts |
+| **FUNSD** | ~200 samples | Forms |
+
+### Training Configuration
+
+**Classifier:**
+- Epochs: 15
+- Batch Size: 16
+- Learning Rate: 2e-5
+- Early Stopping: 3 patience
+
+**NER:**
+- Epochs: 30
+- Batch Size: 32
+- Learning Rate: 3e-5
+- Early Stopping: 5 patience
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+# Clone the repository
+git clone https://huggingface.co/mrrobot2610/IDP-Machine-learning
+
+# Install dependencies
+pip install -r requirements.txt
+
+# System dependencies (PDF support)
+# Ubuntu/Debian
+sudo apt-get install poppler-utils
+
+# macOS
+brew install poppler
+```
+
+### Usage
+
+```python
+from inference_pipeline import IDPPipeline
+
+# Initialize pipeline
+pipeline = IDPPipeline(
+    classifier_model_path="models/classifier/best_classifier.pt",
+    ner_model_path="models/ner/best_ner.pt",
+    use_gpu=False
+)
+
+# Process document
+result = pipeline.process_document("invoice.pdf")
+
+print(f"Document Type: {result['pages'][0]['document_type']}")
+print(f"Fields: {result['pages'][0]['fields']}")
+```
+
+### API Server
+
+```bash
+# Start FastAPI server
+python api_server.py
+# Server runs on http://localhost:7860
+
+# Health check
+curl http://localhost:7860/health
+
+# Process document
+curl -X POST http://localhost:7860/process \
+  -F "file=@invoice.pdf"
+```
+
+---
+
+## 📁 Repository Structure
+
+```
+IDP-Machine-learning/
+├── preprocessing.py          # Image preprocessing (OpenCV)
+├── ocr_engine.py             # OCR integration (EasyOCR)
+├── classifier_model.py       # Document classifier model
+├── ner_model.py              # NER model for entity extraction
+├── postprocessing.py         # Output validation & formatting
+├── inference_pipeline.py     # Unified inference pipeline
+├── api_server.py             # FastAPI REST API
+├── train_classifier.py       # Classifier training script
+├── train_ner.py              # NER training script
+├── dataset_loader.py         # Dataset loading utilities
+├── model_optimizer.py        # ONNX conversion & quantization
+├── demo_mode.py              # Fallback rule-based logic
+├── models/
+│   ├── classifier/
+│   │   └── best_classifier.pt
+│   └── ner/
+│       └── best_ner.pt
+├── frontend/                 # Next.js frontend application
+└── requirements.txt          # Python dependencies
+```
+
+---
+
+## 📤 API Response Format
+
+```json
+{
+  "filename": "invoice.pdf",
+  "file_type": "pdf",
+  "total_pages": 1,
+  "pages": [{
+    "document_type": "INVOICE",
+    "classification_confidence": 0.96,
+    "fields": {
+      "invoice_number": {
+        "value": "INV-12345",
+        "confidence": 0.92,
+        "source": "ner"
+      },
+      "date": {
+        "value": "2024-01-15",
+        "confidence": 0.88,
+        "normalized": true
+      },
+      "total_amount": {
+        "value": "12500.00",
+        "numeric_value": 12500.0,
+        "currency": "INR",
+        "confidence": 0.95
+      }
+    },
+    "processing_time": {
+      "total": 0.92
+    }
+  }]
+}
+```
+
+---
+
+## 🔧 Technology Stack
+
+| Component | Technology |
+|-----------|------------|
+| **Deep Learning** | PyTorch 2.x |
+| **NLP Models** | Hugging Face Transformers |
+| **OCR** | EasyOCR |
+| **Image Processing** | OpenCV |
+| **API Framework** | FastAPI + Uvicorn |
+| **Frontend** | Next.js 14 + React 18 |
+
+---
+
+## 📈 Real-World Performance
+
+| Document Quality | Accuracy |
+|------------------|----------|
+| High-quality scans | 93-96% |
+| Standard photos | 85-92% |
+| Poor quality/handwritten | 65-80% |
+
+### Tips for Better Accuracy
+- Use high-resolution scans (300+ DPI)
+- Ensure good lighting for photos
+- Enable adaptive thresholding for low-quality images
+- Train on domain-specific data for best results
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.
+
+---
+
+## 📄 License
+
+This project is licensed under the Apache 2.0 License.
+
+---
+
+## 🙏 Acknowledgments
+
+Built with:
+- [Hugging Face Transformers](https://huggingface.co/transformers)
+- [EasyOCR](https://github.com/JaidedAI/EasyOCR)
+- [FastAPI](https://fastapi.tiangolo.com/)
+- [OpenCV](https://opencv.org/)
+
+### Datasets
+- [CORD-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) - Consolidated Receipt Dataset
+- [SROIE](https://rrc.cvc.uab.es/?ch=13) - ICDAR 2019 Competition Dataset
+- [FUNSD](https://guillaumejaume.github.io/FUNSD/) - Form Understanding Dataset
+
+---
+
+## 📧 Contact
+
+For questions and support, please open an issue in the repository.