README.md

metadata

title: PDF Parser MCP Server
emoji: 📄
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.44.0
app_file: main.py
pinned: false

PDF Parser MCP Server

A FastAPI-based PDF processing system with MCP (Model Context Protocol) integration for Claude Desktop. Upload PDFs, extract text, generate AI summaries, and interact with documents through Claude Desktop.

🚀 Quick Start

📖 For detailed setup instructions, see SETUP.md

1. Install Dependencies

# Create virtual environment
uv venv
source .venv/bin/activate  # Unix
# or .venv\Scripts\activate  # Windows

# Install all dependencies
uv sync
uv pip install torch torchvision transformers docling-core pdf2image pillow

2. Setup Environment

# Create .env file
touch .env  # Create manually on Windows

# Add your Anthropic API key to .env
ANTHROPIC_API_KEY=your_actual_api_key_here
MAX_TOKENS=180000
CHUNK_SIZE=8000

3. Start the System

# Start FastAPI server with auto-reload
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Test the system
python test_smoldocling.py

4. Configure MCP Server

# Start MCP server (separate terminal)
python mcp_main.py

4. Configure Claude Desktop

1. Open Claude Desktop configuration:

   • Windows: %APPDATA%\Claude\claude_desktop_config.json
   • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
   • Linux: ~/.config/claude/claude_desktop_config.json

2. Add the server configuration:

{
  "mcpServers": {
    "pdf-parser": {
      "command": "uv",
      "args": ["--directory", "/path/to/your/pdf-parser", "run", "mcp_main.py"],
      "cwd": "/path/to/your/pdf-parser"
    }
  }
}

3. Update the paths to match your project location
4. Restart Claude Desktop

5. Upload and Process PDFs

# Upload a PDF
curl -X POST "http://localhost:8000/upload-pdf/" -F "file=@your_document.pdf"

# Check status
curl "http://localhost:8000/status/{file_id}"

# List documents
curl "http://localhost:8000/documents/"

6. Use with Claude Desktop

Once configured, you can interact with your PDFs through Claude Desktop:

You: "List all my documents"
Claude: [Shows all processed PDFs with IDs and status]

You: "What is the summary of document abc-123?"
Claude: [Provides detailed summary of the document]

You: "Search for 'financial projections' in all documents"
Claude: [Searches and shows relevant sections]

You: "What are the key findings in document xyz-456?"
Claude: [Analyzes and provides key insights]

🔧 System Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Claude        │    │   MCP Server    │    │   FastAPI       │
│   Desktop       │◄───┤   (@mcp.tool)   │◄───┤   Server        │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                              │                         │
                              │                         │
                              ▼                         ▼
                    ┌─────────────────┐    ┌─────────────────┐
                    │   Document      │    │   SmolDocling   │
                    │   Storage       │    │   + PyMuPDF     │
                    └─────────────────┘    └─────────────────┘
                              │                         │
                              │                         │
                              ▼                         ▼
                    ┌─────────────────┐    ┌─────────────────┐
                    │   Anthropic     │    │   Smart         │
                    │   Claude API    │    │   Chunking      │
                    └─────────────────┘    └─────────────────┘

                    

🔄 Complete Workflow Pipeline

Visual Overview

graph TD
    A["📄 PDF Upload"] --> B["🔍 File Validation"]
    B --> C["💾 Save to uploads/"]
    C --> D["🚀 Background Processing"]
    
    D --> E["🖼️ PDF → Images<br/>(pdf2image)"]
    E --> F["🤖 SmolDocling Processing<br/>(Vision-Language Model)"]
    F --> G["📝 DocTags Generation<br/>(Layout + Structure)"]
    G --> H["✂️ Text Extraction<br/>(Markdown Format)"]
    
    H --> I["🧮 Token Analysis<br/>(tiktoken)"]
    I --> J["📦 Smart Chunking<br/>(8K tokens max)"]
    J --> K["🎯 Claude 3.5 Sonnet<br/>(Summarization)"]
    
    K --> L["💾 Storage.json<br/>(Document + Chunks + Summary)"]
    L --> M["🔄 MCP Server Update"]
    M --> N["🖥️ Claude Desktop<br/>(Tools Available)"]
    
    N --> O["💬 User Queries"]
    O --> P["🛠️ MCP Tools<br/>(list, search, summarize)"]
    P --> Q["🎯 Intelligent Responses"]
    
    %% Fallback path
    F -.->|"🔄 Fallback"| R["📄 PyMuPDF<br/>(Traditional OCR)"]
    R -.-> H

Phase 1: Document Upload & Initial Processing

📄 PDF Upload (FastAPI)
    ↓
🔍 File Validation (.pdf extension)
    ↓
💾 Save to uploads/ directory with UUID
    ↓
🚀 Background Processing Initiated

Phase 2: Advanced Text Extraction (SmolDocling)

📄 PDF File
    ↓
🖼️ PDF → Images (pdf2image)
    │   ├── Page 1.png
    │   ├── Page 2.png
    │   └── Page N.png
    ↓
🤖 SmolDocling Processing (per page)
    │   ├── Vision-Language Model Analysis
    │   ├── Document Structure Recognition
    │   ├── Table/Code/Formula Detection
    │   └── DocTags Generation
    ↓
📝 DocTags → Structured Text
    │   ├── Layout Preservation
    │   ├── Hierarchy Maintenance
    │   └── Content Organization
    ↓
📋 Consolidated Text Output
    ├── Page 1 Content
    ├── Page 2 Content
    └── Page N Content

🔄 FALLBACK: If SmolDocling fails → PyMuPDF extraction

Phase 3: Intelligent Chunking

📋 Complete Document Text
    ↓
🧮 Token Analysis (tiktoken)
    │   ├── Total token count calculation
    │   ├── Per-page token assessment
    │   └── Chunking strategy determination
    ↓
✂️ Smart Chunking Process
    │   ├── Respect page boundaries
    │   ├── Split oversized pages intelligently
    │   ├── Maintain context windows
    │   └── Preserve document structure
    ↓
📦 Chunk Generation
    │   ├── Chunk 1 (8K tokens max)
    │   ├── Chunk 2 (8K tokens max)
    │   └── Chunk N (remaining content)
    ↓
💾 Store chunks with metadata

Phase 4: AI-Powered Summarization

📦 Document Chunks
    ↓
🎯 Summarization Strategy Selection
    ├── Single Chunk → Direct summarization
    └── Multiple Chunks → Hierarchical approach
    ↓
🤖 Claude 3.5 Sonnet Processing
    │   ├── Individual chunk summaries
    │   ├── Cross-chunk analysis
    │   ├── Overall document synthesis
    │   └── Key insights extraction
    ↓
📄 Summary Generation
    │   ├── Overall summary
    │   ├── Per-chunk summaries
    │   ├── Key findings
    │   └── Important details

Phase 5: Storage & Indexing

📊 Processed Document Data
    ↓
💾 Storage.json Update
    │   ├── Document metadata
    │   ├── Extracted text
    │   ├── Chunk information
    │   ├── Summary data
    │   └── Processing timestamps
    ↓
🔄 MCP Server Synchronization
    │   ├── Update document registry
    │   ├── Enable Claude Desktop access
    │   └── Prepare for querying
    ↓
✅ Processing Complete

Phase 6: Query & Interaction (Claude Desktop)

💬 User Query (Claude Desktop)
    ↓
🛠️ MCP Tool Selection
    │   ├── list_documents()
    │   ├── get_document_summary()
    │   ├── get_document_content()
    │   ├── search_documents()
    │   └── answer_question()
    ↓
📊 Data Retrieval
    │   ├── Document lookup
    │   ├── Content extraction
    │   ├── Context preparation
    │   └── Response formatting
    ↓
🎯 Intelligent Response
    │   ├── Contextual answers
    │   ├── Document citations
    │   ├── Relevant excerpts
    │   └── Follow-up suggestions

🎯 Key Processing Features

SmolDocling Advantages

• 🧠 Intelligent OCR: Understands document layout and structure
• 📊 Table Recognition: Preserves table formatting and relationships
• 💻 Code Detection: Maintains code block formatting and syntax
• 🔢 Formula Processing: Handles mathematical expressions correctly
• 📐 Layout Awareness: Preserves document hierarchy and spacing
• 🖼️ Figure Classification: Identifies and categorizes visual elements

Robust Error Handling

• 🔄 Automatic Fallback: SmolDocling → PyMuPDF if needed
• ⚡ Performance Optimization: GPU acceleration when available
• 💾 Memory Management: Efficient processing for large documents
• 🛡️ Error Recovery: Graceful handling of processing failures

Scalability Features

• 🚀 Background Processing: Non-blocking document processing
• 📦 Efficient Chunking: Token-aware content splitting
• 🔍 Fast Search: Optimized text search across documents
• 💨 Quick Retrieval: Instant access to processed content

📊 Features

• Multi-page PDF Support: Handle 70-80+ page documents
• Advanced Text Extraction: Uses SmolDocling (256M parameter vision-language model) for intelligent document understanding with PyMuPDF fallback
• Layout-Aware Processing: Preserves document structure, tables, code blocks, formulas, and formatting
• AI Summarization: Claude 3.5 Sonnet generates comprehensive summaries
• Token-aware Chunking: Automatically splits large documents respecting token limits
• MCP Integration: Seamless Claude Desktop integration with @mcp.tool() decorators
• Background Processing: Asynchronous PDF processing
• Search & Query: Full-text search across all documents
• RESTful API: Complete REST API for programmatic access

🤖 SmolDocling Integration

This project now uses SmolDocling, a compact 256M parameter vision-language model for advanced document understanding:

Why SmolDocling?

• Better Text Recognition: Understands document layout, tables, code blocks, and mathematical formulas
• Structure Preservation: Maintains document hierarchy and formatting
• Compact Model: Only 256M parameters, efficient for local processing
• Multi-Modal: Processes documents as images for better OCR accuracy

How It Works

1. PDF to Images: Converts PDF pages to images using pdf2image
2. SmolDocling Processing: Each page is processed by the vision-language model
3. DocTags Generation: Creates structured markup preserving layout and content
4. Text Extraction: Converts DocTags to clean text for further processing
5. Fallback: Automatically falls back to PyMuPDF if SmolDocling fails

Requirements

• GPU Recommended: CUDA-compatible GPU for optimal performance
• CPU Fallback: Works on CPU but slower processing
• Memory: ~2GB GPU memory or 4GB RAM for CPU processing

🛠️ MCP Tools Available in Claude Desktop

Tool	Description	Usage
list_documents	List all processed PDFs	"List my documents"
get_document_summary	Get AI-generated summary	"Summarize document abc-123"
get_document_content	Get full or chunked content	"Show content of document xyz-456"
search_documents	Search across all documents	"Search for 'budget' in all docs"
get_document_metadata	Get document metadata	"Show metadata for document abc-123"
answer_question	Answer questions about documents	"What are the main conclusions?"

🔍 Technical Details

PDF Processing

• Library: PyMuPDF (fitz) for robust text extraction
• Multi-page: Handles documents with 70-80+ pages efficiently
• Structure: Preserves page boundaries and formatting

Text Chunking

• Token Counting: Uses tiktoken for accurate token counting
• Smart Splitting: Respects page boundaries when possible
• Large Page Handling: Splits oversized pages intelligently
• Token Limits: Configurable limits (default: 180k tokens)

AI Integration

• Model: Claude 3.5 Sonnet (claude-3-5-sonnet-20241022)
• Hierarchical Summarization: Multi-level summaries for large documents
• Context-aware: Maintains context across chunks

📁 Project Structure

pdf-parser/
├── main.py                 # FastAPI server (uvicorn main:app --reload)
├── mcp_main.py            # MCP server entry point (uv run mcp install)
├── mcp_server.py          # MCP server with @mcp.tool() decorators
├── pdf_processor.py        # PDF text extraction & chunking
├── anthropic_client.py     # Anthropic API integration
├── pyproject.toml         # Project dependencies & MCP config
├── .env.example           # Environment variables template
├── dev_commands.md        # Development commands reference
├── claude_desktop_config.json  # Claude Desktop configuration
├── start_system.py        # System setup helper (optional)
└── README.md             # This file

🚨 Troubleshooting

Common Issues

1. Import Errors: Run uv sync to install dependencies
2. API Key Missing: Add ANTHROPIC_API_KEY to .env file
3. MCP Connection: Check Claude Desktop configuration path
4. File Upload: Ensure sufficient disk space and permissions

Debug Steps

# Check system status
python start_system.py

# Test API endpoint
curl http://localhost:8000/health

# Check MCP server
python start_system.py mcp

📄 License

MIT License - See LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

---

Ready to process your PDFs with AI power! 🚀