README.md

---
title: PDF Parser MCP Server
emoji: 📄
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "4.44.0"
app_file: main.py
pinned: false
---

# PDF Parser MCP Server

A FastAPI-based PDF processing system with MCP (Model Context Protocol) integration for Claude Desktop. Upload PDFs, extract text, generate AI summaries, and interact with documents through Claude Desktop.

## 🚀 Quick Start

> **📖 For detailed setup instructions, see [SETUP.md](SETUP.md)**

### 1. Install Dependencies
```bash
# Create virtual environment
uv venv
source .venv/bin/activate  # Unix
# or .venv\Scripts\activate  # Windows

# Install all dependencies
uv sync
uv pip install torch torchvision transformers docling-core pdf2image pillow
```

### 2. Setup Environment
```bash
# Create .env file
touch .env  # Create manually on Windows

# Add your Anthropic API key to .env
ANTHROPIC_API_KEY=your_actual_api_key_here
MAX_TOKENS=180000
CHUNK_SIZE=8000
```

### 3. Start the System
```bash
# Start FastAPI server with auto-reload
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Test the system
python test_smoldocling.py
```

### 4. Configure MCP Server
```bash
# Start MCP server (separate terminal)
python mcp_main.py
```

### 4. Configure Claude Desktop
1. Open Claude Desktop configuration:
   - **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
   - **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
   - **Linux**: `~/.config/claude/claude_desktop_config.json`

2. Add the server configuration:
```json
{
  "mcpServers": {
    "pdf-parser": {
      "command": "uv",
      "args": ["--directory", "/path/to/your/pdf-parser", "run", "mcp_main.py"],
      "cwd": "/path/to/your/pdf-parser"
    }
  }
}
```

3. Update the paths to match your project location
4. Restart Claude Desktop

### 5. Upload and Process PDFs
```bash
# Upload a PDF
curl -X POST "http://localhost:8000/upload-pdf/" -F "file=@your_document.pdf"

# Check status
curl "http://localhost:8000/status/{file_id}"

# List documents
curl "http://localhost:8000/documents/"
```

### 6. Use with Claude Desktop
Once configured, you can interact with your PDFs through Claude Desktop:

```
You: "List all my documents"
Claude: [Shows all processed PDFs with IDs and status]

You: "What is the summary of document abc-123?"
Claude: [Provides detailed summary of the document]

You: "Search for 'financial projections' in all documents"
Claude: [Searches and shows relevant sections]

You: "What are the key findings in document xyz-456?"
Claude: [Analyzes and provides key insights]
```

## 🔧 System Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Claude        │    │   MCP Server    │    │   FastAPI       │
│   Desktop       │◄───┤   (@mcp.tool)   │◄───┤   Server        │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                              │                         │
                              │                         │
                              ▼                         ▼
                    ┌─────────────────┐    ┌─────────────────┐
                    │   Document      │    │   SmolDocling   │
                    │   Storage       │    │   + PyMuPDF     │
                    └─────────────────┘    └─────────────────┘
                              │                         │
                              │                         │
                              ▼                         ▼
                    ┌─────────────────┐    ┌─────────────────┐
                    │   Anthropic     │    │   Smart         │
                    │   Claude API    │    │   Chunking      │
                    └─────────────────┘    └─────────────────┘

                    
```

## 🔄 Complete Workflow Pipeline

### **Visual Overview**
```mermaid
graph TD
    A["📄 PDF Upload"] --> B["🔍 File Validation"]
    B --> C["💾 Save to uploads/"]
    C --> D["🚀 Background Processing"]
    
    D --> E["🖼️ PDF → Images<br/>(pdf2image)"]
    E --> F["🤖 SmolDocling Processing<br/>(Vision-Language Model)"]
    F --> G["📝 DocTags Generation<br/>(Layout + Structure)"]
    G --> H["✂️ Text Extraction<br/>(Markdown Format)"]
    
    H --> I["🧮 Token Analysis<br/>(tiktoken)"]
    I --> J["📦 Smart Chunking<br/>(8K tokens max)"]
    J --> K["🎯 Claude 3.5 Sonnet<br/>(Summarization)"]
    
    K --> L["💾 Storage.json<br/>(Document + Chunks + Summary)"]
    L --> M["🔄 MCP Server Update"]
    M --> N["🖥️ Claude Desktop<br/>(Tools Available)"]
    
    N --> O["💬 User Queries"]
    O --> P["🛠️ MCP Tools<br/>(list, search, summarize)"]
    P --> Q["🎯 Intelligent Responses"]
    
    %% Fallback path
    F -.->|"🔄 Fallback"| R["📄 PyMuPDF<br/>(Traditional OCR)"]
    R -.-> H
```

### **Phase 1: Document Upload & Initial Processing**
```
📄 PDF Upload (FastAPI)
    ↓
🔍 File Validation (.pdf extension)
    ↓
💾 Save to uploads/ directory with UUID
    ↓
🚀 Background Processing Initiated
```

### **Phase 2: Advanced Text Extraction (SmolDocling)**
```
📄 PDF File
    ↓
🖼️ PDF → Images (pdf2image)
    │   ├── Page 1.png
    │   ├── Page 2.png
    │   └── Page N.png
    ↓
🤖 SmolDocling Processing (per page)
    │   ├── Vision-Language Model Analysis
    │   ├── Document Structure Recognition
    │   ├── Table/Code/Formula Detection
    │   └── DocTags Generation
    ↓
📝 DocTags → Structured Text
    │   ├── Layout Preservation
    │   ├── Hierarchy Maintenance
    │   └── Content Organization
    ↓
📋 Consolidated Text Output
    ├── Page 1 Content
    ├── Page 2 Content
    └── Page N Content

🔄 FALLBACK: If SmolDocling fails → PyMuPDF extraction
```

### **Phase 3: Intelligent Chunking**
```
📋 Complete Document Text
    ↓
🧮 Token Analysis (tiktoken)
    │   ├── Total token count calculation
    │   ├── Per-page token assessment
    │   └── Chunking strategy determination
    ↓
✂️ Smart Chunking Process
    │   ├── Respect page boundaries
    │   ├── Split oversized pages intelligently
    │   ├── Maintain context windows
    │   └── Preserve document structure
    ↓
📦 Chunk Generation
    │   ├── Chunk 1 (8K tokens max)
    │   ├── Chunk 2 (8K tokens max)
    │   └── Chunk N (remaining content)
    ↓
💾 Store chunks with metadata
```

### **Phase 4: AI-Powered Summarization**
```
📦 Document Chunks
    ↓
🎯 Summarization Strategy Selection
    ├── Single Chunk → Direct summarization
    └── Multiple Chunks → Hierarchical approach
    ↓
🤖 Claude 3.5 Sonnet Processing
    │   ├── Individual chunk summaries
    │   ├── Cross-chunk analysis
    │   ├── Overall document synthesis
    │   └── Key insights extraction
    ↓
📄 Summary Generation
    │   ├── Overall summary
    │   ├── Per-chunk summaries
    │   ├── Key findings
    │   └── Important details
```

### **Phase 5: Storage & Indexing**
```
📊 Processed Document Data
    ↓
💾 Storage.json Update
    │   ├── Document metadata
    │   ├── Extracted text
    │   ├── Chunk information
    │   ├── Summary data
    │   └── Processing timestamps
    ↓
🔄 MCP Server Synchronization
    │   ├── Update document registry
    │   ├── Enable Claude Desktop access
    │   └── Prepare for querying
    ↓
✅ Processing Complete
```

### **Phase 6: Query & Interaction (Claude Desktop)**
```
💬 User Query (Claude Desktop)
    ↓
🛠️ MCP Tool Selection
    │   ├── list_documents()
    │   ├── get_document_summary()
    │   ├── get_document_content()
    │   ├── search_documents()
    │   └── answer_question()
    ↓
📊 Data Retrieval
    │   ├── Document lookup
    │   ├── Content extraction
    │   ├── Context preparation
    │   └── Response formatting
    ↓
🎯 Intelligent Response
    │   ├── Contextual answers
    │   ├── Document citations
    │   ├── Relevant excerpts
    │   └── Follow-up suggestions
```

## 🎯 Key Processing Features

### **SmolDocling Advantages**
- **🧠 Intelligent OCR**: Understands document layout and structure
- **📊 Table Recognition**: Preserves table formatting and relationships
- **💻 Code Detection**: Maintains code block formatting and syntax
- **🔢 Formula Processing**: Handles mathematical expressions correctly
- **📐 Layout Awareness**: Preserves document hierarchy and spacing
- **🖼️ Figure Classification**: Identifies and categorizes visual elements

### **Robust Error Handling**
- **🔄 Automatic Fallback**: SmolDocling → PyMuPDF if needed
- **⚡ Performance Optimization**: GPU acceleration when available
- **💾 Memory Management**: Efficient processing for large documents
- **🛡️ Error Recovery**: Graceful handling of processing failures

### **Scalability Features**
- **🚀 Background Processing**: Non-blocking document processing
- **📦 Efficient Chunking**: Token-aware content splitting
- **🔍 Fast Search**: Optimized text search across documents
- **💨 Quick Retrieval**: Instant access to processed content

## 📊 Features

- **Multi-page PDF Support**: Handle 70-80+ page documents
- **Advanced Text Extraction**: Uses SmolDocling (256M parameter vision-language model) for intelligent document understanding with PyMuPDF fallback
- **Layout-Aware Processing**: Preserves document structure, tables, code blocks, formulas, and formatting
- **AI Summarization**: Claude 3.5 Sonnet generates comprehensive summaries
- **Token-aware Chunking**: Automatically splits large documents respecting token limits
- **MCP Integration**: Seamless Claude Desktop integration with @mcp.tool() decorators
- **Background Processing**: Asynchronous PDF processing
- **Search & Query**: Full-text search across all documents
- **RESTful API**: Complete REST API for programmatic access

## 🤖 SmolDocling Integration

This project now uses **SmolDocling**, a compact 256M parameter vision-language model for advanced document understanding:

### Why SmolDocling?
- **Better Text Recognition**: Understands document layout, tables, code blocks, and mathematical formulas
- **Structure Preservation**: Maintains document hierarchy and formatting
- **Compact Model**: Only 256M parameters, efficient for local processing
- **Multi-Modal**: Processes documents as images for better OCR accuracy

### How It Works
1. **PDF to Images**: Converts PDF pages to images using pdf2image
2. **SmolDocling Processing**: Each page is processed by the vision-language model
3. **DocTags Generation**: Creates structured markup preserving layout and content
4. **Text Extraction**: Converts DocTags to clean text for further processing
5. **Fallback**: Automatically falls back to PyMuPDF if SmolDocling fails

### Requirements
- **GPU Recommended**: CUDA-compatible GPU for optimal performance
- **CPU Fallback**: Works on CPU but slower processing
- **Memory**: ~2GB GPU memory or 4GB RAM for CPU processing

## 🛠️ MCP Tools Available in Claude Desktop

| Tool | Description | Usage |
|------|-------------|--------|
| `list_documents` | List all processed PDFs | "List my documents" |
| `get_document_summary` | Get AI-generated summary | "Summarize document abc-123" |
| `get_document_content` | Get full or chunked content | "Show content of document xyz-456" |
| `search_documents` | Search across all documents | "Search for 'budget' in all docs" |
| `get_document_metadata` | Get document metadata | "Show metadata for document abc-123" |
| `answer_question` | Answer questions about documents | "What are the main conclusions?" |

## 🔍 Technical Details

### PDF Processing
- **Library**: PyMuPDF (fitz) for robust text extraction
- **Multi-page**: Handles documents with 70-80+ pages efficiently
- **Structure**: Preserves page boundaries and formatting

### Text Chunking
- **Token Counting**: Uses tiktoken for accurate token counting
- **Smart Splitting**: Respects page boundaries when possible
- **Large Page Handling**: Splits oversized pages intelligently
- **Token Limits**: Configurable limits (default: 180k tokens)

### AI Integration
- **Model**: Claude 3.5 Sonnet (claude-3-5-sonnet-20241022)
- **Hierarchical Summarization**: Multi-level summaries for large documents
- **Context-aware**: Maintains context across chunks

## 📁 Project Structure

```
pdf-parser/
├── main.py                 # FastAPI server (uvicorn main:app --reload)
├── mcp_main.py            # MCP server entry point (uv run mcp install)
├── mcp_server.py          # MCP server with @mcp.tool() decorators
├── pdf_processor.py        # PDF text extraction & chunking
├── anthropic_client.py     # Anthropic API integration
├── pyproject.toml         # Project dependencies & MCP config
├── .env.example           # Environment variables template
├── dev_commands.md        # Development commands reference
├── claude_desktop_config.json  # Claude Desktop configuration
├── start_system.py        # System setup helper (optional)
└── README.md             # This file
```

## 🚨 Troubleshooting

### Common Issues

1. **Import Errors**: Run `uv sync` to install dependencies
2. **API Key Missing**: Add `ANTHROPIC_API_KEY` to `.env` file
3. **MCP Connection**: Check Claude Desktop configuration path
4. **File Upload**: Ensure sufficient disk space and permissions

### Debug Steps

```bash
# Check system status
python start_system.py

# Test API endpoint
curl http://localhost:8000/health

# Check MCP server
python start_system.py mcp
```

## 📄 License

MIT License - See LICENSE file for details.

## 🤝 Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

---

**Ready to process your PDFs with AI power! 🚀**