backend/api/services/INGESTION_SYSTEM.md

# Document Ingestion System

## Overview

The backend now supports a comprehensive document ingestion system that matches the system prompt specification. This allows AI agents to automatically detect and ingest various document types (PDF, DOCX, TXT, URLs, raw text) with full metadata support.

## Endpoints

### 1. Legacy Endpoint (Backward Compatible)
```
POST /rag/ingest
Headers:
  x-tenant-id: <tenant_id>
Body:
  {
    "content": "text content to ingest"
  }
```

### 2. Enhanced Document Ingestion Endpoint
```
POST /rag/ingest-document
Headers:
  x-tenant-id: <tenant_id> (optional if in body)
Body:
  {
    "action": "ingest_document",
    "tenant_id": "<tenant_id>",  // Optional if in header
    "source_type": "pdf | docx | txt | url | raw_text | markdown",  // Auto-detected if not provided
    "content": "text content or URL",
    "metadata": {
      "filename": "document.pdf",
      "url": "https://example.com/doc",
      "doc_id": "unique-document-id"
    }
  }
```

## Features

### Automatic Source Type Detection
- **PDF**: Detected from `.pdf` extension or filename
- **DOCX**: Detected from `.docx` or `.doc` extension
- **TXT**: Detected from `.txt` or `.text` extension
- **Markdown**: Detected from `.md` or `.markdown` extension
- **URL**: Detected from URL in content or metadata
- **Raw Text**: Default fallback for plain text

### URL Processing
- Automatically fetches content from URLs
- Strips HTML tags and scripts
- Normalizes whitespace
- Handles redirects and timeouts

### Text Normalization
- Removes excessive whitespace
- Strips control characters
- Sanitizes input before ingestion

### Metadata Support
- `filename`: Original filename
- `url`: Source URL
- `doc_id`: Unique document identifier (auto-generated if not provided)
- Custom metadata can be added to the metadata object

## Usage Examples

### Example 1: Ingest Raw Text
```json
{
  "action": "ingest_document",
  "tenant_id": "tenant123",
  "source_type": "raw_text",
  "content": "This is a company policy document...",
  "metadata": {
    "filename": "policy.txt",
    "doc_id": "policy-2024-01"
  }
}
```

### Example 2: Ingest from URL
```json
{
  "action": "ingest_document",
  "tenant_id": "tenant123",
  "source_type": "url",
  "content": "https://example.com/documentation",
  "metadata": {
    "url": "https://example.com/documentation",
    "doc_id": "docs-example-com"
  }
}
```

### Example 3: Ingest PDF (with extracted text)
```json
{
  "action": "ingest_document",
  "tenant_id": "tenant123",
  "source_type": "pdf",
  "content": "<extracted PDF text>",
  "metadata": {
    "filename": "manual.pdf",
    "doc_id": "manual-2024"
  }
}
```

## Response Format

```json
{
  "status": "ok",
  "message": "Document ingested successfully. 5 chunk(s) stored.",
  "tenant_id": "tenant123",
  "source_type": "raw_text",
  "doc_id": "policy-2024-01",
  "chunks_stored": 5,
  "metadata": {
    "filename": "policy.txt",
    "doc_id": "policy-2024-01"
  }
}
```

## Integration with AI Agents

The system is designed to work with AI agents that follow the system prompt specification:

1. **Agent detects** document/URL/pasted content
2. **Agent prepares** ingestion payload with proper structure
3. **Agent sends** to `POST /rag/ingest-document`
4. **Backend processes**:
   - Detects/validates source type
   - Fetches URL content if needed
   - Normalizes text
   - Sends to RAG MCP server for chunking/embedding
   - Stores in pgvector
5. **Agent confirms** ingestion to user

## Error Handling

- **400 Bad Request**: Missing tenant_id, invalid payload, empty content
- **500 Internal Server Error**: RAG MCP server error, database error, URL fetch failure

## Notes

- The legacy `/rag/ingest` endpoint remains for backward compatibility
- Source type is auto-detected if not provided
- URL fetching is async and handles timeouts gracefully
- All content is normalized before ingestion
- Metadata is preserved and stored with chunks