Hugging Face's logo

Spaces:
GoutamSachdev
/
Hybrid_Vision_RAG_PDF_Processor
Running on Zero

App Files Community
Hybrid_Vision_RAG_PDF_Processor / README.md
GoutamSachdev's picture
GoutamSachdev
Set sdk_version 5.20.0 for ssr kwarg; drop gradio from requirements to avoid SDK pin conflict
dd3eb25
preview code
|
Raw
History Blame Contribute Delete
8.51 kB

A newer version of the Gradio SDK is available: 6.22.0

Upgrade
metadata 
title: Hybrid Vision RAG PDF Processor
emoji: 📄
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.20.0
app_file: main.py
pinned: false
license: mit

Hybrid Vision RAG PDF Processor

A demo-ready Python project that extracts PDF content, analyzes charts/graphs via a Vision LLM, and answers questions with a Hybrid RAG system (Vector + Knowledge Graph + BM25).

Tech stack

  • Docling — PDF content extraction (text, pages, figures)
  • OpenRouter — Vision LLM for chart/graph analysis
  • Groq — LLM powering the RAG pipeline
  • LlamaIndex — Vector + Knowledge Graph + BM25 retrieval with RRF fusion
  • Gradio — demo UI

What's new

  • ✅ Clean layered architecture — config, models, services, retrieval, session, web
  • ✅ Own session architecture — Session entity + pluggable SessionStore + SessionManager with janitor
  • ✅ Pydantic configuration — all settings loaded from .env with validation
  • ✅ Structured logging — no more raw print() statements
  • ✅ Polished Gradio demo UI — live progress, status badge, commands, and session controls
  • ✅ Pre-flight API validation — OpenRouter + Groq are live-checked before the pipeline starts; failures pop a warning and block processing (see the API Status panel)
  • ✅ Unit tests for session lifecycle and BM25 retrieval
  • ✅ Unified CLI — extract, preprocess, app

Quick start

1. Install dependencies 

uv sync
# or
pip install -e .

2. Configure API keys

Copy .env.example to .env and add your keys:

cp .env.example .env

OPENROUTER_API_KEY=sk-or-v1-...
GROQ_API_KEY=gsk_...

3. Launch the demo 

app
# or
python -m docling_pdf_processor

Open http://localhost:7860 in your browser.

Docker

Run the demo in a container (secrets are injected at runtime — never baked into the image):

# 1. Put real keys in .env (the image does NOT see your host's OS env vars)
#    OPENROUTER_API_KEY=sk-or-v1-...
#    GROQ_API_KEY=gsk_...

# 2. Build & run
docker compose up --build

Then open http://localhost:8010.

Without compose:

docker build -t docling-pdf-processor .
docker run -p 8010:8010 --env-file .env docling-pdf-processor

The compose file mounts a named volume at /app/.cache/huggingface so Docling's model weights download once and persist across restarts.

⚠️ Key source caveat for containers: on your host, GROQ_API_KEY may live in an OS environment variable that overrides the .env placeholder (the API Status panel shows [source: OS env $GROQ_API_KEY]). A container does not inherit host OS env vars — only what you pass via --env-file / env_file. So you must put the real Groq key in .env before docker compose up, or the pre-flight validation will block processing with 401 Invalid API key.

CLI usage 

# Extract images + markdown from a PDF
extract --pdf path/to/doc.pdf --output ./extracted_images

# Preprocess extracted images with a Vision LLM
preprocess --quarters ./extracted_images --output graphs_description.pkl

# Launch the Gradio demo
app

Project architecture 

docling_pdf_processor/
├── config.py              # Pydantic Settings from .env
├── models.py              # Shared dataclasses
├── exceptions.py          # Domain errors
├── logging_config.py      # Structured logging setup
├── cli.py                 # Unified CLI
├── pipeline.py            # Session-aware PipelineOrchestrator
├── services/
│   ├── extractor.py       # Docling PDF extraction
│   ├── vision.py          # OpenRouter Vision-LLM
│   └── rag.py             # Hybrid RAG builder + query
├── retrieval/
│   ├── bm25.py            # BM25 keyword retriever
│   └── hybrid_rrf.py      # RRF fusion retriever
├── session/
│   ├── models.py          # Session + SessionStatus
│   ├── store.py           # SessionStore protocol + implementations
│   └── manager.py         # Lifecycle + janitor
└── web/
    ├── gradio_app.py      # Demo UI
    └── components.py      # UI helpers

Data flow 

User uploads PDF
        │
        ▼
┌─────────────────────┐
│  SessionManager     │  Creates UUID workspace in temp dir
│  (own session arch) │
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│ PipelineOrchestrator│  Streams progress via Queue
│  • ExtractorService   │  Docling → text, pages, figures
│  • VisionService      │  OpenRouter → graph descriptions
│  • RAGService         │  Vector + KG + BM25 indexes
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│   Gradio Chat UI    │  Session-scoped memory
│   !reasoning        │  Inspect retriever contributions
│   !compare          │  Side-by-side retriever comparison
└─────────────────────┘

Session architecture

Every upload creates a Session with:

  • A unique UUID
  • An isolated workspace under tempdir/docling_sessions/{uuid}/
  • A SessionStatus lifecycle: created → extracting → analyzing → indexing → ready
  • A heartbeat timestamp for stale-session cleanup

The SessionStore protocol has two built-in implementations:

  • InMemorySessionStore — default for single-process demos
  • FileSystemSessionStore — persists session metadata to disk

A background janitor cleans sessions idle longer than SESSION_MAX_AGE_SECONDS (default 30 min).

Chat commands

  • !reasoning <query> — show how Vector / KG / BM25 contribute to the final answer
  • !compare <query> — compare Vector, KG, BM25, and Hybrid answers side-by-side

API validation

Before the pipeline runs, the app live-checks the two external APIs it depends on:

API Check Endpoint
OpenRouter (Vision LLM) key + credits GET https://openrouter.ai/api/v1/key
Groq (RAG LLM) key + model actually generate (1-token chat completion) POST https://api.groq.com/openai/v1/chat/completions
  • The API Status panel in the Process PDF tab shows each API as ✅ Valid / ❌ Invalid, with a masked key prefix and the key's source (UI field / OS env $VAR / .env) so it's clear which credential is being tested.
  • Click Validate APIs to re-run the checks on demand; editing any key/model field live-refreshes the panel.
  • When you click Process PDF, validation runs first. If any API fails (bad key → 401, wrong model → 404, no credits → 402), a popup warning shows which API and why, and the pipeline does not start.

Note on key sources: pydantic-settings loads OS environment variables with higher priority than .env. So if a key is set in your shell/system environment, it overrides a placeholder in .env. The source tag in the panel makes this visible — e.g. OS env $GROQ_API_KEY instead of .env.

Validation calls are cheap and run in parallel, typically returning in ~1s.

Production notes

  • Concurrency — Gradio runs with default_concurrency_limit=1 so only one pipeline processes at a time per process instance.
  • Multi-tenant scaling — For true multi-user production, run each session in a separate worker or process.
  • Vector store — Uses LlamaIndex's in-memory SimpleVectorStore (the previous DeepLake store is abandoned and breaks on NumPy 2 / Python 3.13). Indexes are rebuilt from the extracted files each session, which is the existing behavior, so nothing is lost — but for very large corpora consider swapping in a persisted store (FAISS, Chroma, pgvector).
  • API keys — Never commit .env. It is already ignored by .gitignore.
  • Large PDFs — PDFs > 100 MB trigger a warning but are not blocked.

Development

Run tests 

pytest

Run a specific stage from the CLI 

# Extraction only
extract --pdf sample.pdf --output ./out

# Vision analysis only
preprocess --quarters ./out --output ./out/graphs.pkl --api-key $OPENROUTER_API_KEY

License

MIT