README.md

---
title: PaperMate
emoji: 📝
colorFrom: yellow
colorTo: red
sdk: docker
app_port: 7860
pinned: false
---

# PaperMate — AI Research Paper Review System

A web application that automatically evaluates research paper manuscripts using AI. Sign in, upload a PDF, and receive a structured peer-review following the **ACL Rolling Review (ARR)** format with scores across all 7 ARR dimensions.

**Live demo:** [ntphuc149-papermate.hf.space](https://ntphuc149-papermate.hf.space/)

**Pages:** `/app/home.html` (landing) · `/app/upload.html` (submit paper) · `/app/review.html` (view results)

---

## Architecture

```mermaid
flowchart TD
    User(["👤 User"])

    subgraph Frontend["Frontend (Vanilla JS)"]
        UI_Home["home.html\nLanding Page"]
        UI_Upload["upload.html\nUpload Page"]
        UI_Review["review.html\nReview Viewer"]
    end

    subgraph Backend["Backend (FastAPI)"]
        API_Submit["POST /api/submit"]
        API_Review["GET /api/review/{key}"]
        API_Feedback["POST /api/feedback/{key}"]
        Auth["Auth (Supabase GoTrue\n+ Google OAuth)"]
        EmailSvc["Email Service\n(Resend)"]
    end

    subgraph Pipeline["Background Pipeline"]
        G0["0. Guardrails\n(G1a: Abstract? G1b: NLP/CL? G2: ≤8 pages?)"]
        P1["1. PDF → ParsedPaper\n(LandingAI / Docling + VLM enrichment)"]
        P2["2. Extract Title"]
        P3a["3a. Extract Contributions"]
        P3b["3b. Extract Research Topic"]
        P4["4. Generate Search Queries"]
        P5["5. Search Related Papers\n(Tavily — ACL Anthology + arXiv)"]
        P6["6. Fetch Paper Metadata\n(arXiv API)"]
        P7["7. Summarize Related Work"]
        P8["8. Multi-Agent ARR Review\n(LangGraph — ReAct + Gated Pipeline)"]
    end

    subgraph External["External Services"]
        LLM["LLM Provider\nAnthropic / OpenAI\nGemini / OpenRouter"]
        Tavily["Tavily Search API"]
        ArXiv["arXiv API"]
        LandingAI["LandingAI ADE"]
        Docling["Kaggle Docling Server\n(GPU T4 + gpt-4o-mini VLM)"]
        Resend["Resend Email"]
        Supabase["Supabase\n(PostgreSQL + Storage + Auth)"]
    end

    User -->|"Browse"| UI_Home
    UI_Home -->|"CTA"| UI_Upload
    User -->|"Google Login (optional)"| Auth
    User -->|"Upload PDF + Email"| UI_Upload
    UI_Upload -->|"POST /api/submit"| API_Submit
    API_Submit -->|"access_key"| UI_Upload
    API_Submit -->|"create job"| Supabase
    API_Submit -->|"trigger async"| G0
    G0 -->|"PASS"| P1
    G0 -->|"REJECT"| Supabase
    API_Submit -->|"send key email"| EmailSvc

    User -->|"Enter access key"| UI_Review
    UI_Review -->|"poll status"| API_Review
    API_Review -->|"read job"| Supabase

    P1 --> P2 --> P3a & P3b --> P4 --> P5 --> P6 --> P7 --> P8
    P8 -->|"save review"| Supabase
    P8 -->|"notify user"| EmailSvc

    P1 --> LandingAI
    P1 --> Docling
    P3a & P3b & P7 & P8 --> LLM
    P5 --> Tavily
    P6 --> ArXiv
    EmailSvc --> Resend
    Auth --> Supabase
```

---

## Submission Eligibility

Before the pipeline runs, three sequential **guardrails** screen every submission:

| # | Check | Logic | On Fail |
|---|---|---|---|
| G1a | Is it a research paper? | Parse page 1 via Docling; require `section_header` = "Abstract" | Desk-reject |
| G1b | Is it NLP/CL? | Feed abstract text to LLM → classify NLP/Computational Linguistics scope | Desk-reject |
| G2 | Does it follow the page limit? | Reuse full parse; find `section_header` = "References" — must be on page ≤ 9 (≤ 8 content pages) | Desk-reject |

Rejected submissions receive a `status: rejected` record in the database with a vague, non-reversible reason so users cannot infer the guardrail mechanism.

---

## How It Works

1. User signs in (Google OAuth) and uploads a PDF manuscript + email
2. System returns an **access key** immediately (also sent via email)
3. Three guardrails run before the main pipeline — ineligible papers are desk-rejected immediately
4. AI pipeline runs in the background (~5–15 minutes):
   - PDF → **`ParsedPaper`** (structured JSON with labeled elements)
   - Extract title, contributions & research topic from abstract + introduction
   - Generate search queries → find related papers (Tavily: ACL Anthology + arXiv)
   - Summarize related work
   - **Multi-agent review** (LangGraph ReAct + Gated Pipeline — see below)
5. User enters access key on the review page to read results
6. Email notification sent when review is ready

---

## Multi-Agent Review Pipeline (Step 8)

Step 8 uses a **LangGraph StateGraph** combining a **Gated Sequential Pipeline** with a **ReAct orchestrator loop** — mirroring how a senior Area Chair reads a paper: classify first, map claims, then iteratively investigate the riskiest ones.

### Pattern: ReAct + Gated Pipeline (Orchestrator-Subagents)

```mermaid
flowchart TD
    START(["📄 ParsedPaper\n+ Related Summaries"])

    subgraph G1["Gate 1 — Desk Check"]
        DC["Read abstract + intro + conclusion\nClassify paper type\nDetect fatal flaws"]
    end

    subgraph G2["Gate 2 — Claim Mapper"]
        CM["Read results + tables + conclusion\nMap each claim → Table N / Section X.Y\nAssign risk: high / medium / low"]
    end

    DR(["🚫 DESK REJECT\n→ Synthesizer"])

    subgraph G3["Gate 3 — ReAct Deep Dive (max 5 loops)"]
        direction TB
        PL["🧠 Planner — Orchestrator\nTHOUGHT: most important unresolved question?\nACTION: pick tools + focus areas\n─────────────────────────────\nLoop 1–2 · breadth: cover all dimensions\nLoop 3–4 · depth: re-examine weak findings\nLoop 5   · final: novelty deep-dive or presentation"]

        subgraph TOOLS["Subagents — run in parallel via asyncio.gather"]
            T1["AuditMethodologyTool"]
            T2["CheckNoveltyTool"]
            T3["CheckReproducibilityTool"]
            T4["AuditClaimsTool"]
            T5["CheckStatisticalRigorTool"]
            T6["AuditPresentationTool"]
            T7["ResourceQualityTool"]
        end

        PL -->|"tools_to_run"| TOOLS
        TOOLS -->|"findings — OBSERVATION"| PL
    end

    subgraph CP["Compactor"]
        CO["Deduplicate + rank: critical → major → minor\nPreserve verbatim quotes & evidence locations\nKeep ≤ 12 findings"]
    end

    subgraph G4["Gate 4 — Synthesizer"]
        SY["Strengths: scientific judgement — WHY it matters\nWeaknesses: verbatim quote + exact citation\nScore all 7 ARR metrics"]
    end

    FB(["🔄 Fallback\nlegacy review.py"])
    OUT(["✅ ARR Review\n7 metrics + strengths + weaknesses"])

    START --> G1
    DC -->|"DESK REJECT"| DR
    DC -->|"PASS"| G2
    G2 --> G3
    PL -->|"is_last_loop or done"| CP
    CP --> G4
    G4 -->|"success"| OUT
    G4 -->|"error"| FB
    FB --> OUT

    style G1 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G2 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G3 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G4 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style CP fill:#1a1208,stroke:#888,color:#aaa
    style TOOLS fill:#111,stroke:#555,color:#ccc
    style OUT fill:#1a3010,stroke:#2dc653,color:#2dc653
    style DR fill:#2a0a0a,stroke:#e63946,color:#e63946
    style FB fill:#1a1208,stroke:#888,color:#888
```

### Tool Applicability by Paper Type

| Tool | empirical | theoretical | resource | survey/position | reproduction | demo |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| AuditMethodologyTool | ✅ | — | ✅ | — | ✅ | — |
| CheckNoveltyTool | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| CheckReproducibilityTool | ✅ | — | ✅ | — | ✅ | ✅ |
| AuditClaimsTool | ✅ | ✅ | ✅ | — | ✅ | — |
| CheckStatisticalRigorTool | ✅ | — | ✅ | — | ✅ | — |
| AuditPresentationTool | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| ResourceQualityTool | — | — | ✅ | — | — | — |

### Output: Full ACL ARR Review

Every review includes all 7 ARR metrics with correct scales:

| Metric | Scale | Half-points |
|---|---|:---:|
| Soundness | 1–5 | ✅ |
| Excitement | 1–5 | ✅ |
| Reproducibility | 1–5 | — |
| Datasets | 1–5 or N/A | — |
| Software | 1–5 or N/A | — |
| Confidence | 1–5 | — |
| Overall Assessment | 1.0–5.0 | ✅ |

**Overall Assessment labels:**

| Score | Label |
|---|---|
| 5.0 | Award Candidate (top 2.5%) |
| 4.5 | Strong Accept — Conference |
| 4.0 | Accept — Conference |
| 3.5 | Borderline Findings |
| 3.0 | Borderline Findings |
| 2.5 | Borderline Reject |
| 2.0 | Reject |
| 1.5 | Strong Reject |
| 1.0 | Reject — Out of Scope |

---

## Tech Stack

| Layer | Technology |
|---|---|
| Frontend | Plain HTML + Vanilla JS (dark/light theme) |
| Backend | Python FastAPI + BackgroundTasks |
| Database | Supabase (PostgreSQL) — submissions, reviews, pipeline steps, observability |
| File Storage | Supabase Storage — original PDFs + parsed markdown |
| Auth | Supabase GoTrue + Google OAuth (optional; anonymous submit still works) |
| PDF Parsing | LandingAI ADE (cloud) **or** Docling on Kaggle GPU T4 (self-hosted) |
| VLM Enrichment | OpenAI gpt-4o-mini — formulas → LaTeX, tables → markdown, figures → desc |
| LLM Review | Configurable: Anthropic / OpenAI / Gemini / OpenRouter |
| Multi-Agent | LangGraph StateGraph — ReAct + Gated Pipeline (Orchestrator-Subagents) |
| Paper Search | Tavily Search API (ACL Anthology priority + arXiv/Semantic Scholar) |
| Email | Resend (3,000 free/month) |
| Deploy | Hugging Face Docker Spaces |

---

## Project Structure

```
paper_review/
├── backend/
│   ├── main.py               # FastAPI app & endpoints
│   ├── config.py             # Settings from environment variables
│   ├── auth.py               # Supabase GoTrue auth + RBAC (admin / user roles)
│   ├── observability.py      # Pipeline step + LLM call tracking
│   ├── email_service.py      # Resend email integration
│   ├── logger.py             # Per-job stdout + file logger (uvicorn-routed)
│   ├── llm/
│   │   └── client.py         # Unified LLM client (multi-provider)
│   ├── pipeline/
│   │   ├── guardrails.py     # Pre-pipeline eligibility checks (G1a/G1b/G2)
│   │   ├── parsed_paper.py   # ParsedPaper schema + helper functions
│   │   ├── pdf2md.py         # PDF → ParsedPaper (LandingAI or Docling)
│   │   ├── extract.py        # Extract title, contributions & topic
│   │   ├── search.py         # Generate queries + Tavily 2-tier search
│   │   ├── paper_info.py     # arXiv metadata fetching
│   │   ├── summarize.py      # Summarize related papers
│   │   ├── review.py         # Legacy single-pass ARR review (fallback only)
│   │   └── review_agent/     # Multi-agent LangGraph reviewer (primary)
│   │       ├── state.py      # ReviewAgentState + ToolFinding TypedDicts
│   │       ├── prompts.py    # System prompts for all gates, tools & synthesizer
│   │       ├── tools.py      # 7 analysis tools (paper-type gated, parallel)
│   │       ├── nodes.py      # LangGraph node functions (desk_check, planner, ...)
│   │       └── graph.py      # StateGraph wiring + run_review_agent() entry point
│   ├── storage/
│   │   ├── jobs.py           # Storage backend selector
│   │   └── supabase_store.py # Supabase persistence layer
│   └── requirements.txt
├── frontend/
│   ├── home.html             # Landing page (7-metric ARR rubric showcase)
│   ├── upload.html           # Upload page — drag-drop PDF + email
│   ├── login.html            # Google OAuth login
│   ├── review.html           # Review viewer — ARR scores, strengths, weaknesses
│   ├── admin.html            # Admin dashboard
│   ├── css/style.css         # Dark/gold theme, shared across all pages
│   └── js/
│       ├── auth.js           # Supabase auth client
│       ├── upload.js         # File drag-drop, submit, access key display
│       ├── review.js         # Key lookup, poll status, render all 7 ARR chips
│       ├── feedback.js       # Feedback form submission
│       └── theme.js          # Dark/light theme toggle
├── docs/
│   ├── database/             # Supabase schema docs + Google OAuth setup guide
│   └── parser/               # PDF parser evaluation, OCR benchmarks, test outputs
│       ├── scripts/          # Evaluation scripts (run_mvp_eval.py, test_docling_ocr.py)
│       ├── test/             # Parsed PDF JSONs + review outputs for manual evaluation
│       └── vinuni20k-parser-serving.ipynb  # Kaggle Docling server notebook
├── scripts/
│   └── apply_migration.py    # One-time DB migration runner
├── Dockerfile                # Hugging Face Docker Space
├── .env.example              # Template for environment variables
└── run.py                    # Quick-start script (local dev)
```

---

## Setup

### 1. Prerequisites

- Python 3.10+
- Supabase project (free tier works)
- API keys (see below)

### 2. Install dependencies

```bash
cd paper_review
pip install -r backend/requirements.txt
```

### 3. Configure environment variables

```bash
cp .env.example .env
```

Fill in `.env` — minimum required to run:

```env
LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-...

PDF_PARSER=landingai
LANDINGAI_API_KEY=...

TAVILY_API_KEY=tvly-...

RESEND_API_KEY=re_...
RESEND_FROM_EMAIL=onboarding@resend.dev

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJ...
SUPABASE_ANON_KEY=eyJ...
SUPABASE_STORAGE_BUCKET=paper-mate-artifacts

ADMIN_EMAILS=your@email.com
APP_BASE_URL=http://localhost:8000
```

### 4. Set up Supabase

1. Create a project at [supabase.com](https://supabase.com)
2. Run the SQL migrations in order via the Supabase SQL Editor:
   - `scripts/migrations/` — baseline schema, auth/RBAC, RPC functions
   - `supabase/migrations/` — guardrail columns (`rejection_reason`, `rejected` status)
3. Create a Storage bucket named `paper-mate-artifacts` (public)
4. Enable Google OAuth: Authentication → Providers → Google
5. Set Redirect URL: `https://<your-supabase-project>.supabase.co/auth/v1/callback`

### 5. API Keys — Where to Get Them

| Key | Where to get |
|---|---|
| `OPENAI_API_KEY` | [platform.openai.com](https://platform.openai.com) |
| `ANTHROPIC_API_KEY` | [console.anthropic.com](https://console.anthropic.com) |
| `GEMINI_API_KEY` | [aistudio.google.com](https://aistudio.google.com) |
| `OPENROUTER_API_KEY` | [openrouter.ai](https://openrouter.ai) |
| `LANDINGAI_API_KEY` | [va.landing.ai](https://va.landing.ai) |
| `TAVILY_API_KEY` | [tavily.com](https://tavily.com) — free tier available |
| `RESEND_API_KEY` | [resend.com](https://resend.com) — free 3,000 emails/month |
| `SUPABASE_*` | [supabase.com](https://supabase.com) → Project Settings → API |

### 6. Run locally

```bash
cd paper_review
python run.py
```

Or directly with uvicorn:

```bash
cd paper_review
python -m uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000
```

| Page | URL |
|---|---|
| Landing page | http://localhost:8000/app/home.html |
| Login | http://localhost:8000/app/login.html |
| Upload page | http://localhost:8000/app/upload.html |
| Review page | http://localhost:8000/app/review.html |
| API docs | http://localhost:8000/docs |

> **Note:** If using `PDF_PARSER=docling`, start the Kaggle Docling server first (see [Docling on Kaggle](#docling-on-kaggle)), then submit a paper.

---

## Deploy to Hugging Face Spaces

1. Create a new Space (SDK: Docker, template: Blank)
2. Add all env vars as Secrets in Space Settings
3. Push the `deployment` branch:

```bash
git remote add hf https://huggingface.co/spaces/<username>/PaperMate
git push hf origin/deployment:refs/heads/main --force
```

---

## API Reference

| Method | Endpoint | Auth | Description |
|---|---|---|---|
| `POST` | `/api/submit` | Optional | Upload PDF + email → returns `access_key` |
| `GET` | `/api/review/{access_key}` | — | Get job status and review result |
| `GET` | `/api/review/{key}/pdf` | — | Download original PDF |
| `GET` | `/api/review/{key}/markdown` | — | View parsed markdown |
| `POST` | `/api/feedback/{access_key}` | — | Submit feedback on a completed review |
| `GET` | `/api/me` | Required | Get current user profile |
| `GET` | `/api/me/submissions` | Required | List all submissions for the current user (includes `rejection_reason` on rejected submissions) |
| `GET` | `/api/public-config` | — | Supabase config for frontend |
| `GET` | `/health` | — | Health check (`reviewer: multi-agent-v1`) |

**Submission statuses:** `pending` · `processing` · `completed` · `failed` · `rejected`

---

## Switching LLM Provider

```env
LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-4-6

# or
LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini

# or
LLM_PROVIDER=gemini
LLM_MODEL=gemini-1.5-flash

# or
LLM_PROVIDER=openrouter
LLM_MODEL=meta-llama/llama-3.1-8b-instruct
```

---

## Switching PDF Parser

```env
# LandingAI ADE (cloud, simpler setup)
PDF_PARSER=landingai
LANDINGAI_API_KEY=...

# Docling (self-hosted Kaggle notebook — higher quality, structured JSON)
PDF_PARSER=docling
NTFY_TOPIC=papermate_pdf2md
```

### Docling on Kaggle

The Kaggle server runs two stages:
1. **GPU parse** — Docling with `granite-docling-258M` extracts labeled elements (title, section, table, formula, figure...)
2. **VLM enrichment** — all formula/table/figure items enriched via gpt-4o-mini in parallel

**Kaggle secrets required:** `ngrok_auth_token` · `openai_api_key`

---

## GATE 2: First Working Agent (MVP)

| Criteria | Evidence |
|---|---|
| MVP Demo — video 3 phút show user flow end-to-end | [MVP Demo](https://drive.google.com/file/d/1kED_9sBtUXDFKtDzUgGYZK3Jwe_d-wTz/view?usp=drive_link) |
| Architecture diagram — sơ đồ components, data flow | [Arch Diagram](https://drive.google.com/file/d/1q0qt-HZBoXjCGY9jzKYSljUPtNoV6mBO) |
| Repo có >= 10 PR merged | [Proof](https://github.com/AI20K-Build-Cohort-2/C2-App-159/pulls?q=is%3Apr+is%3Aclosed) |
| README.md — setup instructions, env vars, sample queries | You reading this |
| Eval evidences — ít nhất 5 test case manual với output thực tế | [Click here](https://github.com/AI20K-Build-Cohort-2/C2-App-159/tree/main/paper_review/docs) |
| Live demo | [ntphuc149-papermate.hf.space](https://ntphuc149-papermate.hf.space/) |