Spaces:

ntphuc149
/

PaperMate

Running

App Files Files

PaperMate / README.md

ntphuc149

docs: update README — guardrails, sign-in flow, new API endpoints + statuses

6d8a15b 12 days ago

18.7 kB

title: PaperMate
emoji: 📝
colorFrom: yellow
colorTo: red
sdk: docker
app_port: 7860
pinned: false

PaperMate — AI Research Paper Review System

A web application that automatically evaluates research paper manuscripts using AI. Sign in, upload a PDF, and receive a structured peer-review following the ACL Rolling Review (ARR) format with scores across all 7 ARR dimensions.

Live demo: ntphuc149-papermate.hf.space

Pages: /app/home.html (landing) · /app/upload.html (submit paper) · /app/review.html (view results)

Architecture

flowchart TD
    User(["👤 User"])

    subgraph Frontend["Frontend (Vanilla JS)"]
        UI_Home["home.html\nLanding Page"]
        UI_Upload["upload.html\nUpload Page"]
        UI_Review["review.html\nReview Viewer"]
    end

    subgraph Backend["Backend (FastAPI)"]
        API_Submit["POST /api/submit"]
        API_Review["GET /api/review/{key}"]
        API_Feedback["POST /api/feedback/{key}"]
        Auth["Auth (Supabase GoTrue\n+ Google OAuth)"]
        EmailSvc["Email Service\n(Resend)"]
    end

    subgraph Pipeline["Background Pipeline"]
        G0["0. Guardrails\n(G1a: Abstract? G1b: NLP/CL? G2: ≤8 pages?)"]
        P1["1. PDF → ParsedPaper\n(LandingAI / Docling + VLM enrichment)"]
        P2["2. Extract Title"]
        P3a["3a. Extract Contributions"]
        P3b["3b. Extract Research Topic"]
        P4["4. Generate Search Queries"]
        P5["5. Search Related Papers\n(Tavily — ACL Anthology + arXiv)"]
        P6["6. Fetch Paper Metadata\n(arXiv API)"]
        P7["7. Summarize Related Work"]
        P8["8. Multi-Agent ARR Review\n(LangGraph — ReAct + Gated Pipeline)"]
    end

    subgraph External["External Services"]
        LLM["LLM Provider\nAnthropic / OpenAI\nGemini / OpenRouter"]
        Tavily["Tavily Search API"]
        ArXiv["arXiv API"]
        LandingAI["LandingAI ADE"]
        Docling["Kaggle Docling Server\n(GPU T4 + gpt-4o-mini VLM)"]
        Resend["Resend Email"]
        Supabase["Supabase\n(PostgreSQL + Storage + Auth)"]
    end

    User -->|"Browse"| UI_Home
    UI_Home -->|"CTA"| UI_Upload
    User -->|"Google Login (optional)"| Auth
    User -->|"Upload PDF + Email"| UI_Upload
    UI_Upload -->|"POST /api/submit"| API_Submit
    API_Submit -->|"access_key"| UI_Upload
    API_Submit -->|"create job"| Supabase
    API_Submit -->|"trigger async"| G0
    G0 -->|"PASS"| P1
    G0 -->|"REJECT"| Supabase
    API_Submit -->|"send key email"| EmailSvc

    User -->|"Enter access key"| UI_Review
    UI_Review -->|"poll status"| API_Review
    API_Review -->|"read job"| Supabase

    P1 --> P2 --> P3a & P3b --> P4 --> P5 --> P6 --> P7 --> P8
    P8 -->|"save review"| Supabase
    P8 -->|"notify user"| EmailSvc

    P1 --> LandingAI
    P1 --> Docling
    P3a & P3b & P7 & P8 --> LLM
    P5 --> Tavily
    P6 --> ArXiv
    EmailSvc --> Resend
    Auth --> Supabase

Submission Eligibility

Before the pipeline runs, three sequential guardrails screen every submission:

#	Check	Logic	On Fail
G1a	Is it a research paper?	Parse page 1 via Docling; require `section_header` = "Abstract"	Desk-reject
G1b	Is it NLP/CL?	Feed abstract text to LLM → classify NLP/Computational Linguistics scope	Desk-reject
G2	Does it follow the page limit?	Reuse full parse; find `section_header` = "References" — must be on page ≤ 9 (≤ 8 content pages)	Desk-reject

Rejected submissions receive a status: rejected record in the database with a vague, non-reversible reason so users cannot infer the guardrail mechanism.

How It Works

User signs in (Google OAuth) and uploads a PDF manuscript + email
System returns an access key immediately (also sent via email)
Three guardrails run before the main pipeline — ineligible papers are desk-rejected immediately
AI pipeline runs in the background (~5–15 minutes):
- PDF → ParsedPaper (structured JSON with labeled elements)
- Extract title, contributions & research topic from abstract + introduction
- Generate search queries → find related papers (Tavily: ACL Anthology + arXiv)
- Summarize related work
- Multi-agent review (LangGraph ReAct + Gated Pipeline — see below)
User enters access key on the review page to read results
Email notification sent when review is ready

Multi-Agent Review Pipeline (Step 8)

Step 8 uses a LangGraph StateGraph combining a Gated Sequential Pipeline with a ReAct orchestrator loop — mirroring how a senior Area Chair reads a paper: classify first, map claims, then iteratively investigate the riskiest ones.

Pattern: ReAct + Gated Pipeline (Orchestrator-Subagents)

flowchart TD
    START(["📄 ParsedPaper\n+ Related Summaries"])

    subgraph G1["Gate 1 — Desk Check"]
        DC["Read abstract + intro + conclusion\nClassify paper type\nDetect fatal flaws"]
    end

    subgraph G2["Gate 2 — Claim Mapper"]
        CM["Read results + tables + conclusion\nMap each claim → Table N / Section X.Y\nAssign risk: high / medium / low"]
    end

    DR(["🚫 DESK REJECT\n→ Synthesizer"])

    subgraph G3["Gate 3 — ReAct Deep Dive (max 5 loops)"]
        direction TB
        PL["🧠 Planner — Orchestrator\nTHOUGHT: most important unresolved question?\nACTION: pick tools + focus areas\n─────────────────────────────\nLoop 1–2 · breadth: cover all dimensions\nLoop 3–4 · depth: re-examine weak findings\nLoop 5   · final: novelty deep-dive or presentation"]

        subgraph TOOLS["Subagents — run in parallel via asyncio.gather"]
            T1["AuditMethodologyTool"]
            T2["CheckNoveltyTool"]
            T3["CheckReproducibilityTool"]
            T4["AuditClaimsTool"]
            T5["CheckStatisticalRigorTool"]
            T6["AuditPresentationTool"]
            T7["ResourceQualityTool"]
        end

        PL -->|"tools_to_run"| TOOLS
        TOOLS -->|"findings — OBSERVATION"| PL
    end

    subgraph CP["Compactor"]
        CO["Deduplicate + rank: critical → major → minor\nPreserve verbatim quotes & evidence locations\nKeep ≤ 12 findings"]
    end

    subgraph G4["Gate 4 — Synthesizer"]
        SY["Strengths: scientific judgement — WHY it matters\nWeaknesses: verbatim quote + exact citation\nScore all 7 ARR metrics"]
    end

    FB(["🔄 Fallback\nlegacy review.py"])
    OUT(["✅ ARR Review\n7 metrics + strengths + weaknesses"])

    START --> G1
    DC -->|"DESK REJECT"| DR
    DC -->|"PASS"| G2
    G2 --> G3
    PL -->|"is_last_loop or done"| CP
    CP --> G4
    G4 -->|"success"| OUT
    G4 -->|"error"| FB
    FB --> OUT

    style G1 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G2 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G3 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style G4 fill:#1a1208,stroke:#f5a623,color:#f5a623
    style CP fill:#1a1208,stroke:#888,color:#aaa
    style TOOLS fill:#111,stroke:#555,color:#ccc
    style OUT fill:#1a3010,stroke:#2dc653,color:#2dc653
    style DR fill:#2a0a0a,stroke:#e63946,color:#e63946
    style FB fill:#1a1208,stroke:#888,color:#888

Tool Applicability by Paper Type

Tool	empirical	theoretical	resource	survey/position	reproduction	demo
AuditMethodologyTool	✅	—	✅	—	✅	—
CheckNoveltyTool	✅	✅	✅	✅	✅	✅
CheckReproducibilityTool	✅	—	✅	—	✅	✅
AuditClaimsTool	✅	✅	✅	—	✅	—
CheckStatisticalRigorTool	✅	—	✅	—	✅	—
AuditPresentationTool	✅	✅	✅	✅	✅	✅
ResourceQualityTool	—	—	✅	—	—	—

Output: Full ACL ARR Review

Every review includes all 7 ARR metrics with correct scales:

Metric	Scale	Half-points
Soundness	1–5	✅
Excitement	1–5	✅
Reproducibility	1–5	—
Datasets	1–5 or N/A	—
Software	1–5 or N/A	—
Confidence	1–5	—
Overall Assessment	1.0–5.0	✅

Overall Assessment labels:

Score	Label
5.0	Award Candidate (top 2.5%)
4.5	Strong Accept — Conference
4.0	Accept — Conference
3.5	Borderline Findings
3.0	Borderline Findings
2.5	Borderline Reject
2.0	Reject
1.5	Strong Reject
1.0	Reject — Out of Scope

Tech Stack

Layer	Technology
Frontend	Plain HTML + Vanilla JS (dark/light theme)
Backend	Python FastAPI + BackgroundTasks
Database	Supabase (PostgreSQL) — submissions, reviews, pipeline steps, observability
File Storage	Supabase Storage — original PDFs + parsed markdown
Auth	Supabase GoTrue + Google OAuth (optional; anonymous submit still works)
PDF Parsing	LandingAI ADE (cloud) or Docling on Kaggle GPU T4 (self-hosted)
VLM Enrichment	OpenAI gpt-4o-mini — formulas → LaTeX, tables → markdown, figures → desc
LLM Review	Configurable: Anthropic / OpenAI / Gemini / OpenRouter
Multi-Agent	LangGraph StateGraph — ReAct + Gated Pipeline (Orchestrator-Subagents)
Paper Search	Tavily Search API (ACL Anthology priority + arXiv/Semantic Scholar)
Email	Resend (3,000 free/month)
Deploy	Hugging Face Docker Spaces

Project Structure

paper_review/
├── backend/
│   ├── main.py               # FastAPI app & endpoints
│   ├── config.py             # Settings from environment variables
│   ├── auth.py               # Supabase GoTrue auth + RBAC (admin / user roles)
│   ├── observability.py      # Pipeline step + LLM call tracking
│   ├── email_service.py      # Resend email integration
│   ├── logger.py             # Per-job stdout + file logger (uvicorn-routed)
│   ├── llm/
│   │   └── client.py         # Unified LLM client (multi-provider)
│   ├── pipeline/
│   │   ├── guardrails.py     # Pre-pipeline eligibility checks (G1a/G1b/G2)
│   │   ├── parsed_paper.py   # ParsedPaper schema + helper functions
│   │   ├── pdf2md.py         # PDF → ParsedPaper (LandingAI or Docling)
│   │   ├── extract.py        # Extract title, contributions & topic
│   │   ├── search.py         # Generate queries + Tavily 2-tier search
│   │   ├── paper_info.py     # arXiv metadata fetching
│   │   ├── summarize.py      # Summarize related papers
│   │   ├── review.py         # Legacy single-pass ARR review (fallback only)
│   │   └── review_agent/     # Multi-agent LangGraph reviewer (primary)
│   │       ├── state.py      # ReviewAgentState + ToolFinding TypedDicts
│   │       ├── prompts.py    # System prompts for all gates, tools & synthesizer
│   │       ├── tools.py      # 7 analysis tools (paper-type gated, parallel)
│   │       ├── nodes.py      # LangGraph node functions (desk_check, planner, ...)
│   │       └── graph.py      # StateGraph wiring + run_review_agent() entry point
│   ├── storage/
│   │   ├── jobs.py           # Storage backend selector
│   │   └── supabase_store.py # Supabase persistence layer
│   └── requirements.txt
├── frontend/
│   ├── home.html             # Landing page (7-metric ARR rubric showcase)
│   ├── upload.html           # Upload page — drag-drop PDF + email
│   ├── login.html            # Google OAuth login
│   ├── review.html           # Review viewer — ARR scores, strengths, weaknesses
│   ├── admin.html            # Admin dashboard
│   ├── css/style.css         # Dark/gold theme, shared across all pages
│   └── js/
│       ├── auth.js           # Supabase auth client
│       ├── upload.js         # File drag-drop, submit, access key display
│       ├── review.js         # Key lookup, poll status, render all 7 ARR chips
│       ├── feedback.js       # Feedback form submission
│       └── theme.js          # Dark/light theme toggle
├── docs/
│   ├── database/             # Supabase schema docs + Google OAuth setup guide
│   └── parser/               # PDF parser evaluation, OCR benchmarks, test outputs
│       ├── scripts/          # Evaluation scripts (run_mvp_eval.py, test_docling_ocr.py)
│       ├── test/             # Parsed PDF JSONs + review outputs for manual evaluation
│       └── vinuni20k-parser-serving.ipynb  # Kaggle Docling server notebook
├── scripts/
│   └── apply_migration.py    # One-time DB migration runner
├── Dockerfile                # Hugging Face Docker Space
├── .env.example              # Template for environment variables
└── run.py                    # Quick-start script (local dev)

Setup

1. Prerequisites

Python 3.10+
Supabase project (free tier works)
API keys (see below)

2. Install dependencies

cd paper_review
pip install -r backend/requirements.txt

3. Configure environment variables

cp .env.example .env

Fill in .env — minimum required to run:

LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-...

PDF_PARSER=landingai
LANDINGAI_API_KEY=...

TAVILY_API_KEY=tvly-...

RESEND_API_KEY=re_...
RESEND_FROM_EMAIL=onboarding@resend.dev

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=eyJ...
SUPABASE_ANON_KEY=eyJ...
SUPABASE_STORAGE_BUCKET=paper-mate-artifacts

ADMIN_EMAILS=your@email.com
APP_BASE_URL=http://localhost:8000

4. Set up Supabase

Create a project at supabase.com
Run the SQL migrations in order via the Supabase SQL Editor:
- scripts/migrations/ — baseline schema, auth/RBAC, RPC functions
- supabase/migrations/ — guardrail columns (rejection_reason, rejected status)
Create a Storage bucket named paper-mate-artifacts (public)
Enable Google OAuth: Authentication → Providers → Google
Set Redirect URL: https://<your-supabase-project>.supabase.co/auth/v1/callback

5. API Keys — Where to Get Them

Key	Where to get
`OPENAI_API_KEY`	platform.openai.com
`ANTHROPIC_API_KEY`	console.anthropic.com
`GEMINI_API_KEY`	aistudio.google.com
`OPENROUTER_API_KEY`	openrouter.ai
`LANDINGAI_API_KEY`	va.landing.ai
`TAVILY_API_KEY`	tavily.com — free tier available
`RESEND_API_KEY`	resend.com — free 3,000 emails/month
`SUPABASE_*`	supabase.com → Project Settings → API

6. Run locally

cd paper_review
python run.py

Or directly with uvicorn:

cd paper_review
python -m uvicorn backend.main:app --reload --host 0.0.0.0 --port 8000

Page	URL
Landing page	http://localhost:8000/app/home.html
Login	http://localhost:8000/app/login.html
Upload page	http://localhost:8000/app/upload.html
Review page	http://localhost:8000/app/review.html
API docs	http://localhost:8000/docs

Note: If using PDF_PARSER=docling, start the Kaggle Docling server first (see Docling on Kaggle), then submit a paper.

Deploy to Hugging Face Spaces

Create a new Space (SDK: Docker, template: Blank)
Add all env vars as Secrets in Space Settings
Push the deployment branch:

git remote add hf https://huggingface.co/spaces/<username>/PaperMate
git push hf origin/deployment:refs/heads/main --force

API Reference

Method	Endpoint	Auth	Description
`POST`	`/api/submit`	Optional	Upload PDF + email → returns `access_key`
`GET`	`/api/review/{access_key}`	—	Get job status and review result
`GET`	`/api/review/{key}/pdf`	—	Download original PDF
`GET`	`/api/review/{key}/markdown`	—	View parsed markdown
`POST`	`/api/feedback/{access_key}`	—	Submit feedback on a completed review
`GET`	`/api/me`	Required	Get current user profile
`GET`	`/api/me/submissions`	Required	List all submissions for the current user (includes `rejection_reason` on rejected submissions)
`GET`	`/api/public-config`	—	Supabase config for frontend
`GET`	`/health`	—	Health check (`reviewer: multi-agent-v1`)

Submission statuses: pending · processing · completed · failed · rejected

Switching LLM Provider

LLM_PROVIDER=anthropic
LLM_MODEL=claude-sonnet-4-6

# or
LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini

# or
LLM_PROVIDER=gemini
LLM_MODEL=gemini-1.5-flash

# or
LLM_PROVIDER=openrouter
LLM_MODEL=meta-llama/llama-3.1-8b-instruct

Switching PDF Parser

# LandingAI ADE (cloud, simpler setup)
PDF_PARSER=landingai
LANDINGAI_API_KEY=...

# Docling (self-hosted Kaggle notebook — higher quality, structured JSON)
PDF_PARSER=docling
NTFY_TOPIC=papermate_pdf2md

Docling on Kaggle

The Kaggle server runs two stages:

GPU parse — Docling with granite-docling-258M extracts labeled elements (title, section, table, formula, figure...)
VLM enrichment — all formula/table/figure items enriched via gpt-4o-mini in parallel

Kaggle secrets required: ngrok_auth_token · openai_api_key

GATE 2: First Working Agent (MVP)

Criteria	Evidence
MVP Demo — video 3 phút show user flow end-to-end	MVP Demo
Architecture diagram — sơ đồ components, data flow	Arch Diagram
Repo có >= 10 PR merged	Proof
README.md — setup instructions, env vars, sample queries	You reading this
Eval evidences — ít nhất 5 test case manual với output thực tế	Click here
Live demo	ntphuc149-papermate.hf.space