README.md

🔎 HF Space — Automated Requirement Citation Checker

This Space ingests a CSV with columns: Prompt, Rubric, ID, Requirements, Important Evaluation Rules and verifies each atomic claim in the Requirements column against the public web.

It searches via Google Custom Search (CSE), fetches top pages, and asks both OpenAI and DeepSeek LLMs to decide if each claim is SUPPORTED, CONTRADICTED, or UNVERIFIABLE, returning citations and a concise reason. A per-row conclusion is computed from all claims.

Quick Start (Hugging Face Spaces)

1. Create a new Space → SDK: Gradio → Visibility: your choice.
2. Upload these files to the Space root:
   • app.py
   • verifier.py
   • search.py
   • models.py
   • utils.py
   • requirements.txt
   • .env.template (optional for local dev)
3. In the Space page → Settings → Secrets, add:
   • OPENAI_API_KEY — your OpenAI key (you mentioned OAPI1).
   • DEEPSEEK_API_KEY — your DeepSeek key (you mentioned DAPI).
   • GOOGLE_API_KEY — Google API key (you mentioned API1).
   • GOOGLE_CSE_ID — your CSE ID (you mentioned CX1).
   • (Optional) OPENAI_MODEL (default gpt-4o-mini)
   • (Optional) DEEPSEEK_MODEL (default deepseek-chat)
4. Deploy. Open the Space, upload your CSV, and click Run Verification.

Output

• Table view summarizing Supported/Contradicted/Unverifiable counts and overall conclusion per row.
• JSON with full detail (per-claim statuses, reasons, citations, and both models' raw decisions).
• Markdown report with human-readable results and links.

CSV Format

Columns required (case-sensitive):

• Prompt
• Rubric
• ID
• Requirements
• Important Evaluation Rules

Only Requirements is fact-checked; other columns are passed through for context if you want to extend prompts later.

How it works

• utils.split_atomic_claims heuristically splits Requirements text into atomic claims (lists + sentences).
• search.google_cse queries Google CSE (multiple queries per claim), collects unique result links.
• search.fetch_page downloads pages and extracts plain text + titles for LLM review.
• verifier.verify_claim builds a single packed prompt with all sources and asks both OpenAI & DeepSeek to return structured JSON:

  {
    "status": "SUPPORTED|CONTRADICTED|UNVERIFIABLE",
    "reason": "brief rationale",
    "citations": [{"url": "...", "title": "...", "quote": "short"}]
  }
  

• An ensemble chooses the final status; the other model’s output is preserved for transparency.
• verifier.verify_requirement aggregates claim-level results and computes a row-level conclusion:
  • Any CONTRADICTED → CONTRADICTED
  • Else if all supported → SUPPORTED
  • Else if mixed → PARTIALLY SUPPORTED
  • Else → UNVERIFIABLE

Local Development

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.template .env  # fill keys
python app.py

Notes & Tips

• Be conservative: the models are instructed to mark UNVERIFIABLE when evidence is insufficient.
• You can tune the number of search results and page length limits in search.py.
• For highly technical domains, consider whitelisting official domains in your CSE for higher precision.
• If you hit rate limits, Tenacity will retry with exponential backoff.

License

MIT