Create README.md

README.md

@@ -2,176 +2,167 @@
 title: Federal FOIA Intelligence Search
 emoji: 🏛️
 colorFrom: blue
-colorTo: indigo
+colorTo: gray
 sdk: gradio
-sdk_version: 6.3.0
+sdk_version: 4.36.0
 app_file: app.py
 pinned: true
-tags:
-- foia
-- government-transparency
-- public-records
-- journalism
-- open-data
-- legal-tech
-- oversight
 license: mit
-short_description: 'FOIA INTELLIGENCE SEARCH '
+short_description: 'FOIA DECLASSIFIED DOCUMENTS SEARCH '
 ---
 
 # Federal FOIA Intelligence Search
+### Public Electronic Reading Rooms Only
 
-**Public Electronic Reading Rooms Only**
+A **live, robots-compliant search application** for discovering publicly released U.S. government documents available in **FOIA Electronic Reading Rooms**.
 
-This Hugging Face Space provides a federated search and analysis interface for
-**publicly released U.S. Government records made available under the Freedom of Information Act (FOIA)**.
-
-It is designed for **journalists, researchers, legal professionals, oversight bodies, and the public**.
+This Space is designed for **journalists, researchers, historians, attorneys, and the general public** to explore already-public government records — **not to access classified, restricted, or non-public information**.
 
 ---
 
-## 🔍 What This Space Does
+## ✅ What This App Does
 
-- Searches **only publicly accessible FOIA Electronic Reading Rooms**
-- Uses **official agency search endpoints or public landing pages**
-- Enforces **robots.txt, rate limiting, and safe defaults**
-- Provides **semantic search, clustering, and visualization**
-- Generates **court-ready citations and FOIA request packets**
+- 🔍 Searches **public FOIA Electronic Reading Rooms**
+- 🌐 Uses **live queries** only where automated access is explicitly permitted
+- 🤖 Enforces **robots.txt compliance per agency**
+- 🧾 Clearly distinguishes **live results vs. stub (non-live) sources**
+- 📦 Allows export of results for research and reporting
+- ⚖️ Designed to comply with **FOIA, HF policies, and U.S. law**
 
 ---
 
-## 🏛️ Live Public Sources
-
-This Space currently supports live querying or indexed access to the following **public FOIA repositories**:
+## 🚫 What This App Does NOT Do
 
-- **CIA FOIA Electronic Reading Room**
-- **FBI Vault**
-- **NSA FOIA Library**
-- **Department of Defense FOIA Reading Room**
-- **National Reconnaissance Office (NRO) Declassified Releases**
-- **Department of Justice (DOJ) FOIA Library**
-- **Department of Homeland Security (DHS) FOIA Library**
-- **U.S. Department of State FOIA Reading Room**
-
-All access is:
-- Public
-- Unauthenticated
-- Non-privileged
-- Read-only
+- ❌ No scraping of restricted or protected systems
+- ❌ No bypassing of authentication, paywalls, or CAPTCHAs
+- ❌ No access to classified, controlled, or non-public data
+- ❌ No querying of intelligence systems or operational databases
+- ❌ No real-time surveillance or monitoring
 
 ---
 
-## 📂 Hosted Public Collections (Clearly Labeled)
+## 🟢 Live Sources (Queried in Real Time)
 
-Some historically named programs or collections (e.g. **AATIP**, **Special Access Programs**, **Special Activities**) do **not** operate independent FOIA portals.
+These agencies provide **public FOIA search endpoints** and explicitly permit automated access:
 
-When these appear in the interface, they are:
-
-- Explicitly labeled as **Hosted Public Releases**
-- Linked to the **original publishing agency**
-- Limited to **already-declassified, publicly released documents**
-- Included for **historical research and transparency**
+- **CIA** – FOIA Electronic Reading Room
+- **FBI Vault**
+- **Department of Justice (DOJ)** – FOIA Library
+- **Department of Homeland Security (DHS)** – FOIA Library
+- **U.S. Department of State** – FOIA Search
+- **General Services Administration (GSA)** – FOIA Library
 
-No restricted or classified systems are accessed.
+Live sources are:
+- Rate-limited
+- Queried read-only
+- Checked against robots.txt before each use
 
 ---
 
-## 📊 Analytics & Visualization Features
+## 🟡 Stub-Only Sources (Transparency Mode)
 
-- **Real-time agency coverage heatmap** (documents per agency)
-- **Latency & health indicators** per source
-- **Interactive semantic cluster graph** (Plotly)
-- **Timeline views** for document release dates
-- **Result deduplication and clustering**
+Some agencies **do not permit automated querying**, **block access via robots.txt**, or **do not provide a public FOIA search endpoint**.
 
----
+These are represented as **clearly labeled stub adapters** for transparency only.
 
-## 🧠 Semantic Search
+Stub sources **DO NOT perform live queries**.
 
-- Uses **sentence-transformers + FAISS**
-- Supports:
-  - Semantic clustering
-  - Search-within-results
-  - Topic grouping
-- No model training on classified or private data
+### Stub Sources Include:
+- NSA
+- NRO
+- DIA
+- NGA
+- Special Access Programs (SAP)
+- TEN-CAP
+- AATIP
+- Special Activities
 
 ---
 
-## 🧾 Legal & Journalistic Tools
+## ⚠️ Extended Features (User Warning)
 
-- **Court-ready Bluebook citation PDF export**
-- **Journalist ZIP export** (documents + index)
-- **FOIA request packet generator** (PDF / text)
-- **FOIA exemption (b-code) labeling** (heuristic, informational only)
+An optional **Extended Features** mode allows users to include **stub-only results**.
 
-> ⚠️ FOIA requests are **generated only**.  
-> This Space does **not** submit requests on behalf of users.
+Before enabling, users must explicitly acknowledge that:
+- No live queries will be performed
+- Results are informational only
+- Some agencies are restricted by law or policy
+
+This design prevents misuse while maintaining public-interest transparency.
 
 ---
 
-## ⚖️ What This Space Does NOT Do
+## 🛡️ Trust, Safety & Compliance
 
-❌ No classified access  
-❌ No scraping behind authentication  
-❌ No bypassing agency safeguards  
-❌ No intelligence analysis or inference  
-❌ No automated FOIA submissions  
-❌ No user tracking beyond standard HF analytics  
+This Space was designed with the following safeguards:
 
----
+- ✅ **robots.txt enforcement** per adapter
+- ✅ **No authentication or credential use**
+- ✅ **Public-domain / public-release content only**
+- ✅ **Clear labeling of non-live sources**
+- ✅ **User acknowledgment for extended features**
+- ✅ **Read-only access**
+- ✅ **No data retention or tracking**
+
+The application complies with:
+- U.S. Freedom of Information Act (FOIA)
+- Hugging Face Spaces policies
+- Standard web crawling norms
+- Public-interest research standards
 
-## 📜 Legal Basis
+---
 
-All content accessed through:
+## 📦 Export & Research Use
 
-- **5 U.S.C. § 552 (Freedom of Information Act)**
-- Agency-maintained **Electronic Reading Rooms**
-- Public U.S. Government websites
+Users may export search results as a ZIP archive for:
+- Journalism
+- Academic research
+- Legal analysis
+- Historical archiving
 
-This Space operates entirely within:
-- U.S. law
-- Agency publication rules
-- Hugging Face platform policies
+Exports include:
+- Source agency
+- Document title
+- Public URL
+- Snippet / context
 
 ---
 
-## 🛡️ Safety & Governance
+## 👩‍⚖️ Legal & Policy Notes
 
-- Robots.txt enforcement per adapter
-- Per-agency rate limits
-- Per-agency kill switches
-- Health monitoring & auto-disable
-- Adapter compliance tests (CI)
+- This tool does **not replace formal FOIA requests**
+- It only indexes **already-public disclosures**
+- For non-public records, users should submit FOIA requests directly to agencies
+- Inclusion of an agency name does **not imply live access**
 
 ---
 
-## 🎯 Intended Use
-
-This project is intended to support:
+## 🧪 Technical Overview
 
-- Investigative journalism
-- Academic and historical research
-- Legal review and litigation support
-- Government oversight
-- Public transparency initiatives
+- **Frontend:** Gradio (HF Spaces)
+- **Architecture:** Async federated adapters
+- **Safety:** Per-adapter robots enforcement
+- **Design:** Explicit live vs stub separation
+- **Deployment:** Single `app.py` entrypoint
 
 ---
 
-## 📬 Disclaimer
-
-This tool aggregates and analyzes **public information only**.  
-It does not provide legal advice, intelligence assessments, or official interpretations.
+## 🧭 Intended Audience
 
-Users are responsible for verifying primary sources.
+- Journalists
+- Researchers
+- Policy analysts
+- Attorneys
+- Historians
+- Members of the public seeking transparency
 
 ---
 
-## 📦 Open Source & Transparency
+## 📬 Contact / Issues
 
-All adapters, logic, and safety mechanisms are visible in the source code.
-No hidden data sources or privileged access exist.
+This Space is an open, public-interest research tool.  
+For issues, suggestions, or compliance questions, please use the Hugging Face discussion tab.
 
 ---
 
-**Federal FOIA Intelligence Search**  
-*Making public records easier to find, understand, and cite.*
+### 🏛️ Transparency First. Public Records Only.