Upload README.md with huggingface_hub

README.md

@@ -1,6 +1,6 @@
 ---
-title: Global Data Harvester
-emoji: 🌍
+title: Optimized Data Harvester
+emoji: 🚀
 colorFrom: blue
 colorTo: purple
 sdk: streamlit
@@ -10,9 +10,9 @@ pinned: false
 license: mit
 ---
 
-# 🚀 Simplified Data Harvester
+# 🚀 Optimized Data Harvester
 
-One-click data collection from 10 international APIs with automatic storage and real-time analytics.
+Research-verified API endpoints with intelligent data extraction, compression, and optimized database storage.
 
 ## ✨ Features
 
@@ -26,21 +26,21 @@ One-click data collection from 10 international APIs with automatic storage and
 - **Interactive Cards** - Hover effects and modern UI components
 - **Responsive Layout** - Works perfectly on all devices
 
-### 🚀 **Simplified Features**
+### 🚀 **Optimized Features**
 - **One-Click Collection** - Single button fetches from all 10 APIs automatically
-- **Smart Storage** - Automatic deduplication and compression in SQLite
-- **Real-time Progress** - Live updates during data collection
-- **Instant Analytics** - Automatic visualization of collected data
-- **Error Handling** - Graceful failure management with detailed reporting
-- **Database Viewer** - Built-in interface to explore collected data
-
-### 🔧 **Technical Architecture**
-- **Streamlined Design** - Clean, simple interface focused on results
-- **Optimized Database** - SQLite with compression and indexing
-- **Verified Endpoints** - Pre-configured working API endpoints for 2024
-- **Robust Processing** - Error handling and data validation
-- **Smart Authentication** - Bearer tokens and proper headers
-- **Multi-format Support** - JSON, XML, PX-Web parsing
+- **Intelligent Data Extraction** - Custom parsers for each API response format  
+- **Smart Compression** - Automatic gzip compression for data >512 bytes (up to 80% space savings)
+- **Enhanced Database** - SQLite with WAL mode, proper indexing, and performance optimization
+- **Real-time Analytics** - Live performance metrics, compression ratios, and error tracking
+- **Advanced Viewer** - Enhanced database explorer with summary statistics
+
+### 🔧 **Technical Architecture**  
+- **Response-Tested APIs** - All endpoints verified by live testing in 2024
+- **Smart Data Paths** - Configurable extraction using dot notation (e.g., "body._embedded.schoolUnits") 
+- **Performance Database** - WAL mode, 6 strategic indexes, compression support
+- **Error Resilience** - Graceful degradation with detailed error logging
+- **Format Support** - JSON-stat, SDMX-JSON, HAL+JSON, OData, PX-Web
+- **Authentication Handling** - Bearer tokens, API keys, and public endpoints
 
 ## 📊 **API Details**