Spaces:
Sleeping
Sleeping
| title: Optimized Data Harvester | |
| emoji: 🚀 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: streamlit | |
| sdk_version: 1.28.0 | |
| app_file: app_simplified.py | |
| pinned: false | |
| license: mit | |
| # 🚀 Optimized Data Harvester | |
| Research-verified API endpoints with intelligent data extraction, compression, and optimized database storage. | |
| ## ✨ Features | |
| ### 🌍 **10 Data Sources** - Research-verified API endpoints based on 2024 official documentation | |
| - 🇸🇪 **Swedish Government**: Skolverket, SCB, Kolada, Riksbanken, Swecris, CSN | |
| - 🌍 **International**: Eurostat, WHO, OECD, World Bank | |
| ### 🎯 **Modern Interface** | |
| - **Minimalist Design** - Clean, gradient interface with glass morphism | |
| - **Real-time Metrics** - Live status indicators and progress tracking | |
| - **Interactive Cards** - Hover effects and modern UI components | |
| - **Responsive Layout** - Works perfectly on all devices | |
| ### 🚀 **Optimized Features** | |
| - **One-Click Collection** - Single button fetches from all 10 APIs automatically | |
| - **Intelligent Data Extraction** - Custom parsers for each API response format | |
| - **Smart Compression** - Automatic gzip compression for data >512 bytes (up to 80% space savings) | |
| - **Enhanced Database** - SQLite with WAL mode, proper indexing, and performance optimization | |
| - **Real-time Analytics** - Live performance metrics, compression ratios, and error tracking | |
| - **Advanced Viewer** - Enhanced database explorer with summary statistics | |
| ### 🔧 **Technical Architecture** | |
| - **Response-Tested APIs** - All endpoints verified by live testing in 2024 | |
| - **Smart Data Paths** - Configurable extraction using dot notation (e.g., "body._embedded.schoolUnits") | |
| - **Performance Database** - WAL mode, 6 strategic indexes, compression support | |
| - **Error Resilience** - Graceful degradation with detailed error logging | |
| - **Format Support** - JSON-stat, SDMX-JSON, HAL+JSON, OData, PX-Web | |
| - **Authentication Handling** - Bearer tokens, API keys, and public endpoints | |
| ## 📊 **API Details** | |
| ### Swedish Sources (Research-Verified 2024) | |
| - **Skolverket**: Education data, compact school units with coordinates (HAL+JSON v3) | |
| - **SCB**: Population statistics via POST requests (Rate limit: 10/10 sec) | |
| - **Kolada**: Municipal KPIs and regional data (No auth required) | |
| - **Riksbanken**: Latest exchange rate observations (SWEA v1 API with public key) | |
| - **Swecris**: Research projects (Bearer token: VRSwecrisAPI2025-1) | |
| - **CSN**: Student finance statistics (PX-Web API format) | |
| ### International Sources (Research-Verified 2024) | |
| - **Eurostat**: EU demographic density statistics (JSON-stat 2.0 format) | |
| - **WHO**: Global health dimensions via GHO OData API (No auth required) | |
| - **OECD**: Economic data via SDMX-JSON format (stats.oecd.org) | |
| - **World Bank**: Population indicators via API v2 (format=json) | |
| ## 🚀 **Simple Workflow** | |
| ### **One-Click Operation** | |
| 1. **Click Button** - Press "FETCH ALL DATA FROM ALL APIS" | |
| 2. **Watch Progress** - Real-time updates as each API is contacted | |
| 3. **View Results** - Automatic display of success metrics and data preview | |
| 4. **Explore Data** - Built-in database viewer and analytics charts | |
| ### **What Happens Automatically:** | |
| - **API Requests** - All 10 APIs contacted with proper headers and authentication | |
| - **Data Processing** - Responses parsed and meaningful data extracted | |
| - **Storage** - Automatic saving to SQLite with compression and deduplication | |
| - **Analytics** - Instant charts and statistics generated | |
| - **Error Handling** - Failed requests logged with detailed error messages | |
| ## 🎨 **Modern Design** | |
| - **Glass Morphism** - Translucent cards with backdrop blur | |
| - **Gradient Backgrounds** - Beautiful blue-to-purple gradients | |
| - **Status Indicators** - Color-coded API health monitoring | |
| - **Smooth Animations** - CSS transitions and hover effects | |
| - **Responsive Grid** - Adaptive layout for all screen sizes | |
| ## 📈 **Data Visualization** | |
| - **Success Rate Pie Charts** - Visual fetch status overview | |
| - **Records Bar Charts** - Compare data volume by API | |
| - **Real-time Metrics** - Live updating counters and indicators | |
| - **Interactive Tables** - Sortable, filterable data preview | |
| The simplest and most efficient way to collect data from multiple international APIs with a single click. |