README.md

---
title: Optimized Data Harvester
emoji: 🚀
colorFrom: blue
colorTo: purple
sdk: streamlit
sdk_version: 1.28.0
app_file: app_simplified.py
pinned: false
license: mit
---

# 🚀 Optimized Data Harvester

Research-verified API endpoints with intelligent data extraction, compression, and optimized database storage.

## ✨ Features

### 🌍 **10 Data Sources** - Research-verified API endpoints based on 2024 official documentation
- 🇸🇪 **Swedish Government**: Skolverket, SCB, Kolada, Riksbanken, Swecris, CSN
- 🌍 **International**: Eurostat, WHO, OECD, World Bank

### 🎯 **Modern Interface**
- **Minimalist Design** - Clean, gradient interface with glass morphism
- **Real-time Metrics** - Live status indicators and progress tracking  
- **Interactive Cards** - Hover effects and modern UI components
- **Responsive Layout** - Works perfectly on all devices

### 🚀 **Optimized Features**
- **One-Click Collection** - Single button fetches from all 10 APIs automatically
- **Intelligent Data Extraction** - Custom parsers for each API response format  
- **Smart Compression** - Automatic gzip compression for data >512 bytes (up to 80% space savings)
- **Enhanced Database** - SQLite with WAL mode, proper indexing, and performance optimization
- **Real-time Analytics** - Live performance metrics, compression ratios, and error tracking
- **Advanced Viewer** - Enhanced database explorer with summary statistics

### 🔧 **Technical Architecture**  
- **Response-Tested APIs** - All endpoints verified by live testing in 2024
- **Smart Data Paths** - Configurable extraction using dot notation (e.g., "body._embedded.schoolUnits") 
- **Performance Database** - WAL mode, 6 strategic indexes, compression support
- **Error Resilience** - Graceful degradation with detailed error logging
- **Format Support** - JSON-stat, SDMX-JSON, HAL+JSON, OData, PX-Web
- **Authentication Handling** - Bearer tokens, API keys, and public endpoints

## 📊 **API Details**

### Swedish Sources (Research-Verified 2024)
- **Skolverket**: Education data, compact school units with coordinates (HAL+JSON v3)
- **SCB**: Population statistics via POST requests (Rate limit: 10/10 sec)  
- **Kolada**: Municipal KPIs and regional data (No auth required)
- **Riksbanken**: Latest exchange rate observations (SWEA v1 API with public key)
- **Swecris**: Research projects (Bearer token: VRSwecrisAPI2025-1)
- **CSN**: Student finance statistics (PX-Web API format)

### International Sources (Research-Verified 2024)
- **Eurostat**: EU demographic density statistics (JSON-stat 2.0 format)
- **WHO**: Global health dimensions via GHO OData API (No auth required)
- **OECD**: Economic data via SDMX-JSON format (stats.oecd.org)
- **World Bank**: Population indicators via API v2 (format=json)

## 🚀 **Simple Workflow**

### **One-Click Operation**
1. **Click Button** - Press "FETCH ALL DATA FROM ALL APIS"
2. **Watch Progress** - Real-time updates as each API is contacted
3. **View Results** - Automatic display of success metrics and data preview
4. **Explore Data** - Built-in database viewer and analytics charts

### **What Happens Automatically:**
- **API Requests** - All 10 APIs contacted with proper headers and authentication
- **Data Processing** - Responses parsed and meaningful data extracted
- **Storage** - Automatic saving to SQLite with compression and deduplication
- **Analytics** - Instant charts and statistics generated
- **Error Handling** - Failed requests logged with detailed error messages

## 🎨 **Modern Design**

- **Glass Morphism** - Translucent cards with backdrop blur
- **Gradient Backgrounds** - Beautiful blue-to-purple gradients  
- **Status Indicators** - Color-coded API health monitoring
- **Smooth Animations** - CSS transitions and hover effects
- **Responsive Grid** - Adaptive layout for all screen sizes

## 📈 **Data Visualization**

- **Success Rate Pie Charts** - Visual fetch status overview
- **Records Bar Charts** - Compare data volume by API
- **Real-time Metrics** - Live updating counters and indicators
- **Interactive Tables** - Sortable, filterable data preview

The simplest and most efficient way to collect data from multiple international APIs with a single click.