Spaces:
Running
Running
| # Intel store (`backend/intel/`) | |
| This is Phase 0 and Phase 1 of the predictive intelligence system. The design is in | |
| [docs/blueprint/intelligence-system.md](../../docs/blueprint/intelligence-system.md), and the decisions | |
| are in ADR-0017 to ADR-0021. It runs locally against a SQLite file and is not wired into the deployed API. | |
| The only touchpoint is the news → events hook, which does nothing unless `INTEL_DB_PATH` is set. | |
| ## Set up | |
| All commands run from `backend/`. | |
| 1. Install the dependencies: | |
| ```bash | |
| py -m pip install -r requirements.txt -r requirements-intel.txt | |
| ``` | |
| 2. Add the store's path to `backend/.env`: | |
| ``` | |
| INTEL_DB_PATH=intel/data/intel.db | |
| ``` | |
| 3. Download the raw files into `intel/data/raw/`. The folder is gitignored. | |
| - **Boundaries** come from geoBoundaries (CC BY 4.0). Use the simplified NGA ADM1 and ADM2 files, | |
| named `geoBoundaries-NGA-ADM1_simplified.geojson` and `geoBoundaries-NGA-ADM2_simplified.geojson`. | |
| The download links are on <https://www.geoboundaries.org/countryDownloads.html> (NGA). | |
| - **Overture places** (CDLA Permissive 2.0) are extracted with duckdb straight from Overture's | |
| public S3 bucket. It takes about 30 s and produces 156k rows: | |
| ```bash | |
| py -m intel.ingest.overture_extract | |
| ``` | |
| ## Build, in order | |
| ```bash | |
| py -m intel.cli init # schema + source/stream registry | |
| py -m intel.cli locations # gazetteer: 37 states, 774 LGAs, aliases | |
| py -m intel.cli calendar # holidays and seasons, 2000 to today+10y | |
| py -m intel.cli places # Overture → places, subsector mapping, locality aliases | |
| py -m intel.cli weather-forecast # 16-day forecast per state | |
| py -m intel.cli weather-history # resumable backfill from 2006, 4,500 weighted calls per run | |
| ``` | |
| Re-running any of these is safe, because every build is idempotent. The full weather history is about | |
| 20,000 weighted Open-Meteo calls. The free tier allows 10,000 a day, so run `weather-history` once a day | |
| until `--estimate` reports 0. Each run resumes where the last one stopped. | |
| - **Licence warning:** Open-Meteo's free tier is **non-commercial only**. Commercial use needs a paid | |
| plan or ECMWF open data (ADR-0020). | |
| - **One writer at a time:** SQLite takes a single writer, so don't run two commands against the store | |
| at once. | |
| ## Day to day | |
| ```bash | |
| py -m intel.cli status # Layer 1 report; exits 1 if anything isn't fresh | |
| py -m intel.cli resolve "Ajah" # gazetteer lookup → ng.lagos.eti-osa | |
| py -m intel.cli run-daily # forecast + history top-up + status, under one run id | |
| py -m intel.cli export # CSVs + manifest for the Colab notebooks → intel/data/export/ | |
| ``` | |
| `run-daily` is what a scheduler will call. There is no scheduler yet: the design names a GitHub | |
| Actions cron or Supabase `pg_cron`, because the Hugging Face Space sleeps when idle. | |
| ## Layout | |
| | File | Job | | |
| |---|---| | |
| | `db.py` | Connection and schema init (`../sql/intel_schema.sql`) | | |
| | `registry.py` | Sources (cadence, latency, licence) and streams | | |
| | `handoff.py` | Layer names, handoff receipts, `ACTIVE_LAYERS` | | |
| | `freshness.py` | Layer 1: the two-clock source grading and overdue handoffs | | |
| | `geo.py` | Pure-Python centroid, point-in-polygon and polygon index | | |
| | `locations.py` | Gazetteer build, boundary matching, `resolve()` | | |
| | `subsector_map.py` | Overture categories → VenThrive's 49 subsectors | | |
| | `events.py` | News → permanent `events` hook (ADR-0015 addendum) | | |
| | `export.py` | CSV export for `ml/` | | |
| | `ingest/` | One module per source | | |
| ## Tests | |
| ```bash | |
| py -m pytest tests/test_intel.py -q | |
| ``` | |
| The tests are pure and offline: they use an in-memory database and fake HTTP fetches. | |