intel/README.md

Intel store (backend/intel/)

This is Phase 0 and Phase 1 of the predictive intelligence system. The design is in docs/blueprint/intelligence-system.md, and the decisions are in ADR-0017 to ADR-0021. It runs locally against a SQLite file and is not wired into the deployed API. The only touchpoint is the news → events hook, which does nothing unless INTEL_DB_PATH is set.

Set up

All commands run from backend/.

1. Install the dependencies:

   py -m pip install -r requirements.txt -r requirements-intel.txt
   

2. Add the store's path to backend/.env:

   INTEL_DB_PATH=intel/data/intel.db
   

3. Download the raw files into intel/data/raw/. The folder is gitignored.

   • Boundaries come from geoBoundaries (CC BY 4.0). Use the simplified NGA ADM1 and ADM2 files, named geoBoundaries-NGA-ADM1_simplified.geojson and geoBoundaries-NGA-ADM2_simplified.geojson. The download links are on https://www.geoboundaries.org/countryDownloads.html (NGA).

   • Overture places (CDLA Permissive 2.0) are extracted with duckdb straight from Overture's public S3 bucket. It takes about 30 s and produces 156k rows:

     py -m intel.ingest.overture_extract
     

Build, in order

py -m intel.cli init                 # schema + source/stream registry
py -m intel.cli locations            # gazetteer: 37 states, 774 LGAs, aliases
py -m intel.cli calendar             # holidays and seasons, 2000 to today+10y
py -m intel.cli places               # Overture → places, subsector mapping, locality aliases
py -m intel.cli weather-forecast     # 16-day forecast per state
py -m intel.cli weather-history      # resumable backfill from 2006, 4,500 weighted calls per run

Re-running any of these is safe, because every build is idempotent. The full weather history is about 20,000 weighted Open-Meteo calls. The free tier allows 10,000 a day, so run weather-history once a day until --estimate reports 0. Each run resumes where the last one stopped.

• Licence warning: Open-Meteo's free tier is non-commercial only. Commercial use needs a paid plan or ECMWF open data (ADR-0020).
• One writer at a time: SQLite takes a single writer, so don't run two commands against the store at once.

Day to day

py -m intel.cli status               # Layer 1 report; exits 1 if anything isn't fresh
py -m intel.cli resolve "Ajah"       # gazetteer lookup → ng.lagos.eti-osa
py -m intel.cli run-daily            # forecast + history top-up + status, under one run id
py -m intel.cli export               # CSVs + manifest for the Colab notebooks → intel/data/export/

run-daily is what a scheduler will call. There is no scheduler yet: the design names a GitHub Actions cron or Supabase pg_cron, because the Hugging Face Space sleeps when idle.

Layout

File	Job
db.py	Connection and schema init (../sql/intel_schema.sql)
registry.py	Sources (cadence, latency, licence) and streams
handoff.py	Layer names, handoff receipts, ACTIVE_LAYERS
freshness.py	Layer 1: the two-clock source grading and overdue handoffs
geo.py	Pure-Python centroid, point-in-polygon and polygon index
locations.py	Gazetteer build, boundary matching, resolve()
subsector_map.py	Overture categories → VenThrive's 49 subsectors
events.py	News → permanent events hook (ADR-0015 addendum)
export.py	CSV export for ml/
ingest/	One module per source

Tests

py -m pytest tests/test_intel.py -q

The tests are pure and offline: they use an in-memory database and fake HTTP fetches.