Spaces:
Running
Running
File size: 3,745 Bytes
b7c46f4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | # Intel store (`backend/intel/`)
This is Phase 0 and Phase 1 of the predictive intelligence system. The design is in
[docs/blueprint/intelligence-system.md](../../docs/blueprint/intelligence-system.md), and the decisions
are in ADR-0017 to ADR-0021. It runs locally against a SQLite file and is not wired into the deployed API.
The only touchpoint is the news → events hook, which does nothing unless `INTEL_DB_PATH` is set.
## Set up
All commands run from `backend/`.
1. Install the dependencies:
```bash
py -m pip install -r requirements.txt -r requirements-intel.txt
```
2. Add the store's path to `backend/.env`:
```
INTEL_DB_PATH=intel/data/intel.db
```
3. Download the raw files into `intel/data/raw/`. The folder is gitignored.
- **Boundaries** come from geoBoundaries (CC BY 4.0). Use the simplified NGA ADM1 and ADM2 files,
named `geoBoundaries-NGA-ADM1_simplified.geojson` and `geoBoundaries-NGA-ADM2_simplified.geojson`.
The download links are on <https://www.geoboundaries.org/countryDownloads.html> (NGA).
- **Overture places** (CDLA Permissive 2.0) are extracted with duckdb straight from Overture's
public S3 bucket. It takes about 30 s and produces 156k rows:
```bash
py -m intel.ingest.overture_extract
```
## Build, in order
```bash
py -m intel.cli init # schema + source/stream registry
py -m intel.cli locations # gazetteer: 37 states, 774 LGAs, aliases
py -m intel.cli calendar # holidays and seasons, 2000 to today+10y
py -m intel.cli places # Overture → places, subsector mapping, locality aliases
py -m intel.cli weather-forecast # 16-day forecast per state
py -m intel.cli weather-history # resumable backfill from 2006, 4,500 weighted calls per run
```
Re-running any of these is safe, because every build is idempotent. The full weather history is about
20,000 weighted Open-Meteo calls. The free tier allows 10,000 a day, so run `weather-history` once a day
until `--estimate` reports 0. Each run resumes where the last one stopped.
- **Licence warning:** Open-Meteo's free tier is **non-commercial only**. Commercial use needs a paid
plan or ECMWF open data (ADR-0020).
- **One writer at a time:** SQLite takes a single writer, so don't run two commands against the store
at once.
## Day to day
```bash
py -m intel.cli status # Layer 1 report; exits 1 if anything isn't fresh
py -m intel.cli resolve "Ajah" # gazetteer lookup → ng.lagos.eti-osa
py -m intel.cli run-daily # forecast + history top-up + status, under one run id
py -m intel.cli export # CSVs + manifest for the Colab notebooks → intel/data/export/
```
`run-daily` is what a scheduler will call. There is no scheduler yet: the design names a GitHub
Actions cron or Supabase `pg_cron`, because the Hugging Face Space sleeps when idle.
## Layout
| File | Job |
|---|---|
| `db.py` | Connection and schema init (`../sql/intel_schema.sql`) |
| `registry.py` | Sources (cadence, latency, licence) and streams |
| `handoff.py` | Layer names, handoff receipts, `ACTIVE_LAYERS` |
| `freshness.py` | Layer 1: the two-clock source grading and overdue handoffs |
| `geo.py` | Pure-Python centroid, point-in-polygon and polygon index |
| `locations.py` | Gazetteer build, boundary matching, `resolve()` |
| `subsector_map.py` | Overture categories → VenThrive's 49 subsectors |
| `events.py` | News → permanent `events` hook (ADR-0015 addendum) |
| `export.py` | CSV export for `ml/` |
| `ingest/` | One module per source |
## Tests
```bash
py -m pytest tests/test_intel.py -q
```
The tests are pure and offline: they use an in-memory database and fake HTTP fetches.
|