The commodity configs in commodities_kenya.json already carry
kamis_product_id as an integer field (the source of truth), but
_resolve_product_id ignored it and went through string-matching on
name/alias. That worked for three of four commodities but failed for
BEN-YGN because `_norm("Beans (Yellow-Green)")` = "beans (yellow-green)"
is not an alias key (only "beans" is), producing the
"no PRODUCT_IDS mapping for commodity BEN-YGN — skipping" log and
empty bean ingest.
Fix: check `commodity["kamis_product_id"]` first and return it if
present. String-matching fallback stays in place for configs that
predate the explicit field.
Test:
MZE-DRY Dry Maize -> 1
BEN-YGN Beans (Yellow-Green) -> 30 (was None)
POT-RIR Red Irish Potato -> 57
GRG-KEN Green Grams -> 10
32 tests pass.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix model_type label so it reflects which forecast path actually ran (#6)
Two coordinated fixes:
1. ChronosXGBoostForecaster.train() now sets self.model_used = "chronos2"
when _init_chronos() succeeds. The previous refactor set model_used =
"xgboost" from the XGBoost training step and never updated it when
Chronos loaded, so the "intent" reported to the pipeline was wrong
whenever the train path ran (which happens on the first run after
every HF Space rebuild — no joblib on disk → load() raises → train()).
2. pipeline.py recaptures model_type = model.model_used AFTER predict(),
not before. The model.model_used attribute is updated by predict()
based on which branch actually ran; capturing before predict() records
intent, not reality. This guarantees the step_results detail and the
price_forecasts.model_type column reflect the forecast path the
farmer-facing outputs actually came from.
Observed behavior before the fix: forecast step ran in 2.8s (Chronos
range, not the ~9s XGBoost-standalone range), Chronos clearly loaded
from …
Pre-warm Chronos at FastAPI startup to eliminate first-trigger fallback (#5)
The 60s→180s timeout bump (PR #4) helped the slow-download failure mode
but didn't help when Chronos fails via a quick exception (network/HF
Hub error). Observed today: a post-rebuild trigger finished its forecast
step in 9.1s on XGBoost — the timeout didn't fire, Chronos had
errored out fast and fallen through to the fallback.
More reliable fix: kick off Chronos .load() in a daemon thread via a
FastAPI lifespan hook at Space startup. By the time the first pipeline
trigger arrives (GH Action does wake → 90s sleep → health check →
trigger, so ~60-90s minimum after container start), Chronos is either
already loaded or actively loading with its cache being populated. The
subsequent pipeline-side load() hits the disk cache in <1s regardless
of whether prewarm finished.
If prewarm itself fails for any reason, the pipeline's existing
fallback chain (Chronos → XGBoost → seasonal baseline) still covers
the first trigger.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HF Spaces rebuilds wipe /home/appuser/.cache/huggingface even though
the Dockerfile pre-bakes amazon/chronos-bolt-tiny there, forcing a
re-download on the first trigger after each rebuild. Cold-start
download takes ~80-90s, exceeding the prior 60s timeout and falling
through to XGBoost for the first run. Subsequent triggers load
from disk cache in <1s.
Default timeout now 180s — fits the observed cold-start with headroom.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix four correctness bugs surfaced in the post-MOS-rip diagnostic (#3)
1. Unit mismatch: KAMIS publishes KES per kg; pipeline expected per quintal
Result: SMS showed "Net: KES 14/qtl" for a maize farmer because the
optimizer subtracted per-quintal transport costs from a per-kg price.
Fix: normalize to KES/quintal at both KAMIS PriceRecord creation sites
(demo CSV reader and live API) via a named KG_TO_QUINTAL=100 constant.
2. CI bounds missing for 14d/30d horizons
Neon's price_forecasts rows had ci_lower/ci_upper NULL for 14d and 30d.
Root cause: pipeline.py:1075-1088 only copied ci_lower_7d/ci_upper_7d
into the dict sent to save_pipeline_run; the other four CI fields from
the PriceForecast dataclass were dropped. Fix: include all six CI keys.
3. Swahili SMS leaked the example greeting's name
FMR-K0003's SMS started "Habari Wanjiku, Owino Muriuki,". The translation
prompt said "start naturally, like 'Habari Wanjiku...'" — Claude copied
the example name verbatim before the real farmer's name. Fix: rewrite
the prompt to…
Remove orphaned MOS layer — Chronos-2 is the forecasting model (#2)
The XGBoost MOS "bias correction" sitting on top of Chronos-2 wasn't
earning its keep:
- The monthly retrain (scripts/retrain_mos.py) wrote to models/mos_*.json,
which the live pipeline never loaded — it reads models/price_model.joblib
via ChronosXGBoostForecaster.load(). The retrain has been a silent no-op
since it was written.
- The in-pipeline _train_mos path trained on synthetic data every run
(generate_training_data), against residuals that were structurally OK
but contributed no real accumulated learning. Fresh model per run, 20
samples minimum, no eval gate.
- The retrain script itself had three correctness bugs (cartesian join,
current_price = actual label leak, no promotion gate) — all moot now
since the script is deleted.
Chronos-2 is the honest ML story: a foundation model producing zero-shot
probabilistic forecasts with quantile-native confidence intervals. The
XGBoost standalone model remains as a fallback when Chronos fails to
load or a pair has insuffic…
Kenya-first framing in CLAUDE.md; DPI docstrings s/product/module/; drop orphaned requirements-dashboard.txt
Consolidated findings from a three-way review (reuse/quality/efficiency)
on Phase 11 v2 changes. Fixes landed:
Reuse / dedup:
- src/policy/chronos_precompute.py: promoted to a
public function; the same ~20-line JSONL reader was duplicated in
train_dfl.py, train_dfl_monthly.py, and the benchmark adapter.
- src/policy/train_dfl.py + train_dfl_monthly.py:
is now a thin wrapper over the shared one.
- frontend/api/region.ts: imports from instead
of re-implementing the env-var coercion.
Quality:
- frontend/api/_region.ts: dropped the cross-region fallback in
getMandiName / getCommodityName. A raw id in the response surfaces
the config gap; the old fallback masked it with the other region's
name.
- _region.ts::isRegionFarmer: the old misclassified
India's FMR-KUMR persona as Kenya. Tightened to require a digit after
the K (), matching the actual generator output
.
- src/policy/train_dfl_monthly.py: deleted the dead function
(~25 lines; never called, and the stub r…
Production pipeline: point DFL loader at dfl_v2 (was still v1)
Phase 11 v2 committed the dfl_v2.lgbm.txt artifact + ran the benchmark
adapter against it, but missed updating three dfl_v1 references in the
production code path:
- src/policy/dfl_policy.py::DEFAULT_MODEL_PATH: dfl_v1 -> dfl_v2. This
is what pipeline.py loads — live Kenya runs on the HF Space now use
the same booster the paper reports (73.1% hit / 1,469 KES on Kenya
daily v0_1 vs v1's 61.5% / 5,468).
- src/pipeline.py::_compute_dfl_actions source-label string: dfl_v1 ->
dfl_v2 so rec.dfl.source matches the booster actually loaded.
- src/policy/train_dfl.py CLI default model-name: dfl_v1 -> dfl_v2
so a blank retrain no longer clobbers the historical v1 artifact.
Tests: both regions clean (India 47 passed, Kenya 32 passed + 18
India-only skipped). The dfl_v1 artifact stays committed at
src/policy/models/dfl_v1.lgbm.txt as a historical baseline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 11 v2: DFL v2 with real Chronos + monthly booster
Retrains the production DFL policy with two upgrades:
1. **Real Chronos quantile features** replace the Phase 11 v1 AR(1)
forecast proxy. A one-shot precompute pass (src/policy/chronos_precompute.py)
batches Chronos-Bolt over all 18,149 daily DPs and 139 monthly DPs,
writing append-only JSONL caches. train_dfl.py reads from the cache
at feature-build time and falls back to the cheap proxy only for DPs
below the 5-obs history threshold.
2. **Expanded Kenya event catalog** (committed earlier to lastmile-bench):
4 events vs Phase 11 v1's 2. Training now sees the 2024 Long Rains
failure + 2025 short-rains disruption as in-distribution examples.
Plus: a parallel monthly booster (dfl_monthly_v1.lgbm.txt) for the
kenya_maize_monthly_v0_2 panel. HoldSellMonthlyAction action space
(0/30/60/90d), FEATURE_NAMES_MONTHLY with forecast quantiles shifted
to 30/60/90d, separate Chronos cache at prediction_length=90.
- src/policy/features_monthly.py (new, 138 lines): parallel to…
Post-simplify pass fixes that matter for the live Kenya pipeline
and prepare for the DFL v2 retrain with real Chronos quantiles.
- src/ingestion/kamis.py: rewrite _fetch_live() to pull one request
per commodity with county=None + local filter (was per-market-per-
commodity with 3s polite delay). 40 requests -> 4, wall time 120s
-> 13s on a 10-markets x 4-commodities scrape. Pooled module-level
httpx.Client replaces fresh client per retry.
- src/policy/features.py: add seasonal calendars for Beans, Green
Grams, and Red Irish Potato — 3 of the 4 live Kenya commodities
that previously fell through to seasonal_flag=0. Ordered India
pulses first so 'Moong (Green Gram)' labels still match 'moong'
rather than the new 'green gram' key. Calendars derived from
FEWS NET + FAO GIEWS Kenya crop-calendar references.
- markets_kenya.json: drop Kitengela (Kajiado) + Kaanwa (Tharaka
Nithi) — no KAMIS coverage in the benchmark raw cache. Add
Machakos Main Market, Nkubu (Meru), Son…
Applied findings from three parallel code reviews (reuse/quality/
efficiency) on all Phase 11 changes. Fixes that made the cut:
- src/rag/knowledge_base_kenya.py: import KnowledgeChunk from
knowledge_base.py instead of re-defining the same dataclass inline.
Both RAG modules now share one type.
- src/pipeline.py: stop reaching into src.recommendation_agent's
private _ACTIVE_REGION_CONFIG module global; read REGION_CONFIG[REGION]
from the public surface. Drops the leaky-abstraction warning from
the quality review.
- src/pipeline.py: self._dfl_actions was instance state that was
rebuilt every run() and only read once per optimize step. Replaced
with a local in _step_optimize, one fewer instance field to reason
about.
- src/policy/features.py: build_dp_features called _history_prices(history)
twice when spot<=0 (once for the spot fallback, once for rolling
stats). Cache the normalized history once at the top. Trivial, but
this is in the hot training loop (20,330 call…
Replaces the two-stage (forecast -> argmax-revenue) timing ceiling
with a LightGBM classifier trained directly on oracle actions,
weighted by regret spread so high-stakes DPs dominate learning.
The paper's §5.1 diagnosed this ceiling on Indian pulses; DFL
closes it and beats the Chronos+gate baseline on both panels in
leave-one-event-out CV.
- src/policy/features.py: build_dp_features(dp, history, forecast,
exogenous) -> 24-feature flat dict (6 rolling, 4 seasonal, 9
forecast quantiles, 3 exogenous, 2 identity). Stable FEATURE_NAMES
tuple shared between training + inference. No NaN ever produced.
- src/policy/train_dfl.py: CLI trainer. Loads kenya_maize_daily_v0_1
+ india_pulses_v0_3 from the benchmark repo via sys.path hack
(training-time only), computes oracle action via
lastmile_bench.benchmarks.market_intelligence.decision.all_net_revenues,
weights examples by regret spread normalized per panel, runs
LOO-CV per event, fits final booster on full data. Saves
dfl_v1…
Makes the recommendation agent region-parameterized and renames the
India-specific output field recommendation_ta -> recommendation_local
(+ sibling local_language_code: 'ta' | 'sw').
- REGION_CONFIG dict keyed by region with 10 fields (region_name,
currency_symbol, market_type, local_language_name/code,
phone_country_code, local_greeting_example, typical_crops, etc.)
- SYSTEM_PROMPT + TRANSLATION_PROMPT parameterize via .format()
- Claude models unchanged: Sonnet for reasoning, Haiku for translation
- FarmerRecommendation dataclass: recommendation_ta -> recommendation_local
- Rule-based fallback is region-aware (currency, market terminology)
- Delivery SMS: phone country code + language-tagged local text
(+254..., 'SW:' prefix under Kenya; +91..., 'TA:' under India)
- src/api.py demo endpoint emits the new fields
- src/db.py: DB-schema-gap comment at SellRecommendation persistence
(local text lives in full_data JSONB; no dedicated column change)
Co-Authored-By: Claude Opus 4…
Phase 1 foundation: Kenya migration via REGION toggle
Adds Kenya as a sibling region to the existing Tamil Nadu / India
pipeline behind a MARKET_INTEL_REGION env toggle (default: kenya).
- config.py loads markets_kenya.json / commodities_kenya.json /
farmers_kenya.json when REGION=kenya, preserving all India behavior
under REGION=india
- 10 KAMIS wholesale markets (Kibuye, Wakulima Nairobi, Kongowea,
Eldoret, Nakuru Wakulima, Kitengela, Kangemi, Karatina, Kathonzweni,
Kaanwa) with coordinates and commodity routing
- 4 commodities: Dry Maize (KAMIS product_id 1), Beans (30), Red Irish
Potato (57), Green Grams (10). Maize seasonal indices derived from
real 2021-2025 KAMIS cache; others from domain research
- 100 Kenyan farmer personas (mixed Swahili/Kikuyu/Luo/Luhya/Kalenjin
names), deterministic seed=42
- src/ingestion/kamis.py: live scrape + demo snapshot path, ports the
benchmark reference parser (expired-cert handling, 3s polite delay,
retry backoff). Plugs into pipeline._step_ingest as a sibling to
fetch_mandi_pric…
CLAUDE.md and REBUILD.md claimed config.py loaded from markets.json,
commodities.json, and farmers.json, but it was still 400 lines of inline
Python. A forker editing the JSON saw zero effect. Now the claim is true.
- config.py: load MANDIS, COMMODITIES, SEASONAL_INDICES, POST_HARVEST_LOSS,
BASE_PRICES_RS, _CURATED_FARMERS from JSON. Every existing
"from config import" symbol preserved; default Tamil Nadu state is
byte-identical to pre-rewrite (15 mandis, 10 commodities, 100 farmers).
- src/ingestion/base.py: drop dead PRICE_SOURCES registration instruction
(pipeline.py never read that variable); document actual wiring
- CLAUDE.md: tour step count 12 to 9; clarify /health vs /api/health
- REBUILD.md + frontend/src/pages/Pipeline.tsx: add frontend/api/*.ts to
the region-coupled list (previously contradicted CLAUDE.md's geography
table, forkers were told not to touch files holding static mandi name
lookups)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.c…
Flip ingestion default to live mode; demo is now opt-in
Agmarknet and eNAM now default to real API calls. Demo-mode (seed=42
synthetic prices + simulated eNAM conflicts) is only used when
MARKET_INTEL_DEMO_MODE is explicitly set — previously the default was
demo and real mode was opt-in via MARKET_INTEL_USE_REAL_API. In live
mode eNAM returns empty (no real scraper yet), so reconciliation falls
back to the Agmarknet-only path. Pipeline step details now include
data_source_mode so the frontend can show which tier ran.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clean up DPI commit: redundant state, dead field, stringly typing
Follow-up to 8a1097a addressing review findings:
src/dpi/models.py:
- Remove KCCRecord.interest_rate_pct — the field was set in the
simulator but never read anywhere in the codebase. Dead state.
- Add KCCRepaymentStatus = Literal["current", "overdue", "defaulted"]
type alias and use it on KCCRecord.repayment_status to catch
typos at type-check time.
src/dpi/simulator.py:
- Move config + src.geo imports from inside _generate_all to the
module top. There's no circular dependency — config never imports
from src.dpi — so the defensive late-binding was cargo cult.
- Drop the interest_rate_pct assignment in _make_kcc now that the
field is gone.
src/optimizer.py:
- CreditReadiness.kcc_headroom_rs is now a @property computed as
max(0, limit - outstanding) instead of a separately-stored field.
Prior shape created a sync risk where the three KCC fields could
drift out of agreement.
- Extract KCC utilization thresholds to named module-level constants
(KCC_UTILIZATION_HIGH_P…
Add DPI subsystem (Aadhaar + Land Records + KCC) and wire into credit readiness
Ports a focused three-service Digital Public Infrastructure layer from
Weather AI 2's pattern, adapted for Market Intelligence's credit
readiness product. Scope is deliberately narrower than WA2's six-service
aggregation — MI only needs the services that actually affect a
sell-timing + credit recommendation:
- AadhaarProfile -- identity + district + Tamil-script name
- LandRecord -- acreage, soil, irrigation, registered crops,
nearest mandi (computed at load time via haversine)
- KCCRecord -- real credit limit, outstanding, repayment status,
subsidized interest rate
Skipped on purpose: Soil Health Card (WA2 uses it for weather advisories;
MI doesn't need it), PM-KISAN (income support; not relevant to market
timing), PMFBY (crop insurance; MI isn't an insurance product).
src/dpi/models.py:
- Dataclasses with derived properties: FarmerProfile.total_area,
primary_crops, nearest_mandis, grows_commodity(), repayment_o…
Brings Market Intelligence to parity with Weather AI 2 on both
post-pipeline data validation and ML quality evaluation. Before this
commit, MI had no quality checks and no eval scripts — every other
project in the portfolio has both.
src/quality_checks.py (new, ~170 lines):
- Ported from weather AI 2's pattern, adapted for SQLAlchemy sessions
- check_row_count, check_null_rate, check_value_range, check_freshness
- Table/column identifier whitelist prevents SQL injection via f-string
- run_all_checks() is a no-op when DATABASE_URL is unset — safe to call
from any environment
- Runs 12 checks total: row counts on 4 tables, null rates on 3 price
columns, physical bounds on prices and forecast horizons, freshness
on pipeline_runs
src/pipeline.py:
- Wire run_all_checks() into MarketIntelligencePipeline.run() after
finalization. Results land in PipelineRunResult.quality_checks_passed
and .quality_checks_total; failures are logged as warnings but never
block the run — the dashbo…
Scale farmer registry to 100, Haiku for Tamil translation, featured-only recommend loop
Grow SAMPLE_FARMERS from 3 hand-written personas to a 100-farmer
pilot spread across all 15 Tamil Nadu mandis with diverse
commodity/quantity/storage mixes. Generation is deterministic
(seed=42) so the same 100 farmers appear on every restart.
The weekly pipeline only precomputes recommendations for
FEATURED_FARMERS (the original 3) — the other 97 live in the
registry for on-demand computation and to show the pilot scale
in the UI. This keeps the weekly Claude spend pinned to the
demo surface while the capacity story scales to 100.
RECOMMEND agent changes:
- Translation call switched from Sonnet to Haiku 4.5
(short structured translation task, ~3x cheaper, no quality
risk for this use case)
- Main reasoning call stays on Sonnet — the multi-round tool-use
chain benefits from Sonnet's tool capability, and a live A/B
test against Haiku could not be run in this session because
no valid ANTHROPIC_API_KEY was accessible in the sandbox.
Deferred as a follow-up before switching t…
Replace print logging with structured logger; update Vercel custom domain
Convert all print(..., flush=True) calls in the pipeline scheduler and
retrain endpoint to logger.info/error so step status, pipeline results,
and failure traces flow through the standard logging config instead of
raw stdout. Update CLAUDE.md to the new crop-pricing.jeff-levine.com
custom domain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use get_session() instead of manual sessionmaker (match existing db pattern)
- Remove unused DeliveryChannel enum and Recipient dataclass
- Create Twilio client once before loop, not per message
- Skip duplicate format_sms when language is English
- Remove redundant status/channel reassignment in dry-run path
- Add index on delivery_logs.created_at
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Step 7: SMS delivery for sell recommendations
- sell_recommendations.full_data: complete rec dict with all options,
costs, credit readiness
- market_prices.full_data: complete price dict with source breakdown
- pipeline_runs.price_conflicts: full conflict investigation data
- Vercel endpoints use full_data when available, fall back to columns
Next pipeline run will populate the JSONB columns. Dashboard will then
show full cost breakdowns and investigation steps from Neon.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix stale comment: bolt-base → bolt-tiny in model selection docs
- scripts/retrain_mos.py: pulls prediction-vs-actual pairs from Neon,
builds training features, retrains XGBoost MOS correction models
- POST /api/pipeline/retrain-mos endpoint for triggering from GH Action
- Weekly GH Action retrains on 4th Monday of each month (or manual)
- Model improves over time as more real Agmarknet data accumulates
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix iloc/loc mismatch causing Chronos MOS prediction failure
iterrows() yields index labels but iloc expects positional indices.
When the DataFrame index is non-sequential (after filtering), iloc[[i]]
goes out of bounds. Changed to loc[[i]] in all prediction paths.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Widen market_prices.source column to VARCHAR(100)
- /api/pipeline/status endpoint with step-level progress
- HTML status page now shows pipeline steps with live polling
- Green dots for completed, amber pulse for active, gray for pending
- Run Pipeline button with auto-refresh
- Pipeline progress callback from each step to scheduler
Also: chronos-bolt-tiny, 60s timeout, print() logging for HF logs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch to chronos-bolt-tiny, add visible pipeline logging, 60s timeout
- bolt-tiny (8MB) loads in seconds on CPU, bolt-base (700MB) was hanging
- Add print() logging in scheduler so pipeline progress shows in HF logs
- Reduce Chronos load timeout from 300s to 60s for faster fallback
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wrap database calls in try/except so pipeline never hangs on Neon
init_db() and save_pipeline_run() now fail gracefully if Neon is
sleeping or connection is broken. Pipeline completes and serves
data via API regardless of database state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents pipeline from hanging when Neon PostgreSQL connection is
stale. Adds 10s connect_timeout, pool_timeout, and pool_recycle=300s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move model pre-download AFTER pip install so huggingface_hub is
available. Restores bolt-base (not tiny). The Docker build downloads
the 700MB model once; subsequent pipeline runs load from disk.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch to chronos-bolt-tiny (8MB) for fast CPU loading
chronos-bolt-base (700MB) was hanging on download during pipeline runs.
bolt-tiny is 8MB, loads in seconds, still provides neural foundation
model forecasts with XGBoost MOS correction on top.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents pipeline from hanging indefinitely when downloading the
Chronos-2 model on a cold HF Space start. Falls back to XGBoost
if load exceeds 300s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete Claude agents, Chronos-2 forecasting, and frontend polish
Extract market, commodity, and farmer data to JSON files so the project
can be forked for any region. Add PriceSource protocol in
src/ingestion/base.py, frontend regionConfig.ts, and REBUILD.md with
step-by-step adaptation guide and Claude Code prompt.
Follows the same pattern as Weather AI 2's region adaptation system.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align pipeline market_prices output with frontend interface
Add reconciled_price_rs, agmarknet_price_rs, enam_price_rs, category,
price_trend, and date fields to pipeline output so it matches the shape
the frontend expects. Also add mandi_name to forecast output and
back-fill price_trend from forecast direction data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
P0: SPA catch-all returns 404 not None, recommendation_to_dict includes
farmer_id/name/tamil, demo sell options include drive_time_min/storage_cost.
P1: Pipeline forecasts use current_price_rs (matching frontend), model
metrics flattened (r2 not nested under metrics), seasonal_index added to
forecasts, StatusBadge handles active/partial, Forecast chart uses real
CI values not hardcoded scaling, division-by-zero guard added.
P2: Package name health→market-intelligence, health-domain artifacts in
docstrings/comments cleaned, Google Maps reference removed (uses haversine),
deprecated datetime.utcnow→now(timezone.utc), confidence clamped to 0.10+,
dist_bak deleted.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace health supply chain optimizer with market intelligence agent
Complete rewrite from health domain to post-harvest market intelligence
for Tamil Nadu smallholder farmers. Same 6-step pipeline architecture
(ingest → extract → reconcile → forecast → optimize → recommend),
new domain: conflicting mandi price data → reconciled prices →
sell-now-or-wait recommendations with credit readiness assessment.
Backend: 15 real Tamil Nadu mandis, 10 commodities, real Agmarknet API
(data.gov.in), eNAM scraper, NASA POWER weather, XGBoost price forecaster,
Claude agents with rule-based fallbacks, FAISS+BM25 RAG.
Frontend: 5 pages (Markets, Forecast, Sell Advisor, Pipeline, Data),
amber/saffron theme, Tamil Nadu map, Joyride tour, credit readiness UI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix Chronos integration bugs and efficiency issues from /simplify
- Fix silent bug: column names _facility_id/_drug_id (underscore prefix)
in training data — Chronos predictions were silently empty
- Defer `import torch` from module level to _ensure_loaded/predict_batch
— avoids 1-3s PyTorch load on every process start
- Add torch.no_grad() context for inference
- Use module-level singleton for ChronosBoltForecaster in pipeline
— avoids reloading 9M-param model on every pipeline run
- Remove hardcoded Chronos metadata from api.py, use class directly
- Fix Dockerfile: mkdir models/ instead of COPY (models are gitignored)
- Update tests for underscore-prefixed column names + add plain-column test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add Amazon Chronos-Bolt neural foundation model for demand forecasting
Integrates Chronos-Bolt-Tiny (9M params, T5 transformer pre-trained on
100B+ time series) as a zero-shot ensemble partner with XGBoost. Point
forecasts use 65/35 weighted average; prediction intervals take the
wider bounds for conservative coverage.
Key additions:
- src/forecasting/chronos_model.py: ChronosBoltForecaster wrapper +
ensemble_predictions + build_series_from_training_data
- Pipeline Step 4: Chronos runs as Layer 2.5 between XGBoost and
residual correction, ensembles into forecast dicts
- API: /api/model-info now surfaces chronos_bolt in ml_stack
- Frontend: Chronos hero card in Demand "How It Works" tab
- 10 new tests (all passing, 54 total)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Shows facilities, medicines tracked, stockout risks, data reliability,
active stockout risk table, and API endpoint directory — matching the
Climate Risk Engine's status page pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Simplify: fix bugs, remove duplication, use public APIs
- Remove dead `or True` filter in anomaly detector score()
- Remove dead REPORTING_QUALITY_ENC from detector.py
- Add public metrics/feature_importances properties to XGBoostDemandModel
- Add predict_batch() for vectorized predictions
- Fix residual model to use public predict_batch() instead of _model internals
- Reuse training_df in pipeline instead of calling build_training_data twice
- Add exception handling for residual model training failures
- Replace per-item anomaly scoring loop with batch score_batch()
- Use real facility/drug data for anomaly features instead of hardcoded values
- Gate init_db() with flag so DDL only runs once
- Replace duplicate LEAD_TIME_MAP with import from config.LEAD_TIMES
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Expand XGBoost demand model from 13 to 20 features (RMSE 5.8→3.3, R²=0.998)
- Add residual correction model (MOS pattern: model-correcting-model)
- Add isolation forest anomaly detection on consumption patterns
- Upgrade RAG from TF-IDF to sentence-transformers (all-MiniLM-L6-v2)
- Expand knowledge base from 33 to 101 chunks across 15 categories
- Add Neon PostgreSQL persistence layer (5 tables)
- Add evaluation suite (44 pytest tests)
- Wire new components into pipeline and API
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Unify 4 endpoints to use _get() helper instead of inline store/demo branching
- Add _source() helper replacing 9 duplicated ternary expressions
- Rename store.procurement_plan -> procurement_plans (list) to match demo data key
- Add thread-safe double-checked locking on demo cache
- Remove setdefault() calls that mutated shared cache on every request
- _get() now accepts default param for dict-typed fields
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add README.md with HF Space YAML frontmatter (docker SDK, port 7860)
- Add .dockerignore for cleaner builds
- Update Dockerfile: non-root user, healthcheck, full requirements
- Add requirements-dashboard.txt (lightweight, unused for now)
- Fix _demo -> _get_demo() bug in procurement/model/pipeline endpoints
- Restrict CORS to specific HF Space and Vercel origins
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full-stack AI pipeline for health commodity forecasting and procurement optimization.
Backend: FastAPI + Claude agents for extraction, healing, reconciliation, demand forecasting, and procurement.
Frontend: React + Vite + Tailwind dashboard with guided tour.
CI: GitHub Actions for backend/frontend validation + weekly pipeline trigger.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
