Genesis AI — Complete Data Flow

1 Master Flow Diagram

flowchart TD A([User opens Dashboard\nbrowser · localhost:5173]) --> B B{Upload server\nrunning?} B -- YES --> C1[GET /api/data/key x16\nFastAPI reads output/*.json] B -- NO --> C2[GET /data/key.json x16\nVite serves public/data/ static files] C1 --> D([React tabs render\nwith computed data]) C2 --> D style A fill:#1a6b2f,color:#fff,rx:20 style D fill:#1a6b2f,color:#fff,rx:20 style B fill:#fff3e0,color:#333

2 Upload Button → Python Pipeline → Dashboard Refresh

flowchart TD U1([User clicks Upload button\nHeader.jsx]) -->|opens| U2[UploadModal.jsx] U2 -->|POST /api/upload/reset| U3[FastAPI resets job state → idle] U3 --> U4[User drags or picks .xlsx file] U4 -->|POST /api/upload\nmultipart form-data| U5 U5[upload_server.py\n@app.post /api/upload] --> U6[Save file to\n_uploads/ folder] U6 --> U7[Spawn background thread\n_run_pipeline] U7 -->|returns immediately| U8[HTTP 200 · ok: true] U8 --> U9[UploadModal polls every 1.5s\nGET /api/upload/status\nGET /api/upload/logs] U7 -->|subprocess.Popen| P1 P1[pipeline/run.py\nEntry Point] --> P2 P2[pipeline/loader.py\nload_source_data\nread_brand_config] --> P3 P3[pipeline/config.py\nDEFAULT_PACK_ORDER\nconstants only] --> P4 P4[pipeline/modelling.py\nbuild_wide_pivot\nrun_elasticity_models\n343 specs tested] --> P5 P5[pipeline/proxies.py\nassign_proxies 7 grains\ncompute_freq_anchors] --> P6 P6[pipeline/diagnostics.py\nrun_diagnostics\nStep1_Diagnostics.xlsx] --> P7 P7[pipeline/exporters/stats.py] --> J1[models.json\nstats.json\nfreq_anchors.json] P7 --> EX2 EX2[pipeline/exporters/market.py] --> J2[trend.json · ms.json\nvol_salience.json · val_share.json\ncomp_ms.json · vtm.json] EX2 --> EX3 EX3[pipeline/exporters/ppa.py\nreads sheet 3_PPA_MT\nreads sheet 4_PPA_TT] --> J3[ppa_mt.json\nppa_tt.json] EX3 --> EX4 EX4[pipeline/exporters/analytics.py] --> J4[interaction.json\ngrowth_decomp.json\npgi.json] EX4 --> EX5 EX5[pipeline/exporters/recommendations.py] --> J5[recs_full.json] J1 & J2 & J3 & J4 & J5 --> OUT[16 JSON files\nin output/] OUT --> CP[Copied to\npublic/data/ as fallback] OUT --> ST[Job status = done] ST --> PL[UploadModal sees done\nUser clicks Reload Dashboard] PL -->|GET /api/data/key x16| SV[upload_server.py\n@app.get /api/data/key\nreads output/key.json] SV --> RD([React re-renders\nall 9 tabs with fresh data]) style U1 fill:#1a6b2f,color:#fff style RD fill:#1a6b2f,color:#fff style P1 fill:#1565c0,color:#fff style OUT fill:#6a1b9a,color:#fff

3 Step-by-Step Reference

Step	Where	File / Endpoint	What happens	Layer
1	Browser	`Header.jsx`	User clicks ↑ Upload button — UploadModal opens	Frontend
2	Browser	`UploadModal.jsx`	POST /api/upload/reset — clears previous job state on server	API
3	Browser	`UploadModal.jsx`	POST /api/upload — sends .xlsx file as multipart form-data	API
4	Server	`upload_server.py`	Saves file to _uploads/, spawns background thread, returns HTTP 200 immediately	Python
5	Server	`upload_server.py`	subprocess.Popen → runs python -m pipeline.run with the uploaded file	Python
6	Pipeline	`pipeline/loader.py`	Reads sheet 1_Source_Data → 3,207 rows DataFrame. Reads sheet 2_Brand_Config → focal brand + competitors	Python
7	Pipeline	`pipeline/config.py`	Loads DEFAULT_PACK_ORDER and all modelling thresholds (no I/O)	Python
8	Pipeline	`pipeline/modelling.py`	build_wide_pivot() builds monthly CH×Region tables. run_elasticity_models() tests 343 OLS specs per grain, selects best Adj-R²	Python
9	Pipeline	`pipeline/proxies.py`	assign_proxies() fills 7 grains that failed sign check. compute_freq_anchors() finds dominant pack per CH×Region	Python
10	Pipeline	`pipeline/diagnostics.py`	Runs collinearity, RPI, distribution, cannibalization checks → writes Step1_Diagnostics.xlsx	Python
11	Exporters	`exporters/stats.py`	Writes models.json, stats.json, freq_anchors.json to output/	Output
12	Exporters	`exporters/market.py`	Writes trend.json, ms.json, vol_salience.json, val_share.json, comp_ms.json, vtm.json	Output
13	Exporters	`exporters/ppa.py`	Reads Excel sheets 3_PPA_MT and 4_PPA_TT → writes ppa_mt.json, ppa_tt.json	Output
14	Exporters	`exporters/analytics.py`	Writes interaction.json, growth_decomp.json, pgi.json	Output
15	Exporters	`exporters/recommendations.py`	Writes recs_full.json	Output
16	Server	`upload_server.py`	Copies all 16 JSON files from output/ to public/data/ as static fallback. Sets job status = done	Python
17	Browser	`UploadModal.jsx`	Polling detects status = done. User clicks Reload Dashboard	Frontend
18	Browser	`DataContext.jsx`	Fires GET /api/data/{key} × 16 calls to fetch all computed JSON data	API
19	Server	`upload_server.py`	@app.get("/api/data/{key}") reads output/{key}.json → returns JSON to browser	Python
20	Browser	All 9 tab components	React re-renders every tab with fresh pipeline-computed data	Frontend

4 All API Endpoints

Method	Endpoint	Called by	Purpose
POST	`/api/upload`	UploadModal.jsx	Upload .xlsx file → triggers Python pipeline
GET	`/api/upload/status`	UploadModal.jsx (poll)	Returns { status, message, elapsed_s }
GET	`/api/upload/logs`	UploadModal.jsx (poll)	Returns live pipeline stdout as { logs }
POST	`/api/upload/reset`	UploadModal.jsx (on open)	Clears job state back to idle
GET	`/api/data/{key}`	DataContext.jsx	Serves output/{key}.json computed by pipeline
GET	`/api/data`	(debug)	Lists all available data keys and file sizes
GET	`/data/{key}.json`	DataContext.jsx (fallback)	Vite static file — used only when server is down

5 Python Output Files → Dashboard Tabs

models.json

exporters/stats.py → Elasticity Results tab

stats.json

exporters/stats.py → Header KPIs

freq_anchors.json

exporters/stats.py → Methodology tab

trend.json

exporters/market.py → Trends tab

ms.json

exporters/market.py → shared

vol_salience.json

exporters/market.py → shared

val_share.json

exporters/market.py → shared

comp_ms.json

exporters/market.py → shared

vtm.json

exporters/market.py → Brand Interaction tab

ppa_mt.json

exporters/ppa.py → PPA tab (MT)

ppa_tt.json

exporters/ppa.py → PPA tab (TT)

interaction.json

exporters/analytics.py → Brand Interaction tab

growth_decomp.json

exporters/analytics.py → Growth Decomp tab

pgi.json

exporters/analytics.py → Price Gradient tab

recs_full.json

exporters/recommendations.py → Recommendations tab

recs.json

alias of recs_full.json

Genesis AI — Price Elasticity Intelligence