docs/DATA_UPGRADE_PLAN.md

Data Upgrade — Working Plan

#1 goal: the best-quality ThoughtSpot demo possible, built on Demo-to-Win principles (a compelling persona story + discoverable outliers, on realistic, believable data). Quality lives in the fact table (~10k varied rows), not dimension size. Fix the generation system for any prospect; fix the grader where it's misaligned — never pad data to satisfy a bad metric.

---

1. Data Quality (mostly landed — parked)

Systemic realism primitives added to the engine (all committed on develop_dg, tested):

• share_of — a subset/part measure is a fraction of its parent, so part ≤ whole and any rate derived as part/whole is ≤ 1.0 by construction (kills impossible values: won>submitted, bounced>sessions, rates>1).
• per — a measure is a stable per-entity characteristic (drawn once per dimension member), so a product keeps a consistent unit price/cost → believable, non-random margins.
• cost share_of price guidance so gross margins stay realistic (no 85% from a $21 COGS).
• Backed off forced dimension cardinality — no 20–40 mandate; use as many real values as a dimension naturally has, no padding.

Open / later (only if it makes the demo better, not to chase a score):

• Dimensional depth: attribute columns + hierarchies (Category→Product, Region→Store) — ThoughtSpot shows these off well. Keep complexity bounded (1–2 facts, ~5–7 dims).
• Richer time hierarchy + more derived measures, toward the golden demo (Vizio).

2. Data Grader Changes (recommended — not yet applied)

Grader lives in tests/e2e_quality.py (LLM rubric ~line 972). Recommended edits so it measures real Demo-to-Win quality:

• COMPLETENESS (15 pts): drop "dimensions have 20+ distinct members." It penalizes realistic small dims (a company has ~6 channels). Reward a well-populated fact table with variation + realistic dimension cardinality; keep the 0-rows=0 rule and the fake-name ("Product 42") penalty.
• TIME COVERAGE: feed the grader today's date so it stops penalizing current-year (2026) data as "future-dated" — that's the grading LLM's training-cutoff bias, not a flaw.
• Keep STORY POTENTIAL (30) and REALISM (20) — they are the Demo-to-Win core and already the right weight. Optional later: reward persona-relevant, discoverable insights explicitly.

3. App Reliability ← ACTIVE FOCUS

Run 873188da (8 tests on e769a68): 7 F's, 3 distinct root causes. None are data-design problems — they're the pipeline failing to complete.

Cause	Tests hit	Root cause	Fix
A. Authoring: "<2 dimensions" after repair	PNC, Deloitte, FedEx (3)	Blueprint JSON likely truncating at max_tokens=8192 (heavy prompt + old 20–40 value lists) → partial parse.	Softened prompt (94a6dd5, done) reduces output; raise max_tokens; confirm by local repro.
B. ThoughtSpot auth failed (code 10002)	Nike, General Mills, Wells Fargo (3)	The sebe env's trusted-auth key on the Space is bad/missing (surfaced by the se→sebe test switch). Data deployed to Snowflake fine; only TS auth failed.	Revert e2e to se, or fix the sebe key in the Space's HF secrets.
C. share_of validation abort	J.B. Hunt (1)	My share_of validation requires the parent be a base measure; the LLM made GROSS_FREIGHT_REVENUE derived → hard abort. Too strict.	Allow share_of to reference any measure (base or derived); evaluate parents before children.

Note the good news buried in the failures: where authoring + auth succeeded, the data generated correctly (Nike PRODUCT=30/STORE=23, GM/WF facts 10k rows) — the data work is sound; the pipeline just isn't finishing.