| --- |
| title: SGO — Semantic Gradient Optimization |
| emoji: 📊 |
| colorFrom: indigo |
| colorTo: purple |
| sdk: docker |
| app_port: 7860 |
| --- |
| |
| # SGO — Semantic Gradient Optimization |
|
|
| You're launching a product. You think the landing page is good. But **who have you actually asked?** |
|
|
| You could run a survey — but that takes weeks and you'd need to find the right people. You could ask an LLM — but one LLM opinion isn't a market. You could A/B test — but you need traffic first, and you don't know *what* to test. |
|
|
| **SGO lets you ask 50 realistic people what they think — in 3 minutes, for $0.10.** |
|
|
| It builds a representative panel from census-grounded synthetic personas, has each one score your thing from their perspective, then asks *"what would change your mind?"* — producing a priority-ranked list of what to fix first. |
|
|
| ``` |
| You: "Here's my landing page. Here's my target market." |
| |
| SGO: "47 evaluators scored you. Avg 5.3/10. |
| Solo devs love it (7.2). Enterprise is blocked (3.1). |
| #1 concern: no SOC2. #2: no free tier. |
| |
| Gradient: |
| +2.1 Add self-hosted option |
| +1.8 Add free tier ← biggest universal win |
| +1.4 Get SOC2 certified |
| +0.6 Drop price ← not actually the blocker" |
| ``` |
|
|
| --- |
|
|
| ## What Can You Use It For? |
|
|
| Anything someone else evaluates. |
|
|
| | What you're optimizing | Who evaluates it | What you learn | |
| |----------------------|-----------------|---------------| |
| | **Product** — landing page, pricing | Buyer personas by company size, role, budget | Which segments convert, which are blocked, and why | |
| | **Resume** — CV + cover letter | Hiring managers at startups vs. enterprises | What stands out, what's a red flag, what to lead with | |
| | **Pitch** — investor deck | VCs and angels at different stages | Whether the story lands, what questions they'd ask | |
| | **Policy** — proposed regulation | Stakeholders by role, income, geography | Who supports it, who opposes, what compromise works | |
| | **Content** — blog post, video | Readers at different expertise levels | Whether it hits the right level, what's confusing | |
| | **Profile** — professional bio, personal brand | Population sample by age, education, occupation | How different demographics perceive you | |
|
|
| SGO ships with a 1M-person census-grounded dataset ([Nemotron-Personas-USA](https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA)) with structured demographics (age, sex, education, occupation, marital status, US geography) plus rich narrative fields — professional persona, skills and expertise, career goals, hobbies, cultural background, and personality. The narratives naturally encode things like seniority, industry, technical depth, and decision-making style, even though those aren't separate columns. |
|
|
| This means most domains work out of the box — the LLM evaluates from the persona's full context, not just the demographic fields. For highly specialized panels (e.g., Series B VCs, enterprise procurement officers), SGO can generate personas via LLM with explicit stratification constraints. See [limitations](#limitations) on generated vs. census-grounded panels. |
|
|
| In each case, SGO tells you **where you stand**, **what's working**, **what's not**, and **what specific change would help the most** — broken down by audience segment. |
|
|
| --- |
|
|
| ## Quick Start |
|
|
| ```bash |
| git clone https://github.com/xuy/sgo.git && cd sgo |
| cp .env.example .env # Add your LLM API key (any OpenAI-compatible provider) |
| uv sync |
| uv run --extra web python web/app.py |
| # Opens at http://localhost:8000 |
| ``` |
|
|
| The web interface walks you through the full pipeline: describe your entity, build a panel, evaluate, find the highest-impact changes, and audit your panel for cognitive biases. |
|
|
| <details> |
| <summary>Alternative: use as a Claude Code skill</summary> |
|
|
| ```bash |
| git clone https://github.com/xuy/sgo.git ~/.claude/skills/sgo |
| cd ~/.claude/skills/sgo && cp .env.example .env && uv sync |
| ``` |
|
|
| Then run: |
|
|
| ``` |
| /sgo # Interactive — it asks what you're optimizing |
| /sgo entities/my_product.md # Start with an existing entity |
| /sgo "optimize my landing page" # Start from a description |
| ``` |
|
|
| </details> |
|
|
| <details> |
| <summary>CLI-only usage (no web interface)</summary> |
|
|
| ```bash |
| uv run python scripts/setup_data.py # Download Nemotron personas (once, ~2GB) |
| # Then use scripts directly: evaluate.py, counterfactual.py, bias_audit.py, compare.py |
| # See AGENT.md for the full pipeline reference |
| ``` |
|
|
| </details> |
|
|
| --- |
|
|
| ## How It Works |
|
|
| You describe what you're optimizing and what your goal is. SGO builds a diverse panel, has each one react, then focuses on the **persuadable middle** — the people who are *almost* convinced — to find what would tip them toward your goal. |
|
|
| SGO does **not** try to please everyone. People who scored 1–3 are not your audience — their feedback is informational, not actionable. The system focuses on moving the people who are close to yes. |
|
|
| **Five steps:** |
|
|
| 1. **Describe your entity and goal** — what an evaluator would see, and what outcome you're optimizing for |
| 2. **Build a panel** — 30–80 evaluators, stratified to cover the segments that matter |
| 3. **Evaluate** — each evaluator scores 1–10. Results are segmented: champions (8+), persuadable (4–7), not-for-them (1–3) |
| 4. **Find directions for your goal** — the persuadable middle re-evaluates hypothetical changes. With a goal, evaluators are weighted by relevance (VJP) |
| 5. **Act and re-run** — make the top change, re-evaluate against the same panel, track improvement over time |
|
|
| The key insight is step 4. The probe produces a ranked list of changes sorted by how much they'd move the persuadable middle toward your goal. SGO calls this the **semantic gradient** — technically a vector-Jacobian product when a goal is specified. |
|
|
| <details> |
| <summary>Example: what the gradient looks like</summary> |
|
|
| Each row is an evaluator. Each column is a hypothetical change. Each cell is the score delta. |
|
|
| | | Add free tier | Get SOC2 | Self-hosted | Open-core | Case studies | |
| |---|:---:|:---:|:---:|:---:|:---:| |
| | Solo dev | +2 | +1 | 0 | +1 | +3 | |
| | Startup EM | +1 | +3 | -1 | +2 | +4 | |
| | Enterprise CTO | 0 | +1 | +2 | +1 | +2 | |
| | Data analyst | +1 | +2 | 0 | 0 | +3 | |
| | **Average** | **+1.0** | **+1.8** | **+0.3** | **+1.0** | **+3.0** | |
|
|
| The column averages tell you what to fix first. "Case studies" has the highest average impact. "Self-hosted" helps enterprise but slightly hurts startups — a tradeoff, not a pure win. |
|
|
| </details> |
|
|
| ### What makes the panel realistic? |
|
|
| SGO uses [NVIDIA Nemotron-Personas-USA](https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA) — 1 million synthetic Americans whose demographics match real US census distributions. Each persona includes detailed narratives: professional background, skills, career goals, hobbies, cultural background, and personality. |
|
|
| This matters because when you ask an LLM to "generate 50 diverse personas," you get 5–6 archetypes with surface variation — mostly coastal, college-educated, and tech-adjacent. You can't audit what's missing. Census-grounded personas give you the construction worker in suburban Illinois and the quilter in rural Texas, because census data says those people exist. |
|
|
| The principle: **define the population before the measurement, not after.** |
|
|
| ### From general population to any domain |
|
|
| Nemotron covers age, sex, education, occupation, geography, and marital status as structured fields — plus rich narratives about each person's career, skills, values, and lifestyle. That's enough to directly evaluate anything consumer-facing: products, profiles, content, policy. |
|
|
| But what about domains the dataset doesn't explicitly cover — like "enterprise CTOs" or "Series B investors"? There are four ways to get there, from most grounded to most flexible: |
|
|
| **1. Filter by what's already there.** A Nemotron persona with `occupation: software_developer`, `education: graduate`, `age: 38` and a professional narrative describing team leadership *is* a plausible engineering manager evaluating your developer tool. You just filter and let the narrative do the work. |
|
|
| **2. Reframe the evaluation prompt.** Same persona, different lens. Instead of *"would you buy this?"*, ask *"you're evaluating this tool for your team — would you champion it internally?"* The persona's professional context, skills, and decision-making style naturally shape the answer. |
|
|
| **3. Enrich with a situational overlay.** Add context that the persona doesn't have: *"You are [full Nemotron persona]. You work at a 50-person Series A startup. Your team's tooling budget is $2k/month. You've been burned by vendor lock-in before."* The demographic grounding stays real; the professional situation is augmented. |
|
|
| **4. Generate from scratch, using Nemotron as a quality bar.** For truly specialized roles (VC partners, procurement officers, regulatory lawyers), generate personas via LLM — but use Nemotron personas as few-shot examples so the output matches the depth and internal consistency of the dataset. SGO's `generate_cohort.py` does this with an explicit warning about the quality tradeoff. |
|
|
| Each step trades some census grounding for more domain specificity. For most use cases, steps 1–2 are enough. |
|
|
| --- |
|
|
| ## Worked Example |
|
|
| <details> |
| <summary>SaaS product launch — full walkthrough</summary> |
|
|
| ### Setup |
|
|
| A seed-stage startup launching "Acme API," a managed data pipeline tool. The landing page says: 200+ connectors, pay-as-you-go at $0.01/sync, SOC2 pending, $99/mo starter, 3-person team. |
|
|
| ### Panel |
|
|
| 40 buyer personas stratified by company size (solo → enterprise), role (IC engineer → CTO → data analyst), budget, and tech stack. |
|
|
| ### Results |
|
|
| ``` |
| Solo devs: avg 7.2 ← love it |
| Startups: avg 5.8 ← cautious |
| Enterprise: avg 3.1 ← blocked |
| Non-technical: avg 4.5 ← confused |
| ``` |
|
|
| ### Gradient |
|
|
| ``` |
| Rank avg Δ Change |
| 1 +2.1 Add self-hosted / VPC option |
| 2 +1.8 Add free tier (1,000 syncs/mo) |
| 3 +1.4 SOC2 certified (not pending) |
| 4 +1.2 Open-core positioning |
| 5 +1.0 Add 3 named customer case studies |
| 6 +0.6 Drop price to $49/mo |
| ``` |
|
|
| **Insight**: Price isn't the blocker. Trust and deployment model are. |
|
|
| ### Iterate |
|
|
| Ship the free tier. Re-evaluate. Score moves from 5.3 → 6.1. Then get SOC2. Score moves to 7.0. Each step verified against the same panel. |
|
|
| ``` |
| v1 baseline 5.3 avg 0% positive concerns: price, trust |
| v2 + free tier 6.1 avg 12% positive concerns: trust |
| v3 + SOC2 7.0 avg 28% positive concerns: (none) |
| ``` |
|
|
| </details> |
|
|
| --- |
|
|
| ## Bias Auditing & Calibration |
|
|
| LLM evaluators don't exhibit cognitive biases at human-realistic levels — they may be too rational (under-biased) or show biases in the wrong patterns (mis-biased). Since real expert panels *are* biased, matching their behavior means matching their bias profile, not eliminating bias. |
|
|
| SGO includes a bias audit inspired by [CoBRA](https://arxiv.org/abs/2509.13588) (Liu et al., CHI'26 Best Paper), which uses validated social science experiments to measure and control cognitive biases in LLM agents. |
|
|
| ### Measuring bias |
|
|
| `bias_audit.py` runs three probes through the same LLM + persona pipeline SGO uses for evaluation: |
|
|
| | Probe | What it tests | Human baseline | |
| |-------|--------------|----------------| |
| | **Framing** | Same entity, gain-framed vs. loss-framed — do evaluators shift scores based on rhetoric vs. substance? | ~30% shift (Tversky & Kahneman, 1981) | |
| | **Authority** | Entity with/without credibility signals (SOC2, press, logos) — how much do credentials move the needle? | ~20% sensitivity in evaluation contexts | |
| | **Order** | Same entity, sections reordered — does information order anchor scores? | Should be ~0% | |
|
|
| ```bash |
| uv run python scripts/bias_audit.py \ |
| --entity entities/my_product.md \ |
| --cohort data/cohort.json \ |
| --probes framing authority order \ |
| --sample 10 |
| ``` |
|
|
| Output: `results/bias_audit/report.md` — per-probe shift %, gap vs. human baselines, and whether the panel is over-biased, under-biased, or well-calibrated. |
|
|
| ### Calibrating evaluation |
|
|
| If the audit reveals bias gaps, add `--bias-calibration` to your evaluation run: |
|
|
| ```bash |
| uv run python scripts/evaluate.py \ |
| --entity entities/my_product.md \ |
| --cohort data/cohort.json \ |
| --tag calibrated \ |
| --bias-calibration |
| ``` |
|
|
| This appends bias-aware instructions to the evaluation prompt — reducing framing, authority, and order artifacts while preserving realistic human-level biases. The goal is not to eliminate bias but to match the type and magnitude of biases that real expert panels exhibit. |
|
|
| ### The expert panel gap |
|
|
| The gap between SGO and real expert panels has three components: |
|
|
| | Gap | What it means | How SGO addresses it | |
| |-----|--------------|---------------------| |
| | **Knowledge** | Does the LLM know what an expert knows? | Persona enrichment, narrative context | |
| | **Preference** | Does it weight factors correctly? | Stratification, prompt design | |
| | **Bias** | Does it exhibit human-realistic cognitive biases? | Bias audit + calibration (CoBRA-inspired) | |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Directional, not definitive** — this is synthetic research. Treat results as strong hypotheses, not proof. Validate important decisions with real users. |
| - **LLM biases** — evaluators inherit the model's cultural blind spots. Results skew toward what the LLM thinks people think. Use `bias_audit.py` to measure and `--bias-calibration` to mitigate. |
| - **Independent evaluators** — each persona scores in isolation. Real-world opinions are social — people influence each other. SGO doesn't capture herd effects. |
| - **Not all changes add up** — two changes that each score +1.5 might not give +3.0 together. Test combinations explicitly. |
|
|
| --- |
|
|
| <details> |
| <summary>Technical details</summary> |
|
|
| ## The Semantic Gradient |
|
|
| SGO computes a Jacobian matrix of score deltas — how each evaluator's score would shift for each hypothetical change: |
|
|
| $$J_{ij} = f(\theta + \Delta_j, \; x_i) - f(\theta, \; x_i)$$ |
|
|
| ### Goal-weighted gradient (VJP) |
|
|
| The key insight: not all evaluators matter equally. A luxury brand shouldn't optimize for budget shoppers. A dating profile shouldn't optimize for incompatible matches. |
|
|
| SGO uses a **goal vector** `v` that weights each evaluator by their relevance to your objective. The gradient is a vector-Jacobian product: |
|
|
| $$\nabla_j = \sum_{i} v_i \cdot J_{ij}$$ |
|
|
| Where `v_i` is the goal-relevance weight for evaluator `i` (0 = irrelevant, 1 = ideal target). |
|
|
| Without a goal, `v = [1/n, ...]` — uniform weights, optimizing for universal appeal. With a goal like *"close enterprise deals"*, enterprise CTOs get `v ≈ 1` and solo hobbyists get `v ≈ 0`. |
|
|
| The LLM assigns goal-relevance weights automatically by evaluating each persona against your stated objective. This means the gradient tells you *"what changes move you toward your goal"*, not *"what changes make everyone like you more"*. |
|
|
| ### What to probe |
|
|
| Only probe changes you'd actually make: |
|
|
| | Category | Examples | Probe? | |
| |----------|---------|--------| |
| | **Presentation** — framing, tone, emphasis | Rewrite headline, reorder features | Yes | |
| | **Actionable** — real changes with real cost | Add free tier, get SOC2 | Yes | |
| | **Fixed** — can't change | History, sunk costs | No | |
| | **Boundary** — won't change | Values, ethics, mission | No | |
|
|
| ### Notation |
|
|
| | Symbol | Meaning | |
| |--------|---------| |
| | θ | Entity you control | |
| | x | Evaluator persona | |
| | g | Goal — what you're optimizing for | |
| | f(θ, x) | LLM evaluation → score + reasoning | |
| | v_i | Goal-relevance weight for evaluator *i* | |
| | Δⱼ | Hypothetical change | |
| | Jᵢⱼ | Score delta: evaluator *i*, change *j* | |
| | ∇ⱼ | Goal-weighted gradient (VJP): impact of change *j* toward goal *g* | |
| |
| ## Project Structure |
| |
| ``` |
| ├── README.md # This file |
| ├── AGENT.md # Execution guide for AI agents |
| ├── SKILL.md # Claude Code skill definition |
| ├── pyproject.toml # Dependencies |
| ├── .env.example # API key template |
| ├── scripts/ |
| │ ├── setup_data.py # Download Nemotron personas (once) |
| │ ├── persona_loader.py # Load + filter |
| │ ├── stratified_sampler.py |
| │ ├── generate_cohort.py # LLM-generate personas (fallback) |
| │ ├── evaluate.py # Scorer (supports --bias-calibration) |
| │ ├── counterfactual.py # Semantic gradient probe |
| │ ├── bias_audit.py # CoBRA-inspired cognitive bias measurement |
| │ └── compare.py # Cross-run diff |
| ├── web/ |
| │ ├── app.py # FastAPI backend (primary entry point) |
| │ └── static/index.html # Single-page frontend |
| ├── templates/ # Entity + changes templates |
| ├── entities/ # Your documents (gitignored) |
| ├── data/ # Cohorts (gitignored) |
| └── results/ # Run outputs (gitignored) |
| ``` |
| |
| </details> |
| |
| ## License |
| |
| MIT |
| |
