Spaces:
Sleeping
Sleeping
Zeetay commited on
Commit ·
98ee05e
1
Parent(s): ef5d512
style: replace box diagram with a list, normalize unicode to ASCII, trim comments
Browse files- README.md +21 -23
- agent/__init__.py +1 -1
- agent/core.py +5 -18
- agent/memory.py +4 -8
- agent/prompts.py +11 -27
- agent/tools.py +3 -4
- api/main.py +4 -25
- db/store.py +3 -17
- evals/golden.py +4 -5
- evals/judge.py +3 -5
- evals/rubric.py +3 -6
- evals/runner.py +7 -12
- feedback/loop.py +3 -12
- scripts/run_eval.py +4 -4
- seed/loader.py +2 -7
- web/app/page.tsx +3 -3
- web/components/BriefForm.tsx +1 -1
- web/components/HowItWorks.tsx +2 -2
- web/components/StatsStrip.tsx +1 -1
README.md
CHANGED
|
@@ -14,7 +14,7 @@ evaluates its **own** outputs with an LLM-as-judge, stores the results, and
|
|
| 14 |
uses that signal to improve over time. The loop runs end to end:
|
| 15 |
|
| 16 |
```
|
| 17 |
-
retrieve
|
| 18 |
```
|
| 19 |
|
| 20 |
Every good output becomes future few-shot fuel and a regression baseline; every
|
|
@@ -49,7 +49,7 @@ Most "LLM app" demos are a single prompt with no memory and no notion of
|
|
| 49 |
quality. This project is built around three ideas that make it a genuine
|
| 50 |
*self-improving* loop:
|
| 51 |
|
| 52 |
-
1. **It judges itself on multiple axes.** Each output is scored 1
|
| 53 |
independent dimensions (hook strength, brand alignment, clarity, conversion
|
| 54 |
intent) by a separate judge model. Dimension scores are first-class and
|
| 55 |
stored separately; there is no single "vibe" score driving decisions.
|
|
@@ -71,17 +71,15 @@ prompt versioning reinforce each other.
|
|
| 71 |
|
| 72 |
## Architecture
|
| 73 |
|
| 74 |
-
``
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
└────────────────────────────────────────────────────────────────────────┘
|
| 84 |
-
```
|
| 85 |
|
| 86 |
| Layer | File(s) | Responsibility |
|
| 87 |
|------------------|----------------------------------|-------------------------------------------------|
|
|
@@ -175,21 +173,21 @@ flagged).
|
|
| 175 |
|
| 176 |
1. **Retrieve.** The incoming brief is embedded (locally, via
|
| 177 |
`all-MiniLM-L6-v2`) and used to query ChromaDB for the **top-3 most similar
|
| 178 |
-
past outputs that scored
|
| 179 |
|
| 180 |
2. **Generate.** Those examples are injected into the active generation prompt
|
| 181 |
under a *"Past high-performing examples"* section, and the model produces a
|
| 182 |
headline hook, body copy, and a CTA in one structured JSON response.
|
| 183 |
|
| 184 |
3. **Evaluate.** Each of the three variants is immediately scored by the judge
|
| 185 |
-
model on the four rubric dimensions. Scores are clamped to 1
|
| 186 |
average is computed for internal ranking only.
|
| 187 |
|
| 188 |
4. **Log.** The run, every variant, and every dimension score are written to
|
| 189 |
SQLite.
|
| 190 |
|
| 191 |
5. **Improve.** The feedback loop:
|
| 192 |
-
- promotes any output with a weighted average **
|
| 193 |
dataset and ChromaDB memory (so it can be retrieved next time);
|
| 194 |
- flags any output **< 2.5** into the `flagged_outputs` table with the reason
|
| 195 |
*"below quality threshold."*
|
|
@@ -202,7 +200,7 @@ over run; that is the "self-improving" part.
|
|
| 202 |
## Evaluation & regression tests
|
| 203 |
|
| 204 |
### The rubric (`evals/rubric.py`)
|
| 205 |
-
Four dimensions, each 1
|
| 206 |
|
| 207 |
| Dimension | Question | Weight |
|
| 208 |
|--------------------|-----------------------------------------------------|:------:|
|
|
@@ -215,7 +213,7 @@ The weighted average is **internal only**, used for thresholds and ranking.
|
|
| 215 |
All four raw dimensions are always stored separately.
|
| 216 |
|
| 217 |
### Golden dataset (`evals/golden.py`)
|
| 218 |
-
Any output with weighted average **
|
| 219 |
scores, prompt version, timestamp). This is the regression baseline.
|
| 220 |
|
| 221 |
### Regression runner (`evals/runner.py`)
|
|
@@ -245,7 +243,7 @@ spuriously) when `GROQ_API_KEY` is unset or the golden dataset is still empty.
|
|
| 245 |
|
| 246 |
All prompt text lives in `agent/prompts.py`; nothing is hardcoded anywhere
|
| 247 |
else. Each prompt is a named, versioned constant (`GENERATION_PROMPT_V1`,
|
| 248 |
-
`GENERATION_PROMPT_V2`,
|
| 249 |
|
| 250 |
```python
|
| 251 |
ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
|
|
@@ -282,7 +280,7 @@ cleanly with local sentence-transformers embeddings. SQLite alone can't do
|
|
| 282 |
nearest-neighbour search over brief semantics; a hosted vector DB would violate
|
| 283 |
the "local only" constraint and add ops overhead for no benefit at this scale.
|
| 284 |
|
| 285 |
-
**Why dimension scoring over a composite score.** A single 1
|
| 286 |
is unactionable and easy for a judge to anchor on. Scoring four independent
|
| 287 |
dimensions tells you *why* copy is weak (great hook, poor clarity) and makes the
|
| 288 |
signal far more stable and debuggable. We do compute a weighted average, but
|
|
@@ -290,19 +288,19 @@ only for internal thresholds/ranking; the four raw dimensions are always
|
|
| 290 |
stored, so we never lose information by collapsing too early.
|
| 291 |
|
| 292 |
**Why SQLite for eval storage.** Eval results are structured, relational, and
|
| 293 |
-
queryable (runs
|
| 294 |
guarantees, trivial setup, a single-file database, and real SQL, ideal for run
|
| 295 |
history and a golden dataset. It needs no server and ships with Python. A hosted
|
| 296 |
DB would add infrastructure with no upside for a local, single-node harness.
|
| 297 |
|
| 298 |
-
**Why the 4.0 threshold for golden inclusion.** On a 1
|
| 299 |
"clearly good on the weighted blend" without demanding perfection. Set it lower
|
| 300 |
and the golden set fills with mediocre copy, weakening both the regression
|
| 301 |
baseline and the few-shot exemplars. Set it higher (e.g. 4.5) and you rarely
|
| 302 |
capture anything, so the system never accumulates a baseline or improves. 4.0 is
|
| 303 |
the point where entries are good enough to *defend* against regressions and to
|
| 304 |
*teach* future runs. (The retrieval floor is a more permissive 3.5 so memory can
|
| 305 |
-
draw on a slightly wider pool of solid examples, while only the strongest
|
| 306 |
outputs become protected golden baselines.)
|
| 307 |
|
| 308 |
---
|
|
|
|
| 14 |
uses that signal to improve over time. The loop runs end to end:
|
| 15 |
|
| 16 |
```
|
| 17 |
+
retrieve -> generate -> evaluate -> log -> improve -> repeat
|
| 18 |
```
|
| 19 |
|
| 20 |
Every good output becomes future few-shot fuel and a regression baseline; every
|
|
|
|
| 49 |
quality. This project is built around three ideas that make it a genuine
|
| 50 |
*self-improving* loop:
|
| 51 |
|
| 52 |
+
1. **It judges itself on multiple axes.** Each output is scored 1-5 on four
|
| 53 |
independent dimensions (hook strength, brand alignment, clarity, conversion
|
| 54 |
intent) by a separate judge model. Dimension scores are first-class and
|
| 55 |
stored separately; there is no single "vibe" score driving decisions.
|
|
|
|
| 71 |
|
| 72 |
## Architecture
|
| 73 |
|
| 74 |
+
A single run flows through `agent/core.py`:
|
| 75 |
+
|
| 76 |
+
1. Retrieve top-3 high-scoring past outputs (`agent/memory.py`, ChromaDB)
|
| 77 |
+
2. Inject them as few-shot examples (`agent/prompts.py`)
|
| 78 |
+
3. Generate headline / body / cta with Groq (`agent/tools.py`)
|
| 79 |
+
4. Judge each variant on 4 dimensions with Groq (`evals/judge.py`)
|
| 80 |
+
5. Store run + outputs + scores in SQLite (`db/store.py`)
|
| 81 |
+
6. Promote winners (>= 4.0) to golden + memory (`feedback/loop.py`)
|
| 82 |
+
7. Flag losers (< 2.5) for review (`feedback/loop.py`)
|
|
|
|
|
|
|
| 83 |
|
| 84 |
| Layer | File(s) | Responsibility |
|
| 85 |
|------------------|----------------------------------|-------------------------------------------------|
|
|
|
|
| 173 |
|
| 174 |
1. **Retrieve.** The incoming brief is embedded (locally, via
|
| 175 |
`all-MiniLM-L6-v2`) and used to query ChromaDB for the **top-3 most similar
|
| 176 |
+
past outputs that scored >= 3.5/5**. Only proven-good copy is ever retrieved.
|
| 177 |
|
| 178 |
2. **Generate.** Those examples are injected into the active generation prompt
|
| 179 |
under a *"Past high-performing examples"* section, and the model produces a
|
| 180 |
headline hook, body copy, and a CTA in one structured JSON response.
|
| 181 |
|
| 182 |
3. **Evaluate.** Each of the three variants is immediately scored by the judge
|
| 183 |
+
model on the four rubric dimensions. Scores are clamped to 1-5 and a weighted
|
| 184 |
average is computed for internal ranking only.
|
| 185 |
|
| 186 |
4. **Log.** The run, every variant, and every dimension score are written to
|
| 187 |
SQLite.
|
| 188 |
|
| 189 |
5. **Improve.** The feedback loop:
|
| 190 |
+
- promotes any output with a weighted average **>= 4.0** into both the golden
|
| 191 |
dataset and ChromaDB memory (so it can be retrieved next time);
|
| 192 |
- flags any output **< 2.5** into the `flagged_outputs` table with the reason
|
| 193 |
*"below quality threshold."*
|
|
|
|
| 200 |
## Evaluation & regression tests
|
| 201 |
|
| 202 |
### The rubric (`evals/rubric.py`)
|
| 203 |
+
Four dimensions, each 1-5:
|
| 204 |
|
| 205 |
| Dimension | Question | Weight |
|
| 206 |
|--------------------|-----------------------------------------------------|:------:|
|
|
|
|
| 213 |
All four raw dimensions are always stored separately.
|
| 214 |
|
| 215 |
### Golden dataset (`evals/golden.py`)
|
| 216 |
+
Any output with weighted average **>= 4.0** is captured (brief, output, all
|
| 217 |
scores, prompt version, timestamp). This is the regression baseline.
|
| 218 |
|
| 219 |
### Regression runner (`evals/runner.py`)
|
|
|
|
| 243 |
|
| 244 |
All prompt text lives in `agent/prompts.py`; nothing is hardcoded anywhere
|
| 245 |
else. Each prompt is a named, versioned constant (`GENERATION_PROMPT_V1`,
|
| 246 |
+
`GENERATION_PROMPT_V2`, ...). A single constant selects which is live:
|
| 247 |
|
| 248 |
```python
|
| 249 |
ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
|
|
|
|
| 280 |
nearest-neighbour search over brief semantics; a hosted vector DB would violate
|
| 281 |
the "local only" constraint and add ops overhead for no benefit at this scale.
|
| 282 |
|
| 283 |
+
**Why dimension scoring over a composite score.** A single 1-10 "quality" score
|
| 284 |
is unactionable and easy for a judge to anchor on. Scoring four independent
|
| 285 |
dimensions tells you *why* copy is weak (great hook, poor clarity) and makes the
|
| 286 |
signal far more stable and debuggable. We do compute a weighted average, but
|
|
|
|
| 288 |
stored, so we never lose information by collapsing too early.
|
| 289 |
|
| 290 |
**Why SQLite for eval storage.** Eval results are structured, relational, and
|
| 291 |
+
queryable (runs -> outputs -> scores; golden; flagged). SQLite gives ACID
|
| 292 |
guarantees, trivial setup, a single-file database, and real SQL, ideal for run
|
| 293 |
history and a golden dataset. It needs no server and ships with Python. A hosted
|
| 294 |
DB would add infrastructure with no upside for a local, single-node harness.
|
| 295 |
|
| 296 |
+
**Why the 4.0 threshold for golden inclusion.** On a 1-5 scale, 4.0 means
|
| 297 |
"clearly good on the weighted blend" without demanding perfection. Set it lower
|
| 298 |
and the golden set fills with mediocre copy, weakening both the regression
|
| 299 |
baseline and the few-shot exemplars. Set it higher (e.g. 4.5) and you rarely
|
| 300 |
capture anything, so the system never accumulates a baseline or improves. 4.0 is
|
| 301 |
the point where entries are good enough to *defend* against regressions and to
|
| 302 |
*teach* future runs. (The retrieval floor is a more permissive 3.5 so memory can
|
| 303 |
+
draw on a slightly wider pool of solid examples, while only the strongest >=4.0
|
| 304 |
outputs become protected golden baselines.)
|
| 305 |
|
| 306 |
---
|
agent/__init__.py
CHANGED
|
@@ -11,5 +11,5 @@ try: # truststore is optional; ignore if unavailable.
|
|
| 11 |
import truststore
|
| 12 |
|
| 13 |
truststore.inject_into_ssl()
|
| 14 |
-
except Exception: # noqa: BLE001
|
| 15 |
pass
|
|
|
|
| 11 |
import truststore
|
| 12 |
|
| 13 |
truststore.inject_into_ssl()
|
| 14 |
+
except Exception: # noqa: BLE001 - best-effort; fall back to default certs.
|
| 15 |
pass
|
agent/core.py
CHANGED
|
@@ -1,8 +1,6 @@
|
|
| 1 |
"""The main agent loop: retrieve -> generate -> evaluate -> store -> feedback.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
loop together. It is intentionally synchronous and dependency-injected so it
|
| 5 |
-
can be driven from the API, the CLI, or tests with the same code path.
|
| 6 |
"""
|
| 7 |
|
| 8 |
from typing import Any, Optional
|
|
@@ -26,7 +24,6 @@ class Agent:
|
|
| 26 |
self.memory = memory or Memory()
|
| 27 |
self.temperature = temperature
|
| 28 |
|
| 29 |
-
# --------------------------------------------------------------- generate
|
| 30 |
def generate_variants(self, brief: dict[str, Any], few_shot_block: str, prompt_version: str) -> dict[str, str]:
|
| 31 |
"""Call the LLM once and parse out the three ad-copy variants."""
|
| 32 |
prompt = prompts.render_generation_prompt(brief, few_shot_block, prompt_version)
|
|
@@ -38,27 +35,19 @@ class Agent:
|
|
| 38 |
"cta": str(parsed.get("cta", "")).strip(),
|
| 39 |
}
|
| 40 |
|
| 41 |
-
# -------------------------------------------------------------------- run
|
| 42 |
def run(self, brief: dict[str, Any]) -> dict[str, Any]:
|
| 43 |
"""Execute the full loop for one brand brief and return outputs + scores."""
|
| 44 |
-
#
|
| 45 |
-
# nothing from core, but core depends on them at call time).
|
| 46 |
from evals.judge import judge_output
|
| 47 |
from feedback.loop import run_feedback
|
| 48 |
|
| 49 |
prompt_version = prompts.ACTIVE_PROMPT_VERSION
|
| 50 |
|
| 51 |
-
# 1. Retrieve top-3 most relevant high-scoring past outputs.
|
| 52 |
retrieved = self.memory.retrieve(brief, k=3)
|
| 53 |
-
|
| 54 |
-
# 2. Inject them as few-shot examples.
|
| 55 |
few_shot_block = prompts.build_few_shot_block(retrieved)
|
| 56 |
-
|
| 57 |
-
# 3. Generate the three variants.
|
| 58 |
variants = self.generate_variants(brief, few_shot_block, prompt_version)
|
| 59 |
|
| 60 |
-
#
|
| 61 |
-
# 5. Persist the run and each scored output.
|
| 62 |
run_id = self.store.create_run(brief, prompt_version)
|
| 63 |
|
| 64 |
scored_outputs: list[dict[str, Any]] = []
|
|
@@ -74,8 +63,7 @@ class Agent:
|
|
| 74 |
}
|
| 75 |
)
|
| 76 |
|
| 77 |
-
#
|
| 78 |
-
# flag poor ones for review.
|
| 79 |
feedback_summary = run_feedback(
|
| 80 |
store=self.store,
|
| 81 |
memory=self.memory,
|
|
@@ -90,8 +78,7 @@ class Agent:
|
|
| 90 |
"brief": brief,
|
| 91 |
"prompt_version": prompt_version,
|
| 92 |
"retrieved_examples": len(retrieved),
|
| 93 |
-
#
|
| 94 |
-
"retrieved_count": len(retrieved),
|
| 95 |
"outputs": scored_outputs,
|
| 96 |
"feedback": feedback_summary,
|
| 97 |
}
|
|
|
|
| 1 |
"""The main agent loop: retrieve -> generate -> evaluate -> store -> feedback.
|
| 2 |
|
| 3 |
+
Synchronous and dependency-injected so the API, CLI, and tests share one path.
|
|
|
|
|
|
|
| 4 |
"""
|
| 5 |
|
| 6 |
from typing import Any, Optional
|
|
|
|
| 24 |
self.memory = memory or Memory()
|
| 25 |
self.temperature = temperature
|
| 26 |
|
|
|
|
| 27 |
def generate_variants(self, brief: dict[str, Any], few_shot_block: str, prompt_version: str) -> dict[str, str]:
|
| 28 |
"""Call the LLM once and parse out the three ad-copy variants."""
|
| 29 |
prompt = prompts.render_generation_prompt(brief, few_shot_block, prompt_version)
|
|
|
|
| 35 |
"cta": str(parsed.get("cta", "")).strip(),
|
| 36 |
}
|
| 37 |
|
|
|
|
| 38 |
def run(self, brief: dict[str, Any]) -> dict[str, Any]:
|
| 39 |
"""Execute the full loop for one brand brief and return outputs + scores."""
|
| 40 |
+
# imported here to avoid an import cycle (core <-> evals/feedback)
|
|
|
|
| 41 |
from evals.judge import judge_output
|
| 42 |
from feedback.loop import run_feedback
|
| 43 |
|
| 44 |
prompt_version = prompts.ACTIVE_PROMPT_VERSION
|
| 45 |
|
|
|
|
| 46 |
retrieved = self.memory.retrieve(brief, k=3)
|
|
|
|
|
|
|
| 47 |
few_shot_block = prompts.build_few_shot_block(retrieved)
|
|
|
|
|
|
|
| 48 |
variants = self.generate_variants(brief, few_shot_block, prompt_version)
|
| 49 |
|
| 50 |
+
# judge each variant and persist the run
|
|
|
|
| 51 |
run_id = self.store.create_run(brief, prompt_version)
|
| 52 |
|
| 53 |
scored_outputs: list[dict[str, Any]] = []
|
|
|
|
| 63 |
}
|
| 64 |
)
|
| 65 |
|
| 66 |
+
# promote winners to golden + memory, flag the weak ones
|
|
|
|
| 67 |
feedback_summary = run_feedback(
|
| 68 |
store=self.store,
|
| 69 |
memory=self.memory,
|
|
|
|
| 78 |
"brief": brief,
|
| 79 |
"prompt_version": prompt_version,
|
| 80 |
"retrieved_examples": len(retrieved),
|
| 81 |
+
"retrieved_count": len(retrieved), # what the frontend reads
|
|
|
|
| 82 |
"outputs": scored_outputs,
|
| 83 |
"feedback": feedback_summary,
|
| 84 |
}
|
agent/memory.py
CHANGED
|
@@ -1,12 +1,8 @@
|
|
| 1 |
-
"""
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
self-improving loop.
|
| 7 |
-
|
| 8 |
-
Only entries scoring above RETRIEVAL_SCORE_FLOOR (3.5/5) are ever stored or
|
| 9 |
-
returned, so the agent only learns from outputs that actually worked.
|
| 10 |
"""
|
| 11 |
|
| 12 |
import os
|
|
|
|
| 1 |
+
"""Vector memory backed by ChromaDB + sentence-transformers.
|
| 2 |
|
| 3 |
+
Stores high-scoring outputs embedded by their brief, and retrieves the most
|
| 4 |
+
similar past winners as few-shot examples. Only entries >= RETRIEVAL_SCORE_FLOOR
|
| 5 |
+
(3.5/5) are kept or returned.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
"""
|
| 7 |
|
| 8 |
import os
|
agent/prompts.py
CHANGED
|
@@ -1,30 +1,19 @@
|
|
| 1 |
-
"""All prompts live here
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
that constant is the supported way to roll a new prompt forward — and it is
|
| 6 |
-
what the regression check guards (see evals/runner.py).
|
| 7 |
-
|
| 8 |
-
Nothing in the codebase should hardcode prompt text outside this module.
|
| 9 |
"""
|
| 10 |
|
| 11 |
-
#
|
| 12 |
-
# Active version selector
|
| 13 |
-
# ---------------------------------------------------------------------------
|
| 14 |
-
# Change this to promote a new generation prompt. The regression runner is
|
| 15 |
-
# expected to pass against the golden dataset before this is used for real.
|
| 16 |
ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
|
| 17 |
|
| 18 |
|
| 19 |
-
# ---------------------------------------------------------------------------
|
| 20 |
-
# Generation prompts
|
| 21 |
-
# ---------------------------------------------------------------------------
|
| 22 |
GENERATION_PROMPT_V1 = """You are a senior direct-to-consumer (DTC) ad copywriter.
|
| 23 |
|
| 24 |
You will be given a brand brief. Write three distinct pieces of ad copy:
|
| 25 |
-
1. A HEADLINE hook
|
| 26 |
-
2. A BODY copy
|
| 27 |
-
3. A CTA
|
| 28 |
|
| 29 |
Rules:
|
| 30 |
- Match the brief's tone and speak directly to the described audience.
|
|
@@ -52,9 +41,9 @@ GENERATION_PROMPT_V2 = """You are an award-winning DTC performance copywriter wh
|
|
| 52 |
copy that converts cold traffic.
|
| 53 |
|
| 54 |
You will be given a brand brief. Produce three pieces of ad copy:
|
| 55 |
-
1. HEADLINE
|
| 56 |
-
2. BODY
|
| 57 |
-
3. CTA
|
| 58 |
|
| 59 |
Rules:
|
| 60 |
- Lead with benefit, not feature. Speak to the exact audience in the brief.
|
|
@@ -82,9 +71,6 @@ PROMPT_REGISTRY = {
|
|
| 82 |
}
|
| 83 |
|
| 84 |
|
| 85 |
-
# ---------------------------------------------------------------------------
|
| 86 |
-
# Few-shot helpers
|
| 87 |
-
# ---------------------------------------------------------------------------
|
| 88 |
FEW_SHOT_HEADER = "Past high-performing examples (learn from their style, do not copy verbatim):"
|
| 89 |
|
| 90 |
|
|
@@ -127,9 +113,7 @@ def render_generation_prompt(brief: dict, few_shot_block: str, version: str | No
|
|
| 127 |
)
|
| 128 |
|
| 129 |
|
| 130 |
-
#
|
| 131 |
-
# Judge prompt (LLM-as-judge). Versioned alongside generation prompts.
|
| 132 |
-
# ---------------------------------------------------------------------------
|
| 133 |
JUDGE_PROMPT_VERSION = "JUDGE_PROMPT_V1"
|
| 134 |
|
| 135 |
JUDGE_PROMPT_V1 = """You are a strict, fair advertising copy evaluator.
|
|
|
|
| 1 |
+
"""All prompts live here, versioned by string key.
|
| 2 |
|
| 3 |
+
ACTIVE_PROMPT_VERSION selects the live generation prompt; swapping it is guarded
|
| 4 |
+
by the regression check (evals/runner.py). Don't hardcode prompt text elsewhere.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
+
# Run the regression suite before promoting a new version.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
|
| 9 |
|
| 10 |
|
|
|
|
|
|
|
|
|
|
| 11 |
GENERATION_PROMPT_V1 = """You are a senior direct-to-consumer (DTC) ad copywriter.
|
| 12 |
|
| 13 |
You will be given a brand brief. Write three distinct pieces of ad copy:
|
| 14 |
+
1. A HEADLINE hook - one short line that immediately grabs attention.
|
| 15 |
+
2. A BODY copy - 2-3 sentences that build desire and reflect the brand tone.
|
| 16 |
+
3. A CTA - a single short call to action that drives the stated goal.
|
| 17 |
|
| 18 |
Rules:
|
| 19 |
- Match the brief's tone and speak directly to the described audience.
|
|
|
|
| 41 |
copy that converts cold traffic.
|
| 42 |
|
| 43 |
You will be given a brand brief. Produce three pieces of ad copy:
|
| 44 |
+
1. HEADLINE - a scroll-stopping hook of at most 8 words.
|
| 45 |
+
2. BODY - 2-3 sentences leading with the strongest benefit, in the brand's voice.
|
| 46 |
+
3. CTA - an imperative call to action tied directly to the stated goal.
|
| 47 |
|
| 48 |
Rules:
|
| 49 |
- Lead with benefit, not feature. Speak to the exact audience in the brief.
|
|
|
|
| 71 |
}
|
| 72 |
|
| 73 |
|
|
|
|
|
|
|
|
|
|
| 74 |
FEW_SHOT_HEADER = "Past high-performing examples (learn from their style, do not copy verbatim):"
|
| 75 |
|
| 76 |
|
|
|
|
| 113 |
)
|
| 114 |
|
| 115 |
|
| 116 |
+
# Judge prompt, versioned alongside the generation prompts.
|
|
|
|
|
|
|
| 117 |
JUDGE_PROMPT_VERSION = "JUDGE_PROMPT_V1"
|
| 118 |
|
| 119 |
JUDGE_PROMPT_V1 = """You are a strict, fair advertising copy evaluator.
|
agent/tools.py
CHANGED
|
@@ -1,8 +1,7 @@
|
|
| 1 |
-
"""
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
one place.
|
| 6 |
"""
|
| 7 |
|
| 8 |
import json
|
|
|
|
| 1 |
+
"""Groq client wrapper and JSON parsing helpers.
|
| 2 |
|
| 3 |
+
One chat() helper is shared by the generator and the judge so model config
|
| 4 |
+
lives in one place. No LangChain.
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
import json
|
api/main.py
CHANGED
|
@@ -5,10 +5,6 @@ Endpoints:
|
|
| 5 |
- GET /stats -> live counts (runs, golden, flagged) for the frontend strip
|
| 6 |
- POST /run -> full agent loop (rate limited), returns outputs + scores
|
| 7 |
|
| 8 |
-
CORS is configured for the Vercel frontend, the Groq key is protected by a
|
| 9 |
-
per-IP rate limit, and a fresh deployment is seeded so the first visitor sees
|
| 10 |
-
non-zero stats and working retrieval.
|
| 11 |
-
|
| 12 |
Run locally: uvicorn api.main:app --reload
|
| 13 |
"""
|
| 14 |
|
|
@@ -30,10 +26,7 @@ from slowapi.util import get_remote_address # noqa: E402
|
|
| 30 |
from agent.core import Agent # noqa: E402 (after load_dotenv on purpose)
|
| 31 |
from seed.loader import load_seed_if_empty # noqa: E402
|
| 32 |
|
| 33 |
-
#
|
| 34 |
-
# Shared singletons
|
| 35 |
-
# --------------------------------------------------------------------------- #
|
| 36 |
-
# A single shared agent (and therefore shared Store/Memory) for the process.
|
| 37 |
_agent: Agent | None = None
|
| 38 |
|
| 39 |
|
|
@@ -44,9 +37,6 @@ def get_agent() -> Agent:
|
|
| 44 |
return _agent
|
| 45 |
|
| 46 |
|
| 47 |
-
# --------------------------------------------------------------------------- #
|
| 48 |
-
# CORS origins
|
| 49 |
-
# --------------------------------------------------------------------------- #
|
| 50 |
def _cors_origins() -> list[str]:
|
| 51 |
"""Origins from CORS_ORIGINS (comma-separated), always plus localhost:3000.
|
| 52 |
|
|
@@ -61,9 +51,7 @@ def _cors_origins() -> list[str]:
|
|
| 61 |
return origins
|
| 62 |
|
| 63 |
|
| 64 |
-
#
|
| 65 |
-
# Lifespan: seed a fresh deployment on startup
|
| 66 |
-
# --------------------------------------------------------------------------- #
|
| 67 |
@asynccontextmanager
|
| 68 |
async def lifespan(app: FastAPI):
|
| 69 |
try:
|
|
@@ -71,14 +59,11 @@ async def lifespan(app: FastAPI):
|
|
| 71 |
seeded = load_seed_if_empty(agent.store, agent.memory)
|
| 72 |
if seeded:
|
| 73 |
print(f"[startup] Seeded {seeded} golden/memory example(s).")
|
| 74 |
-
except Exception as exc: # noqa: BLE001
|
| 75 |
print(f"[startup] Seed skipped: {exc}")
|
| 76 |
yield
|
| 77 |
|
| 78 |
|
| 79 |
-
# --------------------------------------------------------------------------- #
|
| 80 |
-
# App + rate limiter
|
| 81 |
-
# --------------------------------------------------------------------------- #
|
| 82 |
limiter = Limiter(key_func=get_remote_address)
|
| 83 |
|
| 84 |
app = FastAPI(title="Self-Improving Ad Copy Agent", version="1.0.0", lifespan=lifespan)
|
|
@@ -94,9 +79,6 @@ app.add_middleware(
|
|
| 94 |
)
|
| 95 |
|
| 96 |
|
| 97 |
-
# --------------------------------------------------------------------------- #
|
| 98 |
-
# Schemas
|
| 99 |
-
# --------------------------------------------------------------------------- #
|
| 100 |
class BrandBrief(BaseModel):
|
| 101 |
brand: str = Field(..., examples=["FitFuel"])
|
| 102 |
product: str = Field(..., examples=["High-protein meal replacement shake"])
|
|
@@ -105,9 +87,6 @@ class BrandBrief(BaseModel):
|
|
| 105 |
goal: str = Field(..., examples=["Drive trial purchases"])
|
| 106 |
|
| 107 |
|
| 108 |
-
# --------------------------------------------------------------------------- #
|
| 109 |
-
# Endpoints
|
| 110 |
-
# --------------------------------------------------------------------------- #
|
| 111 |
@app.get("/")
|
| 112 |
def root() -> dict[str, str]:
|
| 113 |
return {"status": "ok", "endpoint": "POST /run with a brand brief"}
|
|
@@ -138,5 +117,5 @@ def run(request: Request, brief: BrandBrief) -> dict[str, Any]:
|
|
| 138 |
return agent.run(brief.model_dump())
|
| 139 |
except RuntimeError as exc: # e.g. missing GROQ_API_KEY
|
| 140 |
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
| 141 |
-
except Exception as exc: # noqa: BLE001
|
| 142 |
raise HTTPException(status_code=502, detail=f"Agent run failed: {exc}") from exc
|
|
|
|
| 5 |
- GET /stats -> live counts (runs, golden, flagged) for the frontend strip
|
| 6 |
- POST /run -> full agent loop (rate limited), returns outputs + scores
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
Run locally: uvicorn api.main:app --reload
|
| 9 |
"""
|
| 10 |
|
|
|
|
| 26 |
from agent.core import Agent # noqa: E402 (after load_dotenv on purpose)
|
| 27 |
from seed.loader import load_seed_if_empty # noqa: E402
|
| 28 |
|
| 29 |
+
# one shared agent (and Store/Memory) per process
|
|
|
|
|
|
|
|
|
|
| 30 |
_agent: Agent | None = None
|
| 31 |
|
| 32 |
|
|
|
|
| 37 |
return _agent
|
| 38 |
|
| 39 |
|
|
|
|
|
|
|
|
|
|
| 40 |
def _cors_origins() -> list[str]:
|
| 41 |
"""Origins from CORS_ORIGINS (comma-separated), always plus localhost:3000.
|
| 42 |
|
|
|
|
| 51 |
return origins
|
| 52 |
|
| 53 |
|
| 54 |
+
# seed golden + memory on a fresh deployment so the first visitor isn't cold
|
|
|
|
|
|
|
| 55 |
@asynccontextmanager
|
| 56 |
async def lifespan(app: FastAPI):
|
| 57 |
try:
|
|
|
|
| 59 |
seeded = load_seed_if_empty(agent.store, agent.memory)
|
| 60 |
if seeded:
|
| 61 |
print(f"[startup] Seeded {seeded} golden/memory example(s).")
|
| 62 |
+
except Exception as exc: # noqa: BLE001 - never block startup on seeding.
|
| 63 |
print(f"[startup] Seed skipped: {exc}")
|
| 64 |
yield
|
| 65 |
|
| 66 |
|
|
|
|
|
|
|
|
|
|
| 67 |
limiter = Limiter(key_func=get_remote_address)
|
| 68 |
|
| 69 |
app = FastAPI(title="Self-Improving Ad Copy Agent", version="1.0.0", lifespan=lifespan)
|
|
|
|
| 79 |
)
|
| 80 |
|
| 81 |
|
|
|
|
|
|
|
|
|
|
| 82 |
class BrandBrief(BaseModel):
|
| 83 |
brand: str = Field(..., examples=["FitFuel"])
|
| 84 |
product: str = Field(..., examples=["High-protein meal replacement shake"])
|
|
|
|
| 87 |
goal: str = Field(..., examples=["Drive trial purchases"])
|
| 88 |
|
| 89 |
|
|
|
|
|
|
|
|
|
|
| 90 |
@app.get("/")
|
| 91 |
def root() -> dict[str, str]:
|
| 92 |
return {"status": "ok", "endpoint": "POST /run with a brand brief"}
|
|
|
|
| 117 |
return agent.run(brief.model_dump())
|
| 118 |
except RuntimeError as exc: # e.g. missing GROQ_API_KEY
|
| 119 |
raise HTTPException(status_code=500, detail=str(exc)) from exc
|
| 120 |
+
except Exception as exc: # noqa: BLE001 - surface generation/judge failures
|
| 121 |
raise HTTPException(status_code=502, detail=f"Agent run failed: {exc}") from exc
|
db/store.py
CHANGED
|
@@ -1,8 +1,4 @@
|
|
| 1 |
-
"""SQLite
|
| 2 |
-
|
| 3 |
-
This is the persistence backbone of the system. Everything else depends on it.
|
| 4 |
-
All access goes through the Store class so the schema lives in exactly one place.
|
| 5 |
-
"""
|
| 6 |
|
| 7 |
import json
|
| 8 |
import os
|
|
@@ -19,11 +15,8 @@ def _utcnow() -> str:
|
|
| 19 |
|
| 20 |
|
| 21 |
class Store:
|
| 22 |
-
"""
|
| 23 |
-
|
| 24 |
-
A single connection is shared with a lock so the same Store can be used
|
| 25 |
-
from FastAPI request handlers and CLI scripts without surprises.
|
| 26 |
-
"""
|
| 27 |
|
| 28 |
def __init__(self, path: str = DEFAULT_SQLITE_PATH):
|
| 29 |
self.path = path
|
|
@@ -33,7 +26,6 @@ class Store:
|
|
| 33 |
self._conn.execute("PRAGMA journal_mode=WAL;")
|
| 34 |
self._init_schema()
|
| 35 |
|
| 36 |
-
# ------------------------------------------------------------------ schema
|
| 37 |
def _init_schema(self) -> None:
|
| 38 |
with self._lock, self._conn:
|
| 39 |
self._conn.executescript(
|
|
@@ -87,7 +79,6 @@ class Store:
|
|
| 87 |
"""
|
| 88 |
)
|
| 89 |
|
| 90 |
-
# -------------------------------------------------------------------- runs
|
| 91 |
def create_run(self, brief: dict[str, Any], prompt_version: str) -> int:
|
| 92 |
with self._lock, self._conn:
|
| 93 |
cur = self._conn.execute(
|
|
@@ -103,7 +94,6 @@ class Store:
|
|
| 103 |
).fetchone()
|
| 104 |
return dict(row) if row else None
|
| 105 |
|
| 106 |
-
# ----------------------------------------------------------------- outputs
|
| 107 |
def add_output(
|
| 108 |
self,
|
| 109 |
run_id: int,
|
|
@@ -144,7 +134,6 @@ class Store:
|
|
| 144 |
).fetchall()
|
| 145 |
return [dict(r) for r in rows]
|
| 146 |
|
| 147 |
-
# ------------------------------------------------------------------ golden
|
| 148 |
def add_golden(
|
| 149 |
self,
|
| 150 |
brief: dict[str, Any],
|
|
@@ -198,7 +187,6 @@ class Store:
|
|
| 198 |
).fetchone()
|
| 199 |
return row is not None
|
| 200 |
|
| 201 |
-
# ----------------------------------------------------------------- flagged
|
| 202 |
def add_flagged(
|
| 203 |
self,
|
| 204 |
brief: dict[str, Any],
|
|
@@ -239,7 +227,6 @@ class Store:
|
|
| 239 |
result.append(d)
|
| 240 |
return result
|
| 241 |
|
| 242 |
-
# ------------------------------------------------------------------ counts
|
| 243 |
def count_runs(self) -> int:
|
| 244 |
with self._lock:
|
| 245 |
row = self._conn.execute("SELECT COUNT(*) AS n FROM runs").fetchone()
|
|
@@ -255,7 +242,6 @@ class Store:
|
|
| 255 |
row = self._conn.execute("SELECT COUNT(*) AS n FROM flagged_outputs").fetchone()
|
| 256 |
return int(row["n"])
|
| 257 |
|
| 258 |
-
# ------------------------------------------------------------------- close
|
| 259 |
def close(self) -> None:
|
| 260 |
with self._lock:
|
| 261 |
self._conn.close()
|
|
|
|
| 1 |
+
"""SQLite store for runs, outputs, the golden dataset, and flagged outputs."""
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
import json
|
| 4 |
import os
|
|
|
|
| 15 |
|
| 16 |
|
| 17 |
class Store:
|
| 18 |
+
"""Local SQLite wrapper. One connection guarded by a lock so the API
|
| 19 |
+
handlers and CLI can share a Store."""
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
def __init__(self, path: str = DEFAULT_SQLITE_PATH):
|
| 22 |
self.path = path
|
|
|
|
| 26 |
self._conn.execute("PRAGMA journal_mode=WAL;")
|
| 27 |
self._init_schema()
|
| 28 |
|
|
|
|
| 29 |
def _init_schema(self) -> None:
|
| 30 |
with self._lock, self._conn:
|
| 31 |
self._conn.executescript(
|
|
|
|
| 79 |
"""
|
| 80 |
)
|
| 81 |
|
|
|
|
| 82 |
def create_run(self, brief: dict[str, Any], prompt_version: str) -> int:
|
| 83 |
with self._lock, self._conn:
|
| 84 |
cur = self._conn.execute(
|
|
|
|
| 94 |
).fetchone()
|
| 95 |
return dict(row) if row else None
|
| 96 |
|
|
|
|
| 97 |
def add_output(
|
| 98 |
self,
|
| 99 |
run_id: int,
|
|
|
|
| 134 |
).fetchall()
|
| 135 |
return [dict(r) for r in rows]
|
| 136 |
|
|
|
|
| 137 |
def add_golden(
|
| 138 |
self,
|
| 139 |
brief: dict[str, Any],
|
|
|
|
| 187 |
).fetchone()
|
| 188 |
return row is not None
|
| 189 |
|
|
|
|
| 190 |
def add_flagged(
|
| 191 |
self,
|
| 192 |
brief: dict[str, Any],
|
|
|
|
| 227 |
result.append(d)
|
| 228 |
return result
|
| 229 |
|
|
|
|
| 230 |
def count_runs(self) -> int:
|
| 231 |
with self._lock:
|
| 232 |
row = self._conn.execute("SELECT COUNT(*) AS n FROM runs").fetchone()
|
|
|
|
| 242 |
row = self._conn.execute("SELECT COUNT(*) AS n FROM flagged_outputs").fetchone()
|
| 243 |
return int(row["n"])
|
| 244 |
|
|
|
|
| 245 |
def close(self) -> None:
|
| 246 |
with self._lock:
|
| 247 |
self._conn.close()
|
evals/golden.py
CHANGED
|
@@ -1,9 +1,8 @@
|
|
| 1 |
-
"""Golden dataset
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
entry against the active prompt and checks for drift.
|
| 7 |
"""
|
| 8 |
|
| 9 |
from typing import Any
|
|
|
|
| 1 |
+
"""Golden dataset: the regression baseline.
|
| 2 |
|
| 3 |
+
Outputs scoring >= GOLDEN_THRESHOLD are captured with their brief, scores, and
|
| 4 |
+
prompt version. The regression runner later re-scores each entry and checks
|
| 5 |
+
for drift.
|
|
|
|
| 6 |
"""
|
| 7 |
|
| 8 |
from typing import Any
|
evals/judge.py
CHANGED
|
@@ -1,9 +1,7 @@
|
|
| 1 |
"""LLM-as-judge scorer.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
a derived weighted average and a one-line rationale. It never collapses the
|
| 6 |
-
output into a single composite the way a naive scorer would.
|
| 7 |
"""
|
| 8 |
|
| 9 |
from typing import Any
|
|
@@ -23,7 +21,7 @@ def judge_output(brief: dict[str, Any], variant_type: str, output: str) -> dict[
|
|
| 23 |
the valid range and default low.
|
| 24 |
"""
|
| 25 |
if not output.strip():
|
| 26 |
-
# Nothing to score
|
| 27 |
scores = rubric.normalize_scores({})
|
| 28 |
scores["rationale"] = "Empty output."
|
| 29 |
return scores
|
|
|
|
| 1 |
"""LLM-as-judge scorer.
|
| 2 |
|
| 3 |
+
Same Groq model as the generator, but a strict, temperature-0 prompt. Returns
|
| 4 |
+
the four dimension scores plus a weighted average and a short rationale.
|
|
|
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
from typing import Any
|
|
|
|
| 21 |
the valid range and default low.
|
| 22 |
"""
|
| 23 |
if not output.strip():
|
| 24 |
+
# Nothing to score - treat as the floor.
|
| 25 |
scores = rubric.normalize_scores({})
|
| 26 |
scores["rationale"] = "Empty output."
|
| 27 |
return scores
|
evals/rubric.py
CHANGED
|
@@ -1,9 +1,6 @@
|
|
| 1 |
-
"""Scoring rubric
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
the four dimensions, their 1-5 scale, and the weighting used to derive a
|
| 5 |
-
single internal number. Per the design, dimension scores are first-class and
|
| 6 |
-
stored separately; the weighted average exists only for ranking/thresholds.
|
| 7 |
"""
|
| 8 |
|
| 9 |
# The four scored dimensions, each on a 1-5 integer scale.
|
|
|
|
| 1 |
+
"""Scoring rubric: four 1-5 dimensions and the weights for the internal
|
| 2 |
+
weighted average. Dimension scores are stored separately; the weighted average
|
| 3 |
+
is only used for ranking/thresholds.
|
|
|
|
|
|
|
|
|
|
| 4 |
"""
|
| 5 |
|
| 6 |
# The four scored dimensions, each on a 1-5 integer scale.
|
evals/runner.py
CHANGED
|
@@ -1,13 +1,8 @@
|
|
| 1 |
-
"""Eval runner:
|
| 2 |
-
for regressions.
|
| 3 |
|
| 4 |
-
For each golden entry
|
| 5 |
-
|
| 6 |
-
the
|
| 7 |
-
(0.5), the run is marked failed.
|
| 8 |
-
|
| 9 |
-
This is what guards a prompt-version swap (see agent/prompts.py) and what the
|
| 10 |
-
pytest regression suite drives.
|
| 11 |
"""
|
| 12 |
|
| 13 |
from dataclasses import dataclass, field
|
|
@@ -108,7 +103,7 @@ def format_report(report: EvalReport) -> str:
|
|
| 108 |
"",
|
| 109 |
]
|
| 110 |
if report.count == 0:
|
| 111 |
-
lines.append("No golden entries yet
|
| 112 |
return "\n".join(lines)
|
| 113 |
|
| 114 |
for i, r in enumerate(report.results, 1):
|
|
@@ -120,10 +115,10 @@ def format_report(report: EvalReport) -> str:
|
|
| 120 |
|
| 121 |
lines.append("")
|
| 122 |
if report.passed:
|
| 123 |
-
lines.append("RESULT: PASS
|
| 124 |
else:
|
| 125 |
lines.append(
|
| 126 |
-
f"RESULT: FAIL
|
| 127 |
f"{REGRESSION_TOLERANCE}. Do not promote this prompt version."
|
| 128 |
)
|
| 129 |
return "\n".join(lines)
|
|
|
|
| 1 |
+
"""Eval runner: re-scores the golden dataset against a prompt version.
|
|
|
|
| 2 |
|
| 3 |
+
For each golden entry, regenerate copy for the same brief + variant, re-judge
|
| 4 |
+
it, and compare to the stored baseline. A drop > REGRESSION_TOLERANCE (0.5)
|
| 5 |
+
fails the run. Guards prompt-version swaps and backs the pytest regression suite.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
"""
|
| 7 |
|
| 8 |
from dataclasses import dataclass, field
|
|
|
|
| 103 |
"",
|
| 104 |
]
|
| 105 |
if report.count == 0:
|
| 106 |
+
lines.append("No golden entries yet - nothing to check. (PASS)")
|
| 107 |
return "\n".join(lines)
|
| 108 |
|
| 109 |
for i, r in enumerate(report.results, 1):
|
|
|
|
| 115 |
|
| 116 |
lines.append("")
|
| 117 |
if report.passed:
|
| 118 |
+
lines.append("RESULT: PASS - no entry regressed beyond tolerance.")
|
| 119 |
else:
|
| 120 |
lines.append(
|
| 121 |
+
f"RESULT: FAIL - {len(report.regressions)} entry(ies) regressed beyond "
|
| 122 |
f"{REGRESSION_TOLERANCE}. Do not promote this prompt version."
|
| 123 |
)
|
| 124 |
return "\n".join(lines)
|
feedback/loop.py
CHANGED
|
@@ -1,14 +1,7 @@
|
|
| 1 |
-
"""
|
|
|
|
| 2 |
|
| 3 |
-
|
| 4 |
-
1. Read the scores from the latest run.
|
| 5 |
-
2. Promote high-scoring outputs (weighted avg >= GOLDEN_THRESHOLD) into the
|
| 6 |
-
golden dataset AND into ChromaDB memory so future runs can retrieve them.
|
| 7 |
-
3. Flag low-scoring outputs (weighted avg < FLAG_THRESHOLD) into the
|
| 8 |
-
flagged_outputs table with a clear reason.
|
| 9 |
-
|
| 10 |
-
This is the closing arc of the self-improving loop: good outputs become future
|
| 11 |
-
few-shot fuel and regression baselines; bad ones get surfaced for review.
|
| 12 |
"""
|
| 13 |
|
| 14 |
from datetime import datetime, timezone
|
|
@@ -48,7 +41,6 @@ def run_feedback(
|
|
| 48 |
scores = item["scores"]
|
| 49 |
weighted = float(scores.get("weighted_average", 0.0))
|
| 50 |
|
| 51 |
-
# 2. Promote high scorers to golden + memory.
|
| 52 |
if weighted >= GOLDEN_THRESHOLD:
|
| 53 |
added = golden.maybe_add(brief, variant_type, content, scores, prompt_version)
|
| 54 |
memory.add(
|
|
@@ -62,7 +54,6 @@ def run_feedback(
|
|
| 62 |
if added:
|
| 63 |
promoted.append(variant_type)
|
| 64 |
|
| 65 |
-
# 3. Flag low scorers for review.
|
| 66 |
elif weighted < FLAG_THRESHOLD:
|
| 67 |
store.add_flagged(
|
| 68 |
brief=brief,
|
|
|
|
| 1 |
+
"""Runs after each generation: promote winners into the golden dataset and
|
| 2 |
+
ChromaDB memory (>= GOLDEN_THRESHOLD), flag the weak ones (< FLAG_THRESHOLD).
|
| 3 |
|
| 4 |
+
This closes the loop - promoted outputs become future few-shot examples.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
from datetime import datetime, timezone
|
|
|
|
| 41 |
scores = item["scores"]
|
| 42 |
weighted = float(scores.get("weighted_average", 0.0))
|
| 43 |
|
|
|
|
| 44 |
if weighted >= GOLDEN_THRESHOLD:
|
| 45 |
added = golden.maybe_add(brief, variant_type, content, scores, prompt_version)
|
| 46 |
memory.add(
|
|
|
|
| 54 |
if added:
|
| 55 |
promoted.append(variant_type)
|
| 56 |
|
|
|
|
| 57 |
elif weighted < FLAG_THRESHOLD:
|
| 58 |
store.add_flagged(
|
| 59 |
brief=brief,
|
scripts/run_eval.py
CHANGED
|
@@ -49,7 +49,7 @@ def cmd_run(args: argparse.Namespace) -> int:
|
|
| 49 |
brief = _load_brief(args.brief)
|
| 50 |
console.print(Panel.fit(json.dumps(brief, indent=2), title="Brand Brief", border_style="cyan"))
|
| 51 |
|
| 52 |
-
with console.status("[bold green]Running agent loop (retrieve
|
| 53 |
agent = Agent()
|
| 54 |
result = agent.run(brief)
|
| 55 |
|
|
@@ -95,7 +95,7 @@ def cmd_regression(args: argparse.Namespace) -> int:
|
|
| 95 |
with console.status("[bold green]Running golden regression eval..."):
|
| 96 |
report = run_golden_eval()
|
| 97 |
|
| 98 |
-
table = Table(title=f"Regression Eval
|
| 99 |
table.add_column("#", justify="right")
|
| 100 |
table.add_column("Variant", style="cyan")
|
| 101 |
table.add_column("Baseline", justify="right")
|
|
@@ -115,11 +115,11 @@ def cmd_regression(args: argparse.Namespace) -> int:
|
|
| 115 |
console.print(table)
|
| 116 |
console.print()
|
| 117 |
if report.passed:
|
| 118 |
-
console.print(Panel.fit("PASS
|
| 119 |
else:
|
| 120 |
console.print(
|
| 121 |
Panel.fit(
|
| 122 |
-
f"FAIL
|
| 123 |
"Do not promote this prompt version.",
|
| 124 |
border_style="red",
|
| 125 |
)
|
|
|
|
| 49 |
brief = _load_brief(args.brief)
|
| 50 |
console.print(Panel.fit(json.dumps(brief, indent=2), title="Brand Brief", border_style="cyan"))
|
| 51 |
|
| 52 |
+
with console.status("[bold green]Running agent loop (retrieve -> generate -> evaluate -> feedback)..."):
|
| 53 |
agent = Agent()
|
| 54 |
result = agent.run(brief)
|
| 55 |
|
|
|
|
| 95 |
with console.status("[bold green]Running golden regression eval..."):
|
| 96 |
report = run_golden_eval()
|
| 97 |
|
| 98 |
+
table = Table(title=f"Regression Eval - {report.prompt_version}", show_lines=True)
|
| 99 |
table.add_column("#", justify="right")
|
| 100 |
table.add_column("Variant", style="cyan")
|
| 101 |
table.add_column("Baseline", justify="right")
|
|
|
|
| 115 |
console.print(table)
|
| 116 |
console.print()
|
| 117 |
if report.passed:
|
| 118 |
+
console.print(Panel.fit("PASS - no entry regressed beyond tolerance.", border_style="green"))
|
| 119 |
else:
|
| 120 |
console.print(
|
| 121 |
Panel.fit(
|
| 122 |
+
f"FAIL - {len(report.regressions)} entry(ies) regressed. "
|
| 123 |
"Do not promote this prompt version.",
|
| 124 |
border_style="red",
|
| 125 |
)
|
seed/loader.py
CHANGED
|
@@ -1,12 +1,7 @@
|
|
| 1 |
"""Startup seed loader.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
that cold-start look, we load a small set of pre-scored example outputs into
|
| 6 |
-
both SQLite (golden) and ChromaDB (memory) the first time the app boots with an
|
| 7 |
-
empty golden table.
|
| 8 |
-
|
| 9 |
-
Idempotent: if the golden dataset already has entries, this does nothing.
|
| 10 |
"""
|
| 11 |
|
| 12 |
import json
|
|
|
|
| 1 |
"""Startup seed loader.
|
| 2 |
|
| 3 |
+
Loads a few pre-scored examples into golden + memory on first boot so a fresh
|
| 4 |
+
deployment isn't cold (zeroed stats, no retrieval). No-op if golden is non-empty.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
"""
|
| 6 |
|
| 7 |
import json
|
web/app/page.tsx
CHANGED
|
@@ -24,9 +24,9 @@ const ORDER: RunResponse["outputs"][number]["variant_type"][] = [
|
|
| 24 |
];
|
| 25 |
|
| 26 |
const LOADING_PHASES = [
|
| 27 |
-
"Retrieving examples
|
| 28 |
-
"Generating variants
|
| 29 |
-
"Scoring with the judge
|
| 30 |
];
|
| 31 |
|
| 32 |
export default function Page() {
|
|
|
|
| 24 |
];
|
| 25 |
|
| 26 |
const LOADING_PHASES = [
|
| 27 |
+
"Retrieving examples...",
|
| 28 |
+
"Generating variants...",
|
| 29 |
+
"Scoring with the judge...",
|
| 30 |
];
|
| 31 |
|
| 32 |
export default function Page() {
|
web/components/BriefForm.tsx
CHANGED
|
@@ -73,7 +73,7 @@ export function BriefForm({
|
|
| 73 |
disabled={loading}
|
| 74 |
className="border border-zinc-300 bg-zinc-100 px-4 py-2 text-sm font-medium text-ink-950 transition-colors hover:bg-white disabled:cursor-not-allowed disabled:border-ink-600 disabled:bg-ink-700 disabled:text-zinc-400"
|
| 75 |
>
|
| 76 |
-
{loading ? "Running
|
| 77 |
</button>
|
| 78 |
</div>
|
| 79 |
</form>
|
|
|
|
| 73 |
disabled={loading}
|
| 74 |
className="border border-zinc-300 bg-zinc-100 px-4 py-2 text-sm font-medium text-ink-950 transition-colors hover:bg-white disabled:cursor-not-allowed disabled:border-ink-600 disabled:bg-ink-700 disabled:text-zinc-400"
|
| 75 |
>
|
| 76 |
+
{loading ? "Running..." : "Generate Copy"}
|
| 77 |
</button>
|
| 78 |
</div>
|
| 79 |
</form>
|
web/components/HowItWorks.tsx
CHANGED
|
@@ -7,12 +7,12 @@ const STEPS = [
|
|
| 7 |
{
|
| 8 |
n: "02",
|
| 9 |
title: "Judge",
|
| 10 |
-
body: "A separate LLM-as-judge scores each variant 1
|
| 11 |
},
|
| 12 |
{
|
| 13 |
n: "03",
|
| 14 |
title: "Remember",
|
| 15 |
-
body: "Outputs scoring
|
| 16 |
},
|
| 17 |
{
|
| 18 |
n: "04",
|
|
|
|
| 7 |
{
|
| 8 |
n: "02",
|
| 9 |
title: "Judge",
|
| 10 |
+
body: "A separate LLM-as-judge scores each variant 1-5 on four dimensions: hook strength, brand alignment, clarity, and conversion intent.",
|
| 11 |
},
|
| 12 |
{
|
| 13 |
n: "03",
|
| 14 |
title: "Remember",
|
| 15 |
+
body: "Outputs scoring >= 4.0 are promoted into a golden dataset and a vector memory; weak ones (< 2.5) get flagged for review.",
|
| 16 |
},
|
| 17 |
{
|
| 18 |
n: "04",
|
web/components/StatsStrip.tsx
CHANGED
|
@@ -5,7 +5,7 @@ export function StatsStrip({ stats }: { stats: Stats | null }) {
|
|
| 5 |
<span className="inline-flex items-baseline gap-1.5">
|
| 6 |
<span className="text-zinc-500">{label}</span>
|
| 7 |
<span className="font-mono text-zinc-200 tabular-nums">
|
| 8 |
-
{value === null ? "
|
| 9 |
</span>
|
| 10 |
</span>
|
| 11 |
);
|
|
|
|
| 5 |
<span className="inline-flex items-baseline gap-1.5">
|
| 6 |
<span className="text-zinc-500">{label}</span>
|
| 7 |
<span className="font-mono text-zinc-200 tabular-nums">
|
| 8 |
+
{value === null ? "-" : value}
|
| 9 |
</span>
|
| 10 |
</span>
|
| 11 |
);
|