Zeetay commited on
Commit
98ee05e
·
1 Parent(s): ef5d512

style: replace box diagram with a list, normalize unicode to ASCII, trim comments

Browse files
README.md CHANGED
@@ -14,7 +14,7 @@ evaluates its **own** outputs with an LLM-as-judge, stores the results, and
14
  uses that signal to improve over time. The loop runs end to end:
15
 
16
  ```
17
- retrieve generate evaluate log improve repeat
18
  ```
19
 
20
  Every good output becomes future few-shot fuel and a regression baseline; every
@@ -49,7 +49,7 @@ Most "LLM app" demos are a single prompt with no memory and no notion of
49
  quality. This project is built around three ideas that make it a genuine
50
  *self-improving* loop:
51
 
52
- 1. **It judges itself on multiple axes.** Each output is scored 15 on four
53
  independent dimensions (hook strength, brand alignment, clarity, conversion
54
  intent) by a separate judge model. Dimension scores are first-class and
55
  stored separately; there is no single "vibe" score driving decisions.
@@ -71,17 +71,15 @@ prompt versioning reinforce each other.
71
 
72
  ## Architecture
73
 
74
- ```
75
- ┌──────────────────────────── agent/core.py ───────────────────────────┐
76
- brand brief ─▶│ 1. retrieve top-3 high-scoring past outputs (agent/memory.py, Chroma)
77
- 2. inject as few-shot examples (agent/prompts.py)
78
- 3. generate headline / body / cta (agent/tools.py → Groq)
79
- 4. judge each variant on 4 dimensions (evals/judge.py → Groq)
80
- 5. store run + outputs + scores (db/store.py → SQLite)
81
- 6. promote winners (4.0) golden + memory (feedback/loop.py)
82
- 7. flag losers (<2.5) for review (feedback/loop.py)
83
- └────────────────────────────────────────────────────────────────────────┘
84
- ```
85
 
86
  | Layer | File(s) | Responsibility |
87
  |------------------|----------------------------------|-------------------------------------------------|
@@ -175,21 +173,21 @@ flagged).
175
 
176
  1. **Retrieve.** The incoming brief is embedded (locally, via
177
  `all-MiniLM-L6-v2`) and used to query ChromaDB for the **top-3 most similar
178
- past outputs that scored 3.5/5**. Only proven-good copy is ever retrieved.
179
 
180
  2. **Generate.** Those examples are injected into the active generation prompt
181
  under a *"Past high-performing examples"* section, and the model produces a
182
  headline hook, body copy, and a CTA in one structured JSON response.
183
 
184
  3. **Evaluate.** Each of the three variants is immediately scored by the judge
185
- model on the four rubric dimensions. Scores are clamped to 15 and a weighted
186
  average is computed for internal ranking only.
187
 
188
  4. **Log.** The run, every variant, and every dimension score are written to
189
  SQLite.
190
 
191
  5. **Improve.** The feedback loop:
192
- - promotes any output with a weighted average ** 4.0** into both the golden
193
  dataset and ChromaDB memory (so it can be retrieved next time);
194
  - flags any output **< 2.5** into the `flagged_outputs` table with the reason
195
  *"below quality threshold."*
@@ -202,7 +200,7 @@ over run; that is the "self-improving" part.
202
  ## Evaluation & regression tests
203
 
204
  ### The rubric (`evals/rubric.py`)
205
- Four dimensions, each 15:
206
 
207
  | Dimension | Question | Weight |
208
  |--------------------|-----------------------------------------------------|:------:|
@@ -215,7 +213,7 @@ The weighted average is **internal only**, used for thresholds and ranking.
215
  All four raw dimensions are always stored separately.
216
 
217
  ### Golden dataset (`evals/golden.py`)
218
- Any output with weighted average ** 4.0** is captured (brief, output, all
219
  scores, prompt version, timestamp). This is the regression baseline.
220
 
221
  ### Regression runner (`evals/runner.py`)
@@ -245,7 +243,7 @@ spuriously) when `GROQ_API_KEY` is unset or the golden dataset is still empty.
245
 
246
  All prompt text lives in `agent/prompts.py`; nothing is hardcoded anywhere
247
  else. Each prompt is a named, versioned constant (`GENERATION_PROMPT_V1`,
248
- `GENERATION_PROMPT_V2`, ). A single constant selects which is live:
249
 
250
  ```python
251
  ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
@@ -282,7 +280,7 @@ cleanly with local sentence-transformers embeddings. SQLite alone can't do
282
  nearest-neighbour search over brief semantics; a hosted vector DB would violate
283
  the "local only" constraint and add ops overhead for no benefit at this scale.
284
 
285
- **Why dimension scoring over a composite score.** A single 110 "quality" score
286
  is unactionable and easy for a judge to anchor on. Scoring four independent
287
  dimensions tells you *why* copy is weak (great hook, poor clarity) and makes the
288
  signal far more stable and debuggable. We do compute a weighted average, but
@@ -290,19 +288,19 @@ only for internal thresholds/ranking; the four raw dimensions are always
290
  stored, so we never lose information by collapsing too early.
291
 
292
  **Why SQLite for eval storage.** Eval results are structured, relational, and
293
- queryable (runs outputs scores; golden; flagged). SQLite gives ACID
294
  guarantees, trivial setup, a single-file database, and real SQL, ideal for run
295
  history and a golden dataset. It needs no server and ships with Python. A hosted
296
  DB would add infrastructure with no upside for a local, single-node harness.
297
 
298
- **Why the 4.0 threshold for golden inclusion.** On a 15 scale, 4.0 means
299
  "clearly good on the weighted blend" without demanding perfection. Set it lower
300
  and the golden set fills with mediocre copy, weakening both the regression
301
  baseline and the few-shot exemplars. Set it higher (e.g. 4.5) and you rarely
302
  capture anything, so the system never accumulates a baseline or improves. 4.0 is
303
  the point where entries are good enough to *defend* against regressions and to
304
  *teach* future runs. (The retrieval floor is a more permissive 3.5 so memory can
305
- draw on a slightly wider pool of solid examples, while only the strongest 4.0
306
  outputs become protected golden baselines.)
307
 
308
  ---
 
14
  uses that signal to improve over time. The loop runs end to end:
15
 
16
  ```
17
+ retrieve -> generate -> evaluate -> log -> improve -> repeat
18
  ```
19
 
20
  Every good output becomes future few-shot fuel and a regression baseline; every
 
49
  quality. This project is built around three ideas that make it a genuine
50
  *self-improving* loop:
51
 
52
+ 1. **It judges itself on multiple axes.** Each output is scored 1-5 on four
53
  independent dimensions (hook strength, brand alignment, clarity, conversion
54
  intent) by a separate judge model. Dimension scores are first-class and
55
  stored separately; there is no single "vibe" score driving decisions.
 
71
 
72
  ## Architecture
73
 
74
+ A single run flows through `agent/core.py`:
75
+
76
+ 1. Retrieve top-3 high-scoring past outputs (`agent/memory.py`, ChromaDB)
77
+ 2. Inject them as few-shot examples (`agent/prompts.py`)
78
+ 3. Generate headline / body / cta with Groq (`agent/tools.py`)
79
+ 4. Judge each variant on 4 dimensions with Groq (`evals/judge.py`)
80
+ 5. Store run + outputs + scores in SQLite (`db/store.py`)
81
+ 6. Promote winners (>= 4.0) to golden + memory (`feedback/loop.py`)
82
+ 7. Flag losers (< 2.5) for review (`feedback/loop.py`)
 
 
83
 
84
  | Layer | File(s) | Responsibility |
85
  |------------------|----------------------------------|-------------------------------------------------|
 
173
 
174
  1. **Retrieve.** The incoming brief is embedded (locally, via
175
  `all-MiniLM-L6-v2`) and used to query ChromaDB for the **top-3 most similar
176
+ past outputs that scored >= 3.5/5**. Only proven-good copy is ever retrieved.
177
 
178
  2. **Generate.** Those examples are injected into the active generation prompt
179
  under a *"Past high-performing examples"* section, and the model produces a
180
  headline hook, body copy, and a CTA in one structured JSON response.
181
 
182
  3. **Evaluate.** Each of the three variants is immediately scored by the judge
183
+ model on the four rubric dimensions. Scores are clamped to 1-5 and a weighted
184
  average is computed for internal ranking only.
185
 
186
  4. **Log.** The run, every variant, and every dimension score are written to
187
  SQLite.
188
 
189
  5. **Improve.** The feedback loop:
190
+ - promotes any output with a weighted average **>= 4.0** into both the golden
191
  dataset and ChromaDB memory (so it can be retrieved next time);
192
  - flags any output **< 2.5** into the `flagged_outputs` table with the reason
193
  *"below quality threshold."*
 
200
  ## Evaluation & regression tests
201
 
202
  ### The rubric (`evals/rubric.py`)
203
+ Four dimensions, each 1-5:
204
 
205
  | Dimension | Question | Weight |
206
  |--------------------|-----------------------------------------------------|:------:|
 
213
  All four raw dimensions are always stored separately.
214
 
215
  ### Golden dataset (`evals/golden.py`)
216
+ Any output with weighted average **>= 4.0** is captured (brief, output, all
217
  scores, prompt version, timestamp). This is the regression baseline.
218
 
219
  ### Regression runner (`evals/runner.py`)
 
243
 
244
  All prompt text lives in `agent/prompts.py`; nothing is hardcoded anywhere
245
  else. Each prompt is a named, versioned constant (`GENERATION_PROMPT_V1`,
246
+ `GENERATION_PROMPT_V2`, ...). A single constant selects which is live:
247
 
248
  ```python
249
  ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
 
280
  nearest-neighbour search over brief semantics; a hosted vector DB would violate
281
  the "local only" constraint and add ops overhead for no benefit at this scale.
282
 
283
+ **Why dimension scoring over a composite score.** A single 1-10 "quality" score
284
  is unactionable and easy for a judge to anchor on. Scoring four independent
285
  dimensions tells you *why* copy is weak (great hook, poor clarity) and makes the
286
  signal far more stable and debuggable. We do compute a weighted average, but
 
288
  stored, so we never lose information by collapsing too early.
289
 
290
  **Why SQLite for eval storage.** Eval results are structured, relational, and
291
+ queryable (runs -> outputs -> scores; golden; flagged). SQLite gives ACID
292
  guarantees, trivial setup, a single-file database, and real SQL, ideal for run
293
  history and a golden dataset. It needs no server and ships with Python. A hosted
294
  DB would add infrastructure with no upside for a local, single-node harness.
295
 
296
+ **Why the 4.0 threshold for golden inclusion.** On a 1-5 scale, 4.0 means
297
  "clearly good on the weighted blend" without demanding perfection. Set it lower
298
  and the golden set fills with mediocre copy, weakening both the regression
299
  baseline and the few-shot exemplars. Set it higher (e.g. 4.5) and you rarely
300
  capture anything, so the system never accumulates a baseline or improves. 4.0 is
301
  the point where entries are good enough to *defend* against regressions and to
302
  *teach* future runs. (The retrieval floor is a more permissive 3.5 so memory can
303
+ draw on a slightly wider pool of solid examples, while only the strongest >=4.0
304
  outputs become protected golden baselines.)
305
 
306
  ---
agent/__init__.py CHANGED
@@ -11,5 +11,5 @@ try: # truststore is optional; ignore if unavailable.
11
  import truststore
12
 
13
  truststore.inject_into_ssl()
14
- except Exception: # noqa: BLE001 best-effort; fall back to default certs.
15
  pass
 
11
  import truststore
12
 
13
  truststore.inject_into_ssl()
14
+ except Exception: # noqa: BLE001 - best-effort; fall back to default certs.
15
  pass
agent/core.py CHANGED
@@ -1,8 +1,6 @@
1
  """The main agent loop: retrieve -> generate -> evaluate -> store -> feedback.
2
 
3
- This ties memory, generation, the eval judge, persistence, and the feedback
4
- loop together. It is intentionally synchronous and dependency-injected so it
5
- can be driven from the API, the CLI, or tests with the same code path.
6
  """
7
 
8
  from typing import Any, Optional
@@ -26,7 +24,6 @@ class Agent:
26
  self.memory = memory or Memory()
27
  self.temperature = temperature
28
 
29
- # --------------------------------------------------------------- generate
30
  def generate_variants(self, brief: dict[str, Any], few_shot_block: str, prompt_version: str) -> dict[str, str]:
31
  """Call the LLM once and parse out the three ad-copy variants."""
32
  prompt = prompts.render_generation_prompt(brief, few_shot_block, prompt_version)
@@ -38,27 +35,19 @@ class Agent:
38
  "cta": str(parsed.get("cta", "")).strip(),
39
  }
40
 
41
- # -------------------------------------------------------------------- run
42
  def run(self, brief: dict[str, Any]) -> dict[str, Any]:
43
  """Execute the full loop for one brand brief and return outputs + scores."""
44
- # Imported here to keep module import order clean (evals/feedback import
45
- # nothing from core, but core depends on them at call time).
46
  from evals.judge import judge_output
47
  from feedback.loop import run_feedback
48
 
49
  prompt_version = prompts.ACTIVE_PROMPT_VERSION
50
 
51
- # 1. Retrieve top-3 most relevant high-scoring past outputs.
52
  retrieved = self.memory.retrieve(brief, k=3)
53
-
54
- # 2. Inject them as few-shot examples.
55
  few_shot_block = prompts.build_few_shot_block(retrieved)
56
-
57
- # 3. Generate the three variants.
58
  variants = self.generate_variants(brief, few_shot_block, prompt_version)
59
 
60
- # 4. Evaluate every variant immediately.
61
- # 5. Persist the run and each scored output.
62
  run_id = self.store.create_run(brief, prompt_version)
63
 
64
  scored_outputs: list[dict[str, Any]] = []
@@ -74,8 +63,7 @@ class Agent:
74
  }
75
  )
76
 
77
- # 6 + 7. Feedback loop: promote good outputs to golden + memory,
78
- # flag poor ones for review.
79
  feedback_summary = run_feedback(
80
  store=self.store,
81
  memory=self.memory,
@@ -90,8 +78,7 @@ class Agent:
90
  "brief": brief,
91
  "prompt_version": prompt_version,
92
  "retrieved_examples": len(retrieved),
93
- # Surfaced to the frontend so it can show the loop is self-improving.
94
- "retrieved_count": len(retrieved),
95
  "outputs": scored_outputs,
96
  "feedback": feedback_summary,
97
  }
 
1
  """The main agent loop: retrieve -> generate -> evaluate -> store -> feedback.
2
 
3
+ Synchronous and dependency-injected so the API, CLI, and tests share one path.
 
 
4
  """
5
 
6
  from typing import Any, Optional
 
24
  self.memory = memory or Memory()
25
  self.temperature = temperature
26
 
 
27
  def generate_variants(self, brief: dict[str, Any], few_shot_block: str, prompt_version: str) -> dict[str, str]:
28
  """Call the LLM once and parse out the three ad-copy variants."""
29
  prompt = prompts.render_generation_prompt(brief, few_shot_block, prompt_version)
 
35
  "cta": str(parsed.get("cta", "")).strip(),
36
  }
37
 
 
38
  def run(self, brief: dict[str, Any]) -> dict[str, Any]:
39
  """Execute the full loop for one brand brief and return outputs + scores."""
40
+ # imported here to avoid an import cycle (core <-> evals/feedback)
 
41
  from evals.judge import judge_output
42
  from feedback.loop import run_feedback
43
 
44
  prompt_version = prompts.ACTIVE_PROMPT_VERSION
45
 
 
46
  retrieved = self.memory.retrieve(brief, k=3)
 
 
47
  few_shot_block = prompts.build_few_shot_block(retrieved)
 
 
48
  variants = self.generate_variants(brief, few_shot_block, prompt_version)
49
 
50
+ # judge each variant and persist the run
 
51
  run_id = self.store.create_run(brief, prompt_version)
52
 
53
  scored_outputs: list[dict[str, Any]] = []
 
63
  }
64
  )
65
 
66
+ # promote winners to golden + memory, flag the weak ones
 
67
  feedback_summary = run_feedback(
68
  store=self.store,
69
  memory=self.memory,
 
78
  "brief": brief,
79
  "prompt_version": prompt_version,
80
  "retrieved_examples": len(retrieved),
81
+ "retrieved_count": len(retrieved), # what the frontend reads
 
82
  "outputs": scored_outputs,
83
  "feedback": feedback_summary,
84
  }
agent/memory.py CHANGED
@@ -1,12 +1,8 @@
1
- """Retrieval and storage logic backed by ChromaDB + sentence-transformers.
2
 
3
- Memory holds past generated outputs that scored well, embedded by their brand
4
- brief. On each new run we retrieve the top-k most similar high-scoring entries
5
- and feed them back as few-shot examples — this is the substrate of the
6
- self-improving loop.
7
-
8
- Only entries scoring above RETRIEVAL_SCORE_FLOOR (3.5/5) are ever stored or
9
- returned, so the agent only learns from outputs that actually worked.
10
  """
11
 
12
  import os
 
1
+ """Vector memory backed by ChromaDB + sentence-transformers.
2
 
3
+ Stores high-scoring outputs embedded by their brief, and retrieves the most
4
+ similar past winners as few-shot examples. Only entries >= RETRIEVAL_SCORE_FLOOR
5
+ (3.5/5) are kept or returned.
 
 
 
 
6
  """
7
 
8
  import os
agent/prompts.py CHANGED
@@ -1,30 +1,19 @@
1
- """All prompts live here and nowhere else.
2
 
3
- Each prompt is versioned with an explicit string key. The single source of
4
- truth for which generation prompt is live is ACTIVE_PROMPT_VERSION. Swapping
5
- that constant is the supported way to roll a new prompt forward — and it is
6
- what the regression check guards (see evals/runner.py).
7
-
8
- Nothing in the codebase should hardcode prompt text outside this module.
9
  """
10
 
11
- # ---------------------------------------------------------------------------
12
- # Active version selector
13
- # ---------------------------------------------------------------------------
14
- # Change this to promote a new generation prompt. The regression runner is
15
- # expected to pass against the golden dataset before this is used for real.
16
  ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
17
 
18
 
19
- # ---------------------------------------------------------------------------
20
- # Generation prompts
21
- # ---------------------------------------------------------------------------
22
  GENERATION_PROMPT_V1 = """You are a senior direct-to-consumer (DTC) ad copywriter.
23
 
24
  You will be given a brand brief. Write three distinct pieces of ad copy:
25
- 1. A HEADLINE hook one short line that immediately grabs attention.
26
- 2. A BODY copy 2-3 sentences that build desire and reflect the brand tone.
27
- 3. A CTA a single short call to action that drives the stated goal.
28
 
29
  Rules:
30
  - Match the brief's tone and speak directly to the described audience.
@@ -52,9 +41,9 @@ GENERATION_PROMPT_V2 = """You are an award-winning DTC performance copywriter wh
52
  copy that converts cold traffic.
53
 
54
  You will be given a brand brief. Produce three pieces of ad copy:
55
- 1. HEADLINE a scroll-stopping hook of at most 8 words.
56
- 2. BODY 2-3 sentences leading with the strongest benefit, in the brand's voice.
57
- 3. CTA an imperative call to action tied directly to the stated goal.
58
 
59
  Rules:
60
  - Lead with benefit, not feature. Speak to the exact audience in the brief.
@@ -82,9 +71,6 @@ PROMPT_REGISTRY = {
82
  }
83
 
84
 
85
- # ---------------------------------------------------------------------------
86
- # Few-shot helpers
87
- # ---------------------------------------------------------------------------
88
  FEW_SHOT_HEADER = "Past high-performing examples (learn from their style, do not copy verbatim):"
89
 
90
 
@@ -127,9 +113,7 @@ def render_generation_prompt(brief: dict, few_shot_block: str, version: str | No
127
  )
128
 
129
 
130
- # ---------------------------------------------------------------------------
131
- # Judge prompt (LLM-as-judge). Versioned alongside generation prompts.
132
- # ---------------------------------------------------------------------------
133
  JUDGE_PROMPT_VERSION = "JUDGE_PROMPT_V1"
134
 
135
  JUDGE_PROMPT_V1 = """You are a strict, fair advertising copy evaluator.
 
1
+ """All prompts live here, versioned by string key.
2
 
3
+ ACTIVE_PROMPT_VERSION selects the live generation prompt; swapping it is guarded
4
+ by the regression check (evals/runner.py). Don't hardcode prompt text elsewhere.
 
 
 
 
5
  """
6
 
7
+ # Run the regression suite before promoting a new version.
 
 
 
 
8
  ACTIVE_PROMPT_VERSION = "GENERATION_PROMPT_V1"
9
 
10
 
 
 
 
11
  GENERATION_PROMPT_V1 = """You are a senior direct-to-consumer (DTC) ad copywriter.
12
 
13
  You will be given a brand brief. Write three distinct pieces of ad copy:
14
+ 1. A HEADLINE hook - one short line that immediately grabs attention.
15
+ 2. A BODY copy - 2-3 sentences that build desire and reflect the brand tone.
16
+ 3. A CTA - a single short call to action that drives the stated goal.
17
 
18
  Rules:
19
  - Match the brief's tone and speak directly to the described audience.
 
41
  copy that converts cold traffic.
42
 
43
  You will be given a brand brief. Produce three pieces of ad copy:
44
+ 1. HEADLINE - a scroll-stopping hook of at most 8 words.
45
+ 2. BODY - 2-3 sentences leading with the strongest benefit, in the brand's voice.
46
+ 3. CTA - an imperative call to action tied directly to the stated goal.
47
 
48
  Rules:
49
  - Lead with benefit, not feature. Speak to the exact audience in the brief.
 
71
  }
72
 
73
 
 
 
 
74
  FEW_SHOT_HEADER = "Past high-performing examples (learn from their style, do not copy verbatim):"
75
 
76
 
 
113
  )
114
 
115
 
116
+ # Judge prompt, versioned alongside the generation prompts.
 
 
117
  JUDGE_PROMPT_VERSION = "JUDGE_PROMPT_V1"
118
 
119
  JUDGE_PROMPT_V1 = """You are a strict, fair advertising copy evaluator.
agent/tools.py CHANGED
@@ -1,8 +1,7 @@
1
- """Agent tools: the LLM client wrapper and the structured-generation helpers.
2
 
3
- We deliberately call Groq directly (no LangChain). A single chat-completion
4
- helper is shared by both the generator and the judge so model/config lives in
5
- one place.
6
  """
7
 
8
  import json
 
1
+ """Groq client wrapper and JSON parsing helpers.
2
 
3
+ One chat() helper is shared by the generator and the judge so model config
4
+ lives in one place. No LangChain.
 
5
  """
6
 
7
  import json
api/main.py CHANGED
@@ -5,10 +5,6 @@ Endpoints:
5
  - GET /stats -> live counts (runs, golden, flagged) for the frontend strip
6
  - POST /run -> full agent loop (rate limited), returns outputs + scores
7
 
8
- CORS is configured for the Vercel frontend, the Groq key is protected by a
9
- per-IP rate limit, and a fresh deployment is seeded so the first visitor sees
10
- non-zero stats and working retrieval.
11
-
12
  Run locally: uvicorn api.main:app --reload
13
  """
14
 
@@ -30,10 +26,7 @@ from slowapi.util import get_remote_address # noqa: E402
30
  from agent.core import Agent # noqa: E402 (after load_dotenv on purpose)
31
  from seed.loader import load_seed_if_empty # noqa: E402
32
 
33
- # --------------------------------------------------------------------------- #
34
- # Shared singletons
35
- # --------------------------------------------------------------------------- #
36
- # A single shared agent (and therefore shared Store/Memory) for the process.
37
  _agent: Agent | None = None
38
 
39
 
@@ -44,9 +37,6 @@ def get_agent() -> Agent:
44
  return _agent
45
 
46
 
47
- # --------------------------------------------------------------------------- #
48
- # CORS origins
49
- # --------------------------------------------------------------------------- #
50
  def _cors_origins() -> list[str]:
51
  """Origins from CORS_ORIGINS (comma-separated), always plus localhost:3000.
52
 
@@ -61,9 +51,7 @@ def _cors_origins() -> list[str]:
61
  return origins
62
 
63
 
64
- # --------------------------------------------------------------------------- #
65
- # Lifespan: seed a fresh deployment on startup
66
- # --------------------------------------------------------------------------- #
67
  @asynccontextmanager
68
  async def lifespan(app: FastAPI):
69
  try:
@@ -71,14 +59,11 @@ async def lifespan(app: FastAPI):
71
  seeded = load_seed_if_empty(agent.store, agent.memory)
72
  if seeded:
73
  print(f"[startup] Seeded {seeded} golden/memory example(s).")
74
- except Exception as exc: # noqa: BLE001 never block startup on seeding.
75
  print(f"[startup] Seed skipped: {exc}")
76
  yield
77
 
78
 
79
- # --------------------------------------------------------------------------- #
80
- # App + rate limiter
81
- # --------------------------------------------------------------------------- #
82
  limiter = Limiter(key_func=get_remote_address)
83
 
84
  app = FastAPI(title="Self-Improving Ad Copy Agent", version="1.0.0", lifespan=lifespan)
@@ -94,9 +79,6 @@ app.add_middleware(
94
  )
95
 
96
 
97
- # --------------------------------------------------------------------------- #
98
- # Schemas
99
- # --------------------------------------------------------------------------- #
100
  class BrandBrief(BaseModel):
101
  brand: str = Field(..., examples=["FitFuel"])
102
  product: str = Field(..., examples=["High-protein meal replacement shake"])
@@ -105,9 +87,6 @@ class BrandBrief(BaseModel):
105
  goal: str = Field(..., examples=["Drive trial purchases"])
106
 
107
 
108
- # --------------------------------------------------------------------------- #
109
- # Endpoints
110
- # --------------------------------------------------------------------------- #
111
  @app.get("/")
112
  def root() -> dict[str, str]:
113
  return {"status": "ok", "endpoint": "POST /run with a brand brief"}
@@ -138,5 +117,5 @@ def run(request: Request, brief: BrandBrief) -> dict[str, Any]:
138
  return agent.run(brief.model_dump())
139
  except RuntimeError as exc: # e.g. missing GROQ_API_KEY
140
  raise HTTPException(status_code=500, detail=str(exc)) from exc
141
- except Exception as exc: # noqa: BLE001 surface generation/judge failures
142
  raise HTTPException(status_code=502, detail=f"Agent run failed: {exc}") from exc
 
5
  - GET /stats -> live counts (runs, golden, flagged) for the frontend strip
6
  - POST /run -> full agent loop (rate limited), returns outputs + scores
7
 
 
 
 
 
8
  Run locally: uvicorn api.main:app --reload
9
  """
10
 
 
26
  from agent.core import Agent # noqa: E402 (after load_dotenv on purpose)
27
  from seed.loader import load_seed_if_empty # noqa: E402
28
 
29
+ # one shared agent (and Store/Memory) per process
 
 
 
30
  _agent: Agent | None = None
31
 
32
 
 
37
  return _agent
38
 
39
 
 
 
 
40
  def _cors_origins() -> list[str]:
41
  """Origins from CORS_ORIGINS (comma-separated), always plus localhost:3000.
42
 
 
51
  return origins
52
 
53
 
54
+ # seed golden + memory on a fresh deployment so the first visitor isn't cold
 
 
55
  @asynccontextmanager
56
  async def lifespan(app: FastAPI):
57
  try:
 
59
  seeded = load_seed_if_empty(agent.store, agent.memory)
60
  if seeded:
61
  print(f"[startup] Seeded {seeded} golden/memory example(s).")
62
+ except Exception as exc: # noqa: BLE001 - never block startup on seeding.
63
  print(f"[startup] Seed skipped: {exc}")
64
  yield
65
 
66
 
 
 
 
67
  limiter = Limiter(key_func=get_remote_address)
68
 
69
  app = FastAPI(title="Self-Improving Ad Copy Agent", version="1.0.0", lifespan=lifespan)
 
79
  )
80
 
81
 
 
 
 
82
  class BrandBrief(BaseModel):
83
  brand: str = Field(..., examples=["FitFuel"])
84
  product: str = Field(..., examples=["High-protein meal replacement shake"])
 
87
  goal: str = Field(..., examples=["Drive trial purchases"])
88
 
89
 
 
 
 
90
  @app.get("/")
91
  def root() -> dict[str, str]:
92
  return {"status": "ok", "endpoint": "POST /run with a brand brief"}
 
117
  return agent.run(brief.model_dump())
118
  except RuntimeError as exc: # e.g. missing GROQ_API_KEY
119
  raise HTTPException(status_code=500, detail=str(exc)) from exc
120
+ except Exception as exc: # noqa: BLE001 - surface generation/judge failures
121
  raise HTTPException(status_code=502, detail=f"Agent run failed: {exc}") from exc
db/store.py CHANGED
@@ -1,8 +1,4 @@
1
- """SQLite interface for runs, scores, golden dataset, flagged outputs, and versions.
2
-
3
- This is the persistence backbone of the system. Everything else depends on it.
4
- All access goes through the Store class so the schema lives in exactly one place.
5
- """
6
 
7
  import json
8
  import os
@@ -19,11 +15,8 @@ def _utcnow() -> str:
19
 
20
 
21
  class Store:
22
- """Thin, explicit wrapper over a local SQLite database.
23
-
24
- A single connection is shared with a lock so the same Store can be used
25
- from FastAPI request handlers and CLI scripts without surprises.
26
- """
27
 
28
  def __init__(self, path: str = DEFAULT_SQLITE_PATH):
29
  self.path = path
@@ -33,7 +26,6 @@ class Store:
33
  self._conn.execute("PRAGMA journal_mode=WAL;")
34
  self._init_schema()
35
 
36
- # ------------------------------------------------------------------ schema
37
  def _init_schema(self) -> None:
38
  with self._lock, self._conn:
39
  self._conn.executescript(
@@ -87,7 +79,6 @@ class Store:
87
  """
88
  )
89
 
90
- # -------------------------------------------------------------------- runs
91
  def create_run(self, brief: dict[str, Any], prompt_version: str) -> int:
92
  with self._lock, self._conn:
93
  cur = self._conn.execute(
@@ -103,7 +94,6 @@ class Store:
103
  ).fetchone()
104
  return dict(row) if row else None
105
 
106
- # ----------------------------------------------------------------- outputs
107
  def add_output(
108
  self,
109
  run_id: int,
@@ -144,7 +134,6 @@ class Store:
144
  ).fetchall()
145
  return [dict(r) for r in rows]
146
 
147
- # ------------------------------------------------------------------ golden
148
  def add_golden(
149
  self,
150
  brief: dict[str, Any],
@@ -198,7 +187,6 @@ class Store:
198
  ).fetchone()
199
  return row is not None
200
 
201
- # ----------------------------------------------------------------- flagged
202
  def add_flagged(
203
  self,
204
  brief: dict[str, Any],
@@ -239,7 +227,6 @@ class Store:
239
  result.append(d)
240
  return result
241
 
242
- # ------------------------------------------------------------------ counts
243
  def count_runs(self) -> int:
244
  with self._lock:
245
  row = self._conn.execute("SELECT COUNT(*) AS n FROM runs").fetchone()
@@ -255,7 +242,6 @@ class Store:
255
  row = self._conn.execute("SELECT COUNT(*) AS n FROM flagged_outputs").fetchone()
256
  return int(row["n"])
257
 
258
- # ------------------------------------------------------------------- close
259
  def close(self) -> None:
260
  with self._lock:
261
  self._conn.close()
 
1
+ """SQLite store for runs, outputs, the golden dataset, and flagged outputs."""
 
 
 
 
2
 
3
  import json
4
  import os
 
15
 
16
 
17
  class Store:
18
+ """Local SQLite wrapper. One connection guarded by a lock so the API
19
+ handlers and CLI can share a Store."""
 
 
 
20
 
21
  def __init__(self, path: str = DEFAULT_SQLITE_PATH):
22
  self.path = path
 
26
  self._conn.execute("PRAGMA journal_mode=WAL;")
27
  self._init_schema()
28
 
 
29
  def _init_schema(self) -> None:
30
  with self._lock, self._conn:
31
  self._conn.executescript(
 
79
  """
80
  )
81
 
 
82
  def create_run(self, brief: dict[str, Any], prompt_version: str) -> int:
83
  with self._lock, self._conn:
84
  cur = self._conn.execute(
 
94
  ).fetchone()
95
  return dict(row) if row else None
96
 
 
97
  def add_output(
98
  self,
99
  run_id: int,
 
134
  ).fetchall()
135
  return [dict(r) for r in rows]
136
 
 
137
  def add_golden(
138
  self,
139
  brief: dict[str, Any],
 
187
  ).fetchone()
188
  return row is not None
189
 
 
190
  def add_flagged(
191
  self,
192
  brief: dict[str, Any],
 
227
  result.append(d)
228
  return result
229
 
 
230
  def count_runs(self) -> int:
231
  with self._lock:
232
  row = self._conn.execute("SELECT COUNT(*) AS n FROM runs").fetchone()
 
242
  row = self._conn.execute("SELECT COUNT(*) AS n FROM flagged_outputs").fetchone()
243
  return int(row["n"])
244
 
 
245
  def close(self) -> None:
246
  with self._lock:
247
  self._conn.close()
evals/golden.py CHANGED
@@ -1,9 +1,8 @@
1
- """Golden dataset manager.
2
 
3
- The golden dataset is the system's regression baseline: outputs that scored
4
- well enough (weighted average >= GOLDEN_THRESHOLD) are captured with their
5
- brief, scores, and prompt version. The regression runner later re-scores each
6
- entry against the active prompt and checks for drift.
7
  """
8
 
9
  from typing import Any
 
1
+ """Golden dataset: the regression baseline.
2
 
3
+ Outputs scoring >= GOLDEN_THRESHOLD are captured with their brief, scores, and
4
+ prompt version. The regression runner later re-scores each entry and checks
5
+ for drift.
 
6
  """
7
 
8
  from typing import Any
evals/judge.py CHANGED
@@ -1,9 +1,7 @@
1
  """LLM-as-judge scorer.
2
 
3
- Uses the same Groq model as the generator (llama-3.3-70b-versatile) but with a
4
- strict, low-temperature judging prompt. Returns the four dimension scores plus
5
- a derived weighted average and a one-line rationale. It never collapses the
6
- output into a single composite the way a naive scorer would.
7
  """
8
 
9
  from typing import Any
@@ -23,7 +21,7 @@ def judge_output(brief: dict[str, Any], variant_type: str, output: str) -> dict[
23
  the valid range and default low.
24
  """
25
  if not output.strip():
26
- # Nothing to score treat as the floor.
27
  scores = rubric.normalize_scores({})
28
  scores["rationale"] = "Empty output."
29
  return scores
 
1
  """LLM-as-judge scorer.
2
 
3
+ Same Groq model as the generator, but a strict, temperature-0 prompt. Returns
4
+ the four dimension scores plus a weighted average and a short rationale.
 
 
5
  """
6
 
7
  from typing import Any
 
21
  the valid range and default low.
22
  """
23
  if not output.strip():
24
+ # Nothing to score - treat as the floor.
25
  scores = rubric.normalize_scores({})
26
  scores["rationale"] = "Empty output."
27
  return scores
evals/rubric.py CHANGED
@@ -1,9 +1,6 @@
1
- """Scoring rubric definitions.
2
-
3
- The rubric is the contract between the judge and the rest of the system:
4
- the four dimensions, their 1-5 scale, and the weighting used to derive a
5
- single internal number. Per the design, dimension scores are first-class and
6
- stored separately; the weighted average exists only for ranking/thresholds.
7
  """
8
 
9
  # The four scored dimensions, each on a 1-5 integer scale.
 
1
+ """Scoring rubric: four 1-5 dimensions and the weights for the internal
2
+ weighted average. Dimension scores are stored separately; the weighted average
3
+ is only used for ranking/thresholds.
 
 
 
4
  """
5
 
6
  # The four scored dimensions, each on a 1-5 integer scale.
evals/runner.py CHANGED
@@ -1,13 +1,8 @@
1
- """Eval runner: runs the golden dataset against any prompt config and checks
2
- for regressions.
3
 
4
- For each golden entry we regenerate copy for the same brief + variant using the
5
- prompt version under test, re-judge it, and compare the new weighted score to
6
- the entry's stored baseline. If any entry drops by more than REGRESSION_TOLERANCE
7
- (0.5), the run is marked failed.
8
-
9
- This is what guards a prompt-version swap (see agent/prompts.py) and what the
10
- pytest regression suite drives.
11
  """
12
 
13
  from dataclasses import dataclass, field
@@ -108,7 +103,7 @@ def format_report(report: EvalReport) -> str:
108
  "",
109
  ]
110
  if report.count == 0:
111
- lines.append("No golden entries yet nothing to check. (PASS)")
112
  return "\n".join(lines)
113
 
114
  for i, r in enumerate(report.results, 1):
@@ -120,10 +115,10 @@ def format_report(report: EvalReport) -> str:
120
 
121
  lines.append("")
122
  if report.passed:
123
- lines.append("RESULT: PASS no entry regressed beyond tolerance.")
124
  else:
125
  lines.append(
126
- f"RESULT: FAIL {len(report.regressions)} entry(ies) regressed beyond "
127
  f"{REGRESSION_TOLERANCE}. Do not promote this prompt version."
128
  )
129
  return "\n".join(lines)
 
1
+ """Eval runner: re-scores the golden dataset against a prompt version.
 
2
 
3
+ For each golden entry, regenerate copy for the same brief + variant, re-judge
4
+ it, and compare to the stored baseline. A drop > REGRESSION_TOLERANCE (0.5)
5
+ fails the run. Guards prompt-version swaps and backs the pytest regression suite.
 
 
 
 
6
  """
7
 
8
  from dataclasses import dataclass, field
 
103
  "",
104
  ]
105
  if report.count == 0:
106
+ lines.append("No golden entries yet - nothing to check. (PASS)")
107
  return "\n".join(lines)
108
 
109
  for i, r in enumerate(report.results, 1):
 
115
 
116
  lines.append("")
117
  if report.passed:
118
+ lines.append("RESULT: PASS - no entry regressed beyond tolerance.")
119
  else:
120
  lines.append(
121
+ f"RESULT: FAIL - {len(report.regressions)} entry(ies) regressed beyond "
122
  f"{REGRESSION_TOLERANCE}. Do not promote this prompt version."
123
  )
124
  return "\n".join(lines)
feedback/loop.py CHANGED
@@ -1,14 +1,7 @@
1
- """Feedback loop: runs after every agent generation.
 
2
 
3
- Responsibilities:
4
- 1. Read the scores from the latest run.
5
- 2. Promote high-scoring outputs (weighted avg >= GOLDEN_THRESHOLD) into the
6
- golden dataset AND into ChromaDB memory so future runs can retrieve them.
7
- 3. Flag low-scoring outputs (weighted avg < FLAG_THRESHOLD) into the
8
- flagged_outputs table with a clear reason.
9
-
10
- This is the closing arc of the self-improving loop: good outputs become future
11
- few-shot fuel and regression baselines; bad ones get surfaced for review.
12
  """
13
 
14
  from datetime import datetime, timezone
@@ -48,7 +41,6 @@ def run_feedback(
48
  scores = item["scores"]
49
  weighted = float(scores.get("weighted_average", 0.0))
50
 
51
- # 2. Promote high scorers to golden + memory.
52
  if weighted >= GOLDEN_THRESHOLD:
53
  added = golden.maybe_add(brief, variant_type, content, scores, prompt_version)
54
  memory.add(
@@ -62,7 +54,6 @@ def run_feedback(
62
  if added:
63
  promoted.append(variant_type)
64
 
65
- # 3. Flag low scorers for review.
66
  elif weighted < FLAG_THRESHOLD:
67
  store.add_flagged(
68
  brief=brief,
 
1
+ """Runs after each generation: promote winners into the golden dataset and
2
+ ChromaDB memory (>= GOLDEN_THRESHOLD), flag the weak ones (< FLAG_THRESHOLD).
3
 
4
+ This closes the loop - promoted outputs become future few-shot examples.
 
 
 
 
 
 
 
 
5
  """
6
 
7
  from datetime import datetime, timezone
 
41
  scores = item["scores"]
42
  weighted = float(scores.get("weighted_average", 0.0))
43
 
 
44
  if weighted >= GOLDEN_THRESHOLD:
45
  added = golden.maybe_add(brief, variant_type, content, scores, prompt_version)
46
  memory.add(
 
54
  if added:
55
  promoted.append(variant_type)
56
 
 
57
  elif weighted < FLAG_THRESHOLD:
58
  store.add_flagged(
59
  brief=brief,
scripts/run_eval.py CHANGED
@@ -49,7 +49,7 @@ def cmd_run(args: argparse.Namespace) -> int:
49
  brief = _load_brief(args.brief)
50
  console.print(Panel.fit(json.dumps(brief, indent=2), title="Brand Brief", border_style="cyan"))
51
 
52
- with console.status("[bold green]Running agent loop (retrieve generate evaluate feedback)..."):
53
  agent = Agent()
54
  result = agent.run(brief)
55
 
@@ -95,7 +95,7 @@ def cmd_regression(args: argparse.Namespace) -> int:
95
  with console.status("[bold green]Running golden regression eval..."):
96
  report = run_golden_eval()
97
 
98
- table = Table(title=f"Regression Eval {report.prompt_version}", show_lines=True)
99
  table.add_column("#", justify="right")
100
  table.add_column("Variant", style="cyan")
101
  table.add_column("Baseline", justify="right")
@@ -115,11 +115,11 @@ def cmd_regression(args: argparse.Namespace) -> int:
115
  console.print(table)
116
  console.print()
117
  if report.passed:
118
- console.print(Panel.fit("PASS no entry regressed beyond tolerance.", border_style="green"))
119
  else:
120
  console.print(
121
  Panel.fit(
122
- f"FAIL {len(report.regressions)} entry(ies) regressed. "
123
  "Do not promote this prompt version.",
124
  border_style="red",
125
  )
 
49
  brief = _load_brief(args.brief)
50
  console.print(Panel.fit(json.dumps(brief, indent=2), title="Brand Brief", border_style="cyan"))
51
 
52
+ with console.status("[bold green]Running agent loop (retrieve -> generate -> evaluate -> feedback)..."):
53
  agent = Agent()
54
  result = agent.run(brief)
55
 
 
95
  with console.status("[bold green]Running golden regression eval..."):
96
  report = run_golden_eval()
97
 
98
+ table = Table(title=f"Regression Eval - {report.prompt_version}", show_lines=True)
99
  table.add_column("#", justify="right")
100
  table.add_column("Variant", style="cyan")
101
  table.add_column("Baseline", justify="right")
 
115
  console.print(table)
116
  console.print()
117
  if report.passed:
118
+ console.print(Panel.fit("PASS - no entry regressed beyond tolerance.", border_style="green"))
119
  else:
120
  console.print(
121
  Panel.fit(
122
+ f"FAIL - {len(report.regressions)} entry(ies) regressed. "
123
  "Do not promote this prompt version.",
124
  border_style="red",
125
  )
seed/loader.py CHANGED
@@ -1,12 +1,7 @@
1
  """Startup seed loader.
2
 
3
- A fresh deployment has an empty golden dataset and empty memory, so the very
4
- first visitor would see zeroed stats and get no few-shot retrieval. To avoid
5
- that cold-start look, we load a small set of pre-scored example outputs into
6
- both SQLite (golden) and ChromaDB (memory) the first time the app boots with an
7
- empty golden table.
8
-
9
- Idempotent: if the golden dataset already has entries, this does nothing.
10
  """
11
 
12
  import json
 
1
  """Startup seed loader.
2
 
3
+ Loads a few pre-scored examples into golden + memory on first boot so a fresh
4
+ deployment isn't cold (zeroed stats, no retrieval). No-op if golden is non-empty.
 
 
 
 
 
5
  """
6
 
7
  import json
web/app/page.tsx CHANGED
@@ -24,9 +24,9 @@ const ORDER: RunResponse["outputs"][number]["variant_type"][] = [
24
  ];
25
 
26
  const LOADING_PHASES = [
27
- "Retrieving examples",
28
- "Generating variants",
29
- "Scoring with the judge",
30
  ];
31
 
32
  export default function Page() {
 
24
  ];
25
 
26
  const LOADING_PHASES = [
27
+ "Retrieving examples...",
28
+ "Generating variants...",
29
+ "Scoring with the judge...",
30
  ];
31
 
32
  export default function Page() {
web/components/BriefForm.tsx CHANGED
@@ -73,7 +73,7 @@ export function BriefForm({
73
  disabled={loading}
74
  className="border border-zinc-300 bg-zinc-100 px-4 py-2 text-sm font-medium text-ink-950 transition-colors hover:bg-white disabled:cursor-not-allowed disabled:border-ink-600 disabled:bg-ink-700 disabled:text-zinc-400"
75
  >
76
- {loading ? "Running" : "Generate Copy"}
77
  </button>
78
  </div>
79
  </form>
 
73
  disabled={loading}
74
  className="border border-zinc-300 bg-zinc-100 px-4 py-2 text-sm font-medium text-ink-950 transition-colors hover:bg-white disabled:cursor-not-allowed disabled:border-ink-600 disabled:bg-ink-700 disabled:text-zinc-400"
75
  >
76
+ {loading ? "Running..." : "Generate Copy"}
77
  </button>
78
  </div>
79
  </form>
web/components/HowItWorks.tsx CHANGED
@@ -7,12 +7,12 @@ const STEPS = [
7
  {
8
  n: "02",
9
  title: "Judge",
10
- body: "A separate LLM-as-judge scores each variant 15 on four dimensions: hook strength, brand alignment, clarity, and conversion intent.",
11
  },
12
  {
13
  n: "03",
14
  title: "Remember",
15
- body: "Outputs scoring 4.0 are promoted into a golden dataset and a vector memory; weak ones (< 2.5) get flagged for review.",
16
  },
17
  {
18
  n: "04",
 
7
  {
8
  n: "02",
9
  title: "Judge",
10
+ body: "A separate LLM-as-judge scores each variant 1-5 on four dimensions: hook strength, brand alignment, clarity, and conversion intent.",
11
  },
12
  {
13
  n: "03",
14
  title: "Remember",
15
+ body: "Outputs scoring >= 4.0 are promoted into a golden dataset and a vector memory; weak ones (< 2.5) get flagged for review.",
16
  },
17
  {
18
  n: "04",
web/components/StatsStrip.tsx CHANGED
@@ -5,7 +5,7 @@ export function StatsStrip({ stats }: { stats: Stats | null }) {
5
  <span className="inline-flex items-baseline gap-1.5">
6
  <span className="text-zinc-500">{label}</span>
7
  <span className="font-mono text-zinc-200 tabular-nums">
8
- {value === null ? "" : value}
9
  </span>
10
  </span>
11
  );
 
5
  <span className="inline-flex items-baseline gap-1.5">
6
  <span className="text-zinc-500">{label}</span>
7
  <span className="font-mono text-zinc-200 tabular-nums">
8
+ {value === null ? "-" : value}
9
  </span>
10
  </span>
11
  );