VibecoderMcSwaggins commited on
Commit
0e1abcc
Β·
1 Parent(s): cb24279

docs: SPEC_10 audit - add 16th location and test updates

Browse files

Senior agent audit findings:
- Added src/agent_factory/judges.py (16th hardcoded location)
- Added domain threading: JudgeHandler β†’ format_user_prompt
- Added tests/e2e/test_simple_mode.py to update list
- Made get_domain_config() accept strings for UI compatibility
- Clarified migration phases with JudgeHandler dependency

docs/specs/SPEC_10_DOMAIN_AGNOSTIC_REFACTOR.md CHANGED
@@ -7,12 +7,12 @@
7
 
8
  ## Problem Statement
9
 
10
- The codebase has "drug repurposing" hardcoded in **15 locations**:
11
 
12
  ```
13
  src/prompts/report.py:11 - SYSTEM_PROMPT
14
  src/prompts/judge.py:5 - SYSTEM_PROMPT
15
- src/prompts/judge.py:140 - Evidence scoring prompt
16
  src/prompts/hypothesis.py:11 - SYSTEM_PROMPT
17
  src/orchestrators/simple.py:476 - Report header
18
  src/orchestrators/simple.py:564 - Report header
@@ -25,10 +25,11 @@ src/mcp_tools.py:27 - Example query
25
  src/mcp_tools.py:116 - Docstring
26
  src/mcp_tools.py:164 - Function docstring
27
  src/mcp_tools.py:167 - Docstring
 
28
  ```
29
 
30
  This violates:
31
- - **DRY** - Same concept repeated 15 times
32
  - **Open/Closed** - Can't add domains without modifying multiple files
33
  - **Flexibility** - Agent is locked to one domain
34
 
@@ -210,7 +211,7 @@ DOMAIN_CONFIGS: dict[ResearchDomain, DomainConfig] = {
210
  DEFAULT_DOMAIN = ResearchDomain.GENERAL
211
 
212
 
213
- def get_domain_config(domain: ResearchDomain | None = None) -> DomainConfig:
214
  """Get configuration for a research domain.
215
 
216
  Args:
@@ -221,6 +222,13 @@ def get_domain_config(domain: ResearchDomain | None = None) -> DomainConfig:
221
  """
222
  if domain is None:
223
  domain = DEFAULT_DOMAIN
 
 
 
 
 
 
 
224
  return DOMAIN_CONFIGS[domain]
225
  ```
226
 
@@ -238,7 +246,7 @@ class Settings(BaseSettings):
238
  research_domain: ResearchDomain = ResearchDomain.GENERAL
239
  ```
240
 
241
- ### 3. Update All 15 Hardcoded Locations
242
 
243
  #### 3.1 Prompts Module
244
 
@@ -250,21 +258,35 @@ def get_system_prompt(domain=None):
250
  config = get_domain_config(domain)
251
  return config.report_system_prompt
252
 
253
- # Keep SYSTEM_PROMPT for backwards compatibility
254
  SYSTEM_PROMPT = get_system_prompt()
255
  ```
256
 
257
  **`src/prompts/judge.py`**:
258
  ```python
259
- from src.config.domain import get_domain_config
260
 
261
  def get_system_prompt(domain=None):
262
  config = get_domain_config(domain)
263
  return config.judge_system_prompt
264
 
265
- def get_scoring_prompt(domain=None):
 
 
 
 
 
 
 
266
  config = get_domain_config(domain)
267
- return config.judge_scoring_prompt
 
 
 
 
 
 
 
268
 
269
  SYSTEM_PROMPT = get_system_prompt()
270
  ```
@@ -280,7 +302,28 @@ def get_system_prompt(domain=None):
280
  SYSTEM_PROMPT = get_system_prompt()
281
  ```
282
 
283
- #### 3.2 Orchestrators
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
284
 
285
  **`src/orchestrators/simple.py`**:
286
  ```python
@@ -288,7 +331,11 @@ from src.config.domain import get_domain_config
288
 
289
  class SimpleOrchestrator:
290
  def __init__(self, domain=None, ...):
 
291
  self.domain_config = get_domain_config(domain)
 
 
 
292
 
293
  def _format_report(self, ...):
294
  return f"""{self.domain_config.report_title}
@@ -308,7 +355,7 @@ async def run_research(..., domain=None):
308
  """
309
  ```
310
 
311
- #### 3.3 Agents
312
 
313
  **`src/agents/magentic_agents.py`**:
314
  ```python
@@ -325,7 +372,7 @@ def create_search_agent(domain=None):
325
  **`src/agents/search_agent.py`** and **`src/agents/tools.py`**:
326
  Similar pattern - inject domain config.
327
 
328
- #### 3.4 MCP Tools
329
 
330
  **`src/mcp_tools.py`**:
331
  ```python
@@ -363,17 +410,17 @@ domain_dropdown = gr.Dropdown(
363
  - [ ] Create `src/config/domain.py` with DomainConfig
364
  - [ ] Add `research_domain` to Settings
365
  - [ ] Update `src/prompts/report.py`
366
- - [ ] Update `src/prompts/judge.py`
367
  - [ ] Update `src/prompts/hypothesis.py`
368
- - [ ] Update `src/orchestrators/simple.py`
 
369
  - [ ] Update `src/orchestrators/advanced.py`
370
  - [ ] Update `src/agents/magentic_agents.py`
371
  - [ ] Update `src/agents/search_agent.py`
372
  - [ ] Update `src/agents/tools.py`
373
  - [ ] Update `src/mcp_tools.py`
374
  - [ ] Add domain selector to Gradio UI
375
- - [ ] Write unit tests for domain config
376
- - [ ] Update CLAUDE.md, AGENTS.md, GEMINI.md
377
 
378
  ## Testing Strategy
379
 
@@ -422,21 +469,13 @@ async def test_simple_mode_respects_domain():
422
 
423
  1. **Phase 1**: Create domain config, add to Settings (no breaking changes)
424
  2. **Phase 2**: Update prompts module to use config (backwards compatible)
425
- 3. **Phase 3**: Update orchestrators (backwards compatible via defaults)
426
- 4. **Phase 4**: Update UI with domain selector
427
- 5. **Phase 5**: Update docs and examples
428
 
429
  ## Success Criteria
430
 
431
- - [ ] Zero hardcoded "drug repurposing" strings in `src/`
432
- - [ ] `grep -r "drug repurposing" src/` returns only `domain.py`
433
- - [ ] All existing tests pass
434
  - [ ] New domain can be added by only modifying `domain.py`
435
- - [ ] Default behavior unchanged (general domain)
436
-
437
- ## Rollback Plan
438
-
439
- All changes are backwards compatible:
440
- - Default domain = GENERAL (similar to current behavior)
441
- - Existing APIs unchanged (domain is optional parameter)
442
- - No database migrations required
 
7
 
8
  ## Problem Statement
9
 
10
+ The codebase has "drug repurposing" hardcoded in **16 locations** (originally identified 15, plus 1 found in audit):
11
 
12
  ```
13
  src/prompts/report.py:11 - SYSTEM_PROMPT
14
  src/prompts/judge.py:5 - SYSTEM_PROMPT
15
+ src/prompts/judge.py:140 - Evidence scoring prompt (inside format_user_prompt)
16
  src/prompts/hypothesis.py:11 - SYSTEM_PROMPT
17
  src/orchestrators/simple.py:476 - Report header
18
  src/orchestrators/simple.py:564 - Report header
 
25
  src/mcp_tools.py:116 - Docstring
26
  src/mcp_tools.py:164 - Function docstring
27
  src/mcp_tools.py:167 - Docstring
28
+ src/agent_factory/judges.py:21 - Imports format_user_prompt (needs update)
29
  ```
30
 
31
  This violates:
32
+ - **DRY** - Same concept repeated 15+ times
33
  - **Open/Closed** - Can't add domains without modifying multiple files
34
  - **Flexibility** - Agent is locked to one domain
35
 
 
211
  DEFAULT_DOMAIN = ResearchDomain.GENERAL
212
 
213
 
214
+ def get_domain_config(domain: ResearchDomain | str | None = None) -> DomainConfig:
215
  """Get configuration for a research domain.
216
 
217
  Args:
 
222
  """
223
  if domain is None:
224
  domain = DEFAULT_DOMAIN
225
+
226
+ if isinstance(domain, str):
227
+ try:
228
+ domain = ResearchDomain(domain)
229
+ except ValueError:
230
+ domain = DEFAULT_DOMAIN
231
+
232
  return DOMAIN_CONFIGS[domain]
233
  ```
234
 
 
246
  research_domain: ResearchDomain = ResearchDomain.GENERAL
247
  ```
248
 
249
+ ### 3. Update All Hardcoded Locations
250
 
251
  #### 3.1 Prompts Module
252
 
 
258
  config = get_domain_config(domain)
259
  return config.report_system_prompt
260
 
261
+ # Keep SYSTEM_PROMPT for backwards compatibility (uses default)
262
  SYSTEM_PROMPT = get_system_prompt()
263
  ```
264
 
265
  **`src/prompts/judge.py`**:
266
  ```python
267
+ from src.config.domain import get_domain_config, ResearchDomain
268
 
269
  def get_system_prompt(domain=None):
270
  config = get_domain_config(domain)
271
  return config.judge_system_prompt
272
 
273
+ def format_user_prompt(
274
+ question: str,
275
+ evidence: list[Evidence],
276
+ iteration: int = 0,
277
+ max_iterations: int = 10,
278
+ total_evidence_count: int | None = None,
279
+ domain: ResearchDomain | None = None, # NEW ARGUMENT
280
+ ) -> str:
281
  config = get_domain_config(domain)
282
+ # ... existing logic ...
283
+
284
+ # Inside f-string:
285
+ return f"""...
286
+ {config.judge_scoring_prompt}
287
+ DO NOT decide "synthesize" vs "continue" - that decision is made by the system.
288
+ ...
289
+ """
290
 
291
  SYSTEM_PROMPT = get_system_prompt()
292
  ```
 
302
  SYSTEM_PROMPT = get_system_prompt()
303
  ```
304
 
305
+ #### 3.2 Judge Factory
306
+
307
+ **`src/agent_factory/judges.py`**:
308
+ ```python
309
+ from src.config.domain import ResearchDomain
310
+
311
+ class JudgeHandler:
312
+ def __init__(self, model: Any = None, domain: ResearchDomain | None = None) -> None:
313
+ self.model = model or get_model()
314
+ self.domain = domain # Store domain
315
+ # ...
316
+
317
+ async def assess(self, ...):
318
+ # ...
319
+ if evidence:
320
+ user_prompt = format_user_prompt(
321
+ ...,
322
+ domain=self.domain # Pass domain
323
+ )
324
+ ```
325
+
326
+ #### 3.3 Orchestrators
327
 
328
  **`src/orchestrators/simple.py`**:
329
  ```python
 
331
 
332
  class SimpleOrchestrator:
333
  def __init__(self, domain=None, ...):
334
+ self.domain = domain
335
  self.domain_config = get_domain_config(domain)
336
+
337
+ # Pass domain to JudgeHandler
338
+ self.judge = JudgeHandler(domain=domain)
339
 
340
  def _format_report(self, ...):
341
  return f"""{self.domain_config.report_title}
 
355
  """
356
  ```
357
 
358
+ #### 3.4 Agents
359
 
360
  **`src/agents/magentic_agents.py`**:
361
  ```python
 
372
  **`src/agents/search_agent.py`** and **`src/agents/tools.py`**:
373
  Similar pattern - inject domain config.
374
 
375
+ #### 3.5 MCP Tools
376
 
377
  **`src/mcp_tools.py`**:
378
  ```python
 
410
  - [ ] Create `src/config/domain.py` with DomainConfig
411
  - [ ] Add `research_domain` to Settings
412
  - [ ] Update `src/prompts/report.py`
413
+ - [ ] Update `src/prompts/judge.py` (Add domain arg to `format_user_prompt`)
414
  - [ ] Update `src/prompts/hypothesis.py`
415
+ - [ ] Update `src/agent_factory/judges.py` (Pass domain to `format_user_prompt`)
416
+ - [ ] Update `src/orchestrators/simple.py` (Pass domain to `JudgeHandler`)
417
  - [ ] Update `src/orchestrators/advanced.py`
418
  - [ ] Update `src/agents/magentic_agents.py`
419
  - [ ] Update `src/agents/search_agent.py`
420
  - [ ] Update `src/agents/tools.py`
421
  - [ ] Update `src/mcp_tools.py`
422
  - [ ] Add domain selector to Gradio UI
423
+ - [ ] **Update Tests**: `tests/e2e/test_simple_mode.py` contains hardcoded "Drug Repurposing" assertions that will fail with default "General" domain.
 
424
 
425
  ## Testing Strategy
426
 
 
469
 
470
  1. **Phase 1**: Create domain config, add to Settings (no breaking changes)
471
  2. **Phase 2**: Update prompts module to use config (backwards compatible)
472
+ 3. **Phase 3**: Update `JudgeHandler` and `format_user_prompt` (requires careful threading of domain)
473
+ 4. **Phase 4**: Update orchestrators and agents
474
+ 5. **Phase 5**: Update UI with domain selector and Fix Tests
475
 
476
  ## Success Criteria
477
 
478
+ - [ ] Zero hardcoded "drug repurposing" strings in `src/` (except `domain.py`)
479
+ - [ ] All existing tests pass (after updates)
 
480
  - [ ] New domain can be added by only modifying `domain.py`
481
+ - [ ] Default behavior is "General Research"