Merge pull request #125 from The-Obstacle-Is-The-Way/feat/smoke-tests

Makefile

@@ -28,9 +28,19 @@ format:
 typecheck:
 	uv run mypy src
 
-check: lint typecheck test-cov
-	@echo "All checks passed!"
+# Run all checks (lint, typecheck, test)
+check: lint typecheck test
+
+# Smoke tests - run against real APIs (slow, not for CI)
+smoke-free:
+	@echo "Running Free Tier smoke test..."
+	uv run python -m pytest tests/e2e/test_smoke.py::test_free_tier_synthesis -v -s
+
+smoke-paid:
+	@echo "Running Paid Tier smoke test (requires OPENAI_API_KEY)..."
+	uv run python -m pytest tests/e2e/test_smoke.py::test_paid_tier_synthesis -v -s
+
+smoke: smoke-free  # Default to free tier
+
+# Clean up cache and artifacts
 
-clean:
-	rm -rf .pytest_cache .mypy_cache .ruff_cache __pycache__ .coverage htmlcov
-	find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true

SPEC_ARCHITECTURAL_DEBT.md

@@ -230,6 +230,10 @@ class WorkflowState:
 
 **CRITICAL**: Each phase MUST pass smoke tests before merge. Unit tests alone are insufficient.
 
+> **Implementation Status**: IMPLEMENTED (PR #125)
+> Smoke tests are now live in `tests/e2e/test_smoke.py` with Makefile targets.
+> Run `make smoke-free` or `make smoke-paid` before any refactoring PR.
+
 ### Smoke Test Infrastructure
 
 Add to `Makefile`:
@@ -339,17 +343,37 @@ Implement **Priority 1, 2, and 3** before merging PR #124.
 ### Phase 2: Future PRs (Separate Tickets)
 Create GitHub issues for Priority 4-8. Do NOT bloat the current bug fix PR.
 
+**IMPORTANT**: Before starting ANY Priority 4-7 refactors, FIRST implement Priority 8 (Smoke Tests).
+This ensures we can detect regressions from refactoring. Sequence:
+
+1. **PR: Smoke Test Infrastructure** (Priority 8) - MUST BE FIRST
+   - Create `tests/e2e/test_smoke.py`
+   - Add `make smoke-free` and `make smoke-paid` to Makefile
+   - Verify both Free Tier and Paid Tier produce synthesis
+
+2. **PR: Dead Config Cleanup** (Priority 4)
+3. **PR: Prompt Unification** (Priority 5)
+4. **PR: Factory Registry Pattern** (Priority 6)
+5. **PR: WorkflowState Dataclass** (Priority 7)
+
 ---
 
-## Appendix: Line Number Reference
-
-| Item | Current Location |
-|------|------------------|
-| `_handle_timeout()` | Lines 201-248 |
-| `_force_synthesis()` | Lines 250-297 |
-| Redundant imports (timeout) | Lines 207-208 |
-| Redundant imports (force) | Lines 257-258 |
-| Magic string detection | Line 385 |
-| `_get_event_type_for_agent()` | Lines 582-602 |
-| Module imports | Lines 18-48 |
-| `run()` method | Lines 299-456 |
+## Appendix: Line Number Reference (Historical)
+
+> **Note**: These line numbers were from BEFORE Phase 1 refactoring.
+> After PR #124 merge, the following methods were consolidated:
+> - `_handle_timeout()` → DELETED (merged into `_synthesize_fallback`)
+> - `_force_synthesis()` → DELETED (merged into `_synthesize_fallback`)
+> - Redundant imports → REMOVED (centralized at module level)
+> - Magic strings → REPLACED with constants
+
+| Item (Pre-Refactor) | Original Location | Status |
+|---------------------|-------------------|--------|
+| `_handle_timeout()` | Lines 201-248 | DELETED |
+| `_force_synthesis()` | Lines 250-297 | DELETED |
+| Redundant imports (timeout) | Lines 207-208 | REMOVED |
+| Redundant imports (force) | Lines 257-258 | REMOVED |
+| Magic string detection | Line 385 | REFACTORED |
+| `_get_event_type_for_agent()` | Lines 582-602 | REFACTORED |
+| Module imports | Lines 18-48 | UPDATED |
+| `run()` method | Lines 299-456 | UPDATED |

tests/e2e/test_smoke.py

@@ -0,0 +1,65 @@
+"""
+Smoke tests for regression prevention.
+
+These tests run against REAL APIs and verify end-to-end functionality.
+They are slow (2-5 minutes) and should NOT run in CI.
+
+Usage:
+    make smoke-free   # Test Free Tier (HuggingFace)
+    make smoke-paid   # Test Paid Tier (OpenAI BYOK)
+"""
+
+import os
+
+import pytest
+
+from src.orchestrators.advanced import AdvancedOrchestrator
+
+
+@pytest.mark.e2e
+@pytest.mark.timeout(600)  # 10 minute timeout for Free Tier
+async def test_free_tier_synthesis():
+    """Verify Free Tier produces actual synthesis (not just 'Research complete.')"""
+    # Use a simple query that is likely to yield results quickly
+    orch = AdvancedOrchestrator(max_rounds=2)
+
+    events = []
+    print("\nRunning Free Tier Smoke Test...")
+    async for event in orch.run("What is libido?"):
+        if event.type == "complete":
+            events.append(event)
+            print(f"Received complete event: {event.message[:50]}...")
+
+    # MUST have a complete event
+    assert len(events) >= 1, "No complete event received"
+
+    # Complete event MUST have substantive content (not just signal)
+    final = events[-1]
+
+    # P2 Bug Regression Check: Ensure content isn't just "Research complete."
+    assert len(final.message) > 100, f"Synthesis too short: {len(final.message)} chars"
+
+    # P1 Bug Regression Check: Ensure we got actual text
+    assert "Research complete." not in final.message or len(final.message) > 50, (
+        "Got empty synthesis signal instead of actual report"
+    )
+
+
+@pytest.mark.e2e
+@pytest.mark.timeout(300)  # 5 minute timeout for Paid Tier
+async def test_paid_tier_synthesis():
+    """Verify Paid Tier (BYOK) produces synthesis."""
+    api_key = os.environ.get("OPENAI_API_KEY")
+    if not api_key:
+        pytest.skip("OPENAI_API_KEY not set")
+
+    orch = AdvancedOrchestrator(max_rounds=2, api_key=api_key)
+
+    events = []
+    print("\nRunning Paid Tier Smoke Test...")
+    async for event in orch.run("What is libido?"):
+        if event.type == "complete":
+            events.append(event)
+
+    assert len(events) >= 1, "No complete event received"
+    assert len(events[-1].message) > 100, "Synthesis too short"