backend/DESIGN.md

# RMI Backend — 2026 Architecture Design

## Why this exists

The current backend (`/root/backend/`) has grown organically:
- `main.py` is 10,305 lines
- 124 router files flat in `app/routers/`
- 14 RAG modules scattered at top of `app/`
- `token_scanner.py` is 4,109 lines
- `x402_tools.py` is 5,817 lines
- Cross-cutting concerns (redis, auth, errors) duplicated across modules
- Domain logic entangled with FastAPI
- No tests, no type safety, no clear boundaries

15 mechanical refactor tasks would patch symptoms. This design fixes the architecture.

## Target Layout

```
/root/backend/
├── pyproject.toml              uv + ruff + mypy + pytest config (single source)
├── .pre-commit-config.yaml     ruff + mypy + size cap + gitleaks
├── Dockerfile
├── alembic/                    async migrations
│
├── app/
│   ├── main.py                 <100 lines: app factory + lifespan + middleware ONLY
│   ├── config.py               pydantic-settings, env loading
│   │
│   ├── core/                   cross-cutting, NO business logic
│   │   ├── logging.py          structlog JSON + correlation ID
│   │   ├── errors.py           AppError hierarchy + FastAPI handlers
│   │   ├── redis.py            async client + get_redis() Depends()
│   │   ├── db.py               async SQLAlchemy session
│   │   ├── auth.py             JWT decode + role guards
│   │   ├── lifespan.py         startup/shutdown
│   │   ├── middleware.py       CORS, rate limit, correlation ID
│   │   ├── websocket.py        WS connection manager
│   │   ├── tracing.py          OpenTelemetry + Langfuse v4 init
│   │   ├── http.py             async httpx client
│   │   ├── pagination.py       cursor-based
│   │   ├── metrics.py          Prometheus /metrics endpoint (P0 v3)
│   │   └── legacy.py           @legacy decorator for deprecation tracking
│   │
│   ├── api/                    HTTP transport, thin routes
│   │   ├── deps.py             shared Depends (current_user, redis, etc)
│   │   ├── v1/
│   │   │   ├── public/         no auth  — scanner, wallet, token, pricing, health
│   │   │   ├── auth/           JWT      — portfolio, alerts, intel, profile
│   │   │   ├── admin/          admin    — users, system, ops
│   │   │   ├── x402/           paid     — tools, tokens, wallets, defi, security
│   │   │   └── mcp/            MCP      — tools.py
│   │   └── ws/                 WebSocket
│   │       └── alerts.py
│   │
│   ├── domain/                 pure business logic, NO FastAPI imports
│   │   ├── scanner/            core + honeypot + rugcheck + holders + contract + deployer + models + service
│   │   ├── wallet/             analyzer + labels + behavior + models + service
│   │   ├── token/              discovery + supply + models + service
│   │   ├── rag/                embeddings + chunking + search + ingest + firehose + feedback + agentic + evaluation + tracing + router + permanence + models + service
│   │   ├── x402/               facilitator + tokens + enforcement + settlement + models + service
│   │   ├── intel/              feeds + narratives + graph + models + service
│   │   ├── scam/               classifier + patterns + models + service
│   │   ├── databus/            client + chains(96) + models + service
│   │   └── bulletin/           board + models + service
│   │
│   ├── infra/                  external integrations
│   │   ├── ollama.py
│   │   ├── langfuse.py
│   │   ├── vector_store.py
│   │   ├── chains/             evm + solana + bitcoin + base + ...
│   │   ├── apis/               coingecko + etherscan + birdeye + goplus + arkham + dune + ...
│   │   └── providers/          ollama + openrouter + huggingface + ...
│   │
│   ├── agents/                 v3 NEW — bounded-task AI agent loops (M1)
│   │   ├── loop.py             frozen: TaskInput/LoopBudget/TaskOutput Pydantic models
│   │   ├── fact_store.py       Redis-backed long-term memory
│   │   ├── kill_switches.py    budget enforcers (iterations, tokens, wall, spend)
│   │   └── executor.py         bounded loop runner
│   │
│   ├── mcp/                    v3 NEW — MCP mesh v2 (M2)
│   │   ├── manifest.py         frozen: MCPToolManifest Pydantic model
│   │   ├── registry.py         FastAPI on :8643 — list/resolve tools
│   │   ├── auth.py             gopass-backed auth scopes
│   │   └── servers/            opencti + dify + n8n FastMCP servers
│   │
│   ├── middleware/             v3 NEW — Starlette middleware classes (M7)
│   │   ├── cost_tracking.py    frozen: per-tenant/per-route cost middleware
│   │   ├── rate_limit_v2.py    per-tier sliding window
│   │   ├── auth_deps.py        JWT/API key on legacy routes
│   │   ├── security_headers.py CSP, HSTS, X-Frame-Options
│   │   └── shadow_traffic.py   v1/v2 response diffing
│   │
│   └── workers/                background jobs (separate from API)
│       ├── firehose.py
│       ├── scanner_queue.py
│       ├── ingest_cron.py
│       └── cleanup.py
│
├── docs/
│   ├── adr/                    v3 NEW — Architecture Decision Records (M6)
│   ├── runbooks/               v3 NEW — incident response (M6)
│   ├── postmortems/            v3 NEW — incident learnings (M6)
│   ├── oncall/                 v3 NEW — rotation doc (M6)
│   ├── load/                   v3 NEW — k6 baseline reports (P2 #26)
│   ├── contract/               v3 NEW — Pact contract specs (P2 #27)
│   └── plans/
│       ├── UNFUCK-V1.md        Frozen — superseded
│       ├── UNFUCK-V2.md        Frozen — superseded
│       ├── UNFUCK-V3.md        ACTIVE — AI-Forward Ship Edition (this document's parent)
│       └── INTEGRATION-LOG.md  append-only worklog of v3 execution
│
└── tests/
    ├── conftest.py
    ├── unit/
    │   ├── core/               33+ core tests
    │   └── domain/             58+ domain tests
    ├── e2e/                    v3 NEW — Playwright (P2 #25)
    ├── contract/               v3 NEW — Pact (P2 #25)
    ├── load/                   v3 NEW — k6 scripts (P2 #25)
    └── rag/
        ├── eval_suite.py       v3 NEW — retrieval quality regression (M3)
        └── prompts/
            └── test_snapshots.py v3 NEW — prompt regression
```

## Key Design Principles

1. **STRICT LAYERING.** `api → domain → infra`. Never reverse. Domain knows nothing about HTTP.
2. **ONE SOURCE OF TRUTH for cross-cutting.** redis/auth/errors/logging live in `core/` exactly once. Routes import, never redefine.
3. **HARD SIZE CAP.** 500 lines per file. Enforced in pre-commit. No 4,109-line `token_scanner.py` ever again.
4. **THIN ROUTES.** Routes parse → call service → return. No business logic in HTTP layer.
5. **DOMAIN = PURE PYTHON.** `domain/scanner/` can be unit tested without spinning up FastAPI. This is the test that proves the architecture.
6. **WORKERS SEPARATED.** Background jobs don't pollute the API. firehose, scanner_queue, ingest_cron live in `workers/`.
7. **PYDANTIC V2 EVERYWHERE.** Every domain has `models.py`. No `dict` types crossing boundaries.
8. **ASYNC-ONLY.** No sync I/O in handlers. Same shape for the whole codebase.
9. **OBSERVABILITY BY DEFAULT.** structlog JSON + correlation ID + OTel + Langfuse in `core/tracing.py`. Every endpoint instrumented without opt-in.
10. **STRANGLER FIG MIGRATION.** New skeleton co-exists with old code. Old `main.py` keeps importing the old routers. New routes added alongside. Per-domain cutover, not big-bang.

## Migration Order

| Order | Domain | Why |
|-------|--------|-----|
| 0 | `rag_engine` shim | unblock prod crash, temp until `app/rag/` lands |
| 1 | `core/` | foundation everyone depends on |
| 2 | `infra/` | external integrations domain depends on |
| 3 | `alerts` | smallest, well-bounded, has WS + JWT + redis — proves full pattern |
| 4 | `wallet` | high-value, used by frontend |
| 5 | `token` | high-value |
| 6 | `scanner` | biggest (4,109 lines), do last when pattern is mature |
| 7 | `x402` | payment system, critical, mature pattern by then |
| 8 | `intel`, `scam`, `databus`, `bulletin` | long tail |
| 9 | `rag` consolidation (was 14 files) | last because it's the most coupled |

---

# THE 14-POINT MODERN BUILDER BAR (v3 — non-negotiable quality gate)

Every file that touches the unfuck MUST clear this. Pre-commit enforces what can be enforced.

```
#  Standard                                  Status  Enforcement
1  Pydantic v2 strict (ConfigDict)            PARTIAL  pre-commit mypy
2  async-only handlers                       DONE    pre-commit AST scan
3  <500 lines/file (pre-commit)              DONE    pre-commit file-size-cap
4  pre-commit hooks (ruff+mypy+gitleaks+
   detect-secrets+bandit+file-size+pytest)   PARTIAL  install on dev + CI
5  typed errors (18/18) — currently 4/18    4/18    pre-commit AST scan
6  1 type per layer (api→domain→core→infra)  PARTIAL  architecture test
7  structlog everywhere                      DONE    pre-commit no-print()
8  event bus (Redis pub/sub)                 PARTIAL  manual review
9  per-domain /health/{domain}              PARTIAL  manual review
10 per-route cost tracking                   NONE    M7 middleware (P2)
11 no from-main-import                       DONE    pre-commit AST scan
12 single get_redis()                        DONE    pre-commit no-new-get-redis
13 OTel trace coverage >85%                  0%      M4 wiring (P1)
14 Langfuse span coverage >95% LLM calls     PARTIAL  M4 wiring (P1)
```

**Items 5, 10, 13, 14 are the v3 additions.**

---

# THE 30 MUST-DO ITEMS (v3 — ACTIVE PLAN)

## P0 — Week 1 (32h total — 16h human + 16h M3)

| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 1 | main.py lifespan merge crash fixed | 1 | human | ✅ DONE | `main.py` (94 lines) |
| 2 | Audit 53 uncommitted modifications | 3 | human | ⏳ TODO | `git diff HEAD` |
| 3 | 65 broken crons → Tailscale IP | 4 | human | ⏳ TODO | crons 167.86.116.51 → 100.100.18.18 |
| 4 | /metrics endpoint | 2 | M3 | ⏳ TODO | `app/core/metrics.py` (NEW) |
| 5 | AlertManager → Telegram | 2 | M3 | ⏳ TODO | `docker-compose.yml` + `alertmanager.yml` |
| 6 | Embedder dual-dim wrapper (M3) | 8 | M3 | ⏳ TODO | `app/rag/embedder.py` (NEW) |
| 7 | WalletService proper async impl | 6 | M3 | ⏳ STUBS DONE, REAL PENDING | `app/domain/wallet/service.py` |
| 8 | Frontend SPA migrate Contabo→netcup | 2 | human | ⏳ TODO | `/var/www/rmi/` |
| 9 | GHOST_ADMIN_KEY from gopass | 1 | human | ⏳ TODO | `.env` |
| 10 | Load avg 14.19 → identify cause | 3 | human | ⏳ TODO | Erigon or Ollama? |

## P1 — Week 2 (37h total — 3h human + 34h M3)

| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 11 | GlitchTip self-host + wire SDK | 4 | M3 | ⏳ TODO | `docker-compose.glitchtip.yml` (NEW) |
| 12 | OTel auto-instrumentation (M4) | 6 | M3 | ⏳ TODO | `app/core/tracing.py` wire into lifespan |
| 13 | GitHub Actions CI | 8 | M3 | ⏳ TODO | `.github/workflows/ci.yml` (NEW) |
| 14 | Renovate + Dependabot | 2 | M3 | ⏳ TODO | `.github/dependabot.yml` (NEW) |
| 15 | Trivy in CI | 2 | M3 | ⏳ TODO | `.github/workflows/ci.yml` |
| 16 | Semgrep SAST in CI | 2 | M3 | ⏳ TODO | `.github/workflows/ci.yml` |
| 17 | Gitleaks + Trufflehog | 2 | M3 | ⏳ TODO | `.pre-commit-config.yaml` + CI |
| 18 | Vault rotation script (gopass) | 3 | human | ⏳ TODO | `scripts/rotate_secrets.sh` (NEW) |
| 19 | Rate limit middleware on all /api/v1/* | 4 | M3 | ⏳ TODO | `app/core/middleware.py` |
| 20 | Auth Depends() on legacy routes | 4 | M3 | ⏳ TODO | `app/core/middleware.py` |

## P2 — Week 3 (46h total — 12h human + 34h M3)

| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 21 | Security headers (CSP, HSTS, X-Frame) | 3 | M3 | ⏳ TODO | `app/core/middleware.py` |
| 22 | Backup restore drill | 3 | human | ⏳ TODO | R2 bucket test |
| 23 | 4 Grafana dashboards | 5 | M3 | ⏳ TODO | `grafana/dashboards/*.json` |
| 24 | Container memory limits (all 36) | 3 | M3 | ⏳ TODO | `docker-compose.yml` |
| 25 | tests/{e2e,contract,load}/ scaffolding | 4 | M3 | ⏳ TODO | `tests/e2e/` `tests/contract/` `tests/load/` |
| 26 | k6 baselines (top 50 routes) | 6 | M3 | ⏳ TODO | `tests/load/baseline.js` |
| 27 | Pact contracts + openapi-generator SDK | 5 | M3 | ⏳ TODO | `tests/contract/` |
| 28 | Top 10 ADRs | 5 | human | ⏳ TODO | `docs/adr/0001-0010.md` |
| 29 | Runbooks + postmortem + oncall | 4 | human | ⏳ TODO | `docs/runbooks/` `docs/postmortems/` `docs/oncall/` |
| 30 | Agent Loop Spec productionization (M1) | 8 | M3 | ⏳ TODO | `app/agents/loop.py` (NEW, frozen on creation) |

## P3 — Defer until revenue

Multi-region read replicas. SOC2 audit. Paid third-party pen-test. SBOM distribution. AsyncAPI WebSocket spec. Frontend RUM (Sentry browser SDK, web vitals). Pyroscope continuous profiling. OWASP ZAP.

---

# THE 8 AI-FORWARD MODULES (v3)

## M1: Agent Loop Productionization (P2 #30)

**Deliverable: `app/agents/loop.py` (frozen on creation)**

```python
from pydantic import BaseModel, Field, ConfigDict

class TaskInput(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True)
    task_id: str = Field(..., pattern=r"^\d+-[a-z0-9-]+$")
    description: str = Field(..., max_length=500)
    context_budget_tokens: int = Field(default=8000, le=32000)
    allowed_tools: list[str] = Field(..., min_length=1)
    success_criteria: str = Field(..., max_length=300)
    verify_command: str = Field(..., max_length=200)

class LoopBudget(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True)
    max_iterations: int = Field(default=20, le=100)
    max_tokens: int = Field(default=50_000, le=500_000)
    max_wallclock_seconds: int = Field(default=900, le=3600)
    max_spend_usd: float = Field(default=5.0, le=50.0)

class TaskOutput(BaseModel):
    task_id: str
    files_touched: list[str]
    lines_added: int
    lines_removed: int
    verify_passed: bool
    worklog_entry: str
    spend_usd: float
    iterations_used: int
```

**Kill switches** (every iteration):
- max_iterations ≤ 20 (max 100)
- max_tokens ≤ 50K (max 500K)
- max_wallclock ≤ 900s (max 3600s)
- max_spend ≤ $5 (max $50)
- verify_command_failed_3x → human review

**fact_store companion** (`app/agents/fact_store.py`):
- Redis key: `fact:{namespace}:{key}`
- TTL: 86400s
- Loaded at loop start, written at loop end
- Prevents re-discovery of known facts

**Acceptance gate: 7 consecutive days zero runaway.**

## M2: MCP Mesh v2 (P2 #24, D9)

**Deliverable: `app/mcp/manifest.py` (frozen on creation)**

```python
from pydantic import BaseModel, Field
from typing import Any

class MCPToolManifest(BaseModel):
    name: str = Field(..., pattern=r"^[a-z_]+:[a-z_]+$")  # e.g. "rmi-netcup:docker_ps"
    version: str = Field(..., pattern=r"^\d+\.\d+\.\d+$")
    server: str = Field(..., pattern=r"^[a-z-]+$")
    description: str = Field(..., max_length=200)
    input_schema: dict[str, Any]
    output_schema: dict[str, Any]
    auth_scope: str = Field(..., pattern=r"^[a-z_]+:[a-z_]+$")  # gopass scope
    deprecated: bool = False
    successor: str | None = None
```

**Mesh registry on netcup `:8643`** with:
- GET /tools (list all with filters)
- GET /tools/{name} (full manifest)
- POST /resolve (name+MAJOR version → executor endpoint)

**3 unwired servers to wire in D9:**
- OpenCTI (40 tools) — wire with FastMCP
- Dify (18 tools) — wire with FastMCP
- n8n (20 tools) — wire with FastMCP

**Acceptance gate: 332+ tools versioned, all gopass-backed auth, OpenCTI+Dify+n8n wired.**

## M3: RAG Dual-Dim Wrapper (P0 #6)

**Deliverable: `app/rag/embedder.py` (wraps frozen `app/crypto_embeddings.py`)**

```python
from enum import Enum

class EmbedderBackend(Enum):
    BGE_M3 = "bge-m3"                   # 1024d, legacy
    QWEN3_4B = "qwen3-embedding:4b"     # 2048d, target

class DualDimEmbedder:
    def __init__(self, backend: EmbedderBackend):
        self.backend = backend
        self.dim = 1024 if backend == EmbedderBackend.BGE_M3 else 2048

    async def embed(self, texts: list[str]) -> list[list[float]]:
        return await self._ollama_embed(texts, self.backend.value)

    async def reindex_collection(self, name: str, target: EmbedderBackend) -> bool:
        old = DualDimEmbedder(EmbedderBackend.BGE_M3)
        new = DualDimEmbedder(target)
        # 1. Read all docs from collection
        # 2. Re-embed with new backend
        # 3. Write to new collection {name}_v2
        # 4. Atomic swap: {name} → {name}_legacy, {name}_v2 → {name}
        # 5. Verify: query 10 known docs, confirm retrieval
        return True
```

**The math:**
| Metric | bge-m3 | qwen3-4b | Delta |
|--------|--------|----------|-------|
| Dimensions | 1024 | 2048 | +2x |
| RAM (loaded) | 1.2GB | 2.0GB | +800MB |
| Embed latency | ~80ms | ~140ms | +60ms |
| Retrieval (nDCG@10) | 0.71 | 0.83 (est) | +12% |
| Reindex time | — | 6h overnight | one-time |
| Storage/coll | ~50MB | ~100MB | +650MB total |

**Strategy: collection-by-collection overnight, smallest first, verify each morning.** If <8%, defer.

**Acceptance gate: All 13 collections reindexed, eval harness +8% nDCG@10, bge-m3 unloaded.**

## M4: Observability Triangle (P1 #11, #12, P2 #21)

**Three pillars wired into the 94-line lifespan:**

```python
# main.py additions (~15 lines to existing 94)
from contextlib import asynccontextmanager
from app.core.tracing import setup_otel
from app.core.langfuse import langfuse
from app.core.errors import init_glitchtip

@asynccontextmanager
async def lifespan(app: FastAPI):
    _otel = setup_otel()        # OTel: httpx, redis, asyncpg, neo4j
    init_glitchtip()             # Sentry-compatible SDK
    await app.state.redis.initialize()
    yield
    _otel.shutdown()
    langfuse.flush()
    await app.state.redis.close()

app = FastAPI(lifespan=lifespan)
```

**Pillars:**
- **OTel**: auto-instrument httpx/redis/asyncpg/neo4j, export to local OTel collector
- **Langfuse**: `@observe()` decorator on every LLM call, >95% coverage
- **GlitchTip**: Sentry-compatible SDK, DSN from gopass, captures every unhandled exception

**Acceptance gate: 7 consecutive days triangle lit.**

## M5: CI/CD Standards (P1 #13-17, P1 #19, P1 #20)

**Pipeline stages:**
```
PRE-COMMIT (local):  ruff + mypy + gitleaks + detect-secrets + bandit + file-size + pytest-smoke
CI (GH Actions):     lint + test-full + build + trivy + semgrep + gitleaks-history + grype
DEPLOY (blue-green): build-green → start → health-gate → swap-router → verify-5xx → keep-blue-10min
ROLLBACK (auto):     swap-back-to-blue on 5xx spike → page → freeze-deploys-1h
```

**Justfile deploy target:**
```makefile
deploy:
    docker compose -f docker-compose.green.yml build
    docker compose -f docker-compose.green.yml up -d
    ./scripts/health-check.sh http://127.0.0.1:8001/health 60
    ./scripts/swap-upstream.sh green
    ./scripts/watch-errors.sh 600 0.01
    docker compose -f docker-compose.blue.yml stop
```

## M6: ADRs + Runbooks + Postmortems + Oncall (P2 #28, #29)

**Top 10 ADRs (write these first):**
1. Why FastAPI over Litestar/Flask/Django
2. Why 5 DBs (Postgres+Redis+CH+Neo4j+Qdrant)
3. Why strangler-fig over rewrite
4. Why bge-m3 → qwen3-embedding:4b (M3)
5. Why self-host GlitchTip over Sentry SaaS
6. Why single-VPS blue-green over k3s
7. Why gopass over Vault/Doppler/AWS SM
8. Why Hermes cron over systemd/Airflow
9. Why MCP mesh over direct API calls
10. Why Tailscale over WireGuard/ZeroTier

**ADR Template:**
```
# ADR-XXXX: [Title]
## Status: Accepted | Rejected | Superseded
## Date: YYYY-MM-DD
## Decider: @cryptorugmunch
## Context: What we're solving
## Decision: What we chose
## Alternatives Considered: What we rejected
## Consequences: Easier / Harder / Mitigations
```

## M7: Cost Tracking + SLOs + Error Budgets (P2 #21)

**Per-tenant cost middleware (`app/middleware/cost_tracking.py`):**
```python
class CostTrackingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        tenant = request.headers.get("X-Tenant-ID", "anonymous")
        t0 = time.monotonic()
        response = await call_next(request)
        elapsed_ms = (time.monotonic() - t0) * 1000
        cost = self._estimate_cost(route, response, elapsed_ms)
        await self._buffer_cost(tenant, route, cost, elapsed_ms)
        return response
```

**SLO Targets:**
| Service | SLO | Error budget |
|---------|-----|--------------|
| /api/v1/* p99 latency | < 500ms | 43min/mo |
| /api/v2/* p99 latency | < 200ms | 43min/mo |
| /health uptime | ≥ 99.5% | 3.6h/mo |
| /api/* error rate (5xx) | < 0.1% | 43min of 5xx/mo |
| RAG retrieval success | > 95% | 43min of failures/mo |
| LLM calls timeout rate | < 1% | 7.2h of timeouts/mo |

**Auto-freeze deploys on 4x burn:**
```yaml
- alert: SLOBurnRate4x
  expr: (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > 0.02
  for: 2m
  annotations:
    webhook: "http://127.0.0.1:8642/ci/freeze"
```

## M8: AI Tool Wiring Matrix

| Task | Best tool | Worst tool |
|------|-----------|------------|
| Bounded code edit (1 file, <50 lines) | **aider** | claude-code |
| Multi-file refactor (3–10 files) | **claude-code** | aider |
| Codebase search | **Bloop** (when installed) | grep |
| Test generation | **claude-code** | aider |
| Doc generation (ADR, runbook) | **GLM-5.2** | aider |
| Security review | **Semgrep** (not LLM) | any LLM |
| Dependency upgrade | **Renovate** (not LLM) | any LLM |
| Prompt engineering | **GLM-5.2 + DSPy** | claude-code |
| RAG eval | Custom harness + LLM-as-judge | any LLM alone |
| Long-context (≥100K) | **Kimi K2.7** (256K) | DeepSeek (slow) |
| Vision | **Gemini 2.5-pro** | text-only LLM |
| Fast small (≤500 tokens) | **Groq** | DeepSeek |
| Autonomous (bounded loop) | **Hermes** | claude-code |

---

# 10 INCIDENT SCENARIOS (v3)

## v2's 7 (preserved)
1. rmi-redis crashes at 3am → auto-restart
2. Disk full on netcup → AlertManager page at 80%
3. Postgres corruption → restore from nightly backup
4. SSL cert expiry → Cloudflare manages, Tailscale renew
5. OOM kill on rmi-backend → Docker limit + py-spy/memray
6. DDoS on rugmunch.io → Cloudflare "Under Attack" mode
7. DeepSeek balance hits $0 → failover to Ollama Cloud/Gemini

## v3's 3 NEW (AI-specific)

### Scenario 8 — Agent Loop Runaway
**Symptom:** LLM spend spikes ($50+/h). Hermes logs show task iterating >50×.
**Detect:** Daily spend alert (M7) at $20/h. Hermes /cron/health shows iterations >20.
**Mitigate:** `pkill -f "hermes.*task_id=X"`. Disable cron job.
**Recover:** Audit input contract — likely missing success_criteria or verify_command.
**Root cause:** vague description or missing verify_command.

### Scenario 9 — RAG Poisoning
**Symptom:** RAG retrieval returns adversarial content. User reports "the AI told me to send funds to X."
**Detect:** Langfuse span shows suspicious source. Eval harness flags quality regression.
**Mitigate:** Disable affected collection. Roll back FAISS index.
**Recover:** Add source allowlist to ingest step.
**Root cause:** open ingestion endpoint or untrusted RSS feed.

### Scenario 10 — MCP Tool Compromise
**Symptom:** Unusual gopass access pattern. MCP tool called from unexpected IP. Lateral movement.
**Detect:** gopass audit log. CrowdSec flags anomalous SSH. Tailscale ACL violation.
**Mitigate:** `gopass rm rmi/infra/compromised/scope` + rotate. Revoke Tailscale node. Block IP.
**Recover:** Audit all gopass accesses in last 30 days. Rotate ALL keys.
**Root cause:** over-broad auth_scope or leaked Tailscale key. SEV1.

---

# FROZEN FILE MANIFEST v3 (14 entries)

| # | File | Why frozen |
|---|------|------------|
| 1 | `main.py` | Front door (was 8475, now 94) |
| 2 | `_legacy_main.py` | Strangler-fig target |
| 3 | `app/core/redis.py` | Single Redis source |
| 4 | `app/core/config.py` | Env var loader |
| 5 | `app/databus/core.py` | 96-chain fetch interface |
| 6 | `app/rag/pipeline.py` | RAG pipeline (frozen for M3) |
| 7 | `app/crypto_embeddings.py` | bge-m3 embedder (replaced via wrapper) |
| 8 | `app/api/v1/__init__.py` | Router registry |
| 9 | `docker-compose.yml` | All 36 containers |
| 10 | `app/core/tracing.py` | OTel (not wired — M4) |
| 11 | `app/agents/loop.py` | M1 deliverable (frozen on creation) |
| 12 | `app/mcp/server.py` | M2 deliverable (frozen on creation) |
| 13 | `app/core/metrics.py` | P0 deliverable (frozen on creation) |
| 14 | `app/middleware/cost_tracking.py` | M7 deliverable (frozen on creation) |

**Rule:** Modified only via bounded-task delegation with Task ID in `/home/z/my-project/worklog.md`. Out-of-process edits reverted.

---

# THE MATH (v3 BRUTAL TRUTH)

```
                              Solo          With M3
P0 + P1 + P2 hours            115h          115h
Available budget (21d×8h)    168h          168h human + 152h M3 = 320h
Buffer (30% incident tax)     −50h          −50h human, −46h M3
Effective capacity           118h          224h
NET (capacity − work)        +3h           +109h
                                       ↑ Phase 4 fits here

Plus 20h passive waits (eval soaks, mesh verifications)
→ Solo: 138h needed vs 118h available — slips to Day 25
→ M3: 138h needed vs 224h available — fits comfortably
```

**Verdict: Solo, v3 is a 25-day plan. With M3 parallel, v3 is a 21-day plan.**

---

# 90-DAY ROADMAP (after the unfuck)

## Month 2 — Agent Mesh + RAG Eval + SDK
- Agent mesh: Hermes orchestrating multiple claude-code/aider in parallel
- RAG eval harness: automated retrieval-quality regression on every push
- SDK v1: TypeScript + Python clients from OpenAPI spec

## Month 3 — Multi-Region + SOC2 + Paid Pilots
- Multi-region read replicas (Contabo → Cloudflare LB)
- SOC2 prep (ADRs + runbooks = 60% of evidence)
- 3 paid pilots, $500/mo each, 60-day commitment

## Month 4+ — Hire or Stay Solo
- If 3 pilots + 1 conversion → hire senior backend engineer
- If pilots but no conversion → focus on conversion
- If no pilots → pivot

---

# What Ships This Pass (Foundation)

1. ✅ Fix crash — `rag_engine` re-export shim, backend healthy (Phase 0)
2. ✅ `pyproject.toml` — uv + ruff + mypy strict + pytest
3. ✅ `.pre-commit-config.yaml` — ruff + mypy + size cap (500) + gitleaks
4. ✅ `app/core/` — 11 modules, each <200 lines
5. ✅ `app/api/v1/__init__.py` — router aggregator
6. ✅ `main.py` — 94 lines (Phase 0 just restored)
7. ✅ Verify: backend boots, all 1249 routes respond, health 200
8. ✅ Commit + deploy (`4a8a16d`)

## What Does NOT Ship This Pass (v3 P0-P2)

- Migrating alerts/wallet/token/scanner to new `domain/`. That's the current unfuck.
- The 15 mechanical refactors. Replaced with the layered architecture.
- Deleting old code. Strangler fig — old stays until domain is migrated.
- **v3 new**: M1 Agent Loop, M2 MCP Mesh, M3 RAG dual-dim, M4 Observability Triangle, M5 CI/CD, M6 ADRs, M7 Cost+SLOs, M8 AI Tool Matrix.

## Phase 2: Alerts Vertical Slice (proves the pattern) — COMPLETE

After foundation lands, migrate `alerts` end-to-end as the reference:
```
app/domain/alerts/
├── models.py      # Alert, AlertRule, Notification — Pydantic v2
├── repository.py  # async SQLAlchemy queries
├── service.py     # business logic, pure Python
└── broadcaster.py # WebSocket broadcast helper

app/api/v1/auth/alerts.py  # thin route: parse → call service → return
```

This proves the pattern works: domain is pure Python, route is <100 lines, can be unit tested without HTTP.