Blame for docs/DESIGN.md

# Notestack: Design Doc

Status: draft v0.1 (2026-09-22)
Owner: Firebird Technologies

> Note: Section 0 of the product brief was not included in the handoff. This doc reconstructs the
> product framing from Sections 5 to 12 plus the kickoff decisions. Replace Section 1 with the
> original Section 0 text when available.

## 1. Product

Notestack turns a writer's archive (Substack, Ghost, Medium, any RSS feed or URL) into a grounded
research notebook and a launch engine. Writers can:

- Ask questions of their own archive and get answers with citations to their posts.
- Generate a two-host audio overview of a post or notebook (ElevenLabs).
- Generate videos (explainer 16:9, vertical short 9:16, audiogram 1:1) rendered with Remotion.
- Generate a Launch Kit per post: threads, LinkedIn, Substack Notes, Bluesky, hooks, SEO pack,
  quote cards, carousels, all in the writer's own voice.
- Schedule everything on the Launchpad calendar and resurface evergreen posts.

Every factual statement Notestack generates must trace back to the writer's own sources.

## 2. Kickoff decisions

| Area | Decision |
| --- | --- |
| Backend | FastAPI + SQLAlchemy 2 + Alembic (same stack as blog2video) |
| Worker | Postgres backed queue: the `jobs` table, claimed with `FOR UPDATE SKIP LOCKED`; progress on the row, streamed as SSE. No Redis. |
| Database | Postgres 16 (no vector extension) |
| Frontend | React + Vite + TypeScript, plain CSS with design tokens |
| Renderer | Node service running Remotion (`renderer/`) |
| Storage | Cloudflare R2 (S3 API) for raw HTML, audio, video, stills, compiled DSPy programs |
| Email | Resend, behind a provider interface (mirrors blog2video `EmailService`) |
| Auth | Google sign-in + email 6-digit code signup + password login, JWT access/refresh with `token_version` revocation (mirrors blog2video) |
| LLM | Provider agnostic through DSPy + LiteLLM. Default Z.ai GLM-5.3 (fast tier GLM-5.3-Flash) via its OpenAI compatible endpoint. Retrieval is agentic search over a file system corpus, not embeddings (see 3.2). Any LiteLLM model string works by env var. |
| Billing | Three tiers defined in code (`app/services/plans.py`). `BILLING_ENABLED=false` for now: everyone gets the top tier limits free, checkout returns 409, checkout is not wired up yet. |
| Blog | Typed `blogPosts.ts` content array, `/blogs` and `/blogs/:slug`, same shape as blog2video |

## 3. Architecture

```
            +-------------------+        SSE / REST        +--------------------+
 browser -->|  frontend (Vite)  |------------------------->|  api (FastAPI)     |
            +-------------------+                          |  auth, notebooks,  |
                     ^                                     |  billing, storage  |
                     | presigned GET                       +---------+----------+
                     |                                               | insert queued job
            +--------+----------+                          +---------v----------+
            | Cloudflare R2     |<-------------------------|  worker            |
            | notestack-assets  |   put objects            |  ingest, corpus,   |
            +--------+----------+                          |  DSPy, TTS         |
                     ^                                     +---------+----------+
                     | upload final render                           | POST /render
            +--------+----------+   progress callbacks     +---------v----------+
            | renderer (Remotion)|------------------------>|  api /internal/... |
            +-------------------+                          +--------------------+
     Postgres: data + job queue (api, worker)
```

### 3.1 Storage pipeline (R2)

All binary and large text payloads live in R2. Postgres stores only keys and metadata.

Key layout (`app/services/storage.py::keys`):

```
ws/{workspace_id}/sources/{source_id}/documents/{document_id}/raw.html
ws/{workspace_id}/corpus/manifest.json                           path -> sha256 of every corpus file
ws/{workspace_id}/corpus/INDEX.md                                one line per post
ws/{workspace_id}/corpus/sources/{site}/{date}-{slug}.md         clean post as markdown
ws/{workspace_id}/uploads/{upload_id}/{filename}
ws/{workspace_id}/artifacts/{artifact_id}/{variant}.{ext}      audio, video, stills
ws/{workspace_id}/tts-cache/{sha256(script|voice|settings)}.mp3
system/dspy/{module}/{version}.json                              compiled programs
```

Flows:

1. **Server writes** (ingestion, TTS, compiled programs): worker calls `storage.put_bytes()`.
2. **Browser uploads** (logos, voice consent samples): `POST /api/storage/uploads` returns a presigned
   PUT URL and an `upload_id`; the browser PUTs directly to R2; `POST /api/storage/uploads/{id}/complete`
   verifies with a HEAD request and records size and content type.
3. **Renderer writes**: renderer receives a presigned PUT URL in the render request, so it never holds
   R2 credentials.
4. **Reads**: `GET /api/storage/objects/{key}` checks workspace ownership and 302 redirects to a
   presigned GET (default 1 hour). Public stills (quote cards) can later move to a public bucket with
   a custom domain.

The storage service is an S3 client with R2's endpoint, so MinIO in docker-compose stands in for R2
locally with zero code changes.

### 3.2 Research corpus: file system instead of embeddings

Each post is one markdown file (front matter, `# Title`, `## Section` headings) and the workspace
has an `INDEX.md` with one line per post. Research is an agent loop (`dspy.ReAct`,
`app/pipeline/research.py`) with three tools scoped to the notebook's files (`app/corpus.py`):

| Tool | Does |
| --- | --- |
| `list_files(contains)` | Index lines for the notebook's posts: date, path, title, headings |
| `search(pattern, path)` | Case insensitive regex grep, returns `path:line: text` |
| `read(path, start_line, end_line)` | Numbered lines, 160 per call |

The model searches with several phrasings, reads around the hits, and answers with citations of
the form `(path, line_start, line_end)`. Before anything reaches the writer, every citation is read
back from the file: citations that point outside the notebook, past the end of a file or at empty
lines are dropped, markers are renumbered, and an answer left with no verified citation is replaced
by "not in your posts". The UI streams each step ("Searching for pricing|paid tier", "Reading On
pricing, lines 1 to 40").

Why this over embeddings:
- One vendor. Z.ai's international API has no embedding model, and there is no second key,
  re-embedding job or dimension lock-in.
- Citations are exact line ranges the model actually read, which is stronger grounding than
  nearest-neighbour chunks.
- Writers' archives are small (hundreds to a few thousand posts, a few MB of text), so grep over
  local files takes milliseconds.
- The corpus is plain markdown, so the same files feed podcast scripts, launch kits and a future
  MCP server without conversion.

Trade offs: each question costs several LLM steps instead of one (mitigated by
`LLM_REASONING_EFFORT=low` and the `RESEARCH_MAX_STEPS` cap), and paraphrase matching depends on
the model trying synonyms. If quality or latency needs it later, a Postgres full text index can be
added as a fourth tool without changing the corpus.

Storage and sync: R2 holds the corpus under `ws/{id}/corpus/` with a `manifest.json` of file hashes.
The worker writes files to R2 and its own cache in one pass. The API calls `Corpus.sync()` before
each research run, which reads the manifest and downloads only changed files into
`CORPUS_CACHE_DIR`. Containers share nothing but R2.

Brief items that assumed embeddings: voice similarity uses the LLM judge alone; the topic map is
built from LLM extracted entities per post (Phase 2); resurfacing and SEO internal links search the
corpus with the same tools.

### 3.3 LLM layer

`app/llm/provider.py` builds a `dspy.LM` from env:

```
LLM_MODEL=openai/glm-5.3               # any LiteLLM model string
LLM_FAST_MODEL=openai/glm-5.3-flash    # judges, cleanup, cheap passes
LLM_API_BASE=https://api.z.ai/api/paas/v4
LLM_API_KEY=...
LLM_REASONING_EFFORT=low               # low | high | max
```

Z.ai considerations baked in:
- OpenAI compatible chat endpoint, so the `openai/` LiteLLM prefix plus `api_base` works.
- GLM-5.x always reasons (thinking cannot be disabled). Every call sends `thinking: enabled` with
  `reasoning_effort=low` by default; modules that benefit (the groundedness judge) ask for
  `main_lm("high")`. Reasoning tokens count against `max_tokens`, so the default is 16K to leave
  room for the structured answer. Z.ai recommends temperature 1.0 for GLM-5.x.
- 1M token context on GLM-5.3 means whole-notebook prompts (summaries, podcast scripts) can read
  the files directly; chat uses the research agent to keep citations tight and cost low.
- Tool calling runs through DSPy ReAct, so it works with any provider, not only ones with native
  function calling.
- Token usage from every call is written to `usage_events` (metering from day one).

Swapping to Anthropic, OpenAI, Gemini or a local model is only an env change.

### 3.4 Jobs and progress (Postgres, no Redis)

The `jobs` table is the queue, the progress record and the SSE source.

- **Enqueue:** the API inserts a `queued` row (`create_job`). Nothing else.
- **Claim:** `python -m app.worker` polls every `WORKER_POLL_SECONDS` and takes the oldest runnable
  row with `SELECT ... FOR UPDATE SKIP LOCKED`, so any number of workers can share the table without
  double running a job. Up to `WORKER_CONCURRENCY` jobs run at once in threads.
- **Retry:** a handler that raises is requeued with backoff (30s, 60s, 120s... capped at 15 min)
  until `max_attempts` (default 3), then marked `failed`.
- **Heartbeat:** every progress update stamps `heartbeat_at`. A `running` job silent for
  `JOB_STALE_SECONDS` (worker crashed, renderer died) is requeued by the stale sweeper.
- **Hand off:** render jobs return `HANDED_OFF`; the renderer's progress callbacks keep the row
  alive and mark it done.
- **Progress to the browser:** `GET /api/jobs/{id}/events` re-reads the row about once a second and
  sends only changes. Latency is at most a second, which is fine for progress bars.
- **Schedules:** the worker runs the stale sweep (1 min), update email batch (5 min, gated to the
  campaign send hour) and daily cleanup. Each takes a Postgres advisory lock, so with several
  workers only one runs a given task.

No request blocks for more than a few seconds. Add Redis back only if job volume makes polling a
measurable load on Postgres; the handler code would not change.

## 4. Data model

Implemented in `backend/app/models` and `alembic/versions/0001_initial.py`.

- users, workspaces, workspace_members, subscriptions
- email_verification_codes, update_emails, update_email_sends
- sources, documents (with corpus `path`), notebooks, notebook_documents
- voice_profiles, voice_consents
- chats, messages, citations (document, path, line range, verified quote)
- artifacts, jobs (covers render_jobs and tts_jobs via `kind`), uploads
- calendar_items, tracked_links, engagement_events
- dspy_traces, usage_events

## 5. DSPy program design

See brief Section 6. Signatures live in `backend/app/llm/signatures.py` with Pydantic output models.
Phase 1 ships CleanAndSegment, ResearchArchive (the file research agent) and BuildVoiceProfile. All grounded outputs cite `SourceRef(path, line_start, line_end)`. The rest are stubs with typed
signatures so the schema contract is fixed early.

## 6. Plans

| | Free | Writer | Studio |
| --- | --- | --- | --- |
| Price (when billing turns on) | $0 | $24.99/mo ($18.99/mo billed annually) | $48.99/mo ($36.99/mo billed annually) |
| Sources | 1 | 3 | 10 |
| Indexed posts | 5 | 500 | 5,000 |
| Audio overview minutes / mo | 3 | 60 | 240 |
| Video render minutes / mo | 1 | 30 | 120 |
| Launch Kits / mo | 2 | 50 | unlimited |
| Voice profile | yes | yes | yes |
| Voice cloning (with consent) | no | yes | yes |
| Brand kit, no watermark | no | yes | yes |

While `BILLING_ENABLED=false`, every workspace resolves to Studio limits.

## 7. Design system

Strict palette: `#000000`, `#217cff`, `#FFFFFF` and opacity variants only. Tokens in
`frontend/src/styles/tokens.css`. Landing page is a night sky: canvas starfield with slow parallax,
small twinkling sparkles, a soft blue nebula haze and a neon horizon line. `prefers-reduced-motion`
renders a static sky. Logo slot (`<Logo />`) is a white tile that holds the black silhouette mark;
drop the final file at `frontend/public/logo.svg`.

Copy rule: no em dashes anywhere in UI, email or generated output. The generation layer strips them
as a last pass (`app/llm/postprocess.py`).

## 8. Security and privacy

- JWT access (72h) + refresh, revocation by bumping `users.token_version`.
- One auth provider per account for life (Google or email), same as blog2video.
- Verification codes hashed, 10 minute TTL, 5 attempts, 60s resend cooldown.
- R2 keys are always workspace scoped; the API checks ownership before presigning.
- No training on user content without opt-in (`workspaces.training_opt_in`, default false).
- Voice cloning requires a stored consent record with the recorded sample key.