docs/SPECIFICATION.md

# Collab Editor - Project Specification

## 1. Overview

A collaborative, real-time scientific article editor deployed as a Hugging Face Space. Users write rich content (math, citations, custom components) in a TipTap-based editor synced via Yjs/Hocuspocus, then publish a self-contained static HTML article. An AI assistant helps with writing and editing.

**Stack**: React 18 + TipTap 3 + Yjs (frontend) / Express + Hocuspocus + Node 20 (backend) / Docker on HF Spaces. No CSS-in-JS - all styling via vanilla CSS custom properties.

**Relationship to `research-article-template`**: the CSS foundation, design tokens, and visual language come from the [research-article-template](https://huggingface.co/spaces/tfrere/research-article-template) project. The editor imports the template's CSS files (`_variables.css`, `_reset.css`, `_base.css`, `_layout.css`, component partials) and the publisher injects them inline into published HTML. The published output is designed to look identical to articles built with the Astro-based template.

---

## 2. Architecture

```mermaid
graph TB
  subgraph browser [Browser]
    SPA["React SPA<br/>TipTap + Yjs"]
  end

  subgraph server [Node Backend - port 8080]
    Express["Express HTTP"]
    Hocuspocus["Hocuspocus<br/>WebSocket"]
    Publisher["Publisher Pipeline<br/>HTML + PDF"]
    Agent["AI Agent<br/>HF Inference"]
    Auth["HF OAuth"]
  end

  subgraph storage [Persistence]
    LocalFS["Local FS<br/>data/*.yjs"]
    HFDataset["HF Dataset<br/>articles/ published/"]
  end

  SPA -->|"WebSocket /collab"| Hocuspocus
  SPA -->|"REST /api/*"| Express
  Hocuspocus -->|"Database ext"| LocalFS
  LocalFS -->|"schedulePush"| HFDataset
  HFDataset -->|"pullDocument"| LocalFS
  Express --> Publisher
  Express --> Agent
  Express --> Auth
  Publisher --> LocalFS
  Publisher -->|"uploadPublishedAssets"| HFDataset
```

**Single process in production**: the backend serves the Vite-built frontend, all REST APIs, the WebSocket collab channel, and static published articles. No reverse proxy needed.

---

## 3. Data Model (Yjs Shared Types)

The entire collaborative state lives in a single `Y.Doc`:

- **`Y.XmlFragment("default")`** - TipTap document content (ProseMirror nodes synced via Collaboration extension)
- **`Y.Map("frontmatter")`** - scalar metadata: `title`, `subtitle`, `description`, `published`, `doi`, `template`, `licence`
- **`Y.Array("frontmatter.authors")`** - `{ name, url?, affiliations: number[] }[]`
- **`Y.Array("frontmatter.affiliations")`** - `{ name, url? }[]`
- **`Y.Map("citations")`** - CSL-JSON entries keyed by citation ID
- **`Y.Map("settings")`** - `citationStyle`, `primaryHue`, and future editor preferences
- **`Y.Map("comments")`** - comment threads keyed by `commentId`, each with author/text/resolved

All types are concurrently editable by multiple users and persist to `data/default.yjs`.

---

## 4. Backend Components

### 4.1 HTTP Routes

| Method | Path | Auth | Purpose |
|--------|------|------|---------|
| `GET` | `/oauth/authorize` | Public | Redirect to HF OAuth |
| `GET` | `/auth/callback` | Public (CSRF state) | Exchange code, set cookie, redirect to `/editor` |
| `GET` | `/api/auth/status` | Cookie | Return `{ authenticated, canEdit, user }` |
| `POST` | `/api/chat` | OAuth (optional) | Stream AI agent responses (HF Inference Providers) |
| `POST` | `/api/publish` | OAuth (canEdit) | Run publish pipeline, generate HTML/PDF |
| `POST` | `/api/admin/reset-document` | OAuth (canEdit) | Delete local `.yjs`, close connections |
| `POST` | `/api/upload` | None (uses cookie for HF) | Upload image (multipart, max 10MB) |
| `POST` | `/api/citations/resolve` | None | Resolve DOI/URL to CSL-JSON |
| `POST` | `/api/citations/format` | None | Format entries to HTML bibliography |
| `POST` | `/api/citations/import-bib` | None | Parse BibTeX to CSL-JSON |
| `GET` | `/editor` | OAuth (canEdit) | Serve SPA (or login page) |
| `GET` | `*` | Public | Serve published article (or login page) |

### 4.2 WebSocket Collaboration

- Upgrade on `/collab` only; all other paths rejected
- Single document: `DEFAULT_DOC_NAME = "default"`
- Hocuspocus `onAuthenticate`: validates OAuth token if enabled, checks `canEdit`
- `Database` extension: `fetch` reads local `.yjs` or pulls from HF; `store` writes local + schedules HF push (10s debounce)

### 4.3 HF Storage

- Dataset ID: `HF_DATASET_ID` or `{SPACE_ID}-data`
- Dataset is created **private by default** (`createRepo({ private: true })`). The OAuth grant needs `manage-repos` on the user's first write; subsequent containers reuse the cached token.
- Token: `HF_TOKEN` (env) or cached OAuth token from last authenticated user
- **Documents**: `articles/<name>.yjs` - debounced push on every Hocuspocus store
- **Published assets**: `published/<name>/{index.html, article.pdf, thumb.jpg, meta.json, llms.txt}`
- **Images**: `images/<uuid-filename>` referenced from articles via `/d/images/...` proxy URLs
- `flushAll()` on `SIGTERM`/`SIGINT` to push pending changes

### 4.3.1 Storage Status & Recovery

The persistence pipeline used to fail silently in multiple places (`createRepo` 403 on a missing scope, `uploadFile` 5xx mid-debounce, `writeFileSync` on a readonly FS, ...) and the editor would happily keep showing "Saved". To make data first-class:

- **In-memory tracker** in `hf-storage.ts` records `datasetReady`, `lastLocalSaveAt`, `lastCloudPushAt`, `pendingPush`, `lastError {stage, message, statusCode, at, docName}`. Every write path updates it; every error path records the failure.
- **`GET /api/storage/status`** exposes the tracker (canEdit-gated). The frontend `SyncIndicator` polls it every 5s and displays a three-state badge: green "Saved" / amber "Saving..." / **red "Storage error"** (pulsing, with the exact reason in the tooltip + actionable hint for the 403 / missing-scope case).
- **Eager `ensureDatasetExists`** on first `/api/auth/status` for a canEdit user. A misconfigured fork now surfaces its error within ~10s of login instead of waiting for an edit + 12s debounce cycle.
- **`beforeunload` guard** on the editor: if a local edit is in flight, a push is armed, the WS is offline, or the tracker reports an error, the browser pops the standard "Leave site?" confirm.
- **`GET /api/admin/export-doc`** (canEdit-gated) streams the on-disk `.yjs` snapshot as a download. The escape hatch for disaster recovery: when the cloud push has been failing and the container is about to rebuild, an admin can grab the doc bytes manually.

### 4.3.2 Dataset Reverse Proxy (`/d/*`)

Since the dataset is private, anonymous viewers of a published article can't fetch its images / PDF / og:image directly from `huggingface.co/datasets/...`. The editor server exposes `GET /d/:path*` as an authenticated forward-proxy:

- **Whitelist**: only `images/` and `published/` are reachable; `articles/` (raw `.yjs` drafts) is **always 404** regardless of caller.
- **Token cascade**: request cookie → cached user token → `HF_TOKEN` env → anonymous fetch. The cookie token is also promoted into the cache opportunistically, so the first signed-in viewer warms the proxy for subsequent anonymous viewers within the same container lifetime.
- **Streaming**: WHATWG body piped straight to the Express response - no buffering of full PDFs in Node memory.
- **Caching**: `images/*` is served as `immutable, max-age=1y` (UUID names, never overwritten); `published/*` as `max-age=300, stale-while-revalidate=60` (re-published in place).
- **Error mapping**: upstream 401/403 collapse to 502 so the browser never gets prompted for credentials it can't supply; upstream 404 passes through.

### 4.4 Publisher Pipeline

```mermaid
flowchart LR
  YDoc["Y.Doc (.yjs)"] --> Extract["extractFromYDoc<br/>frontmatter + JSON"]
  Extract --> GenHTML["generateHTML<br/>@tiptap/html"]
  GenHTML --> PostProc["postProcess<br/>accordion, biblio,<br/>mermaid, htmlEmbed"]
  PostProc --> Render["renderArticleHTML<br/>full HTML page"]
  CSS["loadCSS<br/>template styles"] --> Render
  Render --> LocalWrite["Write local<br/>index.html"]
  Render --> PDF["Playwright<br/>PDF + thumbnail"]
  LocalWrite --> HFUpload["uploadPublishedAssets<br/>HF dataset"]
```

- **CSS loading**: reads template CSS files, resolves `@custom-media` queries via `resolveCustomMedia()`, splits into variables/reset/base/layout/components/article/print
- **Post-processing**: accordion divs to `<details>`, bibliography injection, mermaid to `<pre>`, htmlEmbed to `<iframe>`
- **HTML output**: self-contained page with inline CSS, CDN assets (KaTeX, highlight.js, Mermaid), theme toggle (SVG sun/moon), TOC generation (scroll-based, collapsible), lightbox, footer with citation/BibTeX/DOI
- **PDF**: optional Playwright Chromium headless (1200x630 thumbnail + full PDF)
- **Server extensions**: mirror of frontend TipTap extensions for server-side HTML generation

### 4.5 Auth

- Enabled when `SPACE_ID` + `OAUTH_CLIENT_ID` are set
- OAuth 2.0 flow with HF as provider; cookie `hf_access_token` (httpOnly, secure, sameSite: none)
- `resolveUser`: `whoAmI` via `@huggingface/hub`, then `checkWriteAccess` (Space owner or org member with write/admin role)
- In-memory state map with 10-min TTL for CSRF protection

### 4.6 AI Agent

- Provider: Hugging Face Inference Providers (`https://router.huggingface.co/v1`), default model `openai/gpt-oss-120b`. Model ids may be suffixed with `:<provider>` (e.g. `meta-llama/Llama-3.3-70B-Instruct:together`) to bypass providers that enforce overly strict tool-call validation (notably Groq) or that don't support the `tools` parameter (Nscale, etc.).
- Auth: per-request bearer token resolved from the editor's OAuth cookie when available, falling back to the server-side `HF_TOKEN`. On a HF Space with `inference-api` scope, no extra secret is needed - the logged-in user pays for their own inference under their HF quota.
- Streaming via Vercel AI SDK `streamText` over `@ai-sdk/openai-compatible`
- Reasoning parts from prior assistant turns are stripped before re-sending the history: providers like Cerebras reject `reasoning_content` on round-trip, and the model doesn't need to see its own past reasoning to continue the conversation.
- **Context**: document text, current selection, frontmatter (sent by frontend with each message)
- **Tools** (declarative, executed client-side by the frontend):
  - `replaceSelection` - replace selected text
  - `insertAtCursor` - insert at cursor position
  - `applyDiff` - search/replace in document
  - `updateFrontmatter` - modify metadata fields
  - `addAuthor` / `removeAuthor` - manage author list
- Agent edits are grouped in a single Yjs `UndoManager` batch for Cmd+Z

### 4.7 Citations

- Uses `@citation-js/core` with bibtex, doi, csl plugins
- Resolve: DOI URL or identifier to CSL-JSON entries
- Format: entries + style + locale to HTML bibliography
- Import: BibTeX string to CSL-JSON

---

## 5. Frontend Components

### 5.1 App Shell

- **No router** - single view with conditional rendering
- **Theme**: CSS custom properties with dynamic primary color from `settings.primaryHue` (OKLCH color model, synced via Yjs settings)
- **Layout**: top bar (undo/redo, settings, publish, user chip) + 3-column CSS grid (TOC / editor / comments)
- **Chat**: floating button bottom-left, `ChatPanel` overlay
- **Modals**: comment dialog, settings drawer, publish confirmation

### 5.2 Editor

- Creates `Y.Doc` + `HocuspocusProvider` (WebSocket to `/collab`)
- **Seeding**: after provider `synced` event only, if `Y.XmlFragment("default")` is empty, inserts `DEFAULT_CONTENT` + `seedFrontmatter` + `SEED_CITATIONS`
- **Yjs Maps**: `citations`, `settings`, `comments`, `frontmatter` (via dedicated stores)
- **Image handling**: paste/drop with upload to `/api/upload`

### 5.3 TipTap Extensions

**Built-in (configured)**:
- StarterKit (no codeBlock, no undo), CodeBlockLowlight (all languages), Placeholder, Collaboration, CollaborationCursorV3, Mathematics (KaTeX), Image, Table/Row/Cell/Header

**Custom**:
- `CollaborationUndo` - bridges Yjs UndoManager for agent batch edits
- `Comment` - inline mark with `commentId` + `resolved`
- `SlashCommands` - `/` trigger with suggestion popup
- `ImageUpload` - drag-drop upload node with progress
- `Citation` - inline atomic node (key + label), links to `citationsMap`
- `Bibliography` - block node with rendered HTML from citations
- `Glossary` - inline atomic (term + definition tooltip)
- `Footnote` - inline atomic (content shown in footer)
- `Stack` + `StackColumn` - multi-column layout (2/3/4 cols)

### 5.4 Component System

Registry-based system for MDX-like custom components:

| Component | Kind | Purpose |
|-----------|------|---------|
| `accordion` | wrapper | Collapsible section (details/summary) |
| `note` | wrapper | Info/warning/danger/success callout |
| `quoteBlock` | wrapper | Styled blockquote |
| `wide` | wrapper | Content wider than column |
| `fullWidth` | wrapper | Full viewport width |
| `sidenote` | wrapper | Marginal note |
| `reference` | wrapper | Reference container |
| `htmlEmbed` | atomic | External HTML embed (iframe) |
| `hfUser` | atomic | HF user card |
| `rawHtml` | atomic | Raw HTML injection |
| `mermaid` | atomic | Mermaid diagram (live preview) |

- **Factory**: `createComponentExtension(def)` generates TipTap nodes from registry definitions (handles both wrapper and atomic kinds)
- **NodeViews**: `WrapperView` (editable content area + chrome), `AtomicView` (placeholder + field editor), `MermaidView` (textarea + SVG preview)
- **Slash menu integration**: each component generates a slash menu item via `getComponentSlashItems()`

### 5.5 Frontmatter System

- `FrontmatterStore`: wraps `Y.Map` + `Y.Array` for real-time collaborative metadata editing
- `useFrontmatter` hook: React state synced with Yjs observations
- `FrontmatterHero`: WYSIWYG editable hero section (title, subtitle, authors, affiliations, date, DOI)
- `SettingsDrawer`: template variant, SEO, banner, citation style, primary color hue slider, PDF/TOC/licence toggles
- `HueSlider`: OKLCH hue picker (0-360) with live preview, synced to `settingsMap.primaryHue`

### 5.6 Other UI

- **`TableOfContents`**: extracts headings from TipTap doc, scroll-based active state, collapsible sub-sections
- **`ChatPanel`**: message list + quick actions on selection + input with streaming
- **`CommentPopover`**: positioned comment popover anchored to the active thread (resolve/delete inline)
- **`BubbleToolbar`**: floating toolbar on text selection (bold, italic, link, comment, etc.)
- **`BlockHandle`**: drag handle for block-level nodes

### 5.7 CSS Architecture

```
styles/
  _variables.css       # Template tokens: --primary-color, breakpoints, @custom-media
  _reset.css           # Scoped reset for article content
  _base.css            # Typography, scoped to article content
  _layout.css          # 3-column grid, .wide/.full-width helpers
  _print.css           # Print styles
  _ui.css              # Editor chrome: buttons, dialogs, drawers, spinner
  tokens.css           # Design tokens (light/dark): text, bg, accent, code, danger, shadows
  article.css          # .tiptap content styles (shared editor/published)
  toc.css              # Editor TOC overrides
  editing.css          # Editor-only: layout, cursors, slash menu
  _publisher.css       # Published-only: theme toggle, wide/fullWidth, footer, lightbox
  components/
    _code.css          # Code blocks + syntax highlighting
    _table.css         # Tables
    _tag.css           # Tags
    _card.css          # Cards
    _mermaid.css       # Mermaid diagrams
    _embed.css         # Embed containers
    _embed-studio.css  # Embed studio overlay
    _hero.css          # Hero section (from template)
    _toc.css           # Base TOC styles (from template)
    _button.css        # Buttons (template)
    _form.css          # Form elements (template)
    _footer.css        # Footer (template)
```

The publisher reads these same CSS files server-side and injects them inline into published HTML, using `resolveCustomMedia()` to expand `@custom-media` queries into standard `@media` rules.

---

## 6. Deployment

### 6.1 Docker Build (3-stage)

1. **frontend-build**: `npm install` + `npm run build` (Vite)
2. **backend-build**: `npm install` + `npx tsc`
3. **runtime**: `node:20-slim` + Chromium system deps + `npm install --omit=dev` + Playwright Chromium + copy `frontend-dist/` + copy `frontend/src/styles/` to `frontend-styles/`

**CMD**: `node dist/server.js` on port 8080.

### 6.2 HF Space Configuration (README.md frontmatter)

- SDK: `docker`, port `8080`
- OAuth: `hf_oauth: true`, scopes: `manage-repos`, `inference-api`
- Two git remotes: `space` (tfrere/collab-editor, dev) and `prod` (tfrere/research-article-template-editor, production)

### 6.3 Environment Variables

| Variable | Required | Purpose |
|----------|----------|---------|
| `PORT` | No (default 8080) | HTTP listen port |
| `NODE_ENV` | No | `production` switches to `frontend-dist` path |
| `SPACE_ID` | For OAuth/HF | HF Space identifier, enables OAuth + dataset |
| `SPACE_HOST` | For OAuth | HTTPS callback URL host |
| `OAUTH_CLIENT_ID` | For OAuth | HF OAuth client |
| `OAUTH_CLIENT_SECRET` | For OAuth | HF OAuth secret |
| `OAUTH_SCOPES` | No (default `openid profile`) | OAuth scopes. Add `manage-repos` for dataset persistence and `inference-api` to power AI features with the user's token |
| `HF_DATASET_ID` | No | Override dataset name (default: `{SPACE_ID}-data`) |
| `HF_TOKEN` | For AI chat in local dev | Fallback Hub token for HF API + Inference Providers. Needs the "Make calls to Inference Providers" permission |
| `HF_INFERENCE_MODEL` | No (default `openai/gpt-oss-120b`) | Default chat-completion model id served by HF Inference Providers. May be suffixed with `:<provider>` to pin a specific routing |
| `ENABLE_PDF` | No (default true) | Toggle PDF/thumbnail generation |

### 6.4 Local Development

```bash
# Terminal 1 - Backend
cd backend && npm install && npm run dev
# Starts on http://localhost:8080

# Terminal 2 - Frontend
cd frontend && npm install && npm run dev
# Starts on http://localhost:5678 (proxies /api and /collab to :8080)
```

Create a `.env` file in `backend/` with at minimum `HF_TOKEN` for AI chat (must have the "Make calls to Inference Providers" permission). Without `SPACE_ID`, OAuth is disabled and all users can edit.

---

## 7. Key Data Flows

### 7.1 Collaborative Editing

```mermaid
sequenceDiagram
  participant ClientA as Client A
  participant Server as Hocuspocus
  participant ClientB as Client B
  participant Disk as Local FS
  participant HF as HF Dataset

  ClientA->>Server: WebSocket connect /collab
  Server->>Disk: Database.fetch (load .yjs)
  Server-->>ClientA: sync Y.Doc state
  ClientA->>Server: Y.Doc update (edit)
  Server->>ClientB: broadcast update
  Server->>Disk: Database.store (write .yjs)
  Disk-->>HF: schedulePush (10s debounce)
```

### 7.2 Publish Flow

```mermaid
sequenceDiagram
  participant User as Editor UI
  participant API as POST /api/publish
  participant HP as Hocuspocus
  participant Pub as Publisher
  participant FS as Local FS
  participant HF as HF Dataset

  User->>API: Click Publish
  API->>HP: openDirectConnection
  HP-->>API: Y.Doc snapshot
  API->>FS: Write .yjs snapshot
  API->>Pub: publishDocument()
  Pub->>Pub: extractFromYDoc + loadCSS
  Pub->>Pub: renderArticleHTML + PDF
  Pub->>FS: Write index.html locally
  Pub->>HF: uploadPublishedAssets
  Pub-->>API: { htmlUrl, pdfUrl, success }
  API-->>User: Publish result
```

### 7.3 Published Article Lifecycle (Container Restarts)

HF Spaces containers are ephemeral. The local filesystem is wiped on every restart (git push, Space rebuild, idle timeout). The published article survives via this restore flow:

```mermaid
sequenceDiagram
  participant Container as New Container
  participant FS as Local FS
  participant HF as HF Dataset
  participant Visitor as GET /

  Container->>Container: Server starts
  Container->>HF: ensurePublishedRestored()
  HF-->>FS: Pull index.html, PDF, meta.json
  Note over FS: data/published/default/index.html

  Visitor->>Container: GET /
  Container->>FS: Check published path
  FS-->>Container: index.html exists
  Container-->>Visitor: Serve published article
```

On publish, HTML is **always written locally first** (so `GET /` serves the new version immediately), then uploaded to HF dataset for persistence across restarts.

### 7.4 AI Agent Chat

```mermaid
sequenceDiagram
  participant User as Chat Panel
  participant Hook as useAgentChat
  participant API as POST /api/chat
  participant LLM as HF Inference

  User->>Hook: sendMessage(text)
  Hook->>Hook: Build context (doc, selection, frontmatter)
  Hook->>API: { messages, context }
  API->>LLM: streamText (system prompt + tools)
  LLM-->>API: Stream (text + tool_calls)
  API-->>Hook: SSE stream
  Hook->>Hook: Execute tool calls client-side
  Note over Hook: replaceSelection, applyDiff,<br/>updateFrontmatter, etc.
  Hook->>Hook: UndoManager batch for Cmd+Z
```

---

## 8. Current Limitations and Known Issues

- **Test suite in progress**: P0 tests being added (see `docs/TESTS.md`)
- **Single document**: only `"default"` document supported; no multi-doc
- **Single-user token**: last OAuth token cached globally for all HF API calls
- **No rate limiting** on `/api/chat` or `/api/citations/*`
- **XSS surface**: `meta.licence` and `biblioHtml` not escaped in published HTML
- **WS debug logging**: every WebSocket message logged in production
- **No `.env.example`**: environment variables documented only in code