| # Collab Editor - Project Specification |
|
|
| ## 1. Overview |
|
|
| A collaborative, real-time scientific article editor deployed as a Hugging Face Space. Users write rich content (math, citations, custom components) in a TipTap-based editor synced via Yjs/Hocuspocus, then publish a self-contained static HTML article. An AI assistant helps with writing and editing. |
|
|
| **Stack**: React 18 + TipTap 3 + Yjs (frontend) / Express + Hocuspocus + Node 20 (backend) / Docker on HF Spaces. No CSS-in-JS - all styling via vanilla CSS custom properties. |
|
|
| **Relationship to `research-article-template`**: the CSS foundation, design tokens, and visual language come from the [research-article-template](https://huggingface.co/spaces/tfrere/research-article-template) project. The editor imports the template's CSS files (`_variables.css`, `_reset.css`, `_base.css`, `_layout.css`, component partials) and the publisher injects them inline into published HTML. The published output is designed to look identical to articles built with the Astro-based template. |
|
|
| --- |
|
|
| ## 2. Architecture |
|
|
| ```mermaid |
| graph TB |
| subgraph browser [Browser] |
| SPA["React SPA<br/>TipTap + Yjs"] |
| end |
| |
| subgraph server [Node Backend - port 8080] |
| Express["Express HTTP"] |
| Hocuspocus["Hocuspocus<br/>WebSocket"] |
| Publisher["Publisher Pipeline<br/>HTML + PDF"] |
| Agent["AI Agent<br/>HF Inference"] |
| Auth["HF OAuth"] |
| end |
| |
| subgraph storage [Persistence] |
| LocalFS["Local FS<br/>data/*.yjs"] |
| HFDataset["HF Dataset<br/>articles/ published/"] |
| end |
| |
| SPA -->|"WebSocket /collab"| Hocuspocus |
| SPA -->|"REST /api/*"| Express |
| Hocuspocus -->|"Database ext"| LocalFS |
| LocalFS -->|"schedulePush"| HFDataset |
| HFDataset -->|"pullDocument"| LocalFS |
| Express --> Publisher |
| Express --> Agent |
| Express --> Auth |
| Publisher --> LocalFS |
| Publisher -->|"uploadPublishedAssets"| HFDataset |
| ``` |
|
|
| **Single process in production**: the backend serves the Vite-built frontend, all REST APIs, the WebSocket collab channel, and static published articles. No reverse proxy needed. |
|
|
| --- |
|
|
| ## 3. Data Model (Yjs Shared Types) |
|
|
| The entire collaborative state lives in a single `Y.Doc`: |
|
|
| - **`Y.XmlFragment("default")`** - TipTap document content (ProseMirror nodes synced via Collaboration extension) |
| - **`Y.Map("frontmatter")`** - scalar metadata: `title`, `subtitle`, `description`, `published`, `doi`, `template`, `licence` |
| - **`Y.Array("frontmatter.authors")`** - `{ name, url?, affiliations: number[] }[]` |
| - **`Y.Array("frontmatter.affiliations")`** - `{ name, url? }[]` |
| - **`Y.Map("citations")`** - CSL-JSON entries keyed by citation ID |
| - **`Y.Map("settings")`** - `citationStyle`, `primaryHue`, and future editor preferences |
| - **`Y.Map("comments")`** - comment threads keyed by `commentId`, each with author/text/resolved |
|
|
| All types are concurrently editable by multiple users and persist to `data/default.yjs`. |
|
|
| --- |
|
|
| ## 4. Backend Components |
|
|
| ### 4.1 HTTP Routes |
|
|
| | Method | Path | Auth | Purpose | |
| |--------|------|------|---------| |
| | `GET` | `/oauth/authorize` | Public | Redirect to HF OAuth | |
| | `GET` | `/auth/callback` | Public (CSRF state) | Exchange code, set cookie, redirect to `/editor` | |
| | `GET` | `/api/auth/status` | Cookie | Return `{ authenticated, canEdit, user }` | |
| | `POST` | `/api/chat` | OAuth (optional) | Stream AI agent responses (HF Inference Providers) | |
| | `POST` | `/api/publish` | OAuth (canEdit) | Run publish pipeline, generate HTML/PDF | |
| | `POST` | `/api/admin/reset-document` | OAuth (canEdit) | Delete local `.yjs`, close connections | |
| | `POST` | `/api/upload` | None (uses cookie for HF) | Upload image (multipart, max 10MB) | |
| | `POST` | `/api/citations/resolve` | None | Resolve DOI/URL to CSL-JSON | |
| | `POST` | `/api/citations/format` | None | Format entries to HTML bibliography | |
| | `POST` | `/api/citations/import-bib` | None | Parse BibTeX to CSL-JSON | |
| | `GET` | `/editor` | OAuth (canEdit) | Serve SPA (or login page) | |
| | `GET` | `*` | Public | Serve published article (or login page) | |
|
|
| ### 4.2 WebSocket Collaboration |
|
|
| - Upgrade on `/collab` only; all other paths rejected |
| - Single document: `DEFAULT_DOC_NAME = "default"` |
| - Hocuspocus `onAuthenticate`: validates OAuth token if enabled, checks `canEdit` |
| - `Database` extension: `fetch` reads local `.yjs` or pulls from HF; `store` writes local + schedules HF push (10s debounce) |
|
|
| ### 4.3 HF Storage |
|
|
| - Dataset ID: `HF_DATASET_ID` or `{SPACE_ID}-data` |
| - Dataset is created **private by default** (`createRepo({ private: true })`). The OAuth grant needs `manage-repos` on the user's first write; subsequent containers reuse the cached token. |
| - Token: `HF_TOKEN` (env) or cached OAuth token from last authenticated user |
| - **Documents**: `articles/<name>.yjs` - debounced push on every Hocuspocus store |
| - **Published assets**: `published/<name>/{index.html, article.pdf, thumb.jpg, meta.json, llms.txt}` |
| - **Images**: `images/<uuid-filename>` referenced from articles via `/d/images/...` proxy URLs |
| - `flushAll()` on `SIGTERM`/`SIGINT` to push pending changes |
|
|
| ### 4.3.1 Storage Status & Recovery |
|
|
| The persistence pipeline used to fail silently in multiple places (`createRepo` 403 on a missing scope, `uploadFile` 5xx mid-debounce, `writeFileSync` on a readonly FS, ...) and the editor would happily keep showing "Saved". To make data first-class: |
|
|
| - **In-memory tracker** in `hf-storage.ts` records `datasetReady`, `lastLocalSaveAt`, `lastCloudPushAt`, `pendingPush`, `lastError {stage, message, statusCode, at, docName}`. Every write path updates it; every error path records the failure. |
| - **`GET /api/storage/status`** exposes the tracker (canEdit-gated). The frontend `SyncIndicator` polls it every 5s and displays a three-state badge: green "Saved" / amber "Saving..." / **red "Storage error"** (pulsing, with the exact reason in the tooltip + actionable hint for the 403 / missing-scope case). |
| - **Eager `ensureDatasetExists`** on first `/api/auth/status` for a canEdit user. A misconfigured fork now surfaces its error within ~10s of login instead of waiting for an edit + 12s debounce cycle. |
| - **`beforeunload` guard** on the editor: if a local edit is in flight, a push is armed, the WS is offline, or the tracker reports an error, the browser pops the standard "Leave site?" confirm. |
| - **`GET /api/admin/export-doc`** (canEdit-gated) streams the on-disk `.yjs` snapshot as a download. The escape hatch for disaster recovery: when the cloud push has been failing and the container is about to rebuild, an admin can grab the doc bytes manually. |
|
|
| ### 4.3.2 Dataset Reverse Proxy (`/d/*`) |
| |
| Since the dataset is private, anonymous viewers of a published article can't fetch its images / PDF / og:image directly from `huggingface.co/datasets/...`. The editor server exposes `GET /d/:path*` as an authenticated forward-proxy: |
|
|
| - **Whitelist**: only `images/` and `published/` are reachable; `articles/` (raw `.yjs` drafts) is **always 404** regardless of caller. |
| - **Token cascade**: request cookie → cached user token → `HF_TOKEN` env → anonymous fetch. The cookie token is also promoted into the cache opportunistically, so the first signed-in viewer warms the proxy for subsequent anonymous viewers within the same container lifetime. |
| - **Streaming**: WHATWG body piped straight to the Express response - no buffering of full PDFs in Node memory. |
| - **Caching**: `images/*` is served as `immutable, max-age=1y` (UUID names, never overwritten); `published/*` as `max-age=300, stale-while-revalidate=60` (re-published in place). |
| - **Error mapping**: upstream 401/403 collapse to 502 so the browser never gets prompted for credentials it can't supply; upstream 404 passes through. |
|
|
| ### 4.4 Publisher Pipeline |
|
|
| ```mermaid |
| flowchart LR |
| YDoc["Y.Doc (.yjs)"] --> Extract["extractFromYDoc<br/>frontmatter + JSON"] |
| Extract --> GenHTML["generateHTML<br/>@tiptap/html"] |
| GenHTML --> PostProc["postProcess<br/>accordion, biblio,<br/>mermaid, htmlEmbed"] |
| PostProc --> Render["renderArticleHTML<br/>full HTML page"] |
| CSS["loadCSS<br/>template styles"] --> Render |
| Render --> LocalWrite["Write local<br/>index.html"] |
| Render --> PDF["Playwright<br/>PDF + thumbnail"] |
| LocalWrite --> HFUpload["uploadPublishedAssets<br/>HF dataset"] |
| ``` |
|
|
| - **CSS loading**: reads template CSS files, resolves `@custom-media` queries via `resolveCustomMedia()`, splits into variables/reset/base/layout/components/article/print |
| - **Post-processing**: accordion divs to `<details>`, bibliography injection, mermaid to `<pre>`, htmlEmbed to `<iframe>` |
| - **HTML output**: self-contained page with inline CSS, CDN assets (KaTeX, highlight.js, Mermaid), theme toggle (SVG sun/moon), TOC generation (scroll-based, collapsible), lightbox, footer with citation/BibTeX/DOI |
| - **PDF**: optional Playwright Chromium headless (1200x630 thumbnail + full PDF) |
| - **Server extensions**: mirror of frontend TipTap extensions for server-side HTML generation |
|
|
| ### 4.5 Auth |
|
|
| - Enabled when `SPACE_ID` + `OAUTH_CLIENT_ID` are set |
| - OAuth 2.0 flow with HF as provider; cookie `hf_access_token` (httpOnly, secure, sameSite: none) |
| - `resolveUser`: `whoAmI` via `@huggingface/hub`, then `checkWriteAccess` (Space owner or org member with write/admin role) |
| - In-memory state map with 10-min TTL for CSRF protection |
|
|
| ### 4.6 AI Agent |
|
|
| - Provider: Hugging Face Inference Providers (`https://router.huggingface.co/v1`), default model `openai/gpt-oss-120b`. Model ids may be suffixed with `:<provider>` (e.g. `meta-llama/Llama-3.3-70B-Instruct:together`) to bypass providers that enforce overly strict tool-call validation (notably Groq) or that don't support the `tools` parameter (Nscale, etc.). |
| - Auth: per-request bearer token resolved from the editor's OAuth cookie when available, falling back to the server-side `HF_TOKEN`. On a HF Space with `inference-api` scope, no extra secret is needed - the logged-in user pays for their own inference under their HF quota. |
| - Streaming via Vercel AI SDK `streamText` over `@ai-sdk/openai-compatible` |
| - Reasoning parts from prior assistant turns are stripped before re-sending the history: providers like Cerebras reject `reasoning_content` on round-trip, and the model doesn't need to see its own past reasoning to continue the conversation. |
| - **Context**: document text, current selection, frontmatter (sent by frontend with each message) |
| - **Tools** (declarative, executed client-side by the frontend): |
| - `replaceSelection` - replace selected text |
| - `insertAtCursor` - insert at cursor position |
| - `applyDiff` - search/replace in document |
| - `updateFrontmatter` - modify metadata fields |
| - `addAuthor` / `removeAuthor` - manage author list |
| - Agent edits are grouped in a single Yjs `UndoManager` batch for Cmd+Z |
|
|
| ### 4.7 Citations |
|
|
| - Uses `@citation-js/core` with bibtex, doi, csl plugins |
| - Resolve: DOI URL or identifier to CSL-JSON entries |
| - Format: entries + style + locale to HTML bibliography |
| - Import: BibTeX string to CSL-JSON |
|
|
| --- |
|
|
| ## 5. Frontend Components |
|
|
| ### 5.1 App Shell |
|
|
| - **No router** - single view with conditional rendering |
| - **Theme**: CSS custom properties with dynamic primary color from `settings.primaryHue` (OKLCH color model, synced via Yjs settings) |
| - **Layout**: top bar (undo/redo, settings, publish, user chip) + 3-column CSS grid (TOC / editor / comments) |
| - **Chat**: floating button bottom-left, `ChatPanel` overlay |
| - **Modals**: comment dialog, settings drawer, publish confirmation |
|
|
| ### 5.2 Editor |
|
|
| - Creates `Y.Doc` + `HocuspocusProvider` (WebSocket to `/collab`) |
| - **Seeding**: after provider `synced` event only, if `Y.XmlFragment("default")` is empty, inserts `DEFAULT_CONTENT` + `seedFrontmatter` + `SEED_CITATIONS` |
| - **Yjs Maps**: `citations`, `settings`, `comments`, `frontmatter` (via dedicated stores) |
| - **Image handling**: paste/drop with upload to `/api/upload` |
|
|
| ### 5.3 TipTap Extensions |
|
|
| **Built-in (configured)**: |
| - StarterKit (no codeBlock, no undo), CodeBlockLowlight (all languages), Placeholder, Collaboration, CollaborationCursorV3, Mathematics (KaTeX), Image, Table/Row/Cell/Header |
|
|
| **Custom**: |
| - `CollaborationUndo` - bridges Yjs UndoManager for agent batch edits |
| - `Comment` - inline mark with `commentId` + `resolved` |
| - `SlashCommands` - `/` trigger with suggestion popup |
| - `ImageUpload` - drag-drop upload node with progress |
| - `Citation` - inline atomic node (key + label), links to `citationsMap` |
| - `Bibliography` - block node with rendered HTML from citations |
| - `Glossary` - inline atomic (term + definition tooltip) |
| - `Footnote` - inline atomic (content shown in footer) |
| - `Stack` + `StackColumn` - multi-column layout (2/3/4 cols) |
|
|
| ### 5.4 Component System |
|
|
| Registry-based system for MDX-like custom components: |
|
|
| | Component | Kind | Purpose | |
| |-----------|------|---------| |
| | `accordion` | wrapper | Collapsible section (details/summary) | |
| | `note` | wrapper | Info/warning/danger/success callout | |
| | `quoteBlock` | wrapper | Styled blockquote | |
| | `wide` | wrapper | Content wider than column | |
| | `fullWidth` | wrapper | Full viewport width | |
| | `sidenote` | wrapper | Marginal note | |
| | `reference` | wrapper | Reference container | |
| | `htmlEmbed` | atomic | External HTML embed (iframe) | |
| | `hfUser` | atomic | HF user card | |
| | `rawHtml` | atomic | Raw HTML injection | |
| | `mermaid` | atomic | Mermaid diagram (live preview) | |
|
|
| - **Factory**: `createComponentExtension(def)` generates TipTap nodes from registry definitions (handles both wrapper and atomic kinds) |
| - **NodeViews**: `WrapperView` (editable content area + chrome), `AtomicView` (placeholder + field editor), `MermaidView` (textarea + SVG preview) |
| - **Slash menu integration**: each component generates a slash menu item via `getComponentSlashItems()` |
|
|
| ### 5.5 Frontmatter System |
|
|
| - `FrontmatterStore`: wraps `Y.Map` + `Y.Array` for real-time collaborative metadata editing |
| - `useFrontmatter` hook: React state synced with Yjs observations |
| - `FrontmatterHero`: WYSIWYG editable hero section (title, subtitle, authors, affiliations, date, DOI) |
| - `SettingsDrawer`: template variant, SEO, banner, citation style, primary color hue slider, PDF/TOC/licence toggles |
| - `HueSlider`: OKLCH hue picker (0-360) with live preview, synced to `settingsMap.primaryHue` |
|
|
| ### 5.6 Other UI |
|
|
| - **`TableOfContents`**: extracts headings from TipTap doc, scroll-based active state, collapsible sub-sections |
| - **`ChatPanel`**: message list + quick actions on selection + input with streaming |
| - **`CommentPopover`**: positioned comment popover anchored to the active thread (resolve/delete inline) |
| - **`BubbleToolbar`**: floating toolbar on text selection (bold, italic, link, comment, etc.) |
| - **`BlockHandle`**: drag handle for block-level nodes |
|
|
| ### 5.7 CSS Architecture |
|
|
| ``` |
| styles/ |
| _variables.css # Template tokens: --primary-color, breakpoints, @custom-media |
| _reset.css # Scoped reset for article content |
| _base.css # Typography, scoped to article content |
| _layout.css # 3-column grid, .wide/.full-width helpers |
| _print.css # Print styles |
| _ui.css # Editor chrome: buttons, dialogs, drawers, spinner |
| tokens.css # Design tokens (light/dark): text, bg, accent, code, danger, shadows |
| article.css # .tiptap content styles (shared editor/published) |
| toc.css # Editor TOC overrides |
| editing.css # Editor-only: layout, cursors, slash menu |
| _publisher.css # Published-only: theme toggle, wide/fullWidth, footer, lightbox |
| components/ |
| _code.css # Code blocks + syntax highlighting |
| _table.css # Tables |
| _tag.css # Tags |
| _card.css # Cards |
| _mermaid.css # Mermaid diagrams |
| _embed.css # Embed containers |
| _embed-studio.css # Embed studio overlay |
| _hero.css # Hero section (from template) |
| _toc.css # Base TOC styles (from template) |
| _button.css # Buttons (template) |
| _form.css # Form elements (template) |
| _footer.css # Footer (template) |
| ``` |
|
|
| The publisher reads these same CSS files server-side and injects them inline into published HTML, using `resolveCustomMedia()` to expand `@custom-media` queries into standard `@media` rules. |
|
|
| --- |
|
|
| ## 6. Deployment |
|
|
| ### 6.1 Docker Build (3-stage) |
|
|
| 1. **frontend-build**: `npm install` + `npm run build` (Vite) |
| 2. **backend-build**: `npm install` + `npx tsc` |
| 3. **runtime**: `node:20-slim` + Chromium system deps + `npm install --omit=dev` + Playwright Chromium + copy `frontend-dist/` + copy `frontend/src/styles/` to `frontend-styles/` |
|
|
| **CMD**: `node dist/server.js` on port 8080. |
|
|
| ### 6.2 HF Space Configuration (README.md frontmatter) |
|
|
| - SDK: `docker`, port `8080` |
| - OAuth: `hf_oauth: true`, scopes: `manage-repos`, `inference-api` |
| - Two git remotes: `space` (tfrere/collab-editor, dev) and `prod` (tfrere/research-article-template-editor, production) |
|
|
| ### 6.3 Environment Variables |
|
|
| | Variable | Required | Purpose | |
| |----------|----------|---------| |
| | `PORT` | No (default 8080) | HTTP listen port | |
| | `NODE_ENV` | No | `production` switches to `frontend-dist` path | |
| | `SPACE_ID` | For OAuth/HF | HF Space identifier, enables OAuth + dataset | |
| | `SPACE_HOST` | For OAuth | HTTPS callback URL host | |
| | `OAUTH_CLIENT_ID` | For OAuth | HF OAuth client | |
| | `OAUTH_CLIENT_SECRET` | For OAuth | HF OAuth secret | |
| | `OAUTH_SCOPES` | No (default `openid profile`) | OAuth scopes. Add `manage-repos` for dataset persistence and `inference-api` to power AI features with the user's token | |
| | `HF_DATASET_ID` | No | Override dataset name (default: `{SPACE_ID}-data`) | |
| | `HF_TOKEN` | For AI chat in local dev | Fallback Hub token for HF API + Inference Providers. Needs the "Make calls to Inference Providers" permission | |
| | `HF_INFERENCE_MODEL` | No (default `openai/gpt-oss-120b`) | Default chat-completion model id served by HF Inference Providers. May be suffixed with `:<provider>` to pin a specific routing | |
| | `ENABLE_PDF` | No (default true) | Toggle PDF/thumbnail generation | |
|
|
| ### 6.4 Local Development |
|
|
| ```bash |
| # Terminal 1 - Backend |
| cd backend && npm install && npm run dev |
| # Starts on http://localhost:8080 |
| |
| # Terminal 2 - Frontend |
| cd frontend && npm install && npm run dev |
| # Starts on http://localhost:5678 (proxies /api and /collab to :8080) |
| ``` |
|
|
| Create a `.env` file in `backend/` with at minimum `HF_TOKEN` for AI chat (must have the "Make calls to Inference Providers" permission). Without `SPACE_ID`, OAuth is disabled and all users can edit. |
|
|
| --- |
|
|
| ## 7. Key Data Flows |
|
|
| ### 7.1 Collaborative Editing |
|
|
| ```mermaid |
| sequenceDiagram |
| participant ClientA as Client A |
| participant Server as Hocuspocus |
| participant ClientB as Client B |
| participant Disk as Local FS |
| participant HF as HF Dataset |
| |
| ClientA->>Server: WebSocket connect /collab |
| Server->>Disk: Database.fetch (load .yjs) |
| Server-->>ClientA: sync Y.Doc state |
| ClientA->>Server: Y.Doc update (edit) |
| Server->>ClientB: broadcast update |
| Server->>Disk: Database.store (write .yjs) |
| Disk-->>HF: schedulePush (10s debounce) |
| ``` |
|
|
| ### 7.2 Publish Flow |
|
|
| ```mermaid |
| sequenceDiagram |
| participant User as Editor UI |
| participant API as POST /api/publish |
| participant HP as Hocuspocus |
| participant Pub as Publisher |
| participant FS as Local FS |
| participant HF as HF Dataset |
| |
| User->>API: Click Publish |
| API->>HP: openDirectConnection |
| HP-->>API: Y.Doc snapshot |
| API->>FS: Write .yjs snapshot |
| API->>Pub: publishDocument() |
| Pub->>Pub: extractFromYDoc + loadCSS |
| Pub->>Pub: renderArticleHTML + PDF |
| Pub->>FS: Write index.html locally |
| Pub->>HF: uploadPublishedAssets |
| Pub-->>API: { htmlUrl, pdfUrl, success } |
| API-->>User: Publish result |
| ``` |
|
|
| ### 7.3 Published Article Lifecycle (Container Restarts) |
|
|
| HF Spaces containers are ephemeral. The local filesystem is wiped on every restart (git push, Space rebuild, idle timeout). The published article survives via this restore flow: |
|
|
| ```mermaid |
| sequenceDiagram |
| participant Container as New Container |
| participant FS as Local FS |
| participant HF as HF Dataset |
| participant Visitor as GET / |
| |
| Container->>Container: Server starts |
| Container->>HF: ensurePublishedRestored() |
| HF-->>FS: Pull index.html, PDF, meta.json |
| Note over FS: data/published/default/index.html |
| |
| Visitor->>Container: GET / |
| Container->>FS: Check published path |
| FS-->>Container: index.html exists |
| Container-->>Visitor: Serve published article |
| ``` |
|
|
| On publish, HTML is **always written locally first** (so `GET /` serves the new version immediately), then uploaded to HF dataset for persistence across restarts. |
|
|
| ### 7.4 AI Agent Chat |
|
|
| ```mermaid |
| sequenceDiagram |
| participant User as Chat Panel |
| participant Hook as useAgentChat |
| participant API as POST /api/chat |
| participant LLM as HF Inference |
| |
| User->>Hook: sendMessage(text) |
| Hook->>Hook: Build context (doc, selection, frontmatter) |
| Hook->>API: { messages, context } |
| API->>LLM: streamText (system prompt + tools) |
| LLM-->>API: Stream (text + tool_calls) |
| API-->>Hook: SSE stream |
| Hook->>Hook: Execute tool calls client-side |
| Note over Hook: replaceSelection, applyDiff,<br/>updateFrontmatter, etc. |
| Hook->>Hook: UndoManager batch for Cmd+Z |
| ``` |
|
|
| --- |
|
|
| ## 8. Current Limitations and Known Issues |
|
|
| - **Test suite in progress**: P0 tests being added (see `docs/TESTS.md`) |
| - **Single document**: only `"default"` document supported; no multi-doc |
| - **Single-user token**: last OAuth token cached globally for all HF API calls |
| - **No rate limiting** on `/api/chat` or `/api/citations/*` |
| - **XSS surface**: `meta.licence` and `biblioHtml` not escaped in published HTML |
| - **WS debug logging**: every WebSocket message logged in production |
| - **No `.env.example`**: environment variables documented only in code |
|
|
