docs/SPECIFICATION.md

Collab Editor - Project Specification

1. Overview

A collaborative, real-time scientific article editor deployed as a Hugging Face Space. Users write rich content (math, citations, custom components) in a TipTap-based editor synced via Yjs/Hocuspocus, then publish a self-contained static HTML article. An AI assistant helps with writing and editing.

Stack: React 18 + TipTap 3 + Yjs (frontend) / Express + Hocuspocus + Node 20 (backend) / Docker on HF Spaces. No CSS-in-JS - all styling via vanilla CSS custom properties.

Relationship to research-article-template: the CSS foundation, design tokens, and visual language come from the research-article-template project. The editor imports the template's CSS files (_variables.css, _reset.css, _base.css, _layout.css, component partials) and the publisher injects them inline into published HTML. The published output is designed to look identical to articles built with the Astro-based template.

---

2. Architecture

graph TB
  subgraph browser [Browser]
    SPA["React SPA<br/>TipTap + Yjs"]
  end

  subgraph server [Node Backend - port 8080]
    Express["Express HTTP"]
    Hocuspocus["Hocuspocus<br/>WebSocket"]
    Publisher["Publisher Pipeline<br/>HTML + PDF"]
    Agent["AI Agent<br/>HF Inference"]
    Auth["HF OAuth"]
  end

  subgraph storage [Persistence]
    LocalFS["Local FS<br/>data/*.yjs"]
    HFDataset["HF Dataset<br/>articles/ published/"]
  end

  SPA -->|"WebSocket /collab"| Hocuspocus
  SPA -->|"REST /api/*"| Express
  Hocuspocus -->|"Database ext"| LocalFS
  LocalFS -->|"schedulePush"| HFDataset
  HFDataset -->|"pullDocument"| LocalFS
  Express --> Publisher
  Express --> Agent
  Express --> Auth
  Publisher --> LocalFS
  Publisher -->|"uploadPublishedAssets"| HFDataset

Single process in production: the backend serves the Vite-built frontend, all REST APIs, the WebSocket collab channel, and static published articles. No reverse proxy needed.

---

3. Data Model (Yjs Shared Types)

The entire collaborative state lives in a single Y.Doc:

• Y.XmlFragment("default") - TipTap document content (ProseMirror nodes synced via Collaboration extension)
• Y.Map("frontmatter") - scalar metadata: title, subtitle, description, published, doi, template, licence
• Y.Array("frontmatter.authors") - { name, url?, affiliations: number[] }[]
• Y.Array("frontmatter.affiliations") - { name, url? }[]
• Y.Map("citations") - CSL-JSON entries keyed by citation ID
• Y.Map("settings") - citationStyle, primaryHue, and future editor preferences
• Y.Map("comments") - comment threads keyed by commentId, each with author/text/resolved

All types are concurrently editable by multiple users and persist to data/default.yjs.

---

4. Backend Components

4.1 HTTP Routes

Method	Path	Auth	Purpose
GET	/oauth/authorize	Public	Redirect to HF OAuth
GET	/auth/callback	Public (CSRF state)	Exchange code, set cookie, redirect to /editor
GET	/api/auth/status	Cookie	Return { authenticated, canEdit, user }
POST	/api/chat	OAuth (optional)	Stream AI agent responses (HF Inference Providers)
POST	/api/publish	OAuth (canEdit)	Run publish pipeline, generate HTML/PDF
POST	/api/admin/reset-document	OAuth (canEdit)	Delete local .yjs, close connections
POST	/api/upload	None (uses cookie for HF)	Upload image (multipart, max 10MB)
POST	/api/citations/resolve	None	Resolve DOI/URL to CSL-JSON
POST	/api/citations/format	None	Format entries to HTML bibliography
POST	/api/citations/import-bib	None	Parse BibTeX to CSL-JSON
GET	/editor	OAuth (canEdit)	Serve SPA (or login page)
GET	*	Public	Serve published article (or login page)

4.2 WebSocket Collaboration

• Upgrade on /collab only; all other paths rejected
• Single document: DEFAULT_DOC_NAME = "default"
• Hocuspocus onAuthenticate: validates OAuth token if enabled, checks canEdit
• Database extension: fetch reads local .yjs or pulls from HF; store writes local + schedules HF push (10s debounce)

4.3 HF Storage

• Dataset ID: HF_DATASET_ID or {SPACE_ID}-data
• Dataset is created private by default (createRepo({ private: true })). The OAuth grant needs manage-repos on the user's first write; subsequent containers reuse the cached token.
• Token: HF_TOKEN (env) or cached OAuth token from last authenticated user
• Documents: articles/<name>.yjs - debounced push on every Hocuspocus store
• Published assets: published/<name>/{index.html, article.pdf, thumb.jpg, meta.json, llms.txt}
• Images: images/<uuid-filename> referenced from articles via /d/images/... proxy URLs
• flushAll() on SIGTERM/SIGINT to push pending changes

4.3.1 Storage Status & Recovery

The persistence pipeline used to fail silently in multiple places (createRepo 403 on a missing scope, uploadFile 5xx mid-debounce, writeFileSync on a readonly FS, ...) and the editor would happily keep showing "Saved". To make data first-class:

• In-memory tracker in hf-storage.ts records datasetReady, lastLocalSaveAt, lastCloudPushAt, pendingPush, lastError {stage, message, statusCode, at, docName}. Every write path updates it; every error path records the failure.
• GET /api/storage/status exposes the tracker (canEdit-gated). The frontend SyncIndicator polls it every 5s and displays a three-state badge: green "Saved" / amber "Saving..." / red "Storage error" (pulsing, with the exact reason in the tooltip + actionable hint for the 403 / missing-scope case).
• Eager ensureDatasetExists on first /api/auth/status for a canEdit user. A misconfigured fork now surfaces its error within ~10s of login instead of waiting for an edit + 12s debounce cycle.
• beforeunload guard on the editor: if a local edit is in flight, a push is armed, the WS is offline, or the tracker reports an error, the browser pops the standard "Leave site?" confirm.
• GET /api/admin/export-doc (canEdit-gated) streams the on-disk .yjs snapshot as a download. The escape hatch for disaster recovery: when the cloud push has been failing and the container is about to rebuild, an admin can grab the doc bytes manually.

4.3.2 Dataset Reverse Proxy (/d/*)

Since the dataset is private, anonymous viewers of a published article can't fetch its images / PDF / og:image directly from huggingface.co/datasets/.... The editor server exposes GET /d/:path* as an authenticated forward-proxy:

• Whitelist: only images/ and published/ are reachable; articles/ (raw .yjs drafts) is always 404 regardless of caller.
• Token cascade: request cookie → cached user token → HF_TOKEN env → anonymous fetch. The cookie token is also promoted into the cache opportunistically, so the first signed-in viewer warms the proxy for subsequent anonymous viewers within the same container lifetime.
• Streaming: WHATWG body piped straight to the Express response - no buffering of full PDFs in Node memory.
• Caching: images/* is served as immutable, max-age=1y (UUID names, never overwritten); published/* as max-age=300, stale-while-revalidate=60 (re-published in place).
• Error mapping: upstream 401/403 collapse to 502 so the browser never gets prompted for credentials it can't supply; upstream 404 passes through.

4.4 Publisher Pipeline

flowchart LR
  YDoc["Y.Doc (.yjs)"] --> Extract["extractFromYDoc<br/>frontmatter + JSON"]
  Extract --> GenHTML["generateHTML<br/>@tiptap/html"]
  GenHTML --> PostProc["postProcess<br/>accordion, biblio,<br/>mermaid, htmlEmbed"]
  PostProc --> Render["renderArticleHTML<br/>full HTML page"]
  CSS["loadCSS<br/>template styles"] --> Render
  Render --> LocalWrite["Write local<br/>index.html"]
  Render --> PDF["Playwright<br/>PDF + thumbnail"]
  LocalWrite --> HFUpload["uploadPublishedAssets<br/>HF dataset"]

• CSS loading: reads template CSS files, resolves @custom-media queries via resolveCustomMedia(), splits into variables/reset/base/layout/components/article/print
• Post-processing: accordion divs to <details>, bibliography injection, mermaid to <pre>, htmlEmbed to <iframe>
• HTML output: self-contained page with inline CSS, CDN assets (KaTeX, highlight.js, Mermaid), theme toggle (SVG sun/moon), TOC generation (scroll-based, collapsible), lightbox, footer with citation/BibTeX/DOI
• PDF: optional Playwright Chromium headless (1200x630 thumbnail + full PDF)
• Server extensions: mirror of frontend TipTap extensions for server-side HTML generation

4.5 Auth

• Enabled when SPACE_ID + OAUTH_CLIENT_ID are set
• OAuth 2.0 flow with HF as provider; cookie hf_access_token (httpOnly, secure, sameSite: none)
• resolveUser: whoAmI via @huggingface/hub, then checkWriteAccess (Space owner or org member with write/admin role)
• In-memory state map with 10-min TTL for CSRF protection

4.6 AI Agent

• Provider: Hugging Face Inference Providers (https://router.huggingface.co/v1), default model openai/gpt-oss-120b. Model ids may be suffixed with :<provider> (e.g. meta-llama/Llama-3.3-70B-Instruct:together) to bypass providers that enforce overly strict tool-call validation (notably Groq) or that don't support the tools parameter (Nscale, etc.).
• Auth: per-request bearer token resolved from the editor's OAuth cookie when available, falling back to the server-side HF_TOKEN. On a HF Space with inference-api scope, no extra secret is needed - the logged-in user pays for their own inference under their HF quota.
• Streaming via Vercel AI SDK streamText over @ai-sdk/openai-compatible
• Reasoning parts from prior assistant turns are stripped before re-sending the history: providers like Cerebras reject reasoning_content on round-trip, and the model doesn't need to see its own past reasoning to continue the conversation.
• Context: document text, current selection, frontmatter (sent by frontend with each message)
• Tools (declarative, executed client-side by the frontend):
  • replaceSelection - replace selected text
  • insertAtCursor - insert at cursor position
  • applyDiff - search/replace in document
  • updateFrontmatter - modify metadata fields
  • addAuthor / removeAuthor - manage author list
• Agent edits are grouped in a single Yjs UndoManager batch for Cmd+Z

4.7 Citations

• Uses @citation-js/core with bibtex, doi, csl plugins
• Resolve: DOI URL or identifier to CSL-JSON entries
• Format: entries + style + locale to HTML bibliography
• Import: BibTeX string to CSL-JSON

---

5. Frontend Components

5.1 App Shell

• No router - single view with conditional rendering
• Theme: CSS custom properties with dynamic primary color from settings.primaryHue (OKLCH color model, synced via Yjs settings)
• Layout: top bar (undo/redo, settings, publish, user chip) + 3-column CSS grid (TOC / editor / comments)
• Chat: floating button bottom-left, ChatPanel overlay
• Modals: comment dialog, settings drawer, publish confirmation

5.2 Editor

• Creates Y.Doc + HocuspocusProvider (WebSocket to /collab)
• Seeding: after provider synced event only, if Y.XmlFragment("default") is empty, inserts DEFAULT_CONTENT + seedFrontmatter + SEED_CITATIONS
• Yjs Maps: citations, settings, comments, frontmatter (via dedicated stores)
• Image handling: paste/drop with upload to /api/upload

5.3 TipTap Extensions

Built-in (configured):

• StarterKit (no codeBlock, no undo), CodeBlockLowlight (all languages), Placeholder, Collaboration, CollaborationCursorV3, Mathematics (KaTeX), Image, Table/Row/Cell/Header

Custom:

• CollaborationUndo - bridges Yjs UndoManager for agent batch edits
• Comment - inline mark with commentId + resolved
• SlashCommands - / trigger with suggestion popup
• ImageUpload - drag-drop upload node with progress
• Citation - inline atomic node (key + label), links to citationsMap
• Bibliography - block node with rendered HTML from citations
• Glossary - inline atomic (term + definition tooltip)
• Footnote - inline atomic (content shown in footer)
• Stack + StackColumn - multi-column layout (2/3/4 cols)

5.4 Component System

Registry-based system for MDX-like custom components:

Component	Kind	Purpose
accordion	wrapper	Collapsible section (details/summary)
note	wrapper	Info/warning/danger/success callout
quoteBlock	wrapper	Styled blockquote
wide	wrapper	Content wider than column
fullWidth	wrapper	Full viewport width
sidenote	wrapper	Marginal note
reference	wrapper	Reference container
htmlEmbed	atomic	External HTML embed (iframe)
hfUser	atomic	HF user card
rawHtml	atomic	Raw HTML injection
mermaid	atomic	Mermaid diagram (live preview)

• Factory: createComponentExtension(def) generates TipTap nodes from registry definitions (handles both wrapper and atomic kinds)
• NodeViews: WrapperView (editable content area + chrome), AtomicView (placeholder + field editor), MermaidView (textarea + SVG preview)
• Slash menu integration: each component generates a slash menu item via getComponentSlashItems()

5.5 Frontmatter System

• FrontmatterStore: wraps Y.Map + Y.Array for real-time collaborative metadata editing
• useFrontmatter hook: React state synced with Yjs observations
• FrontmatterHero: WYSIWYG editable hero section (title, subtitle, authors, affiliations, date, DOI)
• SettingsDrawer: template variant, SEO, banner, citation style, primary color hue slider, PDF/TOC/licence toggles
• HueSlider: OKLCH hue picker (0-360) with live preview, synced to settingsMap.primaryHue

5.6 Other UI

• TableOfContents: extracts headings from TipTap doc, scroll-based active state, collapsible sub-sections
• ChatPanel: message list + quick actions on selection + input with streaming
• CommentPopover: positioned comment popover anchored to the active thread (resolve/delete inline)
• BubbleToolbar: floating toolbar on text selection (bold, italic, link, comment, etc.)
• BlockHandle: drag handle for block-level nodes

5.7 CSS Architecture

styles/
  _variables.css       # Template tokens: --primary-color, breakpoints, @custom-media
  _reset.css           # Scoped reset for article content
  _base.css            # Typography, scoped to article content
  _layout.css          # 3-column grid, .wide/.full-width helpers
  _print.css           # Print styles
  _ui.css              # Editor chrome: buttons, dialogs, drawers, spinner
  tokens.css           # Design tokens (light/dark): text, bg, accent, code, danger, shadows
  article.css          # .tiptap content styles (shared editor/published)
  toc.css              # Editor TOC overrides
  editing.css          # Editor-only: layout, cursors, slash menu
  _publisher.css       # Published-only: theme toggle, wide/fullWidth, footer, lightbox
  components/
    _code.css          # Code blocks + syntax highlighting
    _table.css         # Tables
    _tag.css           # Tags
    _card.css          # Cards
    _mermaid.css       # Mermaid diagrams
    _embed.css         # Embed containers
    _embed-studio.css  # Embed studio overlay
    _hero.css          # Hero section (from template)
    _toc.css           # Base TOC styles (from template)
    _button.css        # Buttons (template)
    _form.css          # Form elements (template)
    _footer.css        # Footer (template)

The publisher reads these same CSS files server-side and injects them inline into published HTML, using resolveCustomMedia() to expand @custom-media queries into standard @media rules.

---

6. Deployment

6.1 Docker Build (3-stage)

1. frontend-build: npm install + npm run build (Vite)
2. backend-build: npm install + npx tsc
3. runtime: node:20-slim + Chromium system deps + npm install --omit=dev + Playwright Chromium + copy frontend-dist/ + copy frontend/src/styles/ to frontend-styles/

CMD: node dist/server.js on port 8080.

6.2 HF Space Configuration (README.md frontmatter)

• SDK: docker, port 8080
• OAuth: hf_oauth: true, scopes: manage-repos, inference-api
• Two git remotes: space (tfrere/collab-editor, dev) and prod (tfrere/research-article-template-editor, production)

6.3 Environment Variables

Variable	Required	Purpose
PORT	No (default 8080)	HTTP listen port
NODE_ENV	No	production switches to frontend-dist path
SPACE_ID	For OAuth/HF	HF Space identifier, enables OAuth + dataset
SPACE_HOST	For OAuth	HTTPS callback URL host
OAUTH_CLIENT_ID	For OAuth	HF OAuth client
OAUTH_CLIENT_SECRET	For OAuth	HF OAuth secret
OAUTH_SCOPES	No (default openid profile)	OAuth scopes. Add manage-repos for dataset persistence and inference-api to power AI features with the user's token
HF_DATASET_ID	No	Override dataset name (default: {SPACE_ID}-data)
HF_TOKEN	For AI chat in local dev	Fallback Hub token for HF API + Inference Providers. Needs the "Make calls to Inference Providers" permission
HF_INFERENCE_MODEL	No (default openai/gpt-oss-120b)	Default chat-completion model id served by HF Inference Providers. May be suffixed with :<provider> to pin a specific routing
ENABLE_PDF	No (default true)	Toggle PDF/thumbnail generation

6.4 Local Development

# Terminal 1 - Backend
cd backend && npm install && npm run dev
# Starts on http://localhost:8080

# Terminal 2 - Frontend
cd frontend && npm install && npm run dev
# Starts on http://localhost:5678 (proxies /api and /collab to :8080)

Create a .env file in backend/ with at minimum HF_TOKEN for AI chat (must have the "Make calls to Inference Providers" permission). Without SPACE_ID, OAuth is disabled and all users can edit.

---

7. Key Data Flows

7.1 Collaborative Editing

sequenceDiagram
  participant ClientA as Client A
  participant Server as Hocuspocus
  participant ClientB as Client B
  participant Disk as Local FS
  participant HF as HF Dataset

  ClientA->>Server: WebSocket connect /collab
  Server->>Disk: Database.fetch (load .yjs)
  Server-->>ClientA: sync Y.Doc state
  ClientA->>Server: Y.Doc update (edit)
  Server->>ClientB: broadcast update
  Server->>Disk: Database.store (write .yjs)
  Disk-->>HF: schedulePush (10s debounce)

7.2 Publish Flow

sequenceDiagram
  participant User as Editor UI
  participant API as POST /api/publish
  participant HP as Hocuspocus
  participant Pub as Publisher
  participant FS as Local FS
  participant HF as HF Dataset

  User->>API: Click Publish
  API->>HP: openDirectConnection
  HP-->>API: Y.Doc snapshot
  API->>FS: Write .yjs snapshot
  API->>Pub: publishDocument()
  Pub->>Pub: extractFromYDoc + loadCSS
  Pub->>Pub: renderArticleHTML + PDF
  Pub->>FS: Write index.html locally
  Pub->>HF: uploadPublishedAssets
  Pub-->>API: { htmlUrl, pdfUrl, success }
  API-->>User: Publish result

7.3 Published Article Lifecycle (Container Restarts)

HF Spaces containers are ephemeral. The local filesystem is wiped on every restart (git push, Space rebuild, idle timeout). The published article survives via this restore flow:

sequenceDiagram
  participant Container as New Container
  participant FS as Local FS
  participant HF as HF Dataset
  participant Visitor as GET /

  Container->>Container: Server starts
  Container->>HF: ensurePublishedRestored()
  HF-->>FS: Pull index.html, PDF, meta.json
  Note over FS: data/published/default/index.html

  Visitor->>Container: GET /
  Container->>FS: Check published path
  FS-->>Container: index.html exists
  Container-->>Visitor: Serve published article

On publish, HTML is always written locally first (so GET / serves the new version immediately), then uploaded to HF dataset for persistence across restarts.

7.4 AI Agent Chat

sequenceDiagram
  participant User as Chat Panel
  participant Hook as useAgentChat
  participant API as POST /api/chat
  participant LLM as HF Inference

  User->>Hook: sendMessage(text)
  Hook->>Hook: Build context (doc, selection, frontmatter)
  Hook->>API: { messages, context }
  API->>LLM: streamText (system prompt + tools)
  LLM-->>API: Stream (text + tool_calls)
  API-->>Hook: SSE stream
  Hook->>Hook: Execute tool calls client-side
  Note over Hook: replaceSelection, applyDiff,<br/>updateFrontmatter, etc.
  Hook->>Hook: UndoManager batch for Cmd+Z

---

8. Current Limitations and Known Issues

• Test suite in progress: P0 tests being added (see docs/TESTS.md)
• Single document: only "default" document supported; no multi-doc
• Single-user token: last OAuth token cached globally for all HF API calls
• No rate limiting on /api/chat or /api/citations/*
• XSS surface: meta.licence and biblioHtml not escaped in published HTML
• WS debug logging: every WebSocket message logged in production
• No .env.example: environment variables documented only in code