feat(storage): private dataset by default + /d/* reverse proxy

The dataset that backs persistence was being created `private: false`
on every first write. For org-owned Spaces with strict data
policies (notably `HuggingFaceBio/carbon-tokenization`) this could
either silently fail (org blocks public dataset creation) or be
flat-out wrong (working drafts of an internal research paper
shouldn't be sitting on a public dataset).

Flip the default to `private: true`. The complication this creates
is that published articles' `<img>` tags, og:image and PDF
download all pointed at `huggingface.co/datasets/.../resolve/...`
URLs that would now 401 for anonymous viewers. Solve that with a
small reverse-proxy route on the editor itself.

- New `GET /d/:path*` handler streams from
`huggingface.co/datasets/${id}/resolve/main/${path}` to the
caller, attaching the right HF token along the way:
1. cookie token from the current request (most viewers are
signed in on private Spaces anyway)
2. last-known cached user token (any signed-in request warms
the proxy for subsequent anonymous viewers)
3. `HF_TOKEN` env fallback (operator-set)
4. anonymous (works only when dataset is public)
- Path whitelist hard-coded to `images/` + `published/`. `articles/`
(raw `.yjs` drafts) is *never* exposed through the proxy. Path
traversal sequences (`..`, leading `/`, NUL bytes) are dropped
before the whitelist check.
- Body is piped via `Readable.fromWeb` so large PDFs don't pin Node
heap.
- Cache-Control:
- `images/*` are UUID-named and never overwritten => `immutable,
max-age=1y`. Browsers skip revalidation entirely.
- `published/*` is republished in place => `max-age=300,
stale-while-revalidate=60` so a freshly-published article
propagates within minutes without thundering-herding HF on every
page load.
- Upstream 401/403 collapse to 502 instead of being forwarded, so
the browser never gets prompted for credentials it can't supply.
Upstream 404 passes through.

`hf-storage.ts` changes:
- `createRepo({ private: true })` + clearer log on the 409-vs-real
failure split (now includes the HTTP status, the most common
cause being a missing `manage-repos` OAuth scope).
- `uploadImageToHf` returns `/d/images/<file>` instead of the raw HF
URL.
- `getPublishedAssetUrl` returns
`https://${SPACE_HOST}/d/published/...` (absolute on purpose:
og:image gets fetched by external unfurlers that don't know our
origin; the PDF download link can be hit from other sites too).
Falls back to `http://localhost:${PORT}` in dev.
- `resolveToken` is now exported so the proxy can reuse the same
cascade without duplicating it.

Backward compat: articles that already have hard-coded
`huggingface.co/datasets/.../resolve/...` image URLs in their Y.js
state keep working because their dataset is still public (the
private default only applies to *new* dataset creations - existing
ones aren't migrated). For brand-new deployments (the bio case),
everything is `/d/*` from day one and the dataset is private.

Docs: TESTS.md gains rows 2.3.3 / 2.3.4 (proxy serves images,
proxy whitelist enforced); SPECIFICATION.md 4.3 calls out the
private default + a new 4.3.1 documenting the proxy contract.

Co-authored-by: Cursor <cursoragent@cursor.com>

backend/src/create-app.ts

@@ -15,6 +15,7 @@ import { createAuthRouter, createRequireEditor } from "./routes/auth.js";
 import { createChatRouter } from "./routes/chat.js";
 import { createPublishRouter } from "./routes/publish.js";
 import { createUploadRouter } from "./routes/upload.js";
+import { createDatasetProxyRouter } from "./routes/dataset-proxy.js";
 
 export { debouncedSave, resetSaveTimers, resetPublishedRestored } from "./persistence.js";
 
@@ -408,6 +409,10 @@ export function createApp() {
   app.use("/api/citations", citationsRouter);
   app.use(createPublishRouter({ oauthEnabled, hocuspocus }));
   app.use(createUploadRouter());
+  // Reverse proxy for private-dataset assets. Mounted before any
+  // static serving so `/d/*` always wins, never falls through to a
+  // 404 from express.static.
+  app.use(createDatasetProxyRouter());
 
   // ---------- Collab WebSocket ----------
 

backend/src/hf-storage.ts

@@ -2,6 +2,7 @@ import { uploadFile, downloadFile, createRepo, commit, type CommitFile, type Rep
 import { sanitizeName } from "./utils.js";
 
 const SPACE_ID = process.env.SPACE_ID || "";
+const SPACE_HOST = process.env.SPACE_HOST || "";
 
 // Derive dataset ID: explicit env > "<space-id>-data"
 const HF_DATASET_ID =
@@ -28,13 +29,48 @@ function getToken(explicit?: string): string {
   return explicit || _cachedToken || ENV_TOKEN;
 }
 
+/**
+ * Public wrapper around the same token cascade used internally. The
+ * dataset-proxy route needs to forward the right token to HF when
+ * fetching assets on behalf of a viewer.
+ */
+export function resolveToken(explicit?: string): string {
+  return getToken(explicit);
+}
+
 export function isHfStorageEnabled(): boolean {
   return Boolean(HF_DATASET_ID && (_cachedToken || ENV_TOKEN));
 }
 
 /**
- * Ensure the backing dataset exists on HF (public).
- * Safe to call multiple times; ignores "already exists".
+ * Public-facing base URL the editor is reachable at, used to build
+ * absolute proxy URLs for assets that need to render outside the
+ * Space's own page (og:image consumed by social card unfurlers,
+ * "Download PDF" link from external sites, ...). In dev we don't
+ * have SPACE_HOST so we fall back to localhost.
+ */
+function getPublicBaseUrl(): string {
+  if (SPACE_HOST) return `https://${SPACE_HOST}`;
+  const port = process.env.PORT || "8080";
+  return `http://localhost:${port}`;
+}
+
+/**
+ * Ensure the backing dataset exists on HF.
+ *
+ * The dataset is created **private by default**: most editor
+ * deployments live in organisations where the working drafts must
+ * not leak, and even for solo public projects the published assets
+ * (images, PDFs, thumbnails) flow back through the editor's own
+ * `/d/...` reverse proxy anyway, so anonymous viewers never need
+ * direct dataset access. Keeping it private also avoids running
+ * into orgs that block `private: false` creations by policy.
+ *
+ * Safe to call multiple times; treats 409 as success and caches a
+ * "ready" flag so we don't retry on every push. A non-409 failure
+ * is logged with the HTTP status when available (the most common
+ * cause is a missing `manage-repos` scope on the user's OAuth
+ * grant) and the flag is *not* set, so the next push will retry.
  */
 let _datasetReady = false;
 
@@ -45,13 +81,22 @@ export async function ensureDatasetExists(token?: string): Promise<void> {
   if (!accessToken) return;
 
   try {
-    await createRepo({ repo, private: false, accessToken });
-    console.log(`[hf-storage] created public dataset ${HF_DATASET_ID}`);
+    await createRepo({ repo, private: true, accessToken });
+    console.log(`[hf-storage] created private dataset ${HF_DATASET_ID}`);
   } catch (err: any) {
-    if (err?.statusCode === 409 || String(err?.message).includes("already")) {
+    const statusCode = err?.statusCode ?? err?.status;
+    const message = String(err?.message || err);
+    if (statusCode === 409 || message.includes("already")) {
       console.log(`[hf-storage] dataset ${HF_DATASET_ID} already exists`);
     } else {
-      console.error(`[hf-storage] failed to create dataset:`, err);
+      // Surface enough detail to triage from Space logs: status code
+      // (403 = scope/permission, 422 = invalid name, 5xx = HF down)
+      // and a one-line message. We deliberately omit the full error
+      // object to avoid noisy stack dumps on every retry.
+      console.error(
+        `[hf-storage] failed to create dataset ${HF_DATASET_ID}` +
+          ` (status=${statusCode ?? "unknown"}): ${message}`,
+      );
       return;
     }
   }
@@ -75,7 +120,12 @@ export async function uploadImageToHf(
     commitTitle: `upload image ${filename}`,
   });
 
-  return `https://huggingface.co/datasets/${HF_DATASET_ID}/resolve/main/${path}`;
+  // Relative URL through the editor's reverse proxy. The dataset is
+  // private and the raw `huggingface.co/datasets/...` URL would 401
+  // for anonymous viewers - the proxy attaches a server-side token
+  // and forwards. Relative is fine because images are only ever
+  // rendered from inside the Space's own pages.
+  return `/d/${path}`;
 }
 
 // ---------- Documents ----------
@@ -242,12 +292,18 @@ export async function pullPublishedAssets(
 // ---------- Published Assets ----------
 
 /**
- * Build the public resolve URL for a published asset.
- * Useful to pre-compute og:image / pdf URLs before HTML generation.
+ * Build the proxied URL for a published asset.
+ *
+ * Returns an **absolute** URL because consumers are sometimes
+ * external (Twitter / Slack / Discord card unfurlers reading
+ * og:image, or external links to the PDF download), and a relative
+ * `/d/...` wouldn't resolve there. Inside the Space's own pages a
+ * relative URL would have worked equally well, but emitting the
+ * absolute form keeps the contract uniform.
  */
 export function getPublishedAssetUrl(docName: string, filename: string): string {
   const safeName = sanitizeName(docName);
-  return `https://huggingface.co/datasets/${HF_DATASET_ID}/resolve/main/published/${safeName}/${filename}`;
+  return `${getPublicBaseUrl()}/d/published/${safeName}/${filename}`;
 }
 
 interface PublishedPayload {

backend/src/routes/dataset-proxy.ts

@@ -0,0 +1,162 @@
+import { Router, type Request, type Response } from "express";
+import { Readable } from "stream";
+import { extractToken } from "../auth.js";
+import {
+  getDatasetId,
+  resolveToken,
+  setUserToken,
+} from "../hf-storage.js";
+
+/**
+ * Authenticated reverse proxy for the backing HF dataset.
+ *
+ * Why this exists
+ * ---------------
+ * The dataset that backs the editor is **private by default** (see
+ * `hf-storage.ts::ensureDatasetExists`), but the published article's
+ * `<img>` tags and the og:image / pdf links still need to be
+ * resolvable from a normal browser - including by anonymous viewers
+ * of a public Space. We can't just hand out
+ * `huggingface.co/datasets/.../resolve/...` URLs because the dataset
+ * itself is gated. Instead, the editor server proxies those reads on
+ * the viewer's behalf, attaching a token it has on hand.
+ *
+ * Token resolution cascade (most specific to most ambient):
+ *   1. Cookie from the current request (if the viewer is signed in)
+ *   2. Last-known cached user token (warmed by any prior signed-in
+ *      request - see `hf-storage::setUserToken`)
+ *   3. The server-side `HF_TOKEN` env (when an operator set one)
+ *   4. No token at all (works only for public datasets)
+ *
+ * Path whitelist
+ * --------------
+ * Only `images/` and `published/` may flow through the proxy. We
+ * explicitly never expose `articles/` (raw Y.js `.yjs` snapshots
+ * including unpublished drafts) and never expose the dataset root
+ * either. Anything outside the whitelist gets a 404 - leaking the
+ * proxy's existence is fine, leaking the path layout is not.
+ *
+ * Streaming
+ * ---------
+ * Responses are piped straight from `fetch`'s WHATWG body to the
+ * Express `res`, so we never buffer a full PDF or large image in
+ * Node memory. The cost is a tiny shim through `Readable.fromWeb`.
+ *
+ * Cache-Control
+ * -------------
+ * - `images/*` filenames are UUID-based and content-addressed in
+ *   practice (the editor never overwrites an existing upload), so
+ *   they get `immutable, max-age=1y` - browsers will skip
+ *   revalidation entirely.
+ * - `published/*` is mutable (every republish overwrites it) so we
+ *   cap at a short max-age and tell intermediaries to revalidate.
+ */
+export function createDatasetProxyRouter(): Router {
+  const router = Router();
+
+  router.get(/^\/d\/(.+)$/, async (req: Request, res: Response) => {
+    const datasetId = getDatasetId();
+    if (!datasetId) {
+      res.status(503).json({ error: "Dataset persistence not configured" });
+      return;
+    }
+
+    const relPath = (req.params as { 0?: string })[0] || "";
+    if (!isPathAllowed(relPath)) {
+      // 404, not 403: don't confirm whether the caller "almost" had
+      // the right path. An attacker probing for /d/articles/foo.yjs
+      // gets the same response as one probing for /d/garbage.
+      res.status(404).json({ error: "Not found" });
+      return;
+    }
+
+    // Promote a cookie token to the cache opportunistically: even if
+    // this very request was authenticated, future anonymous requests
+    // from the same Space session benefit from a warm cache.
+    const cookieToken = extractToken(req.headers.cookie);
+    if (cookieToken) setUserToken(cookieToken);
+
+    const token = resolveToken(cookieToken);
+    const upstreamUrl = `https://huggingface.co/datasets/${datasetId}/resolve/main/${encodePath(relPath)}`;
+
+    let upstream: Response | globalThis.Response;
+    try {
+      upstream = await fetch(upstreamUrl, {
+        headers: token ? { Authorization: `Bearer ${token}` } : {},
+      });
+    } catch (err) {
+      console.error(`[dataset-proxy] fetch failed for ${relPath}:`, (err as Error).message);
+      res.status(502).json({ error: "Upstream fetch failed" });
+      return;
+    }
+
+    if (!upstream.ok || !upstream.body) {
+      // Don't reflect upstream 401 directly: the viewer can't do
+      // anything about a missing server-side token, and surfacing
+      // 401 would trigger browser auth prompts in some setups.
+      // Map any non-2xx to a generic 404 + log the real status for
+      // operators reading the Space logs.
+      console.warn(
+        `[dataset-proxy] ${relPath}: upstream status ${upstream.status} (token=${token ? "yes" : "no"})`,
+      );
+      res.status(upstream.status === 404 ? 404 : 502).json({
+        error: upstream.status === 404 ? "Not found" : "Upstream error",
+      });
+      return;
+    }
+
+    // Forward content-type + content-length when present so the
+    // browser renders the right viewer and shows progress for PDFs.
+    const ct = upstream.headers.get("content-type");
+    if (ct) res.setHeader("Content-Type", ct);
+    const cl = upstream.headers.get("content-length");
+    if (cl) res.setHeader("Content-Length", cl);
+
+    res.setHeader("Cache-Control", cacheControlFor(relPath));
+
+    // Stream the WHATWG body straight to the Express response. The
+    // `as any` is required because @types/node's `Readable.fromWeb`
+    // doesn't accept the DOM `ReadableStream` type without a cast.
+    const nodeStream = Readable.fromWeb(upstream.body as any);
+    nodeStream.on("error", (err) => {
+      console.error(`[dataset-proxy] stream error for ${relPath}:`, err.message);
+      if (!res.headersSent) res.status(502).end();
+      else res.end();
+    });
+    nodeStream.pipe(res);
+  });
+
+  return router;
+}
+
+const ALLOWED_PREFIXES = ["images/", "published/"];
+
+function isPathAllowed(path: string): boolean {
+  // Reject path traversal up-front. `decodeURIComponent` would let
+  // an attacker hide `..` segments behind URL encoding, but Express
+  // already decodes the wildcard param for us, so a plain string
+  // check is sufficient.
+  if (path.includes("..") || path.includes("\0") || path.startsWith("/")) {
+    return false;
+  }
+  return ALLOWED_PREFIXES.some((prefix) => path.startsWith(prefix));
+}
+
+/**
+ * Re-encode the path before sending it upstream so spaces / unicode
+ * filenames don't trip HF's URL parser. We split on `/` so the
+ * slashes stay literal.
+ */
+function encodePath(path: string): string {
+  return path.split("/").map(encodeURIComponent).join("/");
+}
+
+function cacheControlFor(path: string): string {
+  if (path.startsWith("images/")) {
+    // UUID names + no overwrite => safe to mark immutable.
+    return "public, max-age=31536000, immutable";
+  }
+  // Published assets get republished in place, so cap aggressively
+  // but allow shared caches to serve while revalidating.
+  return "public, max-age=300, stale-while-revalidate=60";
+}

docs/SPECIFICATION.md

@@ -92,12 +92,23 @@ All types are concurrently editable by multiple users and persist to `data/defau
 ### 4.3 HF Storage
 
 - Dataset ID: `HF_DATASET_ID` or `{SPACE_ID}-data`
+- Dataset is created **private by default** (`createRepo({ private: true })`). The OAuth grant needs `manage-repos` on the user's first write; subsequent containers reuse the cached token.
 - Token: `HF_TOKEN` (env) or cached OAuth token from last authenticated user
 - **Documents**: `articles/<name>.yjs` - debounced push on every Hocuspocus store
-- **Published assets**: `published/<name>/{index.html, article.pdf, thumb.jpg, meta.json}`
-- **Images**: `images/<uuid-filename>` with public resolve URL
+- **Published assets**: `published/<name>/{index.html, article.pdf, thumb.jpg, meta.json, llms.txt}`
+- **Images**: `images/<uuid-filename>` referenced from articles via `/d/images/...` proxy URLs
 - `flushAll()` on `SIGTERM`/`SIGINT` to push pending changes
 
+### 4.3.1 Dataset Reverse Proxy (`/d/*`)
+
+Since the dataset is private, anonymous viewers of a published article can't fetch its images / PDF / og:image directly from `huggingface.co/datasets/...`. The editor server exposes `GET /d/:path*` as an authenticated forward-proxy:
+
+- **Whitelist**: only `images/` and `published/` are reachable; `articles/` (raw `.yjs` drafts) is **always 404** regardless of caller.
+- **Token cascade**: request cookie → cached user token → `HF_TOKEN` env → anonymous fetch. The cookie token is also promoted into the cache opportunistically, so the first signed-in viewer warms the proxy for subsequent anonymous viewers within the same container lifetime.
+- **Streaming**: WHATWG body piped straight to the Express response - no buffering of full PDFs in Node memory.
+- **Caching**: `images/*` is served as `immutable, max-age=1y` (UUID names, never overwritten); `published/*` as `max-age=300, stale-while-revalidate=60` (re-published in place).
+- **Error mapping**: upstream 401/403 collapse to 502 so the browser never gets prompted for credentials it can't supply; upstream 404 passes through.
+
 ### 4.4 Publisher Pipeline
 
 ```mermaid

docs/TESTS.md

@@ -85,8 +85,10 @@ Data loss is game over for a collaborative editor.
 
 | # | Test | Given / When / Then | Priority |
 |---|------|---------------------|----------|
-| 2.3.1 | Upload returns resolve URL | Given a valid image file / When POST /api/upload / Then response contains a public URL for the image | P1 |
+| 2.3.1 | Upload returns proxy URL | Given a valid image file / When POST /api/upload / Then response contains a `/d/images/...` URL routed through the editor's dataset proxy (the underlying HF dataset is private) | P1 |
 | 2.3.2 | Reject oversized file | Given a file > 10MB / When POST /api/upload / Then response is 413 or 400 with error message | P1 |
+| 2.3.3 | Proxy serves images | Given an uploaded image / When GET `/d/images/<file>` / Then 200 + image bytes (proxy attaches a server-side token to fetch the private HF dataset) | P1 |
+| 2.3.4 | Proxy whitelist | Given any path under `/d/articles/...` (raw Y.js drafts) / When GET / Then 404 - never expose drafts via the proxy | P0 |
 
 ---