Upload 7 files

DATASET_COLLECTION_GUIDANCE.md

@@ -465,3 +465,49 @@ dual-source collection:
 | `ai-assistant.js tRecords` | Added `_source: 'contribution'` to each record for self-describing POST payload |
 | `deduplicate_dataset.py` | Post-loop filter removes tombstone winners from clean output; `main()` reports tombstone count separately so duplicate counts are accurate |
 | `DATASET_COLLECTION_GUIDANCE.md` | This document; added §3 Retraction tombstones, §6 tombstone exclusion note, §8 this table |
+
+## 9. Viewing the dataset from the AI panel
+
+The Extended Settings sheet (Endpoint Configuration → **Dataset Endpoint**)
+surfaces the dataset directly in the browser, with no secret ever leaving the
+server.
+
+How the panel resolves the dataset repo (two sources, first match wins):
+
+1. **Explicit** — `ai_assistant_panel_dataset_repo = "org/repo"` in `conf.py`.
+   Highest trust; no network call. Use for offline docs or to pin a specific
+   repo.
+2. **Auto-discovery** — when a training URL is configured, the panel issues
+   `GET {proxyBase}/` and reads `training.dataset_repo` from the JSON. The
+   proxy already publishes this field, so no `conf.py` change is required for
+   standard deployments.
+
+From the repo id the panel builds three clickable links:
+
+| Card | URL |
+|------|-----|
+| Dataset root | `https://huggingface.co/datasets/<repo>` |
+| Feedback records | `https://huggingface.co/datasets/<repo>/tree/main/feedback` |
+| Contributions | `https://huggingface.co/datasets/<repo>/tree/main/contributions` |
+
+**Token posture (read-only).** The same `GET /` response also reports
+`tokens.hf_token_type`, `tokens.hf_write_token_type`, and
+`tokens.least_privilege_mode`. The panel shows these as a short
+read/write/fine-grained summary so operators can confirm least-privilege
+configuration **without** reading Space logs. Token *values* are never sent to
+the browser. When neither source is reachable, the section degrades to a
+"Not configured" hint and the Space repository secret
+(`TRAINING_DATASET_REPO`) continues to drive persistence server-side — both
+the panel and repo-secret approaches are fully supported.
+
+## 10. Summary of Changes (vNEXT)
+
+| Component | Change |
+|-----------|--------|
+| `_static/ai-assistant.js` | Added **Dataset Endpoint** subsection (E) to `_buildEndpointSheet`: proxy-discovery + explicit-config dataset links and read-only HF token-posture row |
+| `_static/ai-assistant.js` | Project Links sheet now renders HuggingFace **Space / Dataset / Active Endpoint** cards (config-driven or auto-derived) |
+| `_static/ai-assistant.js` | New **self-contained URL-hash share** tier (`#ai-share-c1.<fmt>.<base64url>`): a real navigable, server-free, any-device/any-browser permanent link; routed in `_checkShareHash` and used by the permanent-link button (data: URI kept as fallback) |
+| `_static/ai-assistant.css` | Added `.ai-assistant-panel-ep-ext-dataset-*` rules (light + dark) for the Dataset Endpoint section |
+| `__init__.py` | Injects `panelDatasetRepo` + `panelHf*` keys into `window.AI_ASSISTANT_CONFIG` and registers the matching `ai_assistant_panel_*` config values |
+| `_example_conf.py` | Documented examples for every new key |
+| `app.py` | **No change** — `GET /` already exposes `training.*` and `tokens.*` |

_dataset_schema.py

@@ -0,0 +1,872 @@
+# scikitplot/_externals/_sphinx_ext/_sphinx_ai_assistant/_hf_spaces_proxy/dataset_schema.py
+#
+# Authors: The scikit-plots developers
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# Canonical schema and record-normalization for the AI-assistant dataset.
+# Version: 1.0.0
+
+"""
+Canonical schema, normalization, and pandas loading for the AI-assistant dataset.
+
+Background
+----------
+Two independent server endpoints write to the same HuggingFace dataset repo:
+
+* ``POST /v1/feedback``   → ``feedback/TIMESTAMP.jsonl``
+* ``POST /v1/contribute`` → ``contributions/TIMESTAMP.jsonl``
+
+Before this module, these two paths used *different* field names for the same
+logical concept (e.g. ``conversationId`` vs ``_sessionId``, ``page`` vs
+``_page``, ``model`` vs ``_model``), had an inconsistent ``ratingLabel``
+type (snake_case slug for panel feedback; Title Case string for quick 👍/👎
+feedback), and copied the entire raw client payload into feedback records
+(including the legacy ``rating`` alias and unfiltered extra fields).
+
+This module fixes all of those issues by providing a single canonical schema
+that **both** endpoints write.  Every stored JSONL row is an output of
+:func:`normalize_feedback_record` or :func:`normalize_contribution_record`.
+Old records written before this fix can be read through :func:`normalize_record`
+which back-fills the new fields from the legacy ones.
+
+Canonical key order (identical in every row)
+--------------------------------------------
+::
+
+    schemaVersion
+    _source           _ts            _dedup_key
+    conversationId    feedbackId
+    answerIndex       action         prevFeedbackId    status
+    ratingValue       ratingSlug     ratingTitle       ratingMode    message
+    query             answer
+    model
+    page              consentVersion
+    ts
+
+Notes
+-----
+User note
+    Load the full dataset in one line::
+
+        from dataset_schema import load_dataset
+        df = load_dataset("feedback/", "contributions/")
+
+Developer note — Rating vocabulary
+    ``ratingLabel`` is now always a snake_case slug (canonical identifier).
+    ``ratingTitle`` carries the human-readable display string for dashboards.
+    Old quick-feedback records that stored ``ratingLabel = "Not helpful"`` are
+    normalised: the Title Case string is moved to ``ratingTitle`` and a slug
+    derived to populate ``ratingSlug`` / ``ratingLabel``.
+
+Developer note — Model shape
+    Feedback records used ``model: {id, provider, model}`` (3 keys).
+    Contribution records used ``_model: {default, id, label, provider, model,
+    endpoint, info_url, description}`` (8 keys).  Both are now written as
+    ``model`` with all 8 keys present; missing keys are ``None``.
+
+Developer note — Retraction records
+    A retraction payload has ``action="retract"`` and ``prevSessionId`` pointing
+    to the ``sessionId`` (= ``feedbackId``) of the record being invalidated.
+    The normalised form uses ``prevFeedbackId`` for clarity and fills all rating /
+    content / model fields with ``None``.
+
+References
+----------
+See ``DATASET_COLLECTION_GUIDANCE.md`` for the deduplication contract and
+the training-pipeline usage of ``_source``, ``_dedup_key``, and ``status``.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from collections import OrderedDict
+from pathlib import Path
+from typing import Any
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Schema constants
+# ─────────────────────────────────────────────────────────────────────────────
+
+#: Current schema version for records written by this module.
+#: Increment when a breaking field-name change is introduced.
+SCHEMA_VERSION: int = 1
+
+#: Ordered list of canonical column names.  Every stored JSONL row and every
+#: row in the pandas DataFrame will have these columns in exactly this order.
+CANONICAL_COLUMNS: list[str] = [
+    # ── Schema metadata ───────────────────────────────────────────────────────
+    "schemaVersion",
+    # ── Provenance (server-side, mandatory) ──────────────────────────────────
+    "_source",          # "feedback" | "contribution"
+    "_ts",              # server receive time, ms since epoch (int)
+    "_dedup_key",       # "{conversationId}:{answerIndex}"
+    # ── Session identity ──────────────────────────────────────────────────────
+    "conversationId",   # stable per-page-load chat session UUID
+    "feedbackId",       # per-feedback-event idempotency key (None for contributions)
+    # ── Record descriptor ─────────────────────────────────────────────────────
+    "answerIndex",      # 0-based position of answer in the conversation
+    "action",           # "rate" | "retract"
+    "prevFeedbackId",   # for retracts: feedbackId of the record being invalidated
+    "status",           # "active" | "retracted"  (dedup pipeline manages)
+    # ── Rating ────────────────────────────────────────────────────────────────
+    "ratingValue",      # int | None: numeric score (-5..+5 for panel; -1|+1 for quick)
+    "ratingSlug",       # str | None: snake_case canonical slug ("helpful", "mostly_positive")
+    "ratingTitle",      # str | None: human display string ("Helpful", "Mostly yes")
+    "ratingMode",       # str | None: "quick" | "panel"
+    "message",          # str: free-text user comment (empty string when absent)
+    # ── Conversation content ──────────────────────────────────────────────────
+    "query",            # str: user question
+    "answer",           # str: model response
+    # ── Model ────────────────────────────────────────────────────────────────
+    "model",            # dict | None: normalised 8-key model object (see MODEL_KEYS)
+    # ── Context ───────────────────────────────────────────────────────────────
+    "page",             # str: documentation page URL
+    "consentVersion",   # str | None: "v1.0" for contributions; None for feedback
+    # ── Timestamps ───────────────────────────────────────────────────────────
+    "ts",               # int: client-side event time, ms since epoch
+]
+
+#: Required keys for the normalised model sub-object.
+#: Both feedback (3-key shape) and contribution (8-key shape) are expanded to
+#: this full set; keys absent in the source are filled with ``None``.
+MODEL_KEYS: list[str] = [
+    "id",           # canonical model identifier (e.g. "Qwen2.5-Coder-7B-Instruct-hf")
+    "provider",     # inference provider (e.g. "huggingface", "anthropic", "custom")
+    "model",        # HF model path or model string (e.g. "Qwen/Qwen2.5-Coder-7B-Instruct")
+    "label",        # human display name (e.g. "Qwen2.5-Coder-7B-Instruct (Qwen/HuggingFace)")
+    "endpoint",     # inference endpoint URL (None when not configured)
+    "info_url",     # documentation/info link for this model
+    "description",  # short description text
+    "default",      # bool | None: True when this is the default model in the config
+]
+
+# ── Rating vocabulary ─────────────────────────────────────────────────────────
+# The panel feedback 11-point scale.  ``value`` here is the slug stored as
+# ``ratingLabel`` in the JS source (_FEEDBACK_DEFAULTS[idx].value).
+# The numeric rating is carried in ``ratingValue`` (-5 to +5 mapping to index 0..10).
+# fmt: off
+_PANEL_SCALE: list[dict[str, Any]] = [
+    {"slug": "terrible",          "title": "Terrible",   "scale": -5},
+    {"slug": "poor",              "title": "Poor",       "scale": -4},
+    {"slug": "unsatisfied",       "title": "Unsatisfied","scale": -3},
+    {"slug": "negative",          "title": "No",         "scale": -2},
+    {"slug": "slightly_negative", "title": "Not really", "scale": -1},
+    {"slug": "neutral",           "title": "Neutral",    "scale":  0},
+    {"slug": "slightly_positive", "title": "Somewhat",   "scale": +1},
+    {"slug": "mostly_positive",   "title": "Mostly yes", "scale": +2},
+    {"slug": "good",              "title": "Good",       "scale": +3},
+    {"slug": "very_good",         "title": "Very good",  "scale": +4},
+    {"slug": "excellent",         "title": "Excellent!", "scale": +5},
+]
+# fmt: on
+
+# The quick 👍/👎 options.  ``sentiment`` is used as the canonical slug
+# (after the JS-side fix; old records stored ``title`` in ``ratingLabel``).
+_QUICK_OPTS: list[dict[str, Any]] = [
+    {"slug": "not_helpful", "title": "Not helpful", "value": -1, "sentiment": "negative"},
+    {"slug": "helpful",     "title": "Helpful",     "value": +1, "sentiment": "positive"},
+]
+
+# Derived lookup tables.
+_SLUG_TO_TITLE: dict[str, str] = {
+    **{e["slug"]: e["title"] for e in _PANEL_SCALE},
+    **{e["slug"]: e["title"] for e in _QUICK_OPTS},
+    # Sentiment strings also accepted as slugs (old records may use "positive"/"negative").
+    **{e["sentiment"]: e["title"] for e in _QUICK_OPTS},
+}
+_TITLE_TO_SLUG: dict[str, str] = {
+    **{e["title"]: e["slug"] for e in _PANEL_SCALE},
+    **{e["title"]: e["slug"] for e in _QUICK_OPTS},
+}
+_SLUG_TO_SCALE: dict[str, int] = {e["slug"]: e["scale"] for e in _PANEL_SCALE}
+_SCALE_TO_SLUG: dict[int, str] = {e["scale"]: e["slug"] for e in _PANEL_SCALE}
+_VALUE_TO_QUICK: dict[int, dict] = {e["value"]: e for e in _QUICK_OPTS}
+
+#: All known Title Case rating strings (old quick records use these in ratingLabel).
+_KNOWN_TITLES: frozenset[str] = frozenset(_TITLE_TO_SLUG)
+
+#: Regex that matches a valid snake_case slug (all lowercase + underscores).
+_SLUG_RE: re.Pattern[str] = re.compile(r"^[a-z][a-z0-9_]*[a-z0-9]$|^[a-z]$")
+
+#: Regex detecting the quick-feedback sessionId format generated by the JS widget:
+#: ``<conversationUUID>-quick-<answerIndex>-<ms-epoch>``.
+_QUICK_SESSION_RE: re.Pattern[str] = re.compile(r"-quick-\d+-\d+$")
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Model normalization
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+def normalize_model(raw: dict[str, Any] | None) -> dict[str, Any] | None:
+    """Return a normalised model object with all ``MODEL_KEYS`` present.
+
+    Parameters
+    ----------
+    raw : dict or None
+        Raw model dict from either a feedback record (3-key shape:
+        ``{id, provider, model}``) or a contribution record (8-key shape:
+        ``{id, provider, model, label, endpoint, info_url, description, default}``).
+        ``None`` is returned unchanged.
+
+    Returns
+    -------
+    dict or None
+        All eight canonical keys present; absent source keys are ``None``.
+
+    Notes
+    -----
+    Developer note
+        This ensures ``df["model"].apply(lambda m: m["label"])`` works uniformly
+        across rows from both sources without ``KeyError``.
+
+    Examples
+    --------
+    >>> normalize_model({"id": "foo", "provider": "hf", "model": "Org/foo"})
+    {'id': 'foo', 'provider': 'hf', 'model': 'Org/foo', 'label': None,
+     'endpoint': None, 'info_url': None, 'description': None, 'default': None}
+    """
+    if raw is None:
+        return None
+    if not isinstance(raw, dict):
+        return None
+    return {k: raw.get(k) for k in MODEL_KEYS}
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Rating normalization
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+def normalize_rating(
+    rating_value: int | None,
+    rating_label: str | None,
+    *,
+    rating_mode: str | None = None,
+    rating_title: str | None = None,
+    feedback_id: str | None = None,
+) -> dict[str, Any]:
+    """Derive canonical (ratingSlug, ratingTitle, ratingMode) from raw inputs.
+
+    Parameters
+    ----------
+    rating_value : int or None
+        Numeric rating score.  Quick feedback uses -1/+1; panel uses -5..+5.
+    rating_label : str or None
+        Raw ``ratingLabel`` from the client payload.  This may be:
+
+        * A snake_case slug (``"mostly_positive"``): panel feedback and all
+          records written after the JS-side fix.
+        * A Title Case string (``"Not helpful"``): old quick-feedback records
+          written before the JS-side fix.
+        * A sentiment string (``"positive"``/``"negative"``): transitional.
+
+    rating_mode : str or None, optional
+        ``"quick"`` or ``"panel"`` when the JS widget sends the new
+        ``ratingMode`` field.  Autodetected from ``feedback_id`` and
+        ``rating_label`` when absent.
+    rating_title : str or None, optional
+        Human display string when the JS widget sends the new ``ratingTitle``
+        field.  Derived from ``ratingSlug`` when absent.
+    feedback_id : str or None, optional
+        The per-submission ``feedbackId`` / ``sessionId``; used to autodetect
+        quick-feedback records by the ``-quick-`` pattern in older JS versions.
+
+    Returns
+    -------
+    dict
+        Keys: ``ratingSlug``, ``ratingTitle``, ``ratingMode``.
+        All values are ``str`` or ``None``.
+
+    Notes
+    -----
+    Developer note — Detection order:
+
+    1. If ``rating_mode`` is already provided: use it directly.
+    2. If ``feedback_id`` matches ``_QUICK_SESSION_RE``: quick mode.
+    3. If ``rating_label`` is a known Title Case string: quick mode (old record).
+    4. If ``rating_label`` is snake_case slug: panel mode.
+    5. If ``rating_value`` is -1 or +1 and ``rating_label`` is absent: quick mode.
+    6. Otherwise: panel mode (safe default).
+
+    Examples
+    --------
+    >>> normalize_rating(1, "Helpful")                      # old quick record
+    {'ratingSlug': 'helpful', 'ratingTitle': 'Helpful', 'ratingMode': 'quick'}
+    >>> normalize_rating(2, "mostly_positive")              # panel record
+    {'ratingSlug': 'mostly_positive', 'ratingTitle': 'Mostly yes', 'ratingMode': 'panel'}
+    >>> normalize_rating(1, "helpful", rating_mode="quick") # new quick record
+    {'ratingSlug': 'helpful', 'ratingTitle': 'Helpful', 'ratingMode': 'quick'}
+    """
+    label_str: str = (rating_label or "").strip()
+    detected_mode: str | None = rating_mode
+
+    # ── Step 1: Autodetect mode ───────────────────────────────────────────────
+    if not detected_mode:
+        if feedback_id and _QUICK_SESSION_RE.search(feedback_id):
+            detected_mode = "quick"
+        elif label_str in _KNOWN_TITLES:
+            detected_mode = "quick"
+        elif label_str and _SLUG_RE.match(label_str):
+            detected_mode = "panel"
+        elif rating_value in (-1, 1) and not label_str:
+            detected_mode = "quick"
+        else:
+            detected_mode = "panel"
+
+    # ── Step 2: Derive slug ───────────────────────────────────────────────────
+    slug: str | None
+    if detected_mode == "quick":
+        if label_str in _TITLE_TO_SLUG:
+            # Old record: ratingLabel held the Title Case string.
+            slug = _TITLE_TO_SLUG[label_str]
+        elif label_str in _SLUG_TO_TITLE:
+            # New record or sentiment string already slug-like.
+            slug = label_str
+        elif rating_value in _VALUE_TO_QUICK:
+            slug = _VALUE_TO_QUICK[rating_value]["slug"]
+        else:
+            slug = None
+    else:
+        # Panel mode: ratingLabel is already a slug (or empty for retracts).
+        slug = label_str if (label_str and _SLUG_RE.match(label_str)) else None
+        # If slug missing but scale value present, derive from _SCALE_TO_SLUG.
+        if slug is None and rating_value is not None:
+            slug = _SCALE_TO_SLUG.get(rating_value)
+
+    # ── Step 3: Derive title ──────────────────────────────────────────────────
+    title: str | None
+    if rating_title:
+        title = rating_title  # Explicit (new JS sends ratingTitle)
+    elif slug:
+        title = _SLUG_TO_TITLE.get(slug)
+    else:
+        title = None
+
+    return {
+        "ratingSlug":  slug,
+        "ratingTitle": title,
+        "ratingMode":  detected_mode if (slug is not None) else None,
+    }
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Canonical record construction
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+def _ordered(fields: dict[str, Any]) -> dict[str, Any]:
+    """Return ``fields`` re-ordered to match ``CANONICAL_COLUMNS``.
+
+    Parameters
+    ----------
+    fields : dict
+        Record dict with all canonical keys present.
+
+    Returns
+    -------
+    dict
+        Keys in ``CANONICAL_COLUMNS`` order; extra keys appended alphabetically.
+    """
+    ordered: dict[str, Any] = {}
+    for col in CANONICAL_COLUMNS:
+        ordered[col] = fields.get(col)
+    # Preserve any unexpected extra keys after the canonical set (future fields).
+    for k in sorted(fields):
+        if k not in ordered:
+            ordered[k] = fields[k]
+    return ordered
+
+
+def normalize_feedback_record(
+    payload: dict[str, Any],
+    *,
+    server_ts_ms: int,
+) -> dict[str, Any]:
+    """Build a canonical record from a raw ``POST /v1/feedback`` payload.
+
+    Parameters
+    ----------
+    payload : dict
+        Raw JSON body received by the feedback endpoint.  Handles both normal
+        rating records and ``action="retract"`` tombstones.
+    server_ts_ms : int
+        Server receive timestamp in milliseconds since epoch (``int(time.time() * 1000)``).
+        Pass the same value for the entire request to avoid per-call clock drift.
+
+    Returns
+    -------
+    dict
+        Canonical record with all ``CANONICAL_COLUMNS`` keys in order.
+
+    Notes
+    -----
+    Developer note — Security
+        The previous implementation used ``{**payload, ...}`` which forwarded
+        arbitrary client-supplied fields directly into the dataset.  This
+        implementation whitelists only the known fields from the JS schema,
+        discarding unexpected keys.  The legacy ``rating`` alias is dropped
+        (its value was always identical to ``ratingLabel``).
+
+    Developer note — Retract records
+        When ``payload["action"] == "retract"`` the rating / content / model
+        fields are filled with ``None``; ``prevFeedbackId`` is set from
+        ``payload["prevSessionId"]``.
+
+    Developer note — ratingLabel normalization
+        Old quick-feedback records set ``ratingLabel = opt.title`` (Title Case:
+        ``"Not helpful"`` / ``"Helpful"``); the new JS sets
+        ``ratingLabel = opt.slug`` (snake_case: ``"not_helpful"`` / ``"helpful"``).
+        :func:`normalize_rating` handles both transparently.
+
+    Examples
+    --------
+    >>> record = normalize_feedback_record(payload, server_ts_ms=1_700_000_000_000)
+    >>> list(record.keys()) == CANONICAL_COLUMNS
+    True
+    """
+    is_retract: bool = payload.get("action") == "retract"
+
+    # ── Identity ──────────────────────────────────────────────────────────────
+    conversation_id: str = payload.get("conversationId") or ""
+    # Feedback sessionId is the per-submission idempotency key, renamed to
+    # feedbackId to distinguish it from the chat-session conversationId.
+    feedback_id: str | None = payload.get("sessionId") or None
+    answer_index: int | None = payload.get("answerIndex")
+
+    # ── Retract-specific fields ───────────────────────────────────────────────
+    # prevSessionId in the retract payload points to the sessionId (= feedbackId)
+    # of the original record being invalidated.
+    prev_feedback_id: str | None = (
+        payload.get("prevSessionId") or None if is_retract else None
+    )
+
+    # ── Rating (None for retracts) ────────────────────────────────────────────
+    if is_retract:
+        rating_fields: dict[str, Any] = {
+            "ratingSlug": None,
+            "ratingTitle": None,
+            "ratingMode": None,
+        }
+    else:
+        rating_fields = normalize_rating(
+            payload.get("ratingValue"),
+            payload.get("ratingLabel"),
+            rating_mode=payload.get("ratingMode"),      # new JS field (None if old)
+            rating_title=payload.get("ratingTitle"),    # new JS field (None if old)
+            feedback_id=feedback_id,
+        )
+
+    # ── Model (None for quick feedback and retracts) ──────────────────────────
+    raw_model: dict | None = payload.get("model")
+    if isinstance(raw_model, str):
+        # Guard: old or malformed payloads sometimes send model as a bare string.
+        raw_model = {"id": raw_model, "provider": None, "model": raw_model}
+
+    # ── Assemble canonical record ─────────────────────────────────────────────
+    return _ordered({
+        "schemaVersion":  int(payload.get("schemaVersion") or SCHEMA_VERSION),
+        "_source":        "feedback",
+        "_ts":            server_ts_ms,
+        "_dedup_key":     f"{conversation_id}:{answer_index}",
+        "conversationId": conversation_id or None,
+        "feedbackId":     feedback_id,
+        "answerIndex":    int(answer_index) if answer_index is not None else None,
+        "action":         "retract" if is_retract else "rate",
+        "prevFeedbackId": prev_feedback_id,
+        "status":         "active",
+        "ratingValue":    None if is_retract else payload.get("ratingValue"),
+        "ratingSlug":     rating_fields["ratingSlug"],
+        "ratingTitle":    rating_fields["ratingTitle"],
+        "ratingMode":     rating_fields["ratingMode"],
+        "message":        "" if is_retract else (payload.get("message") or ""),
+        "query":          "" if is_retract else (payload.get("query") or ""),
+        "answer":         "" if is_retract else (payload.get("answer") or ""),
+        "model":          None if is_retract else normalize_model(raw_model),
+        "page":           "" if is_retract else (payload.get("page") or ""),
+        "consentVersion": None,   # feedback is implicit consent; no version stored
+        "ts":             payload.get("ts"),
+    })
+
+
+def normalize_contribution_record(
+    rec: dict[str, Any],
+    *,
+    envelope: dict[str, Any],
+    server_ts_ms: int,
+) -> dict[str, Any]:
+    """Build a canonical record from one turn in a ``POST /v1/contribute`` batch.
+
+    Parameters
+    ----------
+    rec : dict
+        A single item from ``payload["records"]`` (one per-turn ``tRecord``).
+    envelope : dict
+        The outer contribution POST body (contains ``sessionId``, ``page``,
+        ``model``, ``consentVersion``, etc.).
+    server_ts_ms : int
+        Server receive timestamp in milliseconds since epoch.  Compute once
+        per request and pass to all calls so every row in the batch has the
+        same ``_ts``.
+
+    Returns
+    -------
+    dict
+        Canonical record with all ``CANONICAL_COLUMNS`` keys in order.
+
+    Notes
+    -----
+    Developer note — Field renaming
+        The previous implementation stored ``_sessionId``, ``_page``,
+        ``_model``, ``_consentVersion`` (underscore-prefixed server-side
+        names).  These are now stored without the prefix (``conversationId``,
+        ``page``, ``model``, ``consentVersion``) matching the feedback schema.
+        ``_source``, ``_ts``, and ``_dedup_key`` keep their underscore prefix
+        because they are universal provenance fields managed exclusively by
+        the server.
+
+    Developer note — ``_ts`` consistency
+        All rows in a single contribute batch share the same ``server_ts_ms``
+        value.  The previous inline ``int(_time.time() * 1000)`` inside a
+        list comprehension produced slightly different ``_ts`` values per row.
+        Callers must compute ``server_ts_ms`` once before iterating.
+
+    Examples
+    --------
+    >>> ts = int(time.time() * 1000)
+    >>> rows = [normalize_contribution_record(r, envelope=payload, server_ts_ms=ts)
+    ...         for r in payload["records"] if isinstance(r, dict)]
+    """
+    # conversation_id is the JS _sessionId (stable per-page-load chat session UUID).
+    # The envelope calls it "sessionId" (without underscore); we rename to conversationId.
+    conversation_id: str = envelope.get("sessionId") or ""
+    answer_index: int | None = rec.get("answerIndex")
+
+    rating_fields = normalize_rating(
+        rec.get("ratingValue"),
+        rec.get("ratingLabel"),
+        rating_mode=rec.get("ratingMode"),     # new JS field (None if old)
+        rating_title=rec.get("ratingTitle"),   # new JS field (None if old)
+        feedback_id=None,                      # no per-event session ID for contributions
+    )
+
+    return _ordered({
+        "schemaVersion":  int(envelope.get("schemaVersion") or SCHEMA_VERSION),
+        "_source":        "contribution",
+        "_ts":            server_ts_ms,
+        "_dedup_key":     f"{conversation_id}:{answer_index}",
+        "conversationId": conversation_id or None,
+        "feedbackId":     None,   # contributions are batch-level; no per-event ID
+        "answerIndex":    int(answer_index) if answer_index is not None else None,
+        "action":         "rate",
+        "prevFeedbackId": None,
+        "status":         "active",
+        "ratingValue":    rec.get("ratingValue"),
+        "ratingSlug":     rating_fields["ratingSlug"],
+        "ratingTitle":    rating_fields["ratingTitle"],
+        "ratingMode":     rating_fields["ratingMode"],
+        "message":        rec.get("message") or "",
+        "query":          rec.get("query") or "",
+        "answer":         rec.get("answer") or "",
+        "model":          normalize_model(envelope.get("model")),
+        "page":           envelope.get("page") or "",
+        "consentVersion": envelope.get("consentVersion"),
+        "ts":             rec.get("ts"),
+    })
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Back-compat normalisation for old records
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+def normalize_record(raw: dict[str, Any]) -> dict[str, Any]:
+    """Normalise any stored JSONL record (old or new) to the canonical schema.
+
+    Handles records written before the schema fix by detecting and mapping
+    legacy field names (``_sessionId``, ``_page``, ``_model``, ``_consentVersion``,
+    ``rating``) to their canonical equivalents.
+
+    Parameters
+    ----------
+    raw : dict
+        A single record dict as loaded from a JSONL file.
+
+    Returns
+    -------
+    dict
+        Canonical record.  Idempotent: already-canonical records pass through
+        unchanged.
+
+    Notes
+    -----
+    Developer note — Priority
+        For any field that has both an old and a new name present in the same
+        raw record, the new canonical name takes precedence.
+
+    Examples
+    --------
+    >>> old_contribution = {"_sessionId": "abc", "_page": "http://...", ...}
+    >>> new_contribution = normalize_record(old_contribution)
+    >>> "conversationId" in new_contribution
+    True
+    >>> "_sessionId" not in new_contribution
+    True
+    """
+    source: str = raw.get("_source", "")
+    out: dict[str, Any] = dict(raw)
+
+    # ── Map legacy contribution field names → canonical ────────────────────���──
+    if "_sessionId" in out and "conversationId" not in out:
+        out["conversationId"] = out.pop("_sessionId")
+    elif "_sessionId" in out:
+        out.pop("_sessionId")  # canonical name already present; drop alias
+
+    if "_page" in out and "page" not in out:
+        out["page"] = out.pop("_page")
+    elif "_page" in out:
+        out.pop("_page")
+
+    if "_model" in out and "model" not in out:
+        out["model"] = out.pop("_model")
+    elif "_model" in out:
+        out.pop("_model")
+
+    if "_consentVersion" in out and "consentVersion" not in out:
+        out["consentVersion"] = out.pop("_consentVersion")
+    elif "_consentVersion" in out:
+        out.pop("_consentVersion")
+
+    # ── Map legacy feedback field names → canonical ───────────────────────────
+    # sessionId in feedback was the per-submission idempotency key (now feedbackId).
+    # Do NOT rename for contribution records (contributions have no sessionId field).
+    if source == "feedback":
+        if "sessionId" in out and "feedbackId" not in out:
+            out["feedbackId"] = out.pop("sessionId")
+        elif "sessionId" in out:
+            out.pop("sessionId")
+
+        # prevSessionId in retract records → prevFeedbackId.
+        if "prevSessionId" in out and "prevFeedbackId" not in out:
+            out["prevFeedbackId"] = out.pop("prevSessionId")
+        elif "prevSessionId" in out:
+            out.pop("prevSessionId")
+
+    # ── Drop legacy aliases ───────────────────────────────────────────────────
+    # ``rating`` was always == ``ratingLabel``; it provides no additional info.
+    out.pop("rating", None)
+
+    # ── Back-fill missing canonical fields ────────────────────────────────────
+    out.setdefault("schemaVersion",  SCHEMA_VERSION)
+    out.setdefault("feedbackId",     None)
+    out.setdefault("action",         "rate")
+    out.setdefault("prevFeedbackId", None)
+    out.setdefault("status",         "active")
+    out.setdefault("consentVersion", None)
+    out.setdefault("message",        "")
+    out.setdefault("query",          "")
+    out.setdefault("answer",         "")
+    out.setdefault("page",           "")
+
+    # ── Normalise model shape ─────────────────────────────────────────────────
+    raw_model = out.get("model")
+    if isinstance(raw_model, dict):
+        out["model"] = normalize_model(raw_model)
+
+    # ── Normalise rating fields ───────────────────────────────────────────────
+    # For old records that don't yet have ratingSlug/ratingTitle/ratingMode.
+    if "ratingSlug" not in out:
+        rf = normalize_rating(
+            out.get("ratingValue"),
+            out.get("ratingLabel"),
+            rating_mode=out.get("ratingMode"),
+            rating_title=out.get("ratingTitle"),
+            feedback_id=out.get("feedbackId"),
+        )
+        out["ratingSlug"]  = rf["ratingSlug"]
+        out["ratingTitle"] = rf["ratingTitle"]
+        out["ratingMode"]  = rf["ratingMode"]
+
+    # Keep ratingLabel in sync with ratingSlug for backward compat readers.
+    if out.get("ratingSlug") and not out.get("ratingLabel"):
+        out["ratingLabel"] = out["ratingSlug"]
+
+    return _ordered(out)
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# I/O helpers
+# ─────────────────────────────────────────────────────────────────────────────
+
+
+def load_jsonl_file(path: str | Path) -> list[dict[str, Any]]:
+    """Load and normalise all records from a single JSONL file.
+
+    Parameters
+    ----------
+    path : str or Path
+        Path to a ``.jsonl`` file (one JSON object per line; blank lines and
+        comment lines starting with ``#`` are skipped).
+
+    Returns
+    -------
+    list of dict
+        Normalised records.  Malformed lines are skipped with a warning printed
+        to stderr.
+
+    Notes
+    -----
+    User note
+        Both ``feedback/TIMESTAMP.jsonl`` and ``contributions/TIMESTAMP.jsonl``
+        files are valid inputs; the normalisation step handles the field-name
+        differences transparently.
+    """
+    import sys
+
+    records: list[dict[str, Any]] = []
+    path = Path(path)
+    with path.open(encoding="utf-8") as fh:
+        for line_no, line in enumerate(fh, 1):
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError as exc:
+                print(
+                    f"WARNING: {path}:{line_no}: JSON decode error — {exc}",
+                    file=sys.stderr,
+                )
+                continue
+            if not isinstance(obj, dict):
+                print(
+                    f"WARNING: {path}:{line_no}: expected JSON object, got "
+                    f"{type(obj).__name__} — skipped",
+                    file=sys.stderr,
+                )
+                continue
+            records.append(normalize_record(obj))
+    return records
+
+
+def load_dataset(
+    feedback_dir: str | Path | None = None,
+    contributions_dir: str | Path | None = None,
+    *,
+    sort_by: str = "_ts",
+    ascending: bool = True,
+) -> "Any":  # -> pd.DataFrame
+    """Load and combine feedback and contribution records into one pandas DataFrame.
+
+    Parameters
+    ----------
+    feedback_dir : str, Path, or None
+        Directory containing ``feedback/*.jsonl`` files, or a single
+        ``feedback.jsonl`` file.  Skipped when ``None``.
+    contributions_dir : str, Path, or None
+        Directory containing ``contributions/*.jsonl`` files, or a single
+        ``contributions.jsonl`` file.  Skipped when ``None``.
+    sort_by : str, optional
+        Column to sort the combined DataFrame by.  Default ``"_ts"`` (server
+        receive time, ascending).
+    ascending : bool, optional
+        Sort direction.  Default ``True``.
+
+    Returns
+    -------
+    pandas.DataFrame
+        Combined, normalised DataFrame with columns in ``CANONICAL_COLUMNS``
+        order.  ``model`` column contains dict values (or ``NaN`` for rows with
+        no model info).  Flat helper columns ``model_id``, ``model_provider``,
+        and ``model_name`` are appended for easy querying.
+
+    Raises
+    ------
+    ImportError
+        When ``pandas`` is not installed.
+
+    Notes
+    -----
+    User note — one-liner::
+
+        df = load_dataset("feedback/", "contributions/")
+        df.groupby("_source")["ratingValue"].mean()
+
+    User note — filtering retractions::
+
+        active = df[df["action"] != "retract"].copy()
+
+    User note — dedup (prefer contribution over feedback)::
+
+        df_deduped = (
+            df.sort_values(["_dedup_key", "_source"], ascending=[True, True])
+              .drop_duplicates(subset=["_dedup_key"], keep="last")
+        )
+
+    Developer note — model column
+        The ``model`` column holds Python dicts (or ``None`` → pandas ``NaN``).
+        For JSON-serialisable storage use
+        ``df["model"] = df["model"].apply(json.dumps)``.
+
+    Examples
+    --------
+    >>> df = load_dataset("feedback/", "contributions/")
+    >>> df.dtypes["ratingValue"]
+    dtype('object')
+    >>> df.dtypes["_ts"]
+    dtype('int64')
+    """
+    try:
+        import pandas as pd
+    except ImportError as exc:
+        raise ImportError(
+            "pandas is required for load_dataset().  "
+            "Install it with: pip install pandas"
+        ) from exc
+
+    all_records: list[dict[str, Any]] = []
+
+    def _collect(directory: str | Path) -> None:
+        p = Path(directory)
+        if p.is_file():
+            all_records.extend(load_jsonl_file(p))
+        elif p.is_dir():
+            for jsonl_file in sorted(p.glob("*.jsonl")):
+                all_records.extend(load_jsonl_file(jsonl_file))
+
+    if feedback_dir is not None:
+        _collect(feedback_dir)
+    if contributions_dir is not None:
+        _collect(contributions_dir)
+
+    if not all_records:
+        # Return empty DataFrame with correct columns and dtypes.
+        return pd.DataFrame(columns=CANONICAL_COLUMNS)
+
+    df = pd.DataFrame(all_records)
+
+    # ── Ensure all canonical columns are present (back-compat) ────────────────
+    for col in CANONICAL_COLUMNS:
+        if col not in df.columns:
+            df[col] = None
+
+    # ── Reorder columns to canonical order ────────────────────────────────────
+    extra_cols = [c for c in df.columns if c not in CANONICAL_COLUMNS]
+    df = df[CANONICAL_COLUMNS + extra_cols]
+
+    # ── Flat model helper columns for easy querying ───────────────────────────
+    def _model_field(m: Any, key: str) -> Any:
+        if isinstance(m, dict):
+            return m.get(key)
+        return None
+
+    df["model_id"]       = df["model"].apply(_model_field, key="id")
+    df["model_provider"] = df["model"].apply(_model_field, key="provider")
+    df["model_name"]     = df["model"].apply(_model_field, key="model")
+
+    # ── Sort ──────────────────────────────────────────────────────────────────
+    if sort_by in df.columns:
+        df = df.sort_values(sort_by, ascending=ascending, ignore_index=True)
+
+    return df

app.py

@@ -135,14 +135,14 @@ from fastapi import Depends, FastAPI, HTTPException, Request
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse, Response, StreamingResponse
 
-# _shared_logic.py and dataset_schema.py must live alongside this file.
+# _shared_logic.py and _dataset_schema.py must live alongside this file.
 try:
-    from .dataset_schema import (  # type: ignore[import]
+    from ._dataset_schema import (  # type: ignore[import]
         normalize_contribution_record,
         normalize_feedback_record,
     )
 except Exception:
-    from dataset_schema import (  # type: ignore[import]
+    from _dataset_schema import (  # type: ignore[import]
         normalize_contribution_record,
         normalize_feedback_record,
     )
@@ -1253,7 +1253,7 @@ async def contribute(request: Request) -> JSONResponse:  # noqa: PLR0912
     # _consentVersion → consentVersion) to the canonical schema shared with the
     # feedback endpoint.  It also normalises ratingLabel (slug vs Title Case),
     # expands the model object to the full 8-key shape, and enforces key order.
-    # See dataset_schema.py for the full canonical column list.
+    # See _dataset_schema.py for the full canonical column list.
     rows_jsonl = "\n".join(
         json.dumps(
             normalize_contribution_record(
@@ -1828,7 +1828,7 @@ async def feedback(request: Request) -> JSONResponse:
             # prevSessionId → prevFeedbackId, and enforces canonical key order.
             # The resulting record is structurally identical to a contribution row
             # so both sources concatenate directly into one pandas DataFrame.
-            # See dataset_schema.py for the full canonical column list.
+            # See _dataset_schema.py for the full canonical column list.
             record = json.dumps(
                 normalize_feedback_record(
                     payload,