# Dataset Collection Guidance **Applies to:** `scikit-plots/ai-assistant-contributions` (HuggingFace Dataset) **Version:** 2.0 **Maintained by:** scikit-plots maintainers --- ## 1. Overview The AI-assistant widget collects fine-tuning data through **two independent paths** that can produce records for the same (conversation, answer) pair: | Path | Endpoint | Folder | Trigger | |------|----------|--------|---------| | **Individual feedback** | `POST /v1/feedback` | `feedback/` | User clicks a rating button immediately after each answer | | **Whole-conversation contribution** | `POST /v1/contribute` | `contributions/` | User clicks the **Contribute** button in the share sheet (GDPR-gated) | When both paths are active (`FEEDBACK_PERSIST_ENABLED=true` **and** the user later clicks Contribute), the **same Q&A pair is stored twice**. Training on duplicated examples artificially amplifies those examples — the dominant contributor problem — so duplicates **must be resolved before any training run**. --- ## 2. Deduplication Key Contract Every JSONL record stored in the dataset carries two mandatory server-assigned provenance fields: | Field | Type | Value | Set by | |-------|------|-------|--------| | `_source` | `"feedback"` \| `"contribution"` | Provenance tag | Server | | `_dedup_key` | `string` | `"{conversationId}:{answerIndex}"` | Server | ### Key format ``` _dedup_key = f"{conversationId}:{answerIndex}" ``` - **`conversationId`** — stable per-page-load UUID (`_sessionId` in the JS widget, set once at module load and never re-generated). Sent as `detail.conversationId` in the feedback POST and as `payload.sessionId` in the contribution POST. Both endpoints use the *same* UUID so equality across folders is sufficient to detect cross-source duplicates. - **`answerIndex`** — zero-based integer position of the answer within the conversation transcript. > **Invariant:** Within a single conversation, `(conversationId, answerIndex)` > is unique. Across conversations, `conversationId` alone is unique (UUID). > Therefore `_dedup_key` is globally unique for a given answer position within > a given conversation session. ### v2 direct foreign key (preferred join key) Since schema version 2, contribution records carry an additional **direct foreign key** alongside `_dedup_key`: | Field | Type | Description | |-------|------|-------------| | `feedbackId` | `string \| null` | `feedbackId` (= `sessionId`) of the matching `feedback/` record, when the user individually rated this answer before clicking Contribute. `null` when they contributed without ever rating the specific answer. | This is a **1-to-1 join key** between `contributions/` rows and `feedback/` rows — use it for precise cross-source joins instead of the coarser `_dedup_key`. --- ## 3. Priority Rule When two records share the same `_dedup_key`, **always keep the `contribution` record and discard the `feedback` record**. Rationale: - The contribution path is gated behind explicit GDPR consent. It represents a deliberate, end-of-session review of the user's ratings and is the higher-quality signal. - The feedback path fires immediately on button click (keepalive, fire-and- forget) — it may capture a transient or misclicked rating that the user later reconsiders before clicking Contribute. ``` Priority: contribution > feedback ``` ### Retraction tombstones When a user edits a previously submitted rating, the browser sends a **retraction tombstone** record before the new rating. Tombstones are stored in the `feedback/` folder alongside normal rating records. Since schema version 2 the canonical field name for the superseded record's identifier is `prevFeedbackId` (previously `prevSessionId`). Both field names are handled by `normalize_record` in `_dataset_schema.py` for backward compatibility. | Field | Value | |-------|-------| | `action` | `"retract"` | | `prevFeedbackId` | `feedbackId` of the original rating record (v2) | | `_dedup_key` | identical to the original record's `_dedup_key` | | `_ts` | server-write timestamp — always later than the original | | `ratingValue` | `null` (tombstones carry no rating) | | `editCount` | `null` (not applicable to a tombstone) | Tombstones participate in the LWW loop inside `deduplicate()` because their `_ts` is later than the original record's, ensuring the original rating is not emitted. They are then **unconditionally removed from the clean output**: a tombstone is never a valid training example. **Normal terminal state** (edit completed, all three records share the same `_dedup_key`): ``` _ts 100 action="rate" ratingValue=+1 prevFeedbackId=null editCount=0 _ts 200 action="retract" ratingValue=null prevFeedbackId= editCount=null _ts 201 action="rate" ratingValue=-1 prevFeedbackId= editCount=1 ``` LWW selects `_ts 201` (most recent `rate` action). No tombstone in output. ✓ **Degenerate case** (tombstone wins — follow-up rating never reached the server, e.g. network failure after the retraction was sent): ``` _ts 100 action="rate" ratingValue=+1 _ts 200 action="retract" ratingValue=null <- LWW winner ``` Net result: the original +1 was explicitly retracted, so **no record is emitted** for this key. Training data is never contaminated. ### Supersession chain resolution (v2) Beyond LWW, `feedbackId` + `prevFeedbackId` form a walkable edit-chain per `(conversationId, answerIndex)`: ``` rating A: feedbackId="id-A" prevFeedbackId=null editCount=0 <- first rating rating B: feedbackId="id-B" prevFeedbackId="id-A" editCount=1 <- edited rating C: feedbackId="id-C" prevFeedbackId="id-B" editCount=2 <- edited again ``` `editCount` gives the chain length without walking it. The record with the highest `editCount` and `action="rate"` is the current active rating for that `(conversationId, answerIndex)`. `deduplicate_dataset.py` resolves this automatically via LWW on `_ts`. --- ## 4. Dataset Folder Structure ``` scikit-plots/ai-assistant-contributions/ |-- contributions/ | `-- {unix_ms}.jsonl # 1 file per /v1/contribute POST | # Each line = 1 Q&A record (canonical schema v2) | # Key fields: conversationId, feedbackId (FK->feedback/), | # answerIndex, query, answer, | # ratingValue, ratingSlug, ratingTitle, ratingMode, | # prevFeedbackId, editCount, message, ts, | # model (8-key), page, consentVersion (null), | # _source="contribution", _dedup_key, _ts `-- feedback/ `-- {unix_ms}.jsonl # 1 file per /v1/feedback POST (when # FEEDBACK_PERSIST_ENABLED=true) # Each file = 1 record (1 rating or 1 retract tombstone) # Key fields: conversationId, feedbackId, # action, answerIndex, # ratingValue, ratingSlug, ratingTitle, ratingMode, # prevFeedbackId, editCount, message, query, answer, # model (8-key), page, consentVersion (null), # _source="feedback", _dedup_key, _ts ``` --- ## 5. Canonical Deduplication Script Run this script **before every training job** to produce a clean, deduplicated NDJSON file. It reads all JSONL files from both folders, deduplicates by `_dedup_key` applying the priority rule, and writes one record per unique key. ```python r""" deduplicate_dataset.py ====================== Canonical deduplication script for scikit-plots/ai-assistant-contributions. Supports schema versions 1 (legacy) and 2 (current). Records are normalised to the canonical v2 schema by ``_dataset_schema.normalize_record`` before deduplication so callers can always expect the full field set. Usage ----- python deduplicate_dataset.py \ --repo-id scikit-plots/ai-assistant-contributions \ --output clean_dataset.jsonl # Use a local pre-downloaded snapshot (faster on re-runs): python deduplicate_dataset.py \ --repo-id scikit-plots/ai-assistant-contributions \ --local-dir /tmp/ai-contributions-snapshot \ --output clean_dataset.jsonl Requirements ------------ huggingface_hub>=0.23,<2 (optional) hf_transfer for faster downloads (optional) _dataset_schema.py (from _hf_spaces_proxy/) for normalization Notes ----- * Priority rule: "contribution" beats "feedback" for the same _dedup_key. * Retraction tombstones (action="retract") are always excluded from the clean output even if they win the LWW race. * Script is idempotent: re-running produces the same output for the same dataset state. * Output records are written with ``sort_keys=True``, so every record's keys (including nested objects) appear in a fixed alphabetical order in clean_dataset.jsonl. * Progress and statistics are emitted via the module ``logging`` logger. INFO-level records route to stdout; WARNING and ERROR records route to stderr — preserving the previous ``print`` / ``print(..., file=sys.stderr)`` split so that callers capturing stdout see only the NDJSON data. * When _dataset_schema is importable, records are normalised from v1 to v2 schema automatically (legacy _sessionId/_page/_model fields mapped to conversationId/page/model; editCount/feedbackId/prevFeedbackId back-filled). When _dataset_schema is not importable (standalone usage), records are used as-is with a warning. """ # noqa: D205, D400 from __future__ import annotations import argparse import json import logging import sys from pathlib import Path from typing import Any logger = logging.getLogger(__name__) # Optional: normalize records from v1 to v2 schema when _dataset_schema is # available alongside this script (standard _hf_spaces_proxy/ deployment). # Falls back to identity function with a warning for standalone usage. try: from _dataset_schema import normalize_record as _normalize_record _SCHEMA_AVAILABLE = True except ImportError: def _normalize_record(raw: dict) -> dict: return raw _SCHEMA_AVAILABLE = False # Priority order: lower index = higher priority. _SOURCE_PRIORITY: dict[str, int] = { "contribution": 0, "feedback": 1, } _DEFAULT_PRIORITY = 99 def _priority(record: dict) -> int: return _SOURCE_PRIORITY.get(record.get("_source", ""), _DEFAULT_PRIORITY) def load_all_records(local_dir: Path) -> list[dict]: """Read every *.jsonl file under local_dir into a flat list. Parameters ---------- local_dir : pathlib.Path Root of the locally downloaded dataset snapshot. Returns ------- list[dict] All JSON-decoded records, normalised to canonical v2 schema when ``_dataset_schema`` is importable. Malformed lines are skipped with a WARNING-level log record. """ records: list[dict] = [] for jsonl_path in sorted(local_dir.rglob("*.jsonl")): with jsonl_path.open(encoding="utf-8") as fh: for lineno, line in enumerate(fh, 1): line = line.strip() # noqa: PLW2901 if not line: continue try: raw = json.loads(line) except json.JSONDecodeError as exc: logger.warning( "Skipping malformed JSON in %s:%d: %s", jsonl_path, lineno, exc, ) continue if not isinstance(raw, dict): logger.warning( "%s:%d: expected JSON object, got %s -- skipped", jsonl_path, lineno, type(raw).__name__, ) continue records.append(_normalize_record(raw)) return records def deduplicate(records: list[dict]) -> list[dict]: """Deduplicate records by _dedup_key applying the priority rule. Parameters ---------- records : list[dict] All raw records from both contributions/ and feedback/ folders, already normalised to v2 schema by load_all_records. Returns ------- list[dict] One record per unique _dedup_key. Records that have no _dedup_key (legacy, pre-v1.0 records) are retained unchanged. Retraction tombstones are excluded from the output. Notes ----- Priority rule For the same _dedup_key, the record with the lowest _SOURCE_PRIORITY value is kept. Ties (same source) are broken by server-write timestamp (_ts), keeping the most recent. Deterministic: given the same input, the output is always the same. Retraction tombstones action="retract" records are still used during the LWW loop because their later _ts must suppress an earlier rate record (correct behaviour). They are removed in the post-loop filter so they cannot leak into clean_dataset.jsonl. Degenerate case -- orphaned tombstone wins: silently discarded. Net effect: the original rating was explicitly retracted, so no record is emitted for that key -- correct for training data quality. feedbackId cross-source linkage (v2) When both a feedback/ record and a contributions/ record exist for the same _dedup_key, the contribution record's feedbackId field points directly to the feedback record's feedbackId (1-to-1 FK). The winning contribution record therefore carries the complete provenance chain without any additional join. """ keyed: dict[str, dict] = {} # _dedup_key -> winning record no_key: list[dict] = [] # legacy records without _dedup_key for rec in records: dk = rec.get("_dedup_key") if dk is None: no_key.append(rec) continue existing = keyed.get(dk) if existing is None: keyed[dk] = rec continue # Compare source priorities; lower = better (contribution > feedback). new_pri = _priority(rec) old_pri = _priority(existing) if new_pri < old_pri: keyed[dk] = rec elif new_pri == old_pri: # noqa: SIM102 # Same source: keep the most recently written record (_ts). if rec.get("_ts", 0) > existing.get("_ts", 0): keyed[dk] = rec # Post-loop: discard retraction tombstones from the winning set. # # Scenario A (normal edit): user rates +1 (_ts=100), edits (tombstone at # _ts=200), then rates -1 (_ts=201). LWW selects -1. No tombstone. OK # # Scenario B (orphaned tombstone): +1 at _ts=100, tombstone at _ts=200, # but follow-up -1 never reached the server. LWW selects the tombstone. # Without this filter, action="retract" with ratingValue=null would corrupt # training. Filter silently drops it. OK clean_keyed = [r for r in keyed.values() if r.get("action") != "retract"] return clean_keyed + no_key def write_output(records: list[dict], output_path: Path) -> None: """Write records to output_path as newline-delimited JSON. Parameters ---------- records : list[dict] Deduplicated records in canonical v2 schema. output_path : pathlib.Path Destination file. Parent directories are created if absent. Notes ----- Each record is serialised with ``sort_keys=True``, so object keys (at every nesting level) are written in a fixed alphabetical order. This keeps the output byte-for-byte reproducible across runs and makes line-level diffs between dataset snapshots meaningful. """ output_path.parent.mkdir(parents=True, exist_ok=True) with output_path.open("w", encoding="utf-8") as fh: for rec in records: fh.write(json.dumps(rec, ensure_ascii=False, sort_keys=True) + "\n") def _report_stats(records: list[dict]) -> dict[str, Any]: """Return summary statistics for a list of records. Parameters ---------- records : list[dict] Records to summarise (raw or deduplicated). Returns ------- dict Counters by source, action, schema version, and FK population. """ by_source: dict[str, int] = {} by_action: dict[str, int] = {} by_schema: dict[Any, int] = {} with_feedback_id = 0 with_prev_feedback = 0 tombstones = 0 for r in records: src = r.get("_source", "unknown") by_source[src] = by_source.get(src, 0) + 1 act = r.get("action", "rate") by_action[act] = by_action.get(act, 0) + 1 sv = r.get("schemaVersion", "?") by_schema[sv] = by_schema.get(sv, 0) + 1 if r.get("feedbackId"): with_feedback_id += 1 if r.get("prevFeedbackId"): with_prev_feedback += 1 if act == "retract": tombstones += 1 return { "total": len(records), "by_source": by_source, "by_action": by_action, "by_schema": by_schema, "with_feedback_id": with_feedback_id, "with_prev_feedback_id": with_prev_feedback, "tombstones": tombstones, } class _MaxLevelFilter(logging.Filter): """Admit only log records whose level is at or below *max_level*. Parameters ---------- max_level : int Maximum ``logging`` level number (inclusive) to pass through. Records with a higher level number are suppressed. Pass ``logging.INFO`` to block WARNING and above. Notes ----- Attached to the stdout handler inside ``_configure_logging`` so that WARNING / ERROR records are handled exclusively by the stderr handler and are not duplicated on stdout. """ def __init__(self, max_level: int) -> None: super().__init__() self.max_level = max_level def filter(self, record: logging.LogRecord) -> bool: # noqa: A003 """Return ``True`` if *record.levelno* is at or below *max_level*. Parameters ---------- record : logging.LogRecord Log record to evaluate. Returns ------- bool ``True`` to emit the record; ``False`` to suppress it. """ return record.levelno <= self.max_level def _configure_logging() -> None: """Attach stdout and stderr handlers to the root logger for CLI use. Routes INFO-level records to stdout with a plain ``%(message)s`` format, and WARNING / ERROR / CRITICAL records to stderr with a ``[%(levelname)s] %(message)s`` format. This preserves the stdout / stderr split that the original ``print`` / ``print(..., file=sys.stderr)`` calls provided: * Callers that capture stdout (e.g. downstream JSONL pipelines) see only the NDJSON data, never progress lines. * Diagnostic warnings and errors still appear on stderr. The function overwrites ``logging.root.handlers`` directly, so it is idempotent: repeated calls replace handlers rather than stacking duplicates. Notes ----- This is a CLI-only helper. Library callers that import the domain functions (``load_all_records``, ``deduplicate``, …) should configure their own logging handlers; this function is only invoked from ``main()``. """ plain_fmt = logging.Formatter("%(message)s") level_fmt = logging.Formatter("[%(levelname)s] %(message)s") out_handler = logging.StreamHandler(sys.stdout) out_handler.setFormatter(plain_fmt) out_handler.setLevel(logging.DEBUG) out_handler.addFilter(_MaxLevelFilter(logging.INFO)) err_handler = logging.StreamHandler(sys.stderr) err_handler.setFormatter(level_fmt) err_handler.setLevel(logging.WARNING) root = logging.getLogger() root.handlers = [out_handler, err_handler] root.setLevel(logging.DEBUG) def main(argv: list[str] | None = None) -> int: """Run Main.""" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--repo-id", required=True, help="HuggingFace dataset repo ID, e.g. scikit-plots/ai-assistant-contributions", ) parser.add_argument( "--output", default="clean_dataset.jsonl", help="Output path for the deduplicated NDJSON file (default: clean_dataset.jsonl)", ) parser.add_argument( "--local-dir", default=None, help="Use a pre-downloaded local snapshot instead of downloading.", ) parser.add_argument( "--token", default=None, help="HuggingFace read token (optional; uses cached token if absent).", ) parser.add_argument( "--stats-only", action="store_true", help="Print dataset statistics without writing an output file.", ) args = parser.parse_args(argv) _configure_logging() if not _SCHEMA_AVAILABLE: logger.warning( "_dataset_schema.py not found on sys.path. Records will not be " "normalised from v1 to v2 schema. Copy _dataset_schema.py from " "_hf_spaces_proxy/ to the same directory as this script for full " "schema normalisation.", ) local_dir: Path if args.local_dir: local_dir = Path(args.local_dir) else: try: from huggingface_hub import snapshot_download # noqa: PLC0415 except ImportError: logger.error( "huggingface_hub is not installed. " "Run: pip install 'huggingface_hub>=0.23,<2'", ) return 1 logger.info("Downloading %s ...", args.repo_id) local_dir = Path( snapshot_download( repo_id=args.repo_id, repo_type="dataset", token=args.token, ) ) logger.info("Reading records from %s ...", local_dir) all_records = load_all_records(local_dir) raw_stats = _report_stats(all_records) logger.info(" %d total records read", raw_stats["total"]) for src, cnt in sorted(raw_stats["by_source"].items()): logger.info(" %s: %d", src, cnt) for act, cnt in sorted(raw_stats["by_action"].items()): logger.info(" action=%r: %d", act, cnt) for sv, cnt in raw_stats["by_schema"].items(): logger.info(" schemaVersion=%s: %d", sv, cnt) logger.info(" feedbackId populated: %d", raw_stats["with_feedback_id"]) logger.info(" prevFeedbackId populated: %d", raw_stats["with_prev_feedback_id"]) if raw_stats["tombstones"]: logger.info( " %d retraction tombstone(s) in raw data " "(always excluded from clean output)", raw_stats["tombstones"], ) if args.stats_only: return 0 clean = deduplicate(all_records) duplicates_removed = raw_stats["total"] - raw_stats["tombstones"] - len(clean) logger.info(" %d duplicate(s) removed (priority rule applied)", duplicates_removed) logger.info(" %d unique records retained", len(clean)) output_path = Path(args.output) write_output(clean, output_path) logger.info("Clean dataset written to %s", output_path) return 0 if __name__ == "__main__": sys.exit(main()) ``` ### Example run ```bash pip install "huggingface_hub>=0.23,<2" # Full pipeline (download + normalise + dedup): python deduplicate_dataset.py \ --repo-id scikit-plots/ai-assistant-contributions \ --output clean_dataset.jsonl \ --token hf_xxxxxxxxxxxxxxxx # Stats-only (no output file written): python deduplicate_dataset.py \ --repo-id scikit-plots/ai-assistant-contributions \ --stats-only \ --token hf_xxxxxxxxxxxxxxxx # Faster re-runs using a pre-downloaded snapshot: python deduplicate_dataset.py \ --repo-id scikit-plots/ai-assistant-contributions \ --local-dir /tmp/ai-contributions-snapshot \ --output clean_dataset.jsonl ``` --- ## 6. Field Reference (Post-Dedup Record, Canonical v2 Schema) Every record in `clean_dataset.jsonl` uses the canonical key order from `_dataset_schema.CANONICAL_COLUMNS`. Old v1 records are normalised to this schema by `normalize_record` in `_dataset_schema.py`. ### Schema metadata | Field | Type | Description | |-------|------|-------------| | `schemaVersion` | `int` | `2` for all records normalised by this pipeline | ### Provenance (server-assigned) | Field | Type | Description | |-------|------|-------------| | `_source` | `string` | `"contribution"` or `"feedback"` | | `_ts` | `int` | Server receive timestamp (ms since epoch) | | `_dedup_key` | `string` | `"{conversationId}:{answerIndex}"` — cross-source dedup key | ### Session identity | Field | Type | Description | |-------|------|-------------| | `conversationId` | `string` | Stable per-page-load UUID (formerly `_sessionId` in contribution records) | | `feedbackId` | `string \| null` | Per-rating event UUID. **Contribution records**: FK pointing at the matching `feedback/` record (`null` when the user contributed without rating individually). **Feedback records**: own idempotency UUID (formerly `sessionId`). | ### Record descriptor | Field | Type | Description | |-------|------|-------------| | `answerIndex` | `int` | Zero-based position of the answer in the conversation | | `action` | `"rate" \| "retract"` | `"retract"` tombstones are always excluded from clean output | | `prevFeedbackId` | `string \| null` | `feedbackId` of the record this one supersedes. Set on `action="rate"` edits and `action="retract"` tombstones. `null` for a first-time rating. | | `editCount` | `int \| null` | `0` for first rating, `+1` per edit. `null` for tombstones. | | `status` | `"active" \| "retracted"` | Managed by the dedup pipeline | ### Rating | Field | Type | Description | |-------|------|-------------| | `ratingValue` | `int \| null` | Signed integer: `-1`/`+1` for quick; `-5`...`+5` for panel | | `ratingSlug` | `string \| null` | Snake_case canonical identifier (e.g. `"helpful"`, `"mostly_positive"`) | | `ratingTitle` | `string \| null` | Human display string (e.g. `"Helpful"`, `"Mostly yes"`) | | `ratingMode` | `"quick" \| "panel" \| null` | Which rating widget produced this record | | `message` | `string` | Free-text comment (empty string when absent) | ### Conversation content | Field | Type | Description | |-------|------|-------------| | `query` | `string` | User question | | `answer` | `string` | Model response that was rated | ### Model attribution (8-key object) | Field | Type | Description | |-------|------|-------------| | `model` | `dict \| null` | Full model attribution; `null` when no model was configured | | `model.id` | `string \| null` | Canonical model identifier | | `model.provider` | `string \| null` | Inference provider (e.g. `"huggingface"`, `"anthropic"`) | | `model.model` | `string \| null` | HF model path or model string | | `model.label` | `string \| null` | Human display name | | `model.endpoint` | `string \| null` | Inference endpoint URL | | `model.info_url` | `string \| null` | Model info/documentation link | | `model.description` | `string \| null` | Short description | | `model.default` | `bool \| null` | `True` when this was the default model | ### Context | Field | Type | Description | |-------|------|-------------| | `page` | `string` | Originating documentation page URL | | `consentVersion` | `null` | Reserved; always `null` while `CONSENT_VERSION_ENABLED = False` | | `ts` | `int` | Client-side event timestamp (ms since epoch) | > **Note** — retraction tombstones (`action="retract"`) are always excluded > from `clean_dataset.jsonl`. The `action` field therefore never appears in > the clean output described by this table. --- ## 7. Enabling Feedback Persistence By default, `POST /v1/feedback` is **log-only** (no dataset writes). To enable dual-source collection: 1. Set environment variable `FEEDBACK_PERSIST_ENABLED=true` in your HF Space (Settings -> Repository secrets). 2. Ensure `TRAINING_DATASET_REPO` and `HF_WRITE_TOKEN` (or `HF_TOKEN`) are also set. 3. Without both of the above, `FEEDBACK_PERSIST_ENABLED` has no effect. > **Important:** enabling this flag means duplicates **will** appear in the > dataset whenever a user both rates answers AND clicks Contribute. Always run > `deduplicate_dataset.py` before training. --- ## 8. Consent Version (Reserved) `consentVersion` is collected for future use but **not currently enforced**. | Setting | Location | Current value | |---------|----------|---------------| | `CONSENT_VERSION_ENABLED` | `_dataset_schema.py` | `False` | | `CONSENT_VERSION_ENFORCEMENT_ENABLED` | `app.py` | `False` | | `RESERVED_CONSENT_VERSION` | `_dataset_schema.py` | `"1.0.0"` | | JS constant | `ai-assistant.js` | commented out (`// var CONSENT_VERSION = '1.0.0'`) | While both flags are `False`, all stored records have `consentVersion: null` regardless of what the client sends, including historical records that stored `"v1.0"`. To activate enforcement, flip both flags to `True`, uncomment the JS constant, and set `TRAINING_CONSENT_VERSION` in `app.py`. --- ## 9. Summary of Changes ### v1.0 (initial) | Component | Change | |-----------|--------| | `app.py` | Added `FEEDBACK_PERSIST_ENABLED` constant | | `app.py /v1/contribute` | Stored JSONL records carry `_source="contribution"` and `_dedup_key` | | `app.py /v1/feedback` | Optional HF persistence; stored records carry `_source="feedback"` and `_dedup_key` | | `app.py /v1/feedback` | Retraction tombstones detected, validated, rate-limit-exempt, committed with distinct message | | `ai-assistant.js` | Added `conversationId: _sessionId` to feedback POST payload | | `ai-assistant.js _feedbackStore` | Added `conversationId` field | | `ai-assistant.js tRecords` | Added `_source: 'contribution'` to each record | | `deduplicate_dataset.py` | Post-loop filter removes tombstone winners from clean output | ### v2.0 (current — additive, backward compatible) | Component | Change | |-----------|--------| | `_dataset_schema.py` | `SCHEMA_VERSION` bumped 1->2; new `editCount` column; `feedbackId` FK on contribution records; `prevFeedbackId` on `action="rate"` edits; `consentVersion` always null; `_safe_id`/`_safe_int` guards; 8-key model shape unified | | `_dataset_schema.py` | `normalize_feedback_record` / `normalize_contribution_record` always write `SCHEMA_VERSION` (2) — fixes `int(1 or 2) = 1` Python short-circuit bug that stored v1 headers on v2-content records | | `app.py` | `TRAINING_CONSENT_VERSION` hardcoded check replaced by `CONSENT_VERSION_ENFORCEMENT_ENABLED = False` flag — eliminates 422 errors for `consentVersion: null` payloads | | `app.py` | `supported_versions` expanded `{1}->{1, 2}` to accept updated JS clients | | `ai-assistant.js` | `schemaVersion: 1->2` in all four payload sites (quick feedback, panel `_rebuildFeedbackFormIn`, panel `_buildFeedbackBlock`, contribution envelope) | | `ai-assistant.js` | `feedbackId`, `prevFeedbackId`, `editCount` forwarded in `tRecords` from `_feedbackStore` | | `ai-assistant.js` | Shared `_buildModelInfo(cfg)` helper produces canonical 8-key model for all sites | | `deduplicate_dataset.py` | Integrates `_dataset_schema.normalize_record` for v1->v2 normalisation; adds `--stats-only` flag; reports schema version breakdown and FK population counts | | `DATASET_COLLECTION_GUIDANCE.md` | Updated to v2; revised field reference; updated folder structure; v2 dedup script | --- ## 10. Viewing the Dataset from the AI Panel The Extended Settings sheet (Endpoint Configuration -> **Dataset Endpoint**) surfaces the dataset directly in the browser, with no secret ever leaving the server. How the panel resolves the dataset repo (two sources, first match wins): 1. **Explicit** — `ai_assistant_panel_dataset_repo = "org/repo"` in `conf.py`. Highest trust; no network call. Use for offline docs or to pin a specific repo. 2. **Auto-discovery** — when a training URL is configured, the panel issues `GET {proxyBase}/` and reads `training.dataset_repo` from the JSON. From the repo id the panel builds three clickable links: | Card | URL | |------|-----| | Dataset root | `https://huggingface.co/datasets/` | | Feedback records | `https://huggingface.co/datasets//tree/main/feedback` | | Contributions | `https://huggingface.co/datasets//tree/main/contributions` |