chore: make working notes private (untrack notes/ + docs/INTERPRETIVE_SIGNALS.md)

These were committed by accident. Untrack via git rm --cached (local copies kept)
and gitignore notes/ + docs/INTERPRETIVE_SIGNALS.md so they stay private going
forward. History retains the old copies; this only stops future tracking.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.gitignore

@@ -49,3 +49,7 @@ whisker-render.mjs
 output/
 public/peer-ranks.json
 **/* (1).tsx
+
+# private working notes / design specs / migration plans — not for publishing
+notes/
+docs/INTERPRETIVE_SIGNALS.md

docs/INTERPRETIVE_SIGNALS.md

@@ -1,622 +0,0 @@
-# EvalCards interpretive signals — frontend implementation spec
-
-**Status:** implemented reference. Backend producer lives in `eval_card_backend`; field shapes below are stable and covered by that backend's test suite.
-
-**Companion docs:**
-- Spec source of truth: *EvalCards Interpretive Signals v1.0* (Anka Reuel, Stanford). Section refs (§3, §4, …) below point at that doc.
-- Backend implementation reference: `eval_card_backend`.
-
----
-
-## 0. What this PR does at a glance
-
-The backend now annotates evaluation records with four interpretive signals:
-
-1. **Reproducibility gap** — *per row.* Was the evaluation documented well enough to be re-run? Surfaced as a missing-fields list (e.g. "missing `max_tokens`").
-2. **Reporting completeness** — *per benchmark.* What fraction of EvalCards-required documentation fields are populated? Surfaced as a `[0, 1]` score with a missing-field breakdown.
-3. **Provenance** — *per row.* Who reported this score (first-party / third-party / collaborative / unspecified), and is it the only source for this `(model, benchmark, metric)` group?
-4. **Comparability** — *per `(model, benchmark, metric)` group.* Two flavors: **variant divergence** (same model, same benchmark, different setups → diverging scores) and **cross-party divergence** (different orgs reporting → diverging scores).
-
-Plus a corpus-level rollup file (`corpus-aggregates.json`) for a stratified analytics page.
-
-The frontend's job: surface these signals **in three places** — row-level badges, per-eval / per-model summary panels, and a corpus dashboard view.
-
----
-
-## 1. Where the new data lives
-
-All fields are new additions to existing artifacts. No artifact is removed or reshaped.
-
-| Artifact | New fields |
-|---|---|
-| `evals/{id}.json` (`HFEvalDetail`) | Per-row `evalcards.annotations` block on every `metrics[].model_results[]` and `subtasks[…].metrics[].model_results[]`. Plus eval-root `evalcards.annotations.reporting_completeness`, `evalcards.annotations.benchmark_comparability`, and three top-level summaries: `reproducibility_summary`, `provenance_summary`, `comparability_summary`. |
-| `models/{id}.json` (`HFModelDetail`) | Per-row `evalcards.annotations` block on every `hierarchy_by_category[*][*].metrics[].model_results[]`. Plus three top-level summaries scoped to that model. |
-| `eval-list.json` / `eval-list-lite.json` (`HFEvalListEntry`) | Three summaries per entry. |
-| `model-cards.json` / `model-cards-lite.json` (`HFModelCardEntry`) | Three summaries per entry. |
-| `hierarchy.json` (`EvalHierarchy`) | Composite and benchmark entries carry the three summaries; the frontend adapter synthesizes family rows for listing from the backend's top-level `composites[]` plus flat `families[]` lookup. |
-| **`corpus-aggregates.json` (NEW FILE)** | Stratified rollups for paper / dashboard use. |
-| `manifest.json` | New entry in `summary_artifacts`: `corpus_aggregates: "corpus-aggregates.json"`. |
-
-`signal_version` (currently `"1.0"`) is present on every annotation. Treat it as opaque; surface only in admin/debug.
-
----
-
-## 2. TypeScript types to add
-
-Add to `lib/backend-artifacts.ts` (preferred — these are pipeline contract types):
-
-```ts
-// Spec §3
-export interface ReproducibilityGap {
-  has_reproducibility_gap: boolean
-  missing_fields: string[]              // e.g. ["max_tokens"]
-  required_field_count: number          // 2 base + 2 if agentic on current runtime
-  populated_field_count: number
-  signal_version: string
-}
-
-// Spec §5
-export type ProvenanceSourceType =
-  | "first_party"
-  | "third_party"
-  | "collaborative"
-  | "unspecified"
-
-export interface Provenance {
-  source_type: ProvenanceSourceType
-  is_multi_source: boolean
-  first_party_only: boolean             // see §6.1 below for caveat
-  distinct_reporting_organizations: number
-  signal_version: string
-}
-
-// Spec §6.1
-export interface VariantDivergence {
-  has_variant_divergence: boolean
-  group_id: string                      // "{model_route_id}__{metric_summary_id}"
-  divergence_magnitude: number
-  threshold_used: number
-  threshold_basis:
-    | "proportion_or_continuous_normalized"
-    | "percent"
-    | "range_5pct"
-    | "fallback_default"
-  differing_setup_fields: Array<{ field: string; values: unknown[] }>
-  scores_in_group: number[]
-  this_triple_score: number | null      // this row's score within the group
-  triple_count_in_group: number
-  score_scale_anomaly: boolean
-  group_variant_breakdown: Array<{ variant_key: string; row_count: number }>
-  signal_version: string
-}
-
-// Spec §6.2
-export interface CrossPartyDivergence {
-  has_cross_party_divergence: boolean
-  group_id: string
-  divergence_magnitude: number
-  threshold_used: number
-  threshold_basis: VariantDivergence["threshold_basis"]
-  scores_by_organization: Record<string, number>   // display org name → score
-  differing_setup_fields: Array<{ field: string; values: unknown[] }>
-  organization_count: number
-  group_variant_breakdown: Array<{ variant_key: string; row_count: number }>
-  signal_version: string
-}
-
-// Per-row annotation block (carried on every model_result row)
-export interface RowAnnotations {
-  reproducibility_gap: ReproducibilityGap | null
-  provenance: Provenance | null
-  variant_divergence: VariantDivergence | null
-  cross_party_divergence: CrossPartyDivergence | null
-}
-
-// Spec §4
-export interface ReportingCompleteness {
-  completeness_score: number            // [0, 1]
-  total_fields_evaluated: number
-  missing_required_fields: string[]     // dotted paths
-  partial_fields: Array<{
-    field_path: string
-    score: number                       // (0, 1) — strictly between
-    populated_subitems: number
-    total_subitems: number
-  }>
-  field_scores: Array<{
-    field_path: string
-    coverage_type: "full" | "partial" | "reserved"
-    score: number                       // [0, 1]
-  }>
-  signal_version: string
-}
-
-export interface BenchmarkComparability {
-  variant_divergence_groups: Array<{
-    group_id: string
-    model_route_id: string
-    divergence_magnitude: number
-    threshold_used: number
-    threshold_basis: VariantDivergence["threshold_basis"]
-    differing_setup_fields: VariantDivergence["differing_setup_fields"]
-  }>
-  cross_party_divergence_groups: Array<{
-    group_id: string
-    model_route_id: string
-    divergence_magnitude: number
-    threshold_used: number
-    threshold_basis: VariantDivergence["threshold_basis"]
-    scores_by_organization: Record<string, number>
-    differing_setup_fields: VariantDivergence["differing_setup_fields"]
-  }>
-}
-
-// Eval-root or model-root annotation block
-export interface EvalcardsAnnotations {
-  reporting_completeness?: ReportingCompleteness
-  benchmark_comparability?: BenchmarkComparability
-}
-
-// Top-level summary blocks (present on eval-list / model-cards / eval / model / hierarchy nodes)
-export interface ReproducibilitySummary {
-  results_total: number
-  has_reproducibility_gap_count: number
-  populated_ratio_avg: number | null    // null when results_total == 0
-}
-
-export interface ProvenanceSummary {
-  total_results: number
-  total_groups: number
-  multi_source_groups: number
-  first_party_only_groups: number
-  source_type_distribution: Record<ProvenanceSourceType, number>
-}
-
-export interface ComparabilitySummary {
-  total_groups: number
-  groups_with_variant_check: number     // eligible groups (>=2 rows, differing setups, >=2 scored)
-  groups_with_cross_party_check: number // eligible groups (>=2 named orgs)
-  variant_divergent_count: number
-  cross_party_divergent_count: number
-}
-
-export interface SignalSummaries {
-  reproducibility_summary?: ReproducibilitySummary
-  provenance_summary?: ProvenanceSummary
-  comparability_summary?: ComparabilitySummary
-}
-
-// corpus-aggregates.json
-export interface CorpusAggregates {
-  generated_at: string
-  signal_version: string
-  stratification_dimensions: ["category"]
-  reproducibility: Stratified<ReproducibilityCorpusBlock>
-  completeness:   Stratified<CompletenessCorpusBlock>
-  provenance:     Stratified<ProvenanceCorpusBlock>
-  comparability:  Stratified<ComparabilityCorpusBlock>
-}
-
-export interface Stratified<T> {
-  overall: T
-  by_category: Record<string, T>        // categories: agentic | general | knowledge | reasoning | safety | other
-}
-
-export interface ReproducibilityCorpusBlock {
-  total_triples: number
-  triples_with_reproducibility_gap: number
-  reproducibility_gap_rate: number | null
-  agentic_triples: number
-  per_field_missingness: Record<string, {
-    missing_count: number
-    missing_rate: number | null
-    denominator: "all_triples" | "agentic_only"
-    denominator_count: number
-  }>
-}
-
-export interface CompletenessCorpusBlock {
-  total_benchmarks: number
-  completeness_score_mean: number | null
-  completeness_score_median: number | null
-  per_field_population: Record<string, {
-    mean_score: number
-    populated_rate: number
-    fully_populated_rate: number
-    benchmark_count: number
-  }>
-}
-
-export interface ProvenanceCorpusBlock {
-  total_triples: number
-  total_groups: number
-  multi_source_groups: number
-  multi_source_rate: number | null
-  first_party_only_groups: number
-  first_party_only_rate: number | null
-  source_type_distribution: Record<ProvenanceSourceType, number>
-}
-
-export interface ComparabilityCorpusBlock {
-  total_groups: number
-  variant_eligible_groups: number
-  variant_divergent_groups: number
-  variant_divergence_rate: number | null
-  cross_party_eligible_groups: number
-  cross_party_divergent_groups: number
-  cross_party_divergence_rate: number | null   // commonly null on current corpus
-}
-```
-
-Then in `lib/hf-data.ts`:
-
-- Extend `HFEvalModelResult` (line ~522) with `evalcards?: { annotations?: RowAnnotations }`.
-- Extend `HFEvalDetail` (line ~556) with `evalcards?: { annotations?: EvalcardsAnnotations }` plus the three summary fields from `SignalSummaries`.
-- Extend `HFEvalListEntry` (line ~475) with `SignalSummaries` fields.
-- Extend `HFModelCardEntry` (line ~439) with `SignalSummaries` fields.
-- Extend `HFModelDetail` (line ~571) with `SignalSummaries` fields.
-- Extend `HFModelHierarchyMetric` (line ~616) — `model_results` already typed as `HFEvalModelResult`, so the per-row annotations propagate automatically.
-
-In `EvalHierarchy` types (`lib/backend-artifacts.ts` line ~54), add `SignalSummaries` to both `HierarchyFamily` and `HierarchyBenchmark`.
-
-All fields are **optional** at the type level — older cached snapshots won't have them, and the frontend should render gracefully when they're absent.
-
----
-
-## 3. Data plumbing
-
-### 3.1 New fetcher + API route for corpus aggregates
-
-In `lib/hf-data.ts`, add after the existing fetchers (~line 866):
-
-```ts
-export async function fetchCorpusAggregates(): Promise<CorpusAggregates | null> {
-  return fetchHFJsonSafe<CorpusAggregates>("corpus-aggregates.json")
-}
-```
-
-Add to `scripts/cache-hf-data.mjs` `CACHE_ROOT_FILES` array: `"corpus-aggregates.json"`. (Mark it optional in `OPTIONAL_CACHE_ROOT_FILES` if shipping while the HF dataset upload is still rolling — once the backend pipeline next runs against the dataset, the file will appear.)
-
-Create `app/api/corpus-aggregates/route.ts`:
-
-```ts
-import { NextResponse } from "next/server"
-import { fetchCorpusAggregates } from "@/lib/hf-data"
-
-export async function GET() {
-  const aggregates = await fetchCorpusAggregates()
-  if (!aggregates) {
-    return NextResponse.json({ error: "Corpus aggregates not available" }, { status: 404 })
-  }
-  return NextResponse.json(aggregates)
-}
-```
-
-### 3.2 Rest of plumbing is automatic
-
-Existing fetchers (`fetchEvalDetail`, `fetchModelDetail`, `fetchEvalList`, `fetchModelCardsList`, `fetchEvalHierarchy`) just pull the raw JSON, so the new fields propagate without code changes once the types above are widened.
-
----
-
-## 4. UX components to build
-
-Build a small set of reusable signal components in `components/signals/`. Each takes one of the typed shapes above and renders a badge / panel. This keeps signal rendering consistent across `eval-detail.tsx`, `benchmark-detail.tsx`, `model-compare-dialog.tsx`, and the new corpus dashboard.
-
-```
-components/signals/
-├── reproducibility-badge.tsx
-├── provenance-badge.tsx          // already partially exists in benchmark-detail.tsx — see §4.2
-├── variant-divergence-badge.tsx
-├── cross-party-divergence-badge.tsx
-├── reproducibility-panel.tsx      // detail view — full missing-fields list
-├── completeness-panel.tsx         // detail view — score bar + missing-field list
-├── comparability-panel.tsx        // detail view — divergent groups list
-├── signals-row-badges.tsx         // composite: renders all four row-level badges with proper spacing
-└── signal-tooltip.tsx             // shared tooltip primitive
-```
-
-All badges should follow the existing tone conventions used by `getRelationshipBadgeTone` ([components/benchmark-detail.tsx:289](../components/benchmark-detail.tsx#L289)) and the `Badge` primitive in [components/ui/badge.tsx](../components/ui/badge.tsx).
-
-### 4.1 Row-level badges — placement
-
-Insert `<SignalsRowBadges annotations={modelResult.evalcards?.annotations} />` next to the score cell in:
-
-- **Eval detail leaderboard table** — [components/eval-detail.tsx:869-871](../components/eval-detail.tsx#L869-L871) (the `<TableCell className="text-right">` containing the score). Render badges below the score on a new line for desktop, hidden on mobile.
-- **Benchmark detail rows** — `components/benchmark-detail.tsx` renders score rows in several places (search for `formatRawScoreValue`); insert the same component.
-- **Model compare dialog** — [components/model-compare-dialog.tsx](../components/model-compare-dialog.tsx) score columns.
-
-**Display rules — only badge for actionable states.** Silence is meaningful here.
-
-| Signal | Show badge when | Hide when |
-|---|---|---|
-| Reproducibility | `has_reproducibility_gap === true` | gap=false, or annotation absent |
-| Provenance | `source_type` ∈ {`first_party`, `third_party`, `collaborative`} | `source_type === "unspecified"` |
-| Variant divergence | `variant_divergence !== null && has_variant_divergence === true` | null (not applicable) or false (checked, fine) |
-| Cross-party divergence | `cross_party_divergence !== null && has_cross_party_divergence === true` | null (almost always on current corpus) or false |
-
-`has_*: false` means "we checked and it's fine" — silent success. `null` means "not applicable / not enough data" — also silent. **Only divergent / gap-positive states warrant pixels.**
-
-**Dedup rule.** `variant_divergence` and `cross_party_divergence` are duplicated onto every row in the same group. If you render three rows from the same `group_id`, render the divergence badge on each row but the *expanded panel* (§4.4) only once at the group header.
-
-### 4.2 Provenance badge — reuse what's there
-
-[components/benchmark-detail.tsx:262-302](../components/benchmark-detail.tsx#L262-L302) already has `getRelationshipShortLabel` and `getRelationshipBadgeTone`. Extract these into `components/signals/provenance-badge.tsx` and import back into `benchmark-detail.tsx`. The new badge should **also** consume the new `Provenance` annotation when present (it carries `is_multi_source` and `first_party_only`, which the current implementation derives row-by-row from `source_metadata` alone).
-
-When `provenance.first_party_only === true`, show a small ⚠ subtle indicator on the first-party badge ("first-party only — no independent replication"). This is the headline use of the signal for policy-mode readers.
-
-### 4.3 Reproducibility badge — content rules
-
-Tooltip content depends on audience mode (`useAudienceMode()` from [components/audience-mode-provider.tsx:40](../components/audience-mode-provider.tsx#L40)):
-
-- Research mode: "Setup not fully documented. Missing: `max_tokens`, `eval_plan`."
-- Policy mode: "This score's setup isn't fully documented, so it can't be re-run as-is."
-
-Always include the count "{populated_field_count} of {required_field_count} setup fields recorded." Don't hardcode "4 fields" — the active runtime checks 2 base fields (`temperature`, `max_tokens`) plus 2 agentic fields (`eval_plan`, `eval_limits`) when the benchmark is agentic. Read counts off the annotation.
-
-### 4.4 Detail panels — placement
-
-#### Reproducibility panel
-The existing "Evaluation Provenance" panel in [components/eval-detail.tsx:952-998](../components/eval-detail.tsx#L952-L998) (rendered when a row is expanded) is the right place for the **per-row** reproducibility breakdown. Add a new `DetailPanel` adjacent to it:
-
-```tsx
-{rowAnnotations?.reproducibility_gap && (
-  <DetailPanel
-    title={isResearchView ? "Reproducibility" : "Re-runnability"}
-    subtitle={
-      isResearchView
-        ? "Whether the setup is documented well enough for someone else to re-run."
-        : "Whether someone could re-run this evaluation with the information available."
-    }
-  >
-    <MetaRow
-      label="Setup fields recorded"
-      value={`${rowAnnotations.reproducibility_gap.populated_field_count} of ${rowAnnotations.reproducibility_gap.required_field_count}`}
-    />
-    {rowAnnotations.reproducibility_gap.missing_fields.length > 0 && (
-      <MetaRow
-        label="Missing"
-        value={rowAnnotations.reproducibility_gap.missing_fields.join(", ")}
-      />
-    )}
-  </DetailPanel>
-)}
-```
-
-#### Completeness panel
-Render at the **eval-detail header level** (above the leaderboard, below the metric specification card). New `<CompletenessPanel completeness={detail.evalcards?.annotations?.reporting_completeness} />`. UI: progress bar showing `completeness_score`, label "{N} of {M} fields populated" where N = sum of `field_scores[].score` rounded, M = `total_fields_evaluated`. Below: collapsible accordions:
-
-- **Missing required fields** (count badge) — list of `missing_required_fields` with friendly labels (see §6.4 for label mapping).
-- **Partially populated** (count badge) — `partial_fields` rendered as "{field}: {populated_subitems}/{total_subitems}".
-
-In policy mode, don't show the dotted-path field names — show friendly labels only. In research mode, show both.
-
-#### Comparability panel
-Also at eval-detail header level. Sourced from `detail.evalcards?.annotations?.benchmark_comparability`. Render as two collapsibles — "Variant divergence ({count})" and "Cross-party divergence ({count})". Each item should link to the relevant model row (use `model_route_id` from each group entry as anchor — add `id={"row-" + model_route_id}` on the leaderboard row).
-
-When both arrays are empty, hide the panel entirely. When `comparability_summary.groups_with_cross_party_check === 0` (the common state), surface a small note: "No third-party reports available for cross-party comparison."
-
-### 4.5 Per-eval header chips
-On the eval-detail page header (next to existing "Measures" / "Source dataset" chips around [components/eval-detail.tsx:486-525](../components/eval-detail.tsx#L486-L525)), add a fourth chip when `evalcards.annotations.reporting_completeness` is present:
-
-> **Documentation**
-> {round(completeness_score * 100)}%
-
-Tooltip: "{N} of {M} EvalCards documentation fields populated for this benchmark."
-
-### 4.6 Per-model card chips
-On `components/eval-card.tsx` and the model card pages, add three chips driven by the model-level summaries. Replace the hand-written hint at [components/eval-card.tsx:250](../components/eval-card.tsx#L250) ("Some results lack generation settings; compare scores with care.") with a data-driven version:
-
-> {has_reproducibility_gap_count} of {results_total} reported scores aren't fully documented.
-
-Show only when `has_reproducibility_gap_count > 0`. The hand-written hint was a placeholder for exactly this signal — wire it up.
-
----
-
-## 5. New page: corpus dashboard
-
-Add `app/corpus/page.tsx` (linked from main navigation [components/navigation.tsx](../components/navigation.tsx)). Server component that calls `fetchCorpusAggregates()` and renders four sections:
-
-### 5.1 Reproducibility section
-- Headline number: `reproducibility_gap_rate` rendered as percentage. Sub-label: "{triples_with_reproducibility_gap} of {total_triples} reported scores."
-- Per-field horizontal bar chart from `per_field_missingness`. **Bar denominator depends on `denominator` field**: agentic-only fields use `agentic_triples`, others use `total_triples`. Label each bar with the denominator type so users understand.
-- Toggle: `overall` ↔ `by_category` (rendered as a small-multiple grid, one panel per category).
-
-### 5.2 Completeness section
-- Headline: `completeness_score_mean` (and median) across `total_benchmarks`.
-- Histogram of per-benchmark scores (pull individual benchmark scores from `eval-list.json` `reporting_completeness.completeness_score`, since corpus-aggregates only carries mean/median).
-- Per-field bar chart from `per_field_population` — three bars per field: `mean_score`, `populated_rate`, `fully_populated_rate`. (See §6.7 for which one to highlight per coverage type.)
-
-### 5.3 Provenance section
-- Stacked bar of `source_type_distribution` (across all triples).
-- Two ratios: `multi_source_rate`, `first_party_only_rate`. Label both: "% of (model, benchmark, metric) groups."
-
-### 5.4 Comparability section
-- Two side-by-side panels: Variant divergence (eligible-aware rate) and Cross-party divergence (often null).
-- **When `cross_party_divergence_rate === null`:** show a "Not enough multi-org coverage to compute" empty state, not "0%". Same for `variant_divergence_rate === null`. This is critical — see §6.5.
-
-All sections support a category toggle (research mode shows category breakdowns by default; policy mode shows overall by default).
-
----
-
-## 6. Caveats and edge cases (read these before implementing)
-
-### 6.1 `first_party_only` semantics
-A row can be `first_party_only: true` even when `is_multi_source: false`. The spec literal: a group with one *named* org reporting first-party gets the badge. **Don't read it as "exclusive coverage"** — read it as "no independent replication." The label suggestion is "First-party only" rather than "Sole source."
-
-If `distinct_reporting_organizations === 0` (all rows have null org), `first_party_only` is `false` even when `source_type === "first_party"`. Render the row's source as "First-party (org unspecified)" in research mode; suppress the first-party-only badge.
-
-### 6.2 Active reproducibility field set is reduced
-The spec describes four base fields (`temperature`, `top_p`, `max_tokens`, `prompt_template`); the active backend currently checks **only `temperature` and `max_tokens`** plus `eval_plan` / `eval_limits` for agentic benchmarks. **Don't hardcode "4 fields" anywhere.** Always read `required_field_count` off the annotation. This is a deliberate spec-author choice and may revert; the field count is the only stable interface.
-
-### 6.3 Missing-field path strings
-`missing_fields` for reproducibility uses bare names (e.g. `"max_tokens"`). `missing_required_fields` for completeness uses dotted paths (e.g. `"autobenchmarkcard.methodology.baseline_results"`). Different conventions, intentional. Build a small label map for completeness paths — paths come from `eval_card_backend/src/eval_card_backend/registry/completeness_fields.json`. Suggested label rules:
-
-- Drop the `autobenchmarkcard.` / `eee_eval.` / `evalcards.` prefix.
-- Replace dots with " / ", underscore with space, title-case.
-- Example: `autobenchmarkcard.methodology.baseline_results` → "Methodology / Baseline results".
-
-### 6.4 `differing_setup_fields[].values` may contain null and mixed types
-Per spec §6.1.4, `null` is a *distinct* value from any explicit setting (comparing "explicit 2048" to "unspecified" is meaningful). Render `null` as "(unspecified)" rather than the string "null". Numeric, string, boolean, and object values can all appear in the same array; render with `JSON.stringify` for objects, plain text otherwise.
-
-### 6.5 `null` rates in comparability are *not* zero
-Eligibility-aware denominators mean `variant_divergence_rate` and `cross_party_divergence_rate` are `null` when no groups were eligible. **Render as "N/A — not enough data" or an empty-state card, never as "0%".** On the current corpus, `cross_party_divergence_rate` will commonly be null (third-party reports are sparse). Treat this as a normal state, not a data-loading error.
-
-### 6.6 Score-scale anomaly flag
-`variant_divergence.score_scale_anomaly === true` indicates the metric was declared `proportion` but scores fell outside [0, 1] — usually a metric-normalization bug upstream. Surface as a small "data quality warning" annotation alongside the divergence number; the divergence is still computed but the threshold may not be apples-to-apples.
-
-### 6.7 `mean_score` vs `populated_rate` for completeness
-Per-field aggregates expose three numbers. Pick which to display based on `coverage_type`:
-
-- **`full` and `reserved` fields** — `mean_score` and `populated_rate` are equal. Show one number labeled "% of benchmarks populating this field."
-- **`partial` fields** — they diverge. `populated_rate` = % of benchmarks with *any* sub-item; `mean_score` = average sub-item population fraction. Show both: "{populated_rate}% have any data, {mean_score}% on average across sub-items."
-
-### 6.8 No `computed_at` on per-record annotations
-Only `signal_version` is on each annotation. For "last computed" UI text, use `manifest.json → generated_at` from the existing `BackendManifest`.
-
-### 6.9 Stratification categories
-`by_category` keys are: `agentic`, `general`, `knowledge`, `reasoning`, `safety`, `other`. Same set as the existing `category` field on evals — reuse whatever color scheme is currently keyed off `inferCategoryFromBenchmark` ([lib/benchmark-schema.ts](../lib/benchmark-schema.ts)).
-
-### 6.10 Annotation block can be `null` or absent
-`evalcards.annotations.{reproducibility_gap,provenance,variant_divergence,cross_party_divergence}` can each be `null` independently, and the entire `evalcards` block may be absent on older cached snapshots. Use optional chaining everywhere; never assume presence. The `RowAnnotations` type intentionally types each subfield as `T | null` (not `T | undefined`) because the backend writes explicit `null`.
-
----
-
-## 7. Suggested implementation order
-
-1. **Types + plumbing** (1–2 hours): types in `backend-artifacts.ts` + `hf-data.ts`, the `fetchCorpusAggregates` fetcher, the API route, and adding `corpus-aggregates.json` to the cache script. No UI yet.
-2. **Row-level badges** (½ day): build `signals/` directory with the four badge components, the dedup-aware `signals-row-badges.tsx`, and wire into eval-detail and benchmark-detail. This is the most visible win.
-3. **Per-eval completeness panel + comparability panel** (½ day): single benchmark, easy to design around. New `CompletenessPanel` is the headline new UX in this set.
-4. **Per-row reproducibility detail panel** (1–2 hours): drops into the existing expanded row layout.
-5. **Per-eval / per-model header chips + replace the hand-written gap hint** (1–2 hours): wires the summary fields into existing card surfaces.
-6. **Corpus dashboard page** (1–2 days): new route, new components, biggest scope. Defer until 1–5 are live and reviewed.
-
-Each step is independently shippable. Steps 1–5 can land before the corpus dashboard is designed.
-
----
-
-## 8. Out of scope (don't do these yet)
-
-- **Filter / sort the eval list by signal state** ("show only benchmarks with completeness > 0.5"). Wait for the dashboard view to land first; users will tell us which filters they actually want.
-- **Side-by-side score comparison with divergence overlay.** The data supports it (`scores_in_group`, `scores_by_organization`) but the design space is large. Hold off until we see the row-level badges in use.
-- **Recompute / verification UI for missing reproducibility fields.** Backend-side; out of scope here.
-- **Per-instance sample-level badges.** Signals operate at row / benchmark level; sample-level instance data is unaffected.
-
----
-
-## 9. Reference: minimal real-shape examples
-
-Per-row `evalcards.annotations` with all four signals populated:
-
-```jsonc
-{
-  "reproducibility_gap": {
-    "has_reproducibility_gap": true,
-    "missing_fields": ["max_tokens"],
-    "required_field_count": 2,
-    "populated_field_count": 1,
-    "signal_version": "1.0"
-  },
-  "provenance": {
-    "source_type": "first_party",
-    "is_multi_source": false,
-    "first_party_only": true,
-    "distinct_reporting_organizations": 1,
-    "signal_version": "1.0"
-  },
-  "variant_divergence": null,
-  "cross_party_divergence": null
-}
-```
-
-Per-eval `evalcards.annotations` with completeness + comparability:
-
-```jsonc
-{
-  "reporting_completeness": {
-    "completeness_score": 0.62,
-    "total_fields_evaluated": 28,
-    "missing_required_fields": [
-      "autobenchmarkcard.methodology.baseline_results",
-      "autobenchmarkcard.methodology.validation",
-      "evalcards.preregistration_url"
-    ],
-    "partial_fields": [
-      { "field_path": "autobenchmarkcard.data", "score": 0.5, "populated_subitems": 2, "total_subitems": 4 }
-    ],
-    "field_scores": [/* 28 entries */],
-    "signal_version": "1.0"
-  },
-  "benchmark_comparability": {
-    "variant_divergence_groups": [
-      {
-        "group_id": "openai__gpt-5__hfopenllm_v2_bbh_accuracy",
-        "model_route_id": "openai__gpt-5",
-        "divergence_magnitude": 0.12,
-        "threshold_used": 0.05,
-        "threshold_basis": "proportion_or_continuous_normalized",
-        "differing_setup_fields": [
-          { "field": "max_tokens", "values": [2048, 4096, 8192] }
-        ]
-      }
-    ],
-    "cross_party_divergence_groups": []
-  }
-}
-```
-
-Top-level `provenance_summary` example:
-
-```jsonc
-{
-  "total_results": 142,
-  "total_groups": 47,
-  "multi_source_groups": 3,
-  "first_party_only_groups": 30,
-  "source_type_distribution": {
-    "first_party": 120,
-    "third_party": 18,
-    "collaborative": 0,
-    "unspecified": 4
-  }
-}
-```
-
-`corpus-aggregates.json` structure (top of file):
-
-```jsonc
-{
-  "generated_at": "2026-04-27T...",
-  "signal_version": "1.0",
-  "stratification_dimensions": ["category"],
-  "reproducibility": { "overall": {/* ReproducibilityCorpusBlock */}, "by_category": { "agentic": {...}, "general": {...}, ... } },
-  "completeness":   { "overall": {/* CompletenessCorpusBlock */},   "by_category": {...} },
-  "provenance":     { "overall": {/* ProvenanceCorpusBlock */},     "by_category": {...} },
-  "comparability":  { "overall": {/* ComparabilityCorpusBlock */},  "by_category": {...} }
-}
-```
-
----
-
-## 10. Audience-mode wording cheatsheet
-
-| Element | Research mode | Policy mode |
-|---|---|---|
-| Reproducibility gap badge | "Reproducibility gap" | "Setup not documented" |
-| Reproducibility tooltip | "Setup not fully documented. Missing: {fields}." | "This score's setup isn't documented, so it can't be re-run as-is." |
-| Reproducibility panel title | "Reproducibility" | "Re-runnability" |
-| Completeness chip label | "Documentation" | "Documentation" |
-| Completeness panel title | "Reporting completeness" | "How well is this benchmark documented?" |
-| Provenance: first-party | "1st party" | "Reported by model developer" |
-| Provenance: first-party only | "1st party only — no replication" | "Only the model developer reported this score" |
-| Provenance: third-party | "3rd party" | "Independently reported" |
-| Provenance: collaborative | "Collaborative" | "Joint report" |
-| Variant divergence badge | "Variant divergence" | "Score depends on setup" |
-| Variant divergence tooltip | "Scores diverge by {magnitude} across different setups: {fields}." | "Different runs of this evaluation produced different scores — the setup matters." |
-| Cross-party divergence badge | "Cross-party divergence" | "Sources disagree" |
-| Cross-party divergence tooltip | "Reports diverge by {magnitude} across organizations." | "Different organizations reported different scores for this same model on this same benchmark." |
-
-Adjust tone but keep the underlying numbers identical across modes — the data is the same, only the framing changes.
-
----
-
-*Last updated 2026-05-04. Maintainer: backend (`eval_card_backend`), frontend (`general-eval-card`). Questions on UX → discuss with @anka-evals + frontend team.*

notes/backend-v2-migration.md

@@ -1,664 +0,0 @@
-# Frontend migration to backend v2 (Stage J view layer)
-
-> **Status:** spec, drafted 2026-05-03 against `eval_card_backend`'s
-> Stage J view-layer contract.
->
-> **Sources:**
-> - Backend spec (the contract this consumes):
->   `../eval_card_backend/notes/08-frontend-view-layer.md`
-> - Canonical schema (audit/debug only; not in hot path):
->   `../eval_card_backend/notes/01-schema-from-frontend.md`
-
----
-
-## Context
-
-The legacy producer (`eval_cards_backend_pipeline`) emitted ten
-parquets where each row carried a `payload_json` VARCHAR with the
-post-TS-adapter shape baked in. The frontend's "DuckDB backend"
-(`lib/duckdb-data.ts`) read these blobs and `JSON.parse`d them — column
-projection, filter pushdown, and type contracts were all forfeited.
-
-The new producer (`eval_card_backend`) emits a typed view layer over
-its canonical normalised tables. Three Parquet files cover every page
-shape, three small JSON sidecars cover corpus-level scalars and the
-hierarchy tree. Column names match the frontend's TS interfaces
-field-for-field, so the row→object cast is a typed spread for most
-accessors. Two interfaces (`ModelResultForBenchmark` and the
-`evaluations_by_category` body of `ModelEvaluationSummary`) require a
-small mechanical reshape over the row, since one nests fields that the
-view stores flat — see the per-accessor sections below. No
-HF-record-to-display adapter logic survives.
-
-This document specifies what changes in `general-eval-card` once
-backend v2 is faithfully implemented. **The visual frontend, page
-renderers, and TS interface shapes do not change.** Only the I/O
-boundary moves.
-
----
-
-## What changes (overview)
-
-| layer | before (v1) | after (v2) |
-|---|---|---|
-| Distribution | `LOCAL_PIPELINE_OUTPUT` env var pointing at a producer output dir; `duckdb/v1/` subpath; implicit "warehouse/latest/" coupling | `SNAPSHOT_URL` env var (file:// or HF dataset URL); one snapshot pinned per deploy |
-| Storage shape | 10 parquets each with one `payload_json` column | 3 typed-column view parquets + 3 JSON sidecars |
-| Read pattern | `SELECT payload_json FROM read_parquet(?) WHERE id = ?`, then `JSON.parse` | `SELECT col1, col2, ... FROM <view> WHERE id = ?`, typed row spread |
-| List vs detail | Separate `*_lite.parquet` files | Column projection on the same parquet |
-| Suite/aggregate dispatch | Eval id prefix (`aggregate__`, `matrix__`) → different parquet | `is_summary_score` flag + `parent_benchmark_id` on `evals_view` |
-| Slug rule | Custom `replace('/', '__')` escapes; per-page slug helpers | Producer-owned RFC 3986 percent-encoded `route_id` / `evaluation_id` / `metric_summary_id`; frontend decodes only on `<Link>` href |
-| Corpus aggregates | `corpus-aggregates.json` over HF JSON loader | `headline.json` sidecar in the snapshot dir |
-| Hierarchy | Synthesised in the producer's `eval_hierarchy` JSON | `hierarchy.json` sidecar |
-| Backend manifest | `manifest.json` fetched from upstream HF dataset root via `lib/hf-data.ts` | `manifest.json` sidecar inside the snapshot dir, read via `SNAPSHOT_URL` |
-
-The TS interfaces (`EvaluationCardData`, `BenchmarkEvalSummary`,
-`ModelEvaluationSummary`, `ModelResultForBenchmark`, `CorpusAggregates`,
-`EvalHierarchy`, `BackendManifest`) stay as-is — the producer agreed to
-emit columns under those exact names.
-
----
-
-## What does not change
-
-- All page components under `app/`. The renderer trees are unchanged.
-- TS interface declarations in `lib/benchmark-schema.ts`,
-  `lib/eval-processing.ts`, `lib/backend-artifacts.ts`. These are now
-  the contract surface — column names match field names by agreement
-  with the producer.
-- Component files under `components/`.
-- `lib/glossary.ts`, `lib/known-issues.ts`, `lib/utils.ts`,
-  `lib/na-utils.ts` — these are pure presentation helpers.
-- `app/api/*/route.ts` handlers stay as thin pass-throughs to
-  `lib/data-backend.ts`.
-
----
-
-## Distribution: `SNAPSHOT_URL`
-
-Frontend reads `SNAPSHOT_URL` from env at process start. One deploy =
-one snapshot. The URL points at a directory containing the six
-artifacts the frontend reads:
-
-```
-$SNAPSHOT_URL/
-├── models_view.parquet
-├── evals_view.parquet
-├── eval_results_view.parquet
-├── headline.json
-├── hierarchy.json
-└── manifest.json
-```
-
-Examples:
-
-- Local dev: `SNAPSHOT_URL=file:///path/to/eval_card_backend/warehouse/2026-05-03T15-48-59Z`
-- Production (pinned snapshot): `SNAPSHOT_URL=https://huggingface.co/datasets/evaleval/eval-cards-data/resolve/<rev>/warehouse/<snapshot_id>`
-- Production (rolling): `SNAPSHOT_URL=https://huggingface.co/datasets/evaleval/eval-cards-data/resolve/main/warehouse/latest`
-
-`LOCAL_PIPELINE_OUTPUT` is removed. The `duckdb/v1/` subpath is
-removed. The producer maintains a `warehouse/latest/` alias that
-points at the most recent snapshot, so deploys can pin either to a
-timestamped snapshot (immutable, redeploy required to roll forward)
-or to `latest` (auto-rolls forward on the next Space rebuild). Within
-a running process the snapshot is still effectively constant — sidecar
-caches in `lib/sidecars.ts` are first-write-wins per process.
-
----
-
-## DuckDB connection lifecycle
-
-`lib/duckdb.ts` (new file; replaces the connection-management portion
-of `lib/duckdb-data.ts`):
-
-```ts
-import "server-only"
-import { DuckDBConnection } from "@duckdb/node-api"
-
-let connectionPromise: Promise<DuckDBConnection> | null = null
-
-const SNAPSHOT_URL = process.env.SNAPSHOT_URL
-if (!SNAPSHOT_URL) {
-  throw new Error("SNAPSHOT_URL must be set; see notes/backend-v2-migration.md")
-}
-
-const VIEWS = {
-  models_view:       `${SNAPSHOT_URL}/models_view.parquet`,
-  evals_view:        `${SNAPSHOT_URL}/evals_view.parquet`,
-  eval_results_view: `${SNAPSHOT_URL}/eval_results_view.parquet`,
-} as const
-
-export async function getConnection(): Promise<DuckDBConnection> {
-  if (!connectionPromise) {
-    connectionPromise = (async () => {
-      const conn = await DuckDBConnection.create()
-      // httpfs is built into duckdb-node-api; no INSTALL needed.
-      // Register each parquet as a view so callers write `FROM models_view`,
-      // not the full URL.
-      for (const [name, path] of Object.entries(VIEWS)) {
-        await conn.run(
-          `CREATE OR REPLACE VIEW ${name} AS SELECT * FROM read_parquet(?)`,
-          [path]
-        )
-      }
-      return conn
-    })()
-  }
-  return connectionPromise
-}
-```
-
-One connection per Node process. Views are registered once at
-startup; subsequent queries write `FROM models_view` rather than
-re-passing the parquet URL. DuckDB's column projection means the cost
-of `SELECT route_id, model_name FROM models_view` is independent of
-how wide `models_view` is.
-
----
-
-## Per-accessor mapping
-
-`lib/data-backend.ts` keeps its current export names. `lib/duckdb-data.ts`
-gets gutted; each function becomes a thin typed `SELECT`. The mapping
-below uses the column names spec'd in
-`../eval_card_backend/notes/08-frontend-view-layer.md` — the row
-returned by DuckDB casts directly to the TS interface.
-
-### Models
-
-```ts
-// getModelCards / getModelCardsLite — list pages
-export async function getModelCards(): Promise<EvaluationCardData[]> {
-  const conn = await getConnection()
-  const reader = await conn.runAndReadAll(`
-    SELECT id, route_id, model_name, model_id, canonical_model_name, developer,
-           evaluations_count, benchmarks_count, variant_count,
-           categories, category_stats, latest_timestamp,
-           evaluator_count, evaluator_names, source_type_count, source_types,
-           evidence_count, missing_generation_config_count,
-           third_party_eval_count, independent_verification_ratio,
-           reproducibility_status, eval_libraries, latest_source_name,
-           params_billions, benchmark_names, score_summary,
-           reproducibility_summary, provenance_summary, comparability_summary,
-           top_scores, source_urls, detail_urls,
-           model_url, release_date, input_modalities, output_modalities,
-           architecture, params, inference_engine, inference_platform
-    FROM models_view
-    ORDER BY latest_timestamp DESC
-  `)
-  return reader.getRowObjects() as EvaluationCardData[]
-}
-
-// "Lite" is just narrower projection — same parquet, fewer columns.
-export async function getModelCardsLite(): Promise<EvaluationCardData[]> {
-  const conn = await getConnection()
-  const reader = await conn.runAndReadAll(`
-    SELECT id, route_id, model_name, model_id, developer,
-           evaluations_count, benchmarks_count, categories,
-           latest_timestamp, third_party_eval_count,
-           independent_verification_ratio, reproducibility_status,
-           latest_source_name, params_billions
-    FROM models_view
-    ORDER BY benchmarks_count DESC, evaluations_count DESC, model_name ASC
-  `)
-  return reader.getRowObjects() as EvaluationCardData[]
-}
-
-// getModelSummaryById — detail page.
-//
-// The row carries the metadata shell (variants[], categories,
-// category_stats, signal summaries, model_family_id, raw_model_ids,
-// total_evaluations, last_updated). The full `ModelEvaluationSummary`
-// also requires `evaluations_by_category: Record<CategoryType,
-// BenchmarkEvaluation[]>`, which is a heavyweight per-cell breakdown —
-// produced by a separate join over `eval_results_view`, see
-// `getModelEvaluationCells` below.
-//
-// Returning a `ModelSummaryShell` (Omit-ed type, defined alongside the
-// existing TS interface) makes the contract explicit and stops the cast
-// from lying. The model-detail page composes the full
-// `ModelEvaluationSummary` from `shell` + `cells`.
-export type ModelSummaryShell = Omit<
-  ModelEvaluationSummary,
-  "evaluations_by_category"
->
-
-export async function getModelSummaryById(routeId: string): Promise<ModelSummaryShell | null> {
-  const conn = await getConnection()
-  const reader = await conn.runAndReadAll(
-    `SELECT * FROM models_view WHERE route_id = ? OR model_family_id = ? LIMIT 1`,
-    [routeId, routeId]
-  )
-  const rows = reader.getRowObjects()
-  if (rows.length === 0) return null
-  return rows[0] as unknown as ModelSummaryShell
-}
-
-// Per-cell reshape helper. `eval_results_view` rows carry the per-cell
-// fields scattered (model_info, score_details, evaluation_timestamp,
-// source_metadata, source_data, metric_*, etc.) rather than under a
-// nested `result: EvaluationResult` STRUCT. Reshape into the
-// `ModelResultForBenchmark` shape the leaderboard / model-detail
-// renderers expect. Single helper; reused by getEvalSummaryById and
-// getModelEvaluationCells. No HF-record-to-display logic survives.
-function reshapeCellToModelResult(row: Record<string, any>): ModelResultForBenchmark {
-  return {
-    model_info:           row.model_info,
-    model_route_id:       row.model_route_id,
-    score:                row.score,
-    score_details:        row.score_details,
-    evaluation_timestamp: row.evaluation_timestamp,
-    source_metadata:      row.source_metadata,
-    source_data:          row.source_data,
-    source_record_url:    row.source_record_url,
-    aggregate_components: row.aggregate_components,
-    result: {
-      evaluation_name:      row.metric_display_name,
-      metric_summary_id:    row.metric_summary_id,
-      metric_key:           row.metric_id,
-      evaluation_timestamp: row.evaluation_timestamp,
-      metric_config:        { lower_is_better: row.lower_is_better, unit: row.metric_unit, /* …denormalised meta… */ },
-      score_details:        row.score_details,
-      evalcards:            row.evalcards_annotations ? { annotations: row.evalcards_annotations } : undefined,
-    },
-  }
-}
-
-// Helper for the model-detail page's evaluations_by_category body.
-// The page groups by `category` in TS after this returns.
-export async function getModelEvaluationCells(modelId: string): Promise<ModelResultForBenchmark[]> {
-  const conn = await getConnection()
-  const reader = await conn.runAndReadAll(
-    `SELECT * FROM eval_results_view WHERE model_id = ? ORDER BY category, percentile DESC`,
-    [modelId]
-  )
-  return reader.getRowObjects().map(reshapeCellToModelResult)
-}
-```
-
-### Evals
-
-```ts
-// getEvalListData / getEvalListLiteData — list pages
-export async function getEvalListData(): Promise<{
-  evals: BenchmarkEvalListItem[]
-  totalModels: number
-}> {
-  const conn = await getConnection()
-  const [evalsReader, modelsReader] = await Promise.all([
-    conn.runAndReadAll(`
-      SELECT evaluation_id, evaluation_name, canonical_display_name,
-             composite_benchmark_key, composite_benchmark_name,
-             benchmark_family_key, benchmark_leaf_key, category,
-             metric_config, models_count, evaluator_names, source_types,
-             latest_source_name, third_party_ratio,
-             missing_generation_config_count, best_model, worst_model,
-             avg_score, avg_score_norm, has_card,
-             is_aggregated, aggregate_sources, tags,
-             metrics_count, metric_names, instance_data, top_score,
-             subtasks_count, is_summary_score, summary_eval_ids,
-             root_metrics, subtasks, leaderboard_metrics,
-             reproducibility_summary, provenance_summary, comparability_summary,
-             source_data
-      FROM evals_view
-      ORDER BY evaluation_name ASC
-    `),
-    conn.runAndReadAll(`SELECT COUNT(*) AS n FROM models_view`),
-  ])
-  return {
-    evals: evalsReader.getRowObjects() as BenchmarkEvalListItem[],
-    totalModels: Number(modelsReader.getRowObjects()[0].n),
-  }
-}
-
-// getEvalSummaryById — detail page.
-//
-// No more aggregate__/matrix__ id-prefix dispatch — `evals_view` is the
-// single source for all eval shapes. Suite-vs-leaf is a column
-// (`is_summary_score`, `is_aggregated`) on the same parquet.
-//
-// `model_results[]` rows go through the same reshape helper as
-// `getModelEvaluationCells` (defined below) — they share the
-// ModelResultForBenchmark target shape, so the eval/metric/cell
-// → BenchmarkEvaluation reshape is one helper, two callers.
-export async function getEvalSummaryById(evalId: string): Promise<BenchmarkEvalSummary | null> {
-  const conn = await getConnection()
-  const [evalReader, cellsReader] = await Promise.all([
-    conn.runAndReadAll(
-      `SELECT * FROM evals_view WHERE evaluation_id = ? LIMIT 1`,
-      [evalId]
-    ),
-    conn.runAndReadAll(
-      `SELECT * FROM eval_results_view
-       WHERE evaluation_id = ?
-         AND metric_id     = (SELECT primary_metric_id FROM evals_view WHERE evaluation_id = ?)
-       ORDER BY position ASC`,
-      [evalId, evalId]
-    ),
-  ])
-  const evalRows = evalReader.getRowObjects()
-  if (evalRows.length === 0) return null
-  return {
-    ...(evalRows[0] as Omit<BenchmarkEvalSummary, "model_results">),
-    model_results: cellsReader.getRowObjects().map(reshapeCellToModelResult),
-  } as BenchmarkEvalSummary
-}
-```
-
-### Developers
-
-```ts
-// getDeveloperList — list page; reads from headline.json (precomputed,
-// including producer-owned route_id, model/benchmark/evaluation counts,
-// and popular_evals). DeveloperListEntry is satisfied directly by the
-// headline entry shape.
-export async function getDeveloperList(): Promise<DeveloperListEntry[]> {
-  const headline = await fetchHeadline()
-  return headline.developers as DeveloperListEntry[]
-}
-
-// getDeveloperSummaryById — detail page; reads models_view filtered by developer.
-// The route_id on headline.developers[] is the canonical lookup key — we don't
-// re-derive `developer` from the URL slug, since percent-decoding may not
-// round-trip exactly to the producer's source string.
-export async function getDeveloperSummaryById(routeId: string) {
-  const headline = await fetchHeadline()
-  const headlineEntry = headline.developers.find((d) => d.route_id === routeId)
-  if (!headlineEntry) return null
-  const conn = await getConnection()
-  const reader = await conn.runAndReadAll(
-    `SELECT * FROM models_view WHERE developer = ?`,
-    [headlineEntry.developer]
-  )
-  const models = reader.getRowObjects() as EvaluationCardData[]
-  return { ...headlineEntry, models }
-}
-```
-
-### Dashboard convenience accessor
-
-```ts
-// Was: { models, evals } over both legacy parquets; same shape, new sources.
-export async function getDashboardData() {
-  const [models, evalListData] = await Promise.all([
-    getModelCards(),
-    getEvalListData(),
-  ])
-  return { models, evals: evalListData.evals }
-}
-```
-
----
-
-## Sidecar fetchers (replace `lib/hf-data.ts` corpus calls)
-
-Three small JSON files live in the snapshot dir alongside the
-parquets. New module `lib/sidecars.ts` exposes typed fetchers.
-`lib/hf-data.ts`'s `fetchCorpusAggregates`, `fetchEvalHierarchy`,
-`fetchBackendManifest`, and `fetchBackendManifestStatus` get their
-implementations replaced — same export names, new sources.
-
-```ts
-// lib/sidecars.ts
-import "server-only"
-import type {
-  CorpusAggregates,
-  EvalHierarchy,
-  BackendManifest,
-} from "@/lib/backend-artifacts"
-
-const SNAPSHOT_URL = process.env.SNAPSHOT_URL!
-
-let cache: {
-  manifest?: Promise<BackendManifest>
-  headline?: Promise<CorpusAggregates>
-  hierarchy?: Promise<EvalHierarchy>
-} = {}
-
-async function fetchJson<T>(name: string): Promise<T> {
-  const url = `${SNAPSHOT_URL}/${name}`
-  const res = url.startsWith("file://")
-    ? await import("fs/promises").then((fs) => fs.readFile(new URL(url), "utf8"))
-    : await fetch(url, { next: { revalidate: 3600 } }).then((r) => r.text())
-  return JSON.parse(typeof res === "string" ? res : res.toString()) as T
-}
-
-export function fetchManifest(): Promise<BackendManifest> {
-  return (cache.manifest ??= fetchJson<BackendManifest>("manifest.json"))
-}
-
-export function fetchHeadline(): Promise<CorpusAggregates> {
-  return (cache.headline ??= fetchJson<CorpusAggregates>("headline.json"))
-}
-
-export function fetchHierarchy(): Promise<EvalHierarchy> {
-  return (cache.hierarchy ??= fetchJson<EvalHierarchy>("hierarchy.json"))
-}
-```
-
-Then in `lib/hf-data.ts`:
-
-```ts
-// fetchBackendManifest: was a fetchHFJsonSafe call; now reads the snapshot sidecar.
-export const fetchBackendManifest = fetchManifest
-export const fetchCorpusAggregates = fetchHeadline
-export const fetchEvalHierarchy = fetchHierarchy
-
-// fetchBackendManifestStatus: simplified — single snapshot pin, no "latest" comparison.
-export async function fetchBackendManifestStatus(): Promise<BackendManifestStatus> {
-  const m = await fetchManifest()
-  return {
-    currentManifest: m,
-    latestManifest: m,                          // no separate "latest" — snapshot is pinned
-    currentManifestSignature: m.generated_at,
-    latestManifestSignature: m.generated_at,
-    updateAvailable: false,
-    refreshing: false,
-    pendingRefreshCount: 0,
-  }
-}
-```
-
----
-
-## What deletes
-
-After v2 is live, the following code is dead and can be removed in a
-follow-up cleanup:
-
-- `lib/duckdb-data.ts` — replaced by typed SELECTs split between
-  `lib/duckdb.ts` (connection) and `lib/data-backend.ts` (queries).
-- The `payload_json` parser helpers (`parsePayload`, `readPayloads`,
-  `readPayloadById`, `assertDeveloperListShape`) — no JSON blobs to
-  parse.
-- The `aggregate__` / `matrix__` eval-id prefix dispatch in
-  `getEvalSummaryByIdFromDuckDB` — the typed view is the only path.
-- `lib/model-data.ts` — most of its functions exist to convert HF
-  JSON records into `BenchmarkEvaluation` / `EvaluationCardData`. Once
-  the producer emits those shapes directly, the adapter logic deletes.
-  Keep only the helpers that don't touch HF records (slug parsing,
-  display formatters).
-- `lib/eval-processing.ts` — the `groupEvaluationsByModel`,
-  `createModelSummary`, `createBenchmarkEvalSummary`, and
-  `inferCategoryFromBenchmark` adapter functions are no longer called
-  in the data path. The exported types stay.
-- `scripts/audit-adapters.mjs`, `scripts/dump-adapter-outputs.mts`,
-  `scripts/compare-data-backends.mjs`, `scripts/refresh-fixtures.mjs`,
-  `scripts/cache-hf-data.mjs` — adapter / parity-check tooling for the
-  legacy pipeline. Delete once v1 is retired.
-- `data/models/`, `data/developers/`, `data/benchmarks.json`,
-  `data/models.json`, `data/developers.json` — bundled snapshots of
-  v1 output for fixture tests. Replace with v2 fixtures if needed.
-- `LOCAL_PIPELINE_OUTPUT` env var, `duckdb/v1/` subpath conventions,
-  and the parity-emitter expectations documented in
-  `lib/duckdb-data.ts`'s preamble.
-- `inferCategoryFromBenchmark` regex chain in
-  `lib/benchmark-schema.ts` — producer is the source of truth for
-  category. Keep the `EVALUATION_CATEGORIES` const + `CategoryType`
-  type; delete the inference function and `BENCHMARK_PRIORITY_RULES`.
-
----
-
-## Slug rule
-
-Producer emits all URL-bearing identifiers in
-RFC 3986 percent-encoded form (`route_id`, `evaluation_id`,
-`metric_summary_id`). Frontend treats them as opaque except for
-`<Link>` href construction:
-
-```tsx
-// Old: href={`/models/${model.route_id}`}  // route_id was already escaped via __ rule
-// New: href={`/models/${model.route_id}`}  // same code; route_id is now percent-encoded
-```
-
-Decode happens inside the route handler when looking up by slug:
-
-```ts
-// app/models/[id]/page.tsx
-export default async function ModelDetailPage({ params }: { params: { id: string } }) {
-  const summary = await getModelSummaryById(params.id)  // pass encoded form straight through
-  ...
-}
-```
-
-`getModelSummaryById` looks up by `route_id = ?` directly without
-decoding — the producer's `route_id` column matches the URL path
-segment byte-for-byte. The legacy `replace('/', '__')` and
-`replace(/\//g, ...)` helpers in `lib/utils.ts` and `lib/model-family.ts`
-become dead code; remove them in the cleanup pass.
-
----
-
-## Migration strategy
-
-A feature flag gates v1 vs v2 during the transition:
-
-```ts
-// lib/data-backend.ts
-const BACKEND_VERSION = process.env.DATA_BACKEND ?? "v1"
-
-export const getModelCards =
-  BACKEND_VERSION === "v2"
-    ? (await import("@/lib/duckdb")).getModelCards
-    : (await import("@/lib/duckdb-data")).getModelCardsFromDuckDB
-// ... same pattern for other accessors
-```
-
-Phase plan:
-
-1. **Producer ships Stage J.** `eval_card_backend` emits the six
-   v2 artifacts in `warehouse/<snapshot_id>/`. Existing canonical
-   parquets stay alongside.
-2. **Frontend lands `lib/duckdb.ts` + `lib/sidecars.ts`** behind the
-   `DATA_BACKEND=v2` flag. CI builds both backends; default stays v1.
-3. **Smoke test in dev with `DATA_BACKEND=v2`,
-   `SNAPSHOT_URL=file://...`.** Verify each page renders identical
-   bytes (modulo source-of-data labels). Where they diverge, file
-   producer issues — do not patch the frontend to paper over.
-4. **Flip the production default to v2.** Keep v1 path compilable but
-   unreachable. Monitor for a release.
-5. **Delete v1 path** (the "What deletes" list above).
-
-The flag is intentionally process-wide, not per-accessor. Mixing
-backends within one render produces inconsistent snapshots.
-
----
-
-## What doesn't move
-
-- **Instance-level data fetching** (`fetchInstanceLevelData` in
-  `lib/hf-data.ts`). Instance JSONL is referenced by URL in
-  `eval_results_view.instance_file_path`; the lazy-load stays. Pointer
-  shape on the row is unchanged from v1.
-- **Benchmark card metadata** lives inside `evals_view.benchmark_card`
-  STRUCT now, not a separate `benchmark_card_*.json` per file. The
-  page reads it from the eval row directly. Adapter-style readers
-  (`fetchBenchmarkMetadataMap`) become a `SELECT benchmark_id, benchmark_card
-  FROM evals_view` aggregation if anything still calls them — most
-  callers should fold into `getEvalSummaryById`.
-- **EvalCards annotations** (`evalcards.annotations`) live on
-  `eval_results_view.evalcards_annotations` per-row. The eval-detail
-  page reads them inline; no separate fetcher.
-
----
-
-## Open questions / risks
-
-- **httpfs cold-start latency.** First query against an HF-hosted
-  parquet pays a round trip per file. Mitigate by pre-registering all
-  three views at process start (above), so the first user query hits
-  warm metadata. Measure on the production HF Space; if too slow,
-  consider downloading the snapshot to local disk at container start
-  (~MB per snapshot).
-- **Connection lifetime in serverless.** Vercel's serverless
-  runtime tears down the Node process per request; the
-  `connectionPromise` cache doesn't help. The HF Space deployment
-  (Docker, long-lived) is unaffected. If we ever target serverless,
-  switch to `duckdb-wasm` in the browser or a separate serving
-  process.
-- **`aggregate_components[]` on `eval_results_view`.** This array is
-  the per-suite-component breakdown for rollup rows. For non-rollup
-  rows it's always empty. If suite rollups grow common, the storage
-  cost of trailing-empty arrays is non-trivial; consider splitting
-  into a dedicated parquet at that point.
-- **Category drift.** Producer's `category_mapping.json` will lag real
-  benchmark tag changes. The mapping is producer-owned, so the
-  frontend can't patch around drift — this is a feature, not a bug,
-  but it requires operator discipline. Surface "uncategorised
-  benchmark count" in the producer's run summary and the home-page
-  manifest banner.
-- **Type widening for `score_summary` etc.** The producer emits these
-  as DuckDB STRUCTs; the TS interface declares them as nested
-  `{ count, min, max, average }`. `runAndReadAll` returns nested
-  STRUCTs as plain JS objects, so the cast works — but if duckdb-node
-  changes its STRUCT serialisation, audit the `as` casts here. Add a
-  dev-only validator that runs `EvaluationCardData`'s shape check at
-  the row level on the first `getModelCards()` call after process
-  start.
-
-## model-resolution-rework — view-layer contract additions
-
-This is the producer↔frontend column contract for the model-resolution
-rework (proposal §10.5, contract surface 3). The frontend interface field
-names below MUST match the producer `models_view` / `eval_results_view`
-column names exactly. All fields are **additive and nullable** — the
-frontend conditionally renders them and falls back gracefully when null,
-so producer and frontend can roll out independently.
-
-### New columns the frontend reads
-
-| view column | frontend field | interface | meaning |
-| --- | --- | --- | --- |
-| `inference_platform` | `inference_platform` | `ModelInfo`, `EvaluationCardData` | FK to `inference_platforms.id`; "served by" provenance. Already present pre-rework; now populated by the resolver output. |
-| `model_family_id` | `model_family_id` | `EvaluationCardData`, `ModelEvaluationSummary`, `ModelResultForBenchmark`, `BenchmarkLeaderboardRow` | family canonical id (grouping root). Replaces the deleted client-side family computation (`lib/model-family.ts`). Routing fallback when `model_route_id` is absent. |
-| `lineage_origin_model_id` | `lineage_origin_model_id` | `EvaluationCardData`, `ModelSummaryCore` | deepest non-variant ancestor (base model for finetunes/merges/quants). Surfaced as "Base model" on the model detail page. |
-| `resolution_source` | `resolution_source` | `EvaluationCardData`, `ModelSummaryCore` | enum `hf \| models_dev \| curated \| inferred \| none`. Surfaced as "Resolved via". |
-| `resolution_granularity` | `resolution_granularity` | `EvaluationCardData`, `ModelSummaryCore` | enum `variant \| group \| family`. Surfaced as "Granularity". |
-
-These read off `SELECT *`-backed accessors (`getModelSummaryById`) and the
-per-cell reshape, so they flow through automatically once the producer view
-emits them. The explicit `MODEL_CARD_COLUMNS` projection in
-`lib/view-data.ts` is intentionally NOT extended with the three pure-display
-fields until the producer view layer ships them (an explicit SELECT of a
-non-existent column errors). Add them to that projection in lockstep with the
-producer change.
-
-### Renames the frontend does NOT read (cosmetic)
-
-`root_model_id → model_group_id` and `lineage_origin_org_id →
-lineage_origin_model_org_id` are renamed in the registry/producer data layer.
-The frontend view-layer interfaces never read either name (verified by grep;
-the only `root_model_id` reference is in the redirect *generator*, which reads
-the registry `baseline_resolution.json`, not the view layer). No frontend
-change is required for these renames.
-
-### `canonical_id` flip + URL redirects (breaking, data)
-
-Post-flip, `canonical_id` is the **leaf**; ~297 model URLs change (204 flips +
-93 casing re-keys). Old bookmarked URLs are 301-redirected by `middleware.ts`
-using the generated `lib/model-url-redirects.ts` map. The map is regenerated
-from the registry `baseline_resolution.json` via
-`scripts/generate-model-redirects.ts`. **The committed map is PROVISIONAL** —
-built from the pre-final spec-dir baseline — and MUST be regenerated from the
-post-M9 baseline at integration. The redirect preserves query params and the
-ids round-trip through percent-encoding (RFC 3986, `/ → %2F`), matching the
-producer `route_id` form.

notes/merge-cheatsheet-backend-v2.md

@@ -1,193 +0,0 @@
-# Merge cheatsheet: pulling `main` into `feat/use-new-backend-data`
-
-> Drafted 2026-05-04, before pulling. Companion to `backend-v2-migration.md`
-> (which is the design doc). This file is just a per-file conflict guide.
->
-> Branch: `feat/use-new-backend-data` (2 commits ahead of `main`:
-> `7635aee` Integrate with test backend data, `bfce8f2` Drop
-> input/output_modalities from MODEL_CARD_COLUMNS).
-
-## Triage at a glance
-
-| File | Risk | Strategy |
-|---|---|---|
-| `lib/data-backend.ts` | **High** | Keep ours wholesale; re-port any new accessors main added |
-| `lib/backend-artifacts.ts` | **High** | Keep our schema renames; reconcile any *new* main-side fields against producer output |
-| `components/signals/corpus-dashboard.tsx` | **Med** | Keep main's UI structure; rewire data fields to v2 names |
-| `components/signals/corpus-signals-strip.tsx` | **Med** | Same as above |
-| `lib/hf-data.ts` | **Med** | Keep `useViewLayerBackend()` short-circuits at top of 5 fetchers |
-| `Dockerfile` | **Med** | Keep our `DATA_BACKEND=v2` + `SNAPSHOT_URL` wiring; layer main's other changes on top |
-| `lib/benchmark-schema.ts` | **Low** | Trivial 1-line addition (`num_few_shot?`) |
-| `app/page.tsx` | **Low** | One-line copy change (`corpus-aggregates.json` → `headline.json`) |
-
-New files (no conflict possible): `lib/view-data.ts`, `lib/duckdb.ts`,
-`lib/sidecars.ts`, `tests/view-data.test.ts`,
-`notes/backend-v2-migration.md`.
-
----
-
-## `lib/data-backend.ts` — High
-
-**What we did:** Replaced static re-exports from `lib/duckdb-data` with
-a `BACKEND_VERSION` env-flag dispatcher. Each accessor now branches on
-`useViewLayerBackend()` (true when `DATA_BACKEND=v2` or `stage-j`) and
-lazy-imports either `@/lib/view-data` or `@/lib/duckdb-data`.
-Manifest/hierarchy accessors branch between `@/lib/sidecars` and
-`@/lib/hf-data`.
-
-**Reconcile:**
-- Conflict almost certain if main touched any export wiring here.
-- **Keep our file as-is.** The dispatcher pattern is load-bearing.
-- If main added a new accessor (e.g. `getFooBar`), add a new dispatcher
-  function following the same pattern — only the legacy branch needs
-  to be wired immediately; v2 branch can throw `Not implemented` until
-  `lib/view-data.ts` adds it.
-
----
-
-## `lib/backend-artifacts.ts` — High
-
-**What we did:** Renamed corpus-block fields to match what the v2
-producer emits:
-
-| Block | v1 (main) | v2 (ours) |
-|---|---|---|
-| Completeness | `total_benchmarks`, `completeness_score_mean`, `completeness_score_median`, `per_field_population{}` | `total_triples`, `completeness_avg`, `completeness_min`, `completeness_max` |
-| Provenance | `multi_source_groups`, `multi_source_rate`, `first_party_only_groups`, `first_party_only_rate`, `total_groups` | `multi_source_triples`, `first_party_only_triples`, `total_triples` (rates dropped — derived in components via local `rate()` helper) |
-| Comparability | `variant_eligible_groups`, `variant_divergent_groups`, `variant_divergence_rate`, `cross_party_eligible_groups`, `cross_party_divergent_groups`, `cross_party_divergence_rate`, `total_groups` | `total_triples`, `variant_divergent_count`, `cross_party_divergent_count`, `groups_with_variant_check`, `groups_with_cross_party_check` |
-
-Also added: `DeveloperListEntry` interface, optional
-`developers/families/categories` arrays on `CorpusAggregates`,
-optional `eval_hierarchy` key in `BackendManifest.summary_artifacts`.
-
-**Reconcile:**
-- Producer is the source of truth for v2 field names — do **not** add
-  back v1 names to satisfy a main-side change. If main added a field
-  the v2 producer doesn't emit, either drop it or check
-  `eval_card_backend/notes/08-frontend-view-layer.md` first.
-- Keep all three new optional sections on `CorpusAggregates`
-  (developers, families, categories) — they back the new
-  developer-list path.
-- The `summary_artifacts.eval_hierarchy` key is additive; safe to keep
-  alongside whatever main added there.
-
----
-
-## `components/signals/corpus-dashboard.tsx` — Medium
-
-**What we did:** Mechanical rewrite of every field reference in this
-file to use the v2 names from `lib/backend-artifacts.ts` (above).
-Removed the `per_field_population` per-field grid and replaced it with
-a `min / avg / max` MiniMetric trio. Added a local `rate(num, denom)`
-helper (returns null if either side is null/zero) since v2 stores
-counts, not pre-computed rates. Title-cased `CATEGORY_ORDER`
-(`"Agentic"`, `"General"`, …) and made the keys-to-render set extend
-gracefully to unknown categories.
-
-**Reconcile:**
-- If main touched this file for design/UX reasons, **prefer main's
-  visual structure** — but keep our field accessors. The recipe is:
-  - Anywhere main reads `multi_source_rate`, replace with `rate(prov.multi_source_triples, prov.total_triples)`.
-  - Anywhere main reads `completeness_score_mean`, replace with `comp.completeness_avg`.
-  - Anywhere main reads `*_eligible_groups` / `*_divergent_groups`, swap to `groups_with_*_check` / `*_divergent_count`.
-  - Drop any new code that reads `per_field_population` — gone in v2.
-- Keep the local `rate()` helper at the bottom of the file.
-- Category lookup must use the new title-cased keys (or stay tolerant
-  via the `available` set logic we added).
-
----
-
-## `components/signals/corpus-signals-strip.tsx` — Medium
-
-**What we did:** Same field renames as above, same local `rate()`
-helper added. Headline copy updated from "groups" → "triples" where
-the underlying unit changed.
-
-**Reconcile:** Apply the same recipe as `corpus-dashboard.tsx`. The
-two files share field names and the `rate()` helper.
-
----
-
-## `lib/hf-data.ts` — Medium
-
-**What we did:** Added an early-return guard at the top of five
-functions:
-- `fetchBackendManifestStatus` — synthesizes a status from the v2 manifest sidecar
-- `fetchBenchmarkMetadataMap` — delegates to `view-data.getBenchmarkMetadataMap`
-- `fetchBackendManifest` — delegates to `sidecars.fetchManifest`
-- `fetchEvalHierarchy` — delegates to `sidecars.fetchHierarchy` (still wraps in `adaptEvalHierarchy`)
-- `fetchCorpusAggregates` — delegates to `sidecars.fetchHeadline`
-
-Plus a module-level `useViewLayerBackend()` helper and a lazy
-`fetchSnapshotSidecars()` importer near the top of the file.
-
-**Reconcile:**
-- These are all additive guards at the start of existing functions —
-  conflicts are likely only if main re-shaped the same function
-  bodies.
-- Pattern: `if (useViewLayerBackend()) { return <v2 path> }` then fall
-  through to the existing v1 implementation untouched.
-- If main renamed one of these functions, port the guard into the
-  renamed version. Don't drop the guard.
-
----
-
-## `Dockerfile` — Medium
-
-**What we did:**
-- Default `ARG DATA_BACKEND` flipped from `duckdb` → `v2` in **both**
-  stages (builder and runner).
-- Added `ARG SNAPSHOT_URL` + `ENV SNAPSHOT_URL` in both stages,
-  defaulting to a pinned `evaleval/eval-cards-data` warehouse path.
-- Comment block rewritten to reflect v2 + legacy coexistence.
-- Kept legacy `LOCAL_PIPELINE_OUTPUT`, `HF_DATA_LOCAL_DIR`,
-  `HF_DATA_OFFLINE=1` envs intact (legacy backend still compilable).
-
-**Uncommitted tweak (working tree):** `SNAPSHOT_URL` default points at
-`j-chim/temp_evalcard_backend` instead of `evaleval/eval-cards-data` —
-this is the dev/test dataset for the temp HF Space deploy. Do **not**
-commit this override; revert before merging to main, or keep it only
-on local working copy.
-
-**Reconcile:**
-- Keep our `DATA_BACKEND=v2` default and `SNAPSHOT_URL` plumbing.
-- Layer main's non-data changes (base image bumps, `pnpm` version,
-  build commands) on top.
-
----
-
-## `lib/benchmark-schema.ts` — Low
-
-**What we did:** Added one optional field, `num_few_shot?: number`, on
-`GenerationConfig`. That's it.
-
-**Reconcile:** Trivially additive. Keep our line; merge tool should
-handle it cleanly unless main touched the same struct.
-
----
-
-## `app/page.tsx` — Low
-
-**What we did:** One-line copy change in the empty-state banner —
-`corpus-aggregates.json` → `headline.json` (the v2 sidecar name).
-
-**Reconcile:** Trivial. Keep ours.
-
----
-
-## Order of operations after `git pull`
-
-1. Resolve `lib/backend-artifacts.ts` first — it's the schema source
-   of truth that the components depend on.
-2. Resolve `lib/data-backend.ts` and `lib/hf-data.ts` — backend wiring.
-3. Resolve the two `components/signals/*` files using the rename recipe.
-4. Resolve `Dockerfile` — keep our v2 envs.
-5. `app/page.tsx` and `lib/benchmark-schema.ts` — should auto-merge or
-   be trivial.
-6. Run `pnpm tsc --noEmit` (or whatever the project's typecheck is) to
-   catch any v1 field references main introduced that didn't conflict
-   textually but break against our renamed types.
-7. Run `pnpm test` — `tests/view-data.test.ts` and
-   `tests/duckdb-data.test.ts` should both still pass.
-8. Smoke test with `DATA_BACKEND=v2 SNAPSHOT_URL=file://…` and again
-   without (legacy path) — both must render.

notes/migration-plan.md

@@ -1,117 +0,0 @@
-# TS→pipeline migration plan
-
-Drafted 2026-04-27. Companion to `notes/ts-to-pipeline-migration.md` (the original 20-item catalog) and `notes/testing-strategy.md` (the safety net we're building before doing more deletions).
-
-## What's already shipped
-
-See `notes/ts-to-pipeline-migration.md` "Status" section for full detail. Briefly:
-
-- **#4 source-metadata synthesis fallback** — deleted. Pipeline emits on every row; runtime `assertSourceMetadata` guard added at read sites; 86 183/86 183 production rows now show real first/third-party badges (was the explicit goal).
-- **#11 category — partial** — extended `PIPELINE_CATEGORY_MAP` with the 3 missing pipeline keys (all → "General"); removed the regex fallback from `mapHFCategories`. Reverted the more-aggressive call-site changes after subagent audit found 1 500+ Safety rows would have been silently General-ified. Left as TS code path until pipeline-side category accuracy improves (84% of evals currently emit `category: "other"`).
-
-## Framing: TS is the current source of truth
-
-Every TS transformation in `lib/` exists because the pipeline didn't do it (yet) — token canonicalization, variant grouping, source-metadata defaults, category inference, score normalization, etc. The migration is to lift each transformation upstream so the pipeline emits canonical data and every consumer reads instead of re-deriving.
-
-Two failure modes to avoid (per the Phase 3 category regression and the 2026-04-28 v/V finding):
-
-- **Treating a transformation as if it were a default.** TS rules that *normalize* (always overwrite) must be implemented as normalization in pipeline; rules that *fill in defaults* (only when value missing) must be defaults. Misclassifying causes silent data shifts. Each spec calls out which kind it is.
-- **Deleting TS before verifying pipeline matches across the full corpus.** `pnpm audit-adapters --diff` is the verification gate. No deletion ships until pipeline-side output is byte-identical (or differences are explicitly accepted in writing).
-
-## Data direction: cleaning upstream, reshape in SQL, presentation stays in TS
-
-Standing principle for this migration and beyond. Every TS transformation belongs in one of three places:
-
-- **Data cleaning / standardization → pipeline** (changes what the data *is*; canonical form every consumer would want). License strings, developer names, identity tokens, timestamp formats, category labels, source-metadata defaults. Pipeline emits canonical values; every consumer reads. The per-item workflow below is built for this class.
-- **Reshape / dedup / aggregate / sort / filter → DuckDB SQL** (derived view over universal data; every consumer would compute the same thing). Variant dedup, "freshest among candidates", per-category counts, hierarchy flattening, composite rollups. Two valid landing spots: materialized into the pipeline's parquet output (read by DuckDB without further computation) or expressed as SQL at query time. Favor materialization when the answer is identical for every consumer; favor query-time SQL when consumers slice differently.
-- **UI / presentation policy → stays in TS** (app-specific curation choices; different consumers would have different opinions). Card layout, color choices, sort orders for category displays, icon assignments, which fields to surface vs hide. These encode product judgments specific to this app — pipeline-emitting them would inflict this UI's choices on every other consumer of the dataset. Keep in TS, treat as out of scope for the migration.
-
-The test for which category an item belongs in: *would every reasonable consumer want the same answer?* If yes, it's cleaning or reshape (depending on whether it's a value transform or a derived view). If no, it's UI policy and stays.
-
-A trap to watch for: an item can *look* like UI policy because the TS code encodes opinion (e.g. a regex map that ranks things), but actually the curation already lives in the pipeline and the TS code is dead/passthrough. **Always trace which code path actually runs in production before classifying — grep for the function name across `app/`/`components/`/`scripts/`, then chain through caller graphs.** "TS file exists" doesn't mean "TS is doing the work."
-
-The current DuckDB backend is mid-migration scaffolding. The parquet schema (see `pipeline.py write_experimental_parquet_table`) has 11 typed metadata columns (`record_type`, `model_route_id`, `model_family_id`, `eval_summary_id`, `developer_route_id`, `developer`, `category`, `benchmark_family_key`, `models_count`, `total_evaluations`, `last_updated`) plus a `payload_json VARCHAR` column. DuckDB queries today use the metadata columns for routing (`WHERE eval_summary_id = ?`) and always select `payload_json` for the substantive data — variants, model_results, scores, retrieved_timestamps are all nested inside the blob. SQL can route, but it can't reshape what it can't see.
-
-Two open design questions for the reshape class, both legitimate, neither decided:
-
-- **How relational does parquet need to go?** Promoting nested fields to columns (e.g. one row per `(eval_summary_id, variant_key, retrieved_timestamp, source_metadata)` for the variant dedup case) lets SQL do the work directly: `SELECT … QUALIFY ROW_NUMBER() OVER (PARTITION BY variant_key ORDER BY retrieved_timestamp DESC) = 1`. But it's a substantial schema change negotiated with the pipeline owner.
-- **Materialize the answer upstream vs. compute at query time?** Materialize when the answer is identical for every consumer (variant dedup, per-category counts). Compute at query time when consumers slice differently (filtered top-N, user-selected category aggregates). The split is per-case judgment.
-
-What the principle rules out: keeping reshape work in TS adapters after data cleaning moves upstream. "Spec done, pipeline matches, TS deleted" is only complete if the work that's left is presentation, not computation. When specing each item, classify it: cleaning (per-item workflow) or reshape (queue for the parquet-schema / SQL design conversation).
-
-## Per-item workflow
-
-For each remaining item, the work is the same shape:
-
-1. **Spec the transformation** — write `notes/transformations/NN-<name>.md` per the template in `notes/transformations/README.md`. Capture every rule branch. Two required classifications: (a) default-vs-normalization (see Framing section above), (b) cleaning-vs-reshape (see Data direction section above). If it's reshape, capture the *operation* and flag for the SQL design conversation rather than writing a line-by-line TS translation. Document detected divergences against pipeline.
-2. **Test the transformation** — write `tests/transformations/<name>.test.ts` with parameterized tests sourced from the spec table. These double as the executable acceptance criterion for pipeline.
-3. **Hand to pipeline** — file the spec + tests with the pipeline owner (an issue/PR in `eval_cards_backend_pipeline`). Reference the unit tests as the contract.
-4. **Verify cross-corpus match** — once pipeline ships, run the verification script (`scripts/verify-identity.mjs` for #1, similar scripts per item) against the full live cache. Zero divergences before proceeding.
-5. **Pre-process in this repo (optional, if needed)** — only when the gap between TS and pipeline is intolerable to wait through. Not the default. A one-shot Python pre-processor at data ingestion time is acceptable; on-the-fly TS computation is what we're trying to leave behind.
-6. **Delete TS** — remove the implementation; update callers to read pipeline fields directly. This is its own task in the tasklist (e.g. #5b for #1), gated on step 4.
-
-## The 18 remaining items
-
-| Item | Transformation (TS location) | Status | Spec |
-|---|---|---|---|
-| #1 | Identity canonicalization (`lib/model-family.ts`) | spec written 2026-04-28; awaiting pipeline | [01-identity-canonicalization.md](transformations/01-identity-canonicalization.md) |
-| #2 | Setup-alias merging (`lib/eval-processing.ts:371-434`) | not yet specced | — |
-| #3 | Hierarchy flatten + family summary (`lib/hf-data.ts flattenModelEvaluations`, `lib/eval-processing.ts createModelFamilySummary`) | not yet specced; structural decision pending (consumer rewrite vs pipeline-emit-flat-list) | — |
-| #5 | Composite eval rollup (`lib/model-data.ts:874-1041 aggregateBenchmarkSummaries`, ~170 lines) | not yet specced | — |
-| #6 | Matrix leaderboard synthesis (`lib/model-data.ts:1043-1214`, ~170 lines) | not yet specced | — |
-| #7 | Per-instance JSONL normalization (`lib/hf-data.ts:919-1029`, ~110 lines of heuristics) | not yet specced; biggest brittleness | — |
-| #8 | Benchmark display names (`BENCHMARK_NAMES` in `lib/model-data.ts:92`, `SUITE_DISPLAY_NAMES`, etc.) | not yet specced | — |
-| #9 | Developer name canonicalization (`KNOWN_DEVELOPER_NAMES` in `lib/model-data.ts:201-228`) | not yet specced | — |
-| #10 | Metric display-name expansion (`GENERIC_EVALUATION_NAMES` + `prefersBenchmarkName` heuristic) | not yet specced | — |
-| #11 | Category inference (`inferCategoryFromBenchmark` regex in `lib/benchmark-schema.ts:182-206`) | partially handled; pipeline category is too noisy ("other" 84% of evals) — TS regex is the more accurate spec until pipeline improves | — |
-| #12 | Params parsing (`parseParamsBillions` in `lib/model-data.ts:296-338` + dups) | not yet specced | — |
-| #13 | Timestamp normalization (`toComparableTimestamp` + ~5 dups) | not yet specced; small | — |
-| #14 | Score summary stats (`groupEvaluationsByBenchmark` finalisation) | not yet specced | — |
-| #16 | Per-category benchmark counts (`hfModelCardToEvaluationCardData` proportional split) | not yet specced; today TS uses `Math.floor(total / categories.length)` as a fake distribution | — |
-| #17 | Benchmark-card attachment (`attachBenchmarkCardToSummary` + 3-candidate retry) | not yet specced | — |
-| #18 | License canonicalization (`LICENSE_COLORS`/`shortenLicense` in `components/eval-card.tsx:22-48`) | not yet specced | — |
-| #19 | Slug candidate generation (`getModelDetailSlugCandidates`/`getDeveloperSlugCandidates`) | not yet specced; the 6-spelling retry in `getModelSummaryById` is the symptom | — |
-| #20 | Dataset URL synthesis (`components/eval-card.tsx:81-89`) | not yet specced; tiny | — |
-
-The pipeline lives at `/Users/jchim/projects/eval_cards_backend_pipeline`. See its `AGENTS.md` for Python conventions, run instructions, and the `EXPORT_EXPERIMENTAL_PARQUET=1` flag. Pipeline changes are full-rebuild — `output/` is wiped and rewritten each run.
-
-## Recommended order
-
-1. **Testing harness** — done 2026-04-27 (Tier A/B/C + fixtures + audit script).
-2. **#1 identity canonicalization** — spec written 2026-04-28, awaiting pipeline implementation.
-3. **#2 setup-alias merging** — same shape as #1; spec next.
-4. **Small wins #16, #18, #19, #20** — each is a contained transformation. Spec, hand off, batch them on the pipeline side.
-5. **#3 hierarchy flatten** — structural decision needed first (consumer-rewrite vs pipeline-emit-flat-list). Bigger work.
-6. **#5, #6, #14** — composites + matrix + summary stats. Bigger pipeline work.
-7. **#11 category** — pipeline must improve `category` accuracy (84% currently emit `other`); until then TS regex is the spec.
-8. **#7 per-instance JSONL** — biggest brittleness, biggest payoff. Defer until others are done.
-
-Items #8, #9, #10, #12, #13, #17 fold into the natural sweep around #3 or #14 — they share consumers.
-
-## What can be parallelized right now
-
-If multiple agents/sessions run in parallel:
-
-- **Agent 1 (this repo):** build `tests/fixtures/` + `tests/pipeline-contract.test.ts` (Tier A from testing-strategy.md)
-- **Agent 2 (this repo):** build `scripts/audit-adapters.mjs` (Tier C from testing-strategy.md)
-- **Agent 3 (pipeline repo):** sweep through #16, #18, #19, #20 — small Python emissions
-
-Tier B (snapshot tests) needs the fixture set first, so it serializes after Agent 1.
-
-After tests are in: TS deletions #1 and #2 can each be a parallel agent.
-
-## Cross-repo coordination
-
-When pipeline-side work happens, it changes the contract our TS depends on. Sequence:
-
-1. Pipeline emits new field, runs `EXPORT_EXPERIMENTAL_PARQUET=1` locally to materialize.
-2. Pipeline ships; user re-publishes to HF.
-3. Our repo: `pnpm cache-hf-data` to refresh local cache.
-4. Our repo: `pnpm refresh-fixtures` to pin new shape.
-5. Our repo: `pnpm test` to confirm contracts pass.
-6. Our repo: do the TS deletion.
-7. Our repo: `pnpm test` again — snapshots will diff if behavior changes.
-
-For the local development loop, both repos can be exercised against `eval_cards_backend_pipeline/output/` directly using the `HF_DATA_LOCAL_DIR` + `HF_DATA_OFFLINE` env vars from `notes/ts-to-pipeline-migration.md`. No HF round-trip needed for testing.
-
-For the full picture of how upstream changes propagate (drift detection, scenario matrix, cross-repo workflow), see `notes/testing-strategy.md` § "How upstream changes propagate" and § "Workflows".

notes/testing-strategy.md

@@ -1,353 +0,0 @@
-# Testing strategy for the TS→pipeline migration
-
-Drafted 2026-04-27. The motivation is the 2026-04-27 review session: subagent audits caught two regressions the parity harness missed (a 22% category change and a `coding: "Reasoning"` mistake based on a substring fallacy). Both required full-production-cache analysis to surface. Subagent audits are not a sustainable workflow.
-
-## Design principle: separate code drift from upstream drift
-
-Upstream data (the published `evaleval/card_backend` HF dataset) is our best guess at a source of truth, but it isn't immutable. Pipeline-side relabeling, registry updates, and schema changes happen. If our regression tests run against live data, every upstream update lights up the test suite and we can't tell "I broke something" from "upstream changed something I happen to consume."
-
-The fix: **pin tests to a committed snapshot of upstream data**. Refresh the snapshot deliberately (script + commit), and the snapshot diff + test diff arrive together for review. Live-data drift detection is a separate, opt-in concern.
-
-```
-                 ┌── tests run against ──┐
-   live cache ───┤                       ├──→ pinned fixtures ──→ tests
-                 └── refresh script ─────┘    (committed)
-                     (manual, reviewed)
-```
-
-Live cache drift is checked by an opt-in audit, not by the test suite.
-
-## The three tiers
-
-### Tier A — Pipeline contract tests
-**Catches:** "pipeline upstream silently dropped a field we depend on." Three repeated manual checks (`source_metadata`, `category`, hierarchy keys) motivated automating this.
-
-**Mechanic:** vitest file that walks every fixture file and asserts presence/shape of fields the TS code depends on. Each contract is a field-level invariant.
-
-**File:** `tests/pipeline-contract.test.ts`
-
-**Initial contract set** (every one corresponds to a real failure mode):
-- `every model_result has source_metadata` (we deleted the synthesis fallback assuming this)
-- `every model_result.source_metadata has evaluator_relationship in {first_party, third_party, other}`
-- `every eval-detail has category as a non-empty string`
-- `every eval-detail has eval_summary_id, benchmark, benchmark_leaf_name`
-- `every model card has model_family_id matching pipelineSlugify(model_family_id)`
-- `every hierarchy_by_category key is one of the 9 known pipeline categories`
-- `every BenchmarkEvaluation produced by flattenModelEvaluations has source_metadata` (cross-check: contract + adapter together)
-- `every model card has total_evaluations as a number`
-- `every model_result.retrieved_timestamp parses as a valid Date`
-
-**Exit criteria:** all contracts pass against pinned fixtures. Each contract should fail loudly with the offending file path + key path when violated.
-
-**Acceptance:** runs in `pnpm test`. Takes <2s. Adding a new contract is 5 lines.
-
-### Tier B — Adapter snapshot tests
-**Catches:** "I changed TS code and didn't realize it changes the output for some input shape." This is the bulk of regression-detection.
-
-**Mechanic:** vitest snapshot tests. Each adapter × each fixture → snapshot. Regenerate via `vitest --update-snapshots` when changes are intentional; review the snapshot diff alongside the code diff.
-
-**Files:**
-- `tests/adapters/hf-eval-detail-to-summary.test.ts`
-- `tests/adapters/hf-model-card-to-evaluation-card-data.test.ts`
-- `tests/adapters/flatten-model-evaluations.test.ts`
-- `tests/adapters/hf-developer-detail-to-summary.test.ts`
-- `tests/adapters/hf-eval-entry-to-list-item.test.ts`
-- `tests/adapters/build-benchmark-leaderboard-matrix.test.ts`
-- `tests/adapters/build-single-metric-suite-matrix-summary.test.ts`
-- `tests/adapters/aggregate-benchmark-summaries.test.ts`
-
-**Snapshot format:** `tests/__snapshots__/<test>.snap` (vitest default). Commit them.
-
-**Acceptance:** `pnpm test` runs all snapshots, reports any diff, exit non-zero on diff. Adding a new fixture is one line of `test.each`.
-
-### Tier C — Full-cache differential audit
-**Catches:** "what is the *full* impact of my code change across all 5 830 production models?" Used for big migration items where snapshot fixtures can't enumerate every shape.
-
-**Mechanic:** a Node script that runs all adapters against either pinned fixtures or the live cache, produces a deterministic JSON digest (per-output hash + value distributions + invariant violation counts), and supports diff mode.
-
-**File:** `scripts/audit-adapters.mjs`
-
-**Output digest shape:**
-```json
-{
-  "version": 1,
-  "source": ".cache/hf-data",
-  "generated_at": "2026-04-27T22:00:00Z",
-  "adapters": {
-    "hfModelCardToEvaluationCardData": {
-      "outputs_count": 5830,
-      "outputs_hash": "sha256:...",  // hash of all outputs concatenated
-      "field_distributions": {
-        "developer": { "OpenAI": 12, "Anthropic": 8, ... },
-        "categories.length": { "1": 100, "2": 2000, "3": 3000, ... },
-        "evaluator_count": { "0": 200, "1": 1500, ... }
-      }
-    },
-    "flattenModelEvaluations": {
-      "outputs_count": 86183,
-      "outputs_hash": "sha256:...",
-      "invariant_violations": []
-    }
-  }
-}
-```
-
-**Modes:**
-- `node scripts/audit-adapters.mjs --output baseline.json` → write digest
-- `node scripts/audit-adapters.mjs --output candidate.json` → write digest after change
-- `node scripts/audit-adapters.mjs --diff baseline.json candidate.json` → human-readable diff
-- `node scripts/audit-adapters.mjs --against tests/fixtures` → use pinned set instead of live cache
-- `node scripts/audit-adapters.mjs --live --against .cache/hf-data` → drift check against live data
-
-**Acceptance:** runs in <30s against full live cache. Diff mode highlights field-distribution shifts, output-hash changes, and new invariant violations with sample paths.
-
-## Fixture management
-
-### Source
-
-Fixtures are pinned copies of files in `.cache/hf-data/` at a moment in time. They are committed JSON. Reviewers can see them in PR diffs.
-
-### Layout
-
-```
-tests/fixtures/
-  manifest.json        ← list of fixture IDs + source-cache snapshot ts
-  evals/
-    helm_classic_truthfulqa.json
-    helm_safety.json
-    apex_v1.json                        ← first-party (Mercor)
-    artificial_analysis_*_aime.json     ← third-party (AA)
-    helm_capabilities.json              ← composite
-    helm_lite_narrativeqa.json          ← subtask
-    rewardbench2_chat.json              ← coding key in hierarchy
-    ...
-  models/
-    openai__gpt-5.json                  ← multiple variants
-    anthropic__claude-opus-4-5.json     ← typical
-    google__gemini-3-flash.json         ← already in the parity test
-    ...
-  developers/
-    openai.json
-    anthropic.json
-    ...
-```
-
-### Curation criteria
-
-Every fixture earns its place by exercising a specific code path. Avoid random sampling.
-
-Required edge cases:
-- A model with multiple variants (`openai__gpt-5`)
-- A model with subtask hierarchy (helm_lite, helm_classic)
-- A first-party eval (Mercor ACE/APEX)
-- A third-party eval (Artificial Analysis)
-- A composite eval (helm_capabilities)
-- A matrix eval id pattern (synthetic, but the adapter handles it)
-- An eval with `category: "other"` (most of the corpus)
-- An eval that the regex `inferCategoryFromBenchmark` and the pipeline category disagree on (truthfulqa, helm_safety)
-- A model with setup-alias merging (multiple "prompt"/"fc" variants of same release)
-- An ABC-only benchmark (if any are exposed in eval-list)
-- An aggregate eval URL pattern (`aggregate__<suite>`)
-
-Aim for ~25-35 fixtures total. Small enough to review, broad enough to catch the patterns we know about.
-
-### Refresh workflow
-
-```bash
-pnpm refresh-fixtures          # copies tests/fixtures/manifest.json IDs
-                               # from .cache/hf-data/ into tests/fixtures/
-                               # bumps manifest.json snapshot_ts
-git diff tests/fixtures/       # review what upstream changed
-pnpm test                      # snapshot tests will probably diff
-pnpm test -- -u                # update snapshots if intentional
-git diff tests/__snapshots__/  # review what adapter outputs changed
-git add ...                    # commit fixtures + snapshots together
-```
-
-The diff in `tests/fixtures/` shows raw upstream changes. The diff in `tests/__snapshots__/` shows what changes when you feed the new data through the adapters. Both belong in the same commit.
-
-### Refresh cadence
-
-Manual, on demand. Recommended triggers:
-- Before starting a new migration item (to work against current upstream)
-- After observing a discrepancy between live cache and pinned fixtures
-- Periodically (~monthly) to keep fixtures from drifting
-
-There is no auto-refresh. The whole point is that upstream changes are reviewed.
-
-### Live-data drift detection
-
-Separate from regression tests. A vitest file `tests/upstream-drift.test.ts` runs Tier-A contracts against the LIVE cache and reports violations. Run it manually (`pnpm test:drift`); not part of `pnpm test`. If contracts fail there but pass on fixtures, upstream has drifted and someone should refresh fixtures + investigate.
-
-## How upstream changes propagate
-
-Three independent data layers, each updated by a different command:
-
-```
-huggingface.co/datasets/evaleval/card_backend       ← truth (changes when pipeline publishes)
-                       │  pnpm cache-hf-data        ← user-triggered download
-                       ▼
-.cache/hf-data/                                      ← live local cache (mutable)
-                       │  pnpm refresh-fixtures     ← user-triggered re-pin
-                       ▼
-tests/fixtures/                                      ← committed pinned snapshots
-                       │  pnpm test (adapter outputs)
-                       ▼
-tests/__snapshots__/                                 ← committed expected outputs
-```
-
-Default `pnpm test` only sees the pinned bottom two layers, so upstream churn never flaps the regression suite by accident. Each upstream change is observed *deliberately* by re-pinning and reviewing the diff.
-
-### Scenario matrix — what each layer reports
-
-| What changed upstream | `pnpm test` | `pnpm test:drift` (live cache contracts) | `pnpm refresh-fixtures && pnpm test` (snapshot diff) | `pnpm audit-adapters --diff baseline.json candidate.json` |
-|---|---|---|---|---|
-| Pure data refresh, no shape change | ✅ | ✅ | ❌ snapshots diff (timestamps, scores) | hash flips for affected adapters |
-| Additive (new field that no adapter consumes) | ✅ | ✅ | ✅ (raw fixture diff visible, snapshots stable) | distributions stable |
-| New enum value (e.g. `evaluator_relationship: "fourth_party"`) | ✅ | ❌ unknown-value contract | ✅ unless consumed | distribution gains a key |
-| Drops a required field (e.g. `source_metadata`) | ✅ | ❌ contract violation with N/M count | ❌ contracts now fail on pinned data too | `throws` count rises |
-| Reclassifies an existing value (e.g. `category: "other"` → `"safety"`) | ✅ | ✅ (still a known string) | ❌ snapshots diff for that fixture | hash flips |
-| Renames a field | ✅ | varies | ❌ snapshot diff + likely contract failure | hash + throws change |
-| Rewrites the schema (breaking) | ✅ | ❌ multiple contracts | ❌ contracts + snapshots both fail | many hash flips |
-
-The "✅" in `pnpm test` for every row is intentional: by design, default tests only fail when *our code* drifts from a pinned baseline. Upstream drift is reported by the opt-in `pnpm test:drift` and by the snapshot diff that lands the moment fixtures are re-pinned.
-
-### Drift-triage decision tree
-
-A `pnpm test:drift` failure means live cache no longer satisfies a contract our deletions assumed. Three possibilities:
-
-1. **Pipeline regressed (e.g. dropped `source_metadata` on some rows)** — coordinate with the pipeline owner to restore. Don't refresh fixtures yet; the regression would propagate into our pinned set. The runtime `assertSourceMetadata` guards (lib/hf-data.ts, lib/model-data.ts) would also start firing in production, providing a second signal.
-2. **Pipeline emitted a new value our enum doesn't recognise (e.g. new `evaluator_relationship`)** — extend the corresponding `KNOWN_*` set in `tests/upstream-drift.test.ts` and `tests/pipeline-contract.test.ts` AND any consumer code that branches on the old set.
-3. **Pipeline made a schema-level change** — review the upstream commit log (`git -C ../eval_cards_backend_pipeline log`) for context, decide if our consumer needs updates, then refresh fixtures.
-
-A snapshot diff after `pnpm refresh-fixtures` always means *some* output changed. Read the fixture diff and snapshot diff side-by-side:
-
-- Fixture diff explains *what* upstream changed (raw data shift)
-- Snapshot diff explains *how* the adapter projected that change into user-visible output
-- Together → review and decide if the new output is correct (`pnpm test -- -u`) or a regression to fix
-
-### Known gaps in drift coverage
-
-1. **Stale `.cache/hf-data/`**: `pnpm test:drift` runs against whatever is on disk; it doesn't auto-refresh from huggingface.co. If `pnpm cache-hf-data` hasn't been run recently, "drift" reports stale-cache-vs-fixtures, not upstream-vs-fixtures. Fix: run `pnpm cache-hf-data` before `pnpm test:drift` when you care about true upstream.
-2. **Hand-edited fixtures aren't detected**: nothing checks that `tests/fixtures/X.json` matches what `pnpm refresh-fixtures` would produce. If someone edits a fixture for debugging and forgets to restore, tests stay green against the mutation. Mitigation would be a content-hash entry per fixture in `manifest.json`; defer until it's actually a problem.
-3. **Drift covers Tier A invariants only, not Tier B snapshots**: a value-reclassification (Scenario "reclassifies an existing value" above) is invisible to drift. Detection requires `pnpm refresh-fixtures` (snapshot diff) or `pnpm audit-adapters --live --diff` against an older baseline. By design — running snapshots against live data would flap on every refresh.
-4. **`pnpm test:drift` is opt-in, not scheduled**: nobody runs it unless prompted. A CI nightly cron (or `pnpm test:drift` in a weekly task) would catch upstream contract breaks earlier; currently you discover them only when you next run drift.
-5. **Audit script doesn't check Tier A contracts**: if a row violates a contract, the audit reports it indirectly via increased `throws` count (the runtime guards fire) but you'd need `pnpm test:drift` for the exact contract message and per-row locator.
-
-## Build order
-
-Tier A first (smallest, foundational). Tier B next (replaces subagent audits for normal regression detection). Tier C last (heaviest tooling).
-
-Each tier is independently usable, so they can be built in parallel by different agents:
-
-| Tier | Estimated effort | Depends on | Parallelizable? |
-|---|---|---|---|
-| A — contract tests | 1-2h | nothing | yes |
-| B — snapshot tests | 2-3h | fixture set (shared) | mostly |
-| C — audit script | 2-3h | nothing | yes |
-| Fixture set (~25 files) | 1h | curation decisions | shared dep |
-
-Recommended: build the fixture set + Tier A in series (one agent), Tier B and Tier C in parallel after fixtures are in.
-
-## Test-additions deferred to specific migration items
-
-The original Tier B plan listed 8 adapters; 4 are built. The remaining 4 (`hfEvalEntryToListItem`, `aggregateBenchmarkSummaries`, `buildSingleMetricSuiteMatrixSummary`, `createModelFamilySummary`) are deferred to the migration items that touch them — adding fixtures + snapshots speculatively now would be testing-for-testing's-sake. Specifically:
-
-- **`hfEvalEntryToListItem` snapshot** — add when starting #1 (identity parsing) or #2 (setup-alias). Needs an `eval_list_entries` fixture group extracted from `.cache/hf-data/eval-list.json`. Cover at least: a typical entry, one with `display_name` starting with "accuracy on " (triggers `prefersBenchmarkName`), one with `display_name` containing "for scorer", one with a missing `display_name`.
-- **Setup-alias collision fixture** — add when starting #2. Pick a model with `additional_details.mode` ∈ {"prompt", "fc", "thinking"} appearing across multiple submissions for the same model_id. `openai__gpt-5.2` model card has thinking variants; find a corresponding model detail file.
-- **`aggregate__<suite>` pattern** — add when starting #5 (composites) or #6 (matrix synthesis). The aggregate URL pattern is synthetic, not on disk; the test would call `aggregateBenchmarkSummaries` directly with a curated input set. Defer until that adapter is actually being touched.
-- **`createModelFamilySummary` snapshot** — add when starting #3. The flatten + family-summary chain is what `getModelSummaryById` returns; snapshotting `createModelFamilySummary(flattenModelEvaluations(model))` locks the full surface before the refactor.
-
-## Reshape-class items: testing addendum (added 2026-04-28)
-
-The Tier B snapshot framework above assumes the migration target is "pipeline emits the value, TS reads it." That works for cleaning-class items. For **reshape-class** items (#3 hierarchy flatten, #5 composite rollup, #6 matrix synthesis, #14 score summary stats, #16 per-category counts; plus the reshape halves of #2 and #13), the migration target is different: pipeline emits relational rows, **DuckDB SQL** does the dedup/groupby/aggregate. See `notes/migration-plan.md` § "Data direction" for framing.
-
-This shifts what the test set has to verify:
-
-- **Tier A contracts gain a parquet schema dimension.** Today's contracts assert JSON field invariants on `.cache/hf-data/**`. When the parquet schema goes more relational (e.g. one row per `(eval_summary_id, variant_key, retrieved_timestamp)` for the variant dedup case), Tier A grows a parallel set of contracts asserting the new typed columns are present and well-typed. File: `tests/parquet-contract.test.ts` (new, parallel to `tests/pipeline-contract.test.ts`).
-- **Tier B snapshots become parity gates, not destinations.** Today, `tests/adapters/flatten-model-evaluations.test.ts` snapshots the TS reshape output. Once SQL replaces the TS, the same snapshot becomes a TS-vs-SQL parity assertion: run both, diff. The snapshot is committed; the SQL output is computed at test time; equality is the gate. Reshape-class snapshots stay green during the migration *exactly because* they assert behavior preservation, not implementation. Don't delete them on TS removal — convert them.
-- **Tier C audit script grows a backend dimension.** `scripts/audit-adapters.mjs` currently runs adapters against the live cache. Add `--backend duckdb` so the same adapter contract is exercised against the DuckDB read path, producing a digest that diffs against the JSON-backend digest. This is the full-corpus generalization of `scripts/compare-data-backends.mjs`, but at the adapter-output level rather than the HTTP-endpoint level.
-- **Five of the eight planned Tier B adapters are reshape-class:** `flattenModelEvaluations`, `buildBenchmarkLeaderboardMatrix`, `buildSingleMetricSuiteMatrixSummary`, `aggregateBenchmarkSummaries`, `createModelFamilySummary`. Their snapshots are the contract the SQL replacement must match. Build them when migrating each item — the snapshots gate the deletion.
-
-What this doesn't change: cleaning-class items (the 12 that aren't reshape) work exactly as the existing framework describes — refresh fixtures → snapshot diff → review → ship. No structural test changes needed for cleaning items.
-
-## What this DOESN'T cover
-
-- **End-to-end UI tests.** No clicking through pages. Adapter snapshots are a proxy.
-- **Performance regression.** No timing assertions.
-- **Pipeline-side correctness.** Pipeline has its own tests in the sibling repo. Our contracts assert what we *consume*, not what's *correct upstream*.
-- **The DuckDB shadow read.** That's covered by the existing `scripts/compare-data-backends.mjs` parity harness — at the HTTP-endpoint level. The adapter-level parity for reshape items (TS reshape output vs SQL reshape output) is the addendum above.
-
-## Workflows
-
-### Migration workflow (TS deletion against current upstream)
-
-Use this for items #1, #2, #3 and any pipeline-side change that flows back into deletions in this repo.
-
-```bash
-# 1. Sync to current upstream so the work is against fresh data
-pnpm cache-hf-data
-pnpm test:drift                                  # does upstream still satisfy our contracts?
-                                                 # if no → triage per "Drift-triage decision tree" first
-
-# 2. Re-pin fixtures to current upstream
-pnpm refresh-fixtures
-pnpm test                                        # any pre-deletion snapshot diffs?
-                                                 # if yes → review, then `pnpm test -- -u`, separate commit
-                                                 # so the pin-update is isolated from the deletion
-
-# 3. Capture a full-cache baseline so we can diff the impact of the change
-pnpm audit-adapters --output /tmp/baseline.json --live
-
-# 4. Make the deletion (or refactor)
-
-# 5. Verify
-pnpm test                                        # snapshots flag any unexpected output change
-pnpm audit-adapters --output /tmp/candidate.json --live
-pnpm audit-adapters --diff /tmp/baseline.json /tmp/candidate.json   # full-cache impact
-pnpm compare-data-backends --json-base http://localhost:3001 --duckdb-base http://localhost:3002
-
-# 6. Review snapshot diff alongside code diff
-#    - intentional behaviour change: `pnpm test -- -u`, document the why in the commit
-#    - unintentional: fix the code
-
-# 7. Ship
-```
-
-Each step covers a distinct failure mode; nothing duplicates. Steps 3, 5b, 5c (the audit captures) are skippable for tiny changes — start with `pnpm test` alone and escalate if you want fuller coverage.
-
-### Light-touch workflow (small change, no upstream sync needed)
-
-```bash
-pnpm test                                        # baseline green
-# make the change
-pnpm test                                        # snapshots flag any output change
-# review snapshot diff, `pnpm test -- -u` if intentional
-pnpm compare-data-backends ...
-```
-
-### Drift-only workflow (you suspect upstream changed)
-
-```bash
-pnpm cache-hf-data                               # ensure local cache is current
-pnpm test:drift                                  # 5 contracts against full live cache
-# if green: upstream still satisfies our deletions' assumptions
-# if red: triage per "Drift-triage decision tree"
-```
-
-### Cross-repo workflow (pipeline-side change first, TS deletion later)
-
-```bash
-# In ../eval_cards_backend_pipeline
-uv run --with huggingface_hub --no-project python -m scripts.pipeline --dry-run \
-    -e EXPORT_EXPERIMENTAL_PARQUET=1
-# verify output/ has the new field
-
-# Back in this repo
-pnpm cache-hf-data                               # picks up the new published artifact
-pnpm test:drift                                  # do we now have a NEW contract we want to assert?
-                                                 # if yes: extend tests/pipeline-contract.test.ts + drift
-pnpm refresh-fixtures
-pnpm test                                        # snapshots reflect the new field if any adapter consumes it
-# now eligible to delete the TS code that the pipeline emission obviates
-```

notes/transformations/01-identity-canonicalization.md

@@ -1,244 +0,0 @@
-# Identity canonicalization
-
-Drafted 2026-04-28. Migration item #1 in `notes/migration-plan.md`.
-
-## Rule
-
-Given a `ModelInfo` (`{id, name, developer?}`) emitted by some upstream source, derive a **canonical identity tuple** the rest of the app uses for routing, display, and grouping:
-
-```
-{
-  namespace,           // "anthropic" — owner segment, lowercase
-  rawHandle,           // "claude-opus-4.5" — model segment as received
-  normalizedHandle,    // "claude-opus-4.5" — separators collapsed to "-", lowercased
-  familySlug,          // "claude-opus-4.5" — handle minus version-date suffix
-  familyId,            // "anthropic/claude-opus-4.5"
-  familyName,          // "Claude Opus 4.5" — title-cased, with v/V rule
-  variantKey,          // "base" if no date pattern, else "<YYYYMMDD>" or "<YYYYMMDD>-<qualifier>"
-  variantLabel,        // "Current" if base, else "<YYYY-MM-DD>" or "<YYYY-MM-DD> · <Qualifier>"
-  variantDisplayName,  // familyName if base, else "<familyName> (<variantLabel>)"
-  versionDate?,        // "YYYY-MM-DD" if a date pattern was detected
-  versionQualifier?,   // humanized qualifier suffix, if present
-}
-```
-
-## Classification
-
-- **Unconditional normalization** for casing rules (token case map, v/V handling) — when the upstream `name` is present, the canonicalizer ignores it for `familyName` and re-derives from the `id`. Pipeline-side fix: pipeline applies these rules once at emission time; no consumer should re-derive.
-- **Default-only** does not apply to this transformation. Every output field is computed unconditionally.
-- **Cleaning → pipeline.** Both outputs (`model_family_id`, `model_family_name`) are value transforms on a single record. No record merging or aggregation. Migration target: pipeline emits canonical values; TS logic deletes.
-
-## Inputs and expected outputs
-
-The full table below is the executable spec. Every row corresponds to a parameterized test case in `tests/transformations/identity-canonicalization.test.ts`.
-
-### Group A — Token case map
-
-The TS implementation maintains a hand-curated `TOKEN_CASE_MAP` for tokens that deviate from naive title-casing. Pipeline must produce identical outputs for every token below — no improvements or additions without first updating this spec, the unit tests, and the verification script (in that order). "TS is the spec" — see `notes/transformations/README.md`.
-
-| Token (lower) | Canonical |
-|---|---|
-| ai | AI |
-| coder | Coder |
-| command | Command |
-| chat | Chat |
-| claude | Claude |
-| gemini | Gemini |
-| gemma | Gemma |
-| gpt | GPT |
-| haiku | Haiku |
-| instruct | Instruct |
-| instant | Instant |
-| llama | Llama |
-| max | Max |
-| mini | Mini |
-| mistral | Mistral |
-| opus | Opus |
-| phi | Phi |
-| plus | Plus |
-| preview | Preview |
-| pro | Pro |
-| qwen | Qwen |
-| reasoning | Reasoning |
-| sonnet | Sonnet |
-| thinking | Thinking |
-| turbo | Turbo |
-| yi | Yi |
-
-**Detected divergence (2026-04-28):** pipeline emits "Minicpm3 4B FC", "Xlam 2 1B FC R", "Xlam 2 32B FC R" (3 distinct slugs, 7 cards total) which the TS map does NOT have an entry for ("fc"). TS title-cases to "Fc". Either the TS map needs an `fc → FC` entry, OR pipeline already does the right thing here and TS has a gap. **Decision needed: do we want `fc → FC` added?** If yes, both sides update; spec adds the row.
-
-### Group B — v/V version-token rule
-
-For any token matching `/^v\d/i` (e.g. `v3`, `v3.1`, `V0`), the `v` is lowercased.
-
-| Token in handle | Token in name |
-|---|---|
-| v3 | v3 |
-| V3 | v3 |
-| v0.1 | v0.1 |
-| V1.0 | v1.0 |
-
-**Detected divergence (2026-04-28):** pipeline emits 1,253 cards with capital `V` (e.g. "Deepseek V3", "Mistral 7B Instruct V0.3", "Mixtral 8x22b Instruct V0.1", "Nova Lite V1.0"). TS rule lowercases. Pipeline must apply this rule before emitting `model_family_name`.
-
-### Group C — Date and qualifier extraction
-
-`splitVersionParts` operates on a handle that has already been through `normalizeHandle` (Group D). For a normalized handle matching `/^(.*?)-((?:19|20)\d{6})(?:-(.+))?$/`:
-
-| normalizedHandle (input to splitVersionParts) | familySlug | versionDate | versionQualifier | variantKey | variantLabel |
-|---|---|---|---|---|---|
-| claude-3.5-sonnet | claude-3.5-sonnet | _none_ | _none_ | base | Current |
-| claude-3.5-sonnet-20240620 | claude-3.5-sonnet | 2024-06-20 | _none_ | 20240620 | 2024-06-20 |
-| claude-3.5-sonnet-20240620-thinking | claude-3.5-sonnet | 2024-06-20 | Thinking | 20240620-thinking | 2024-06-20 · Thinking |
-| claude-3.5-sonnet-20240620-thinking-high | claude-3.5-sonnet | 2024-06-20 | Thinking High | 20240620-thinking-high | 2024-06-20 · Thinking High |
-
-Note the **dotted** `3.5` not dashed — the upstream call to `normalizeHandle` collapses `(\d)-(?=\d(?:-|$))` to `(\d).` (so `3-5-sonnet` → `3.5-sonnet`). Pipeline-side implementations must apply that collapse before invoking `splitVersionParts`-equivalent logic, otherwise the regex match for the date will be off.
-
-The qualifier is humanized via the same token-case + v/V rules used for `familyName`.
-
-**Important date-pattern caveat:** the date regex requires 8 contiguous digits (`(?:19|20)\d{6}`). A dashed form like `2025-12-11` (which appears in some pipeline IDs like `openai/gpt-5-2025-12-11-thinking-high`) does NOT match — those ten-character dashed dates pass through `normalizeHandle` unchanged (no internal-digit-dash-digit pattern triggers the collapse) and `splitVersionParts` returns `base`/`Current`. This is preserved as-is; do not "fix" the regex to accept dashed dates without checking what relies on the current behaviour.
-
-### Group D — Handle normalization
-
-Raw handles arrive from `getRawHandle()` (Group E) — they do NOT contain the namespace. `normalizeHandle` then applies the rules below in order:
-
-| rawHandle | normalizedHandle | Rule fired |
-|---|---|---|
-| Claude_Opus_4.5 | claude-opus-4.5 | lowercase + underscore→dash |
-| claude opus 4.5 | claude-opus-4.5 | space→dash |
-| --claude--opus-- | claude-opus | leading/trailing/repeated dash collapse |
-| claude-3-5 | claude-3.5 | digit-dash-digit collapses (5 is at end → matches lookahead) |
-| claude-3-5-sonnet | claude-3.5-sonnet | same: 5 followed by `-` → matches |
-| gpt-5 | gpt-5 | no digit-dash-digit pattern |
-| claude-3-5-sonnet-20240620 | claude-3.5-sonnet-20240620 | "3-5" collapses; "20240620" is one token (no internal dashes) so no collapse there |
-| openai/foo (called via pipeline → never happens) | not applicable | namespace is always split off in Group E before normalize is called |
-
-The digit-dash-digit rule is the subtle one: `/(\d)-(?=\d(?:-|$))/g → "$1."` matches a digit-dash-digit pattern only when the right-hand digit is at end-of-string OR followed by another dash. So `3-5-x` becomes `3.5-x`, `3-5` (at end) becomes `3.5`, but `3-5x` (followed by a non-dash) is left alone. Inside `20240620` there are no dashes, so the regex doesn't fire on the date itself.
-
-### Group E — Namespace and rawHandle extraction
-
-The `id` field's first slash splits namespace from handle. If no slash, `developer` field is used as namespace (slug-cased: spaces → dashes, lowercased).
-
-| input.id | input.developer | namespace | rawHandle |
-|---|---|---|---|
-| anthropic/claude-opus-4-5 | (any) | anthropic | claude-opus-4-5 |
-| openai/gpt-5 | (any) | openai | gpt-5 |
-| Claude Opus 4.5 | Anthropic | anthropic | Claude Opus 4.5 (then normalized) |
-| gpt-5 | OpenAI | openai | gpt-5 |
-| (empty) | OpenAI | openai | (empty → falls back to name) |
-
-When `id` lacks a slash, `rawHandle` falls back to `stripNamespace(name, namespace)` then to `name.trim()`.
-
-### Group F — `familyId` and `model_route_id`
-
-```
-familyId      = `${namespace}/${familySlug}`
-model_route_id = familyId.replace(/\//g, "__")
-```
-
-**Pipeline status (2026-04-28):** `model_family_id` matches TS-computed `familyId` for **5,830 / 5,830** cards. `model_route_id` matches `model_family_id.replace(/\//g, "__")` for **5,830 / 5,830** cards. ✅
-
-This is the part that's already safe to delete on the TS side; the rest is not.
-
-## Current TS implementation
-
-The transformation lives in TWO places:
-
-### Primary — `lib/model-family.ts` (consumed at request time)
-
-| Concern | Location |
-|---|---|
-| Top-level entry point | `lib/model-family.ts:165-190` (`getCanonicalModelIdentity`) |
-| Token case map | `lib/model-family.ts:17-44` (`TOKEN_CASE_MAP`) |
-| Token case helper | `lib/model-family.ts:100-123` (`titleCaseToken`) |
-| v/V rule | `lib/model-family.ts:118-120` (inside `titleCaseToken`) |
-| Handle normalization | `lib/model-family.ts:82-90` (`normalizeHandle`) |
-| Date format helper | `lib/model-family.ts:92-98` (`formatVersionDate`) |
-| Date + qualifier extraction | `lib/model-family.ts:139-163` (`splitVersionParts`) |
-| Family name humanization | `lib/model-family.ts:125-137` (`humanizeHandle`) |
-| Namespace extraction | `lib/model-family.ts:62-69` (`getNamespace`) |
-| Raw handle extraction | `lib/model-family.ts:71-80` (`getRawHandle`) |
-| `route_id` derivation | `lib/model-family.ts:220-225` (`getModelFamilyRouteId`) |
-
-Total: ~200 lines. Two exported entry points consumed at six call sites in `lib/model-data.ts`, `lib/hf-data.ts`, `components/eval-detail.tsx`. (A third export, `normalizeModelInfo`, was deleted during the 2026-04-28 orphan sweep — zero callers.)
-
-### Secondary — `scripts/cache-hf-data.mjs` (cache-build-time duplicate)
-
-The cache-build script independently re-implements five of the helpers above (presumably to avoid importing TS into the build script). When `pnpm cache-hf-data` runs, it post-processes downloaded model-cards to apply the same canonicalization that the runtime would. Duplicated helpers:
-
-| Concern | Cache-script location | Equivalent in `lib/model-family.ts` |
-|---|---|---|
-| Handle normalization | `scripts/cache-hf-data.mjs:141-149` (`normalizeHandle`) | identical to lib version |
-| Date format helper | `scripts/cache-hf-data.mjs:151-157` (`formatVersionDate`) | identical |
-| Token case helper | `scripts/cache-hf-data.mjs:159-177` (`titleCaseToken`) | identical |
-| Family name humanization | `scripts/cache-hf-data.mjs:179-185` (`humanizeHandle`) | identical |
-| Family info extractor (smaller version) | `scripts/cache-hf-data.mjs:187-197` (`getCanonicalFamilyInfo`) | subset of `getCanonicalModelIdentity` (returns only familyId + familyName) |
-
-These five functions in `scripts/cache-hf-data.mjs` ALSO need to be deleted in the same migration cleanup, since pipeline-emitted canonical fields obviate both the runtime AND build-time canonicalization paths.
-
-## Pipeline status
-
-Verified against full `.cache/hf-data/model-cards.json` (5,830 entries) on 2026-04-28:
-
-| Field | Pipeline match | Notes |
-|---|---|---|
-| `model_family_id` | 5830 / 5830 ✅ | Matches `${namespace}/${familySlug}` exactly. |
-| `model_route_id` | 5830 / 5830 ✅ | Matches `model_family_id.replace(/\//g, "__")`. |
-| `model_family_name` | 4,570 / 5,830 ❌ | 1,260 disagreements — see "Divergences detected" below. |
-| `family_slug` | not emitted | Pipeline doesn't surface this field. |
-| `version_date` | not emitted | |
-| `version_qualifier` | not emitted | |
-| `variant_key` | partial | Emitted on per-variant entries inside `model_card.variants[]`; not on top-level card. Match status not yet audited. |
-| `variant_label` | partial | Same as `variant_key`. |
-| `variant_display_name` | not emitted | |
-
-## Divergences detected
-
-Sourced from `scripts/verify-identity.mjs` against the full live cache (run 2026-04-28).
-
-### Bucket 1: v/V capitalization (1,253 cards)
-
-Pipeline does not lowercase the `v` in version tokens. Examples:
-
-| Pipeline `model_family_name` | TS-computed `familyName` |
-|---|---|
-| Deepseek V3 | Deepseek v3 |
-| Deepseek V3.1 | Deepseek v3.1 |
-| Mistral 7B Instruct V0.3 | Mistral 7B Instruct v0.3 |
-| Mixtral 8x7b Instruct V0.1 | Mixtral 8x7b Instruct v0.1 |
-| Mixtral 8x22b Instruct V0.1 | Mixtral 8x22b Instruct v0.1 |
-| Nova Lite V1.0 | Nova Lite v1.0 |
-| Nova Micro V1.0 | Nova Micro v1.0 |
-| Nova Pro V1.0 | Nova Pro v1.0 |
-
-### Bucket 2: missing TOKEN_CASE_MAP entries (7 cards, 3 distinct slugs)
-
-Pipeline emits known acronyms in upper-case that TS doesn't know about. Examples:
-
-| Pipeline `model_family_name` | TS-computed `familyName` | Token at issue |
-|---|---|---|
-| Minicpm3 4B FC | Minicpm3 4B Fc | `fc` (function-calling) |
-| Xlam 2 1B FC R | Xlam 2 1B Fc R | `fc` |
-| Xlam 2 32B FC R | Xlam 2 32B Fc R | `fc` |
-
-**Open product question:** TS map is missing `fc → FC`. Either it's an oversight in TS (we should add `fc`) or pipeline is right and TS is wrong. Resolve with product owner before pipeline-side change. Likely the right fix is to add `fc` to the canonical map and have pipeline emit it.
-
-### Other (0 cards)
-
-`date_format`, `qualifier_humanize`, `other` buckets all came back empty. Date/qualifier handling matches pipeline, which is reassuring.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/identity-canonicalization.test.ts`)
-- [ ] `fc → FC` token map decision (product owner)
-- [ ] Filed with pipeline owner (link to issue/PR in `eval_cards_backend_pipeline`)
-- [ ] Pipeline emits `model_family_name` matching this spec across full corpus
-- [ ] Pipeline emits `family_slug`, `version_date`, `version_qualifier`, `variant_key`, `variant_label`, `variant_display_name` on top-level card (or we accept they live only on per-variant entries)
-- [ ] TS deleted; callers read pipeline fields directly. Includes both `lib/model-family.ts:1-225` (full file) AND the 5 duplicated helpers in `scripts/cache-hf-data.mjs:141-197`.
-
-## Notes for pipeline implementer
-
-- The `getCanonicalModelIdentity` function is pure (no I/O, no globals beyond `TOKEN_CASE_MAP`); a Python port should be a direct line-by-line translation.
-- The unit tests in `tests/transformations/identity-canonicalization.test.ts` are the acceptance criteria. A Python equivalent (with the same input/output examples) is the simplest verification path.
-- The audit script `scripts/verify-identity.mjs` already runs TS-vs-pipeline diff across the full cache; once pipeline ships, run it to confirm zero mismatches before deleting TS.
-- The `fc → FC` question above is the only ruleset gap; everything else is "pipeline doesn't apply the rule yet." Resolve `fc` first if possible.

notes/transformations/02-setup-alias-merging.md

@@ -1,155 +0,0 @@
-# Setup-alias variant merging
-
-Drafted 2026-04-28. Migration item #2 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are **refactoring for UI efficiency**, not fixing data correctness. Original TS behaviour is the canonical spec — including its quirks — because that's what users see today. Pipeline must reproduce TS output exactly when this transformation moves upstream. If anyone wants to change *what* gets merged (vs the current TS rules), that's a separate product decision deferred until later.
-
-## Rule (as TS implements it today)
-
-A model card's `variants[]` list contains entries keyed by `variant_key`. The runtime normalizer (`lib/hf-data.ts:normalizeSingleModelCardEntry`, line 750) walks each variant and may transform `variant_key`/`variant_label` based on whether the variant looks like a setup-alias of an underlying date-based variant.
-
-Algorithm (verbatim from current code):
-
-1. If `variant_key === "base"` → rename to `"default"` / `"Default"`.
-2. If `variant_key === "default"` → keep as-is.
-3. Otherwise, build a synthetic identity by feeding `${familyId}-${variant_key}` through `getCanonicalModelIdentity` (the same canonicalizer as migration item #1).
-4. Inspect `syntheticIdentity.versionDate` and `syntheticIdentity.versionQualifier`:
-   - If `versionDate` is set AND `isSetupAliasQualifier(versionQualifier)` returns true → rewrite to `versionDate` (ISO `YYYY-MM-DD` format) for both key and label.
-   - Otherwise → use `syntheticIdentity.variantKey` and `syntheticIdentity.variantLabel`.
-5. After all variants are processed, deduplicate by normalized `variant_key`. Duplicates merge: `evaluation_count` summed, `raw_model_ids` unioned + sorted, `last_updated` taken as the latest timestamp.
-
-`isSetupAliasQualifier` (`lib/hf-data.ts:712`) returns true when the normalized qualifier (lowercased, separators → `-`) matches:
-
-- exactly `prompt`
-- exactly `fc`
-- exactly `function-calling`
-- starts with `thinking` (so `thinking`, `thinking-1k`, `thinking-medium`, `thinking-32k`, etc. all match)
-
-The "starts with thinking" prefix-match is intentionally broad and aggregates all thinking-budget variants (`thinking-1k`, `thinking-medium`, `thinking-32k`, etc.) into a single date-only entry — a deliberate UI-side aggregation choice that prioritizes cross-model comparison readability over per-condition granularity.
-
-## Classification
-
-- **Unconditional normalization.** When inputs match the rule, TS overwrites whatever upstream emitted. Pipeline-side implementation must apply the same overwrite.
-- **Dual class — cleaning and reshape:**
-  - **Cleaning → pipeline** (key derivation): the alias qualifier rules (`isSetupAliasQualifier`) that determine *which bucket* a raw result lands in are value transforms on a single record. Pipeline-side fix: emit a canonical `setup_alias_key` field per result row.
-  - **Reshape → DuckDB SQL** (bucket reduction): collapsing multiple result rows with the same `setup_alias_key` into a single variant entry (taking the latest timestamp, merging evaluation results) is a `GROUP BY setup_alias_key` aggregation. Migration target once the key is emitted upstream: SQL `SELECT … MAX(retrieved_timestamp) … GROUP BY setup_alias_key` rather than TS reduce logic.
-
-## Inputs and expected outputs
-
-These are the rules as TS executes them today. Pipeline must produce identical outputs.
-
-### Group A — `isSetupAliasQualifier` truth table
-
-| Input qualifier | Normalized | Returns |
-|---|---|---|
-| `prompt` | `prompt` | `true` |
-| `Prompt` | `prompt` | `true` (case-insensitive) |
-| `fc` | `fc` | `true` |
-| `function-calling` | `function-calling` | `true` |
-| `function calling` | `function-calling` | `true` (whitespace → dash) |
-| `function_calling` | `function-calling` | `true` (underscore → dash) |
-| `thinking` | `thinking` | `true` |
-| `thinking-1k` | `thinking-1k` | `true` (prefix match) |
-| `thinking-medium` | `thinking-medium` | `true` (prefix match) |
-| `thinking_xhigh` | `thinking-xhigh` | `true` (prefix match after normalization) |
-| `Thinking 1K` | `thinking-1k` | `true` |
-| `high` | `high` | `false` |
-| `medium` | `medium` | `false` |
-| `low` | `low` | `false` |
-| `minimal` | `minimal` | `false` |
-| `8k` | `8k` | `false` |
-| (empty / null / undefined) | `""` | `false` |
-
-### Group B — End-to-end variant normalization
-
-`getCanonicalModelIdentity`'s date regex requires 8 contiguous digits (`(?:19|20)\d{6}`). Dashed-date variant_keys do NOT match — they fall through to the `versionDate = undefined` branch, which sends them to `syntheticIdentity.variantKey === "base"`. **This is part of TS's current behaviour and must be preserved.**
-
-| Input variant_key | TS-observed output `variant_key` | TS-observed output `variant_label` | Notes |
-|---|---|---|---|
-| `default` | `default` | (unchanged) | passthrough |
-| `base` | `default` | `Default` | rename |
-| `20251101` | `20251101` | `2025-11-01` | YYYYMMDD date-only — preserved as raw token, ISO label |
-| `2025-11-01` | `base` | `Current` | dashed date-only — falls through to base because regex doesn't match (NB: by TS quirk, a future product call may decide to align this) |
-| `20240620-thinking` | `2024-06-20` | `2024-06-20` | YYYYMMDD + thinking → merge to ISO date |
-| `20240620-thinking-1k` | `2024-06-20` | `2024-06-20` | YYYYMMDD + thinking-1k → merge (startsWith match) |
-| `20240620-thinking-medium` | `2024-06-20` | `2024-06-20` | merge (startsWith match) — all thinking-N variants for this date collapse together |
-| `20240620-fc` | `2024-06-20` | `2024-06-20` | merge |
-| `20240620-prompt` | `2024-06-20` | `2024-06-20` | merge |
-| `20240620-high` | `20240620-high` | `2024-06-20 · High` | non-alias qualifier preserved |
-| `2025-12-11-thinking-medium` | `base` | `Current` | dashed date — regex doesn't match, falls through |
-| `2025-12-11-thinking-1k` | `base` | `Current` | same — dashed date passes through to base |
-| `2025-12-11-fc` | `base` | `Current` | same |
-| `2025-12-11-high` | `base` | `Current` | same |
-| `2025-08-07-low` | `base` | `Current` | same |
-| `gpt-foo-bar` | `base` | `Current` | no date detected anywhere |
-
-### Group C — Multi-variant deduplication after normalization
-
-When multiple input variants normalize to the same `variant_key`, they merge:
-- `raw_model_ids`: union, deduped, sorted
-- `evaluation_count`: sum
-- `last_updated`: maximum (latest)
-
-This is what produces, for example, the user-visible behaviour for `openai/gpt-5.2`: the cache file has 7 distinct variants, but TS normalization collapses 6 of the 7 (everything except `default`) into a single `base` bucket — because all 6 use dashed dates and fall through to `base`.
-
-## Current TS implementation
-
-| Concern | Location |
-|---|---|
-| Runtime normalizer (active) | `lib/hf-data.ts:750-812` (`normalizeSingleModelCardEntry`) |
-| Setup-alias qualifier check (runtime) | `lib/hf-data.ts:712-720` (`isSetupAliasQualifier`) |
-| Qualifier normalizer (runtime) | `lib/hf-data.ts:708-710` (`normalizeSetupAliasQualifier`) |
-| Cache-time normalizer | `scripts/cache-hf-data.mjs:213-246` (`getNormalizedVariantMeta`) |
-| Setup-alias qualifier check (cache) | `scripts/cache-hf-data.mjs:203-211` |
-| Qualifier normalizer (cache) | `scripts/cache-hf-data.mjs:199-201` |
-| Mode-based path (dead, ignore) | `lib/eval-processing.ts:371-434` |
-
-The mode-based path reads `model_info.additional_details.mode` which is empty on every production model_result row (verified 2026-04-28: 0 of 86,183). It runs but never fires for any input. Pipeline-side implementation should NOT reproduce it; it's load-bearing only against test fixtures that may carry the field.
-
-## Pipeline status — known divergences
-
-Pipeline (`eval_cards_backend_pipeline/scripts/pipeline.py`) implements its own variant aggregation in `aggregated_display_identity` (line 1724). It uses a **different qualifier set and a different input field** than TS:
-
-| Aspect | TS (this spec) | Pipeline today | Result |
-|---|---|---|---|
-| Input field | `variant_key` from variant entry | `model_info.additional_details.mode` from raw eval | Pipeline merges submissions early; TS re-merges at variant level |
-| Qualifier set | `prompt`, `fc`, `function-calling`, plus *any* `thinking*` (prefix) | exact set: `{prompt, fc, function-calling, thinking, prompt-thinking, fc-thinking, function-calling-thinking}` | TS aggregates more aggressively for `thinking-N` variants; pipeline keeps each thinking budget separate |
-| Date-format handling | Only YYYYMMDD recognized; dashed dates fall through to `base` | N/A — pipeline operates on `mode` field, not `variant_key` |  |
-
-**Concrete example:** for `openai/gpt-5.2` with submissions across 7 setups (default + base 2025-12-11 + 5 thinking budgets):
-- Pipeline emits 7 distinct variants (it merges fc/prompt into the date-only base via `mode`-field check, but keeps thinking-{none,low,medium,high,xhigh} separate because those exact strings aren't in pipeline's set)
-- TS normalizes pipeline's 7 variants → 2 (`default` + `base`, all dashed-date variants collapsed)
-
-**The user-visible state today** is whatever TS produces (TS runs on every API request). So users see the post-TS-normalization view: 2 variants for that card, not 7.
-
-When the migration moves this transformation upstream, pipeline must produce TS's 2-variant view, not pipeline's current 7-variant view. Pipeline-side options:
-
-1. Add a second-pass normalization to pipeline output that mirrors TS's rules (prefix-match `thinking*`, only YYYYMMDD dates trigger merge).
-2. Drop pipeline's existing `aggregated_display_identity` mode-based check and replace it entirely with the variant_key-based rule.
-
-Option 2 is cleaner if pipeline owners agree.
-
-## Notes for pipeline implementer
-
-- **Reproduce the prefix-match `thinking*` exactly.** This is the largest TS-vs-pipeline divergence. Don't tighten it to an exact set without a product call.
-- **Reproduce the dashed-date fall-through behaviour.** `2025-12-11-thinking-medium` produces `variant_key: base` in TS today. Whether that's "right" or "wrong" is not for this migration to decide.
-- **Ignore the `mode` field.** It's empty in production. Pipeline's current `aggregated_display_identity` reads it; the replacement should not.
-- **Re-run `scripts/verify-setup-alias.mjs`** against pipeline output once the change ships. Goal: zero divergence between TS-as-is output and pipeline-emitted output for the full corpus.
-
-## Migration checklist
-
-- [x] Spec written (TS-as-is, including quirks)
-- [x] Tests cover each rule branch (`tests/transformations/setup-alias-merging.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits variants matching this spec across the full corpus (verified by `scripts/verify-setup-alias.mjs`)
-- [ ] TS deleted; callers read pipeline-emitted variants directly. Files to delete:
-  - `lib/hf-data.ts:750-812` (`normalizeSingleModelCardEntry`)
-  - `lib/hf-data.ts:708-720` (`normalizeSetupAliasQualifier`, `isSetupAliasQualifier`)
-  - `scripts/cache-hf-data.mjs:199-246` (cache-time mirror)
-  - `lib/eval-processing.ts:371-434` (the dead mode-based path)
-
-## Future product decision (deferred)
-
-Whether the current TS aggregation is the *right* product behaviour is open. The dashed-date fall-through and the prefix-match `thinking*` together produce a heavily-aggregated view (2 variants for `openai/gpt-5.2` instead of 7). If the team later decides users would benefit from per-thinking-budget granularity, the transformation can be redesigned in pipeline (where it's cheaper to change than in TS at runtime). That's explicitly out of scope for this refactor.

notes/transformations/03-license-normalization.md

@@ -1,147 +0,0 @@
-# License normalization
-
-Drafted 2026-04-28. Migration item #18 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency, not fixing data. TS-as-is is the canonical spec. If the truncation behaviour or rule coverage looks suboptimal, that's a deferred product decision (see end of doc). Do not "improve" the rule when porting upstream.
-
-## Rule (as TS implements it today)
-
-`shortenLicense` (`components/eval-card.tsx:38-48`) takes a free-text license string from `benchmark_card.ethical_and_legal_considerations.data_licensing` and returns a short display label. Algorithm:
-
-1. If empty or `"Not specified"` → return `""`.
-2. If lowercased license includes `"creative commons attribution 4"` → `"CC BY 4.0"`.
-3. Else if includes `"creative commons zero"` → `"CC0"`.
-4. Else if includes `"apache license 2"` OR `"apache 2"` → `"Apache 2.0"`.
-5. Else if includes `"mit license"` → `"MIT"`.
-6. Else if includes `"cc-by-sa"` → `"CC BY-SA"`.
-7. Else if length > 24 → return `${license.slice(0, 22)}…` (truncate to 22 chars + ellipsis).
-8. Else → return the input unchanged.
-
-Rules are applied in this order (first match wins). The truncation target (`length > 24`, slice to 22) is asymmetric on purpose — one character of headroom to avoid truncating 24-char strings.
-
-The companion function `licenseBadgeClass` in the same file is **purely presentational** (CSS color mapping) and stays in the component. NOT in scope for this migration.
-
-## Classification
-
-- **Unconditional normalization.** Always overwrite whatever upstream emitted.
-- The output field on the spec side could be named `license_short` and live alongside `data_licensing` in the benchmark card. The original `data_licensing` (free text) stays available for tooltips/details views.
-- **Cleaning → pipeline.** Pure value transform on a single field. No aggregation or record merging. Migration target: pipeline emits `license_short`; TS `shortenLicense` deletes.
-
-## Inputs and expected outputs
-
-Each row corresponds to a parameterized test case in `tests/transformations/license-normalization.test.ts`. Pipeline must produce identical outputs for every case below.
-
-### Group A — Rule firing order (which rule wins)
-
-| Input | Output | Rule fired |
-|---|---|---|
-| `"Apache License 2.0"` | `"Apache 2.0"` | rule 4 (matches "apache license 2") |
-| `"Apache 2.0"` | `"Apache 2.0"` | rule 4 (matches "apache 2") |
-| `"MIT License"` | `"MIT"` | rule 5 |
-| `"Creative Commons Attribution 4.0"` | `"CC BY 4.0"` | rule 2 |
-| `"Creative Commons Zero v1.0 Universal"` | `"CC0"` | rule 3 |
-| `"cc-by-sa-3.0"` | `"CC BY-SA"` | rule 6 |
-| `"Open Data Commons Attribution License"` | `"Open Data Commons Attr…"` | rule 7 (truncate, length > 24) |
-| `"The dataset is made available under a CC BY license."` | `"The dataset is made av…"` | rule 7 (truncate; the prose form bypasses rule 2 because it doesn't contain "creative commons attribution 4") |
-| `"apache-2.0"` | `"apache-2.0"` | rule 8 (passthrough; SPDX-style hyphen-lowercase doesn't match "apache license 2" or "apache 2") |
-| `"other"` | `"other"` | rule 8 (passthrough, length ≤ 24) |
-| `"unknown"` | `"unknown"` | rule 8 (passthrough, length ≤ 24) |
-| `"Not specified"` | `""` | rule 1 |
-| `""` | `""` | rule 1 |
-| `null` / `undefined` | `""` | rule 1 (falsy short-circuit) |
-
-### Group B — Edge cases of the truncation rule
-
-| Input | Output | Notes |
-|---|---|---|
-| 24-char string (no other rule matches) | (input unchanged) | length ≤ 24 → passthrough |
-| 25-char string | first-22-chars + `…` | length > 24 → truncate |
-| String beginning `"MIT-like license that is custom"` | `"MIT"` | rule 5 fires before truncation (substring match) |
-| String beginning `"some apache 2 thing"` | `"Apache 2.0"` | rule 4 fires (substring match) |
-
-### Group C — Case sensitivity
-
-All match-rules call `.toLowerCase()` before substring testing. Inputs:
-
-| Input | Output | Notes |
-|---|---|---|
-| `"APACHE LICENSE 2.0"` | `"Apache 2.0"` | case-insensitive match |
-| `"creative commons attribution 4.0"` | `"CC BY 4.0"` | already lowercase |
-| `"CREATIVE COMMONS ZERO v1.0"` | `"CC0"` | uppercase still matches |
-| `"Mit License"` | `"MIT"` | mixed case |
-
-## Current TS implementation
-
-The same `shortenLicense` function is duplicated in TWO files:
-
-| Concern | Location |
-|---|---|
-| Function (eval-card list render path) | `components/eval-card.tsx:38-48` (`shortenLicense`) |
-| Caller | `components/eval-card.tsx:63` |
-| Function (eval-list page render path — duplicate copy) | `app/evals/page.tsx:24-41` (`shortenLicense`) |
-| Caller | `app/evals/page.tsx:1720` |
-| CSS class mapping (NOT in scope, stays in UI) | `components/eval-card.tsx:22-36` (`LICENSE_COLORS`, `licenseBadgeClass`) AND `app/evals/page.tsx:43-?` (duplicate) |
-
-The two function bodies are **functionally identical** (same output for every input) but textually slightly different — `app/evals/page.tsx` uses template literal `` `${license.slice(0, 22)}…` ``, `components/eval-card.tsx` uses concatenation `license.slice(0, 22) + "…"`. Both run at request time on every render. Pipeline-side emission of `license_short` would eliminate both per-render calls; the deletion task must update BOTH files.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where the transformation lives | `components/eval-card.tsx:shortenLicense` (runs at every render) | not implemented | TS runs at request time; pipeline ships free-text `data_licensing` only |
-| Field consumed | `benchmark_card.ethical_and_legal_considerations.data_licensing` | (same — unchanged) | — |
-| Output field | local variable `shortLicense` (passed to badge as label text) | none | UI shows TS's output; pipeline doesn't expose a short form anywhere |
-| Rule coverage | 5 explicit aliases + truncate fallback | n/a | n/a |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/benchmark-metadata.json` (production cache):
-
-- 85 of 85 benchmark cards have a `data_licensing` field
-- 11 distinct license strings appear across the corpus
-- `shortenLicense` produces the following distribution:
-  - 33 → `""` (cards with `"Not specified"`)
-  - 16 → `"Apache 2.0"`
-  - 9 → `"MIT"`
-  - 8 → `"Open Data Commons Attr…"` (truncated)
-  - 6 → `"CC BY 4.0"`
-  - 3 → `"CC BY-SA"`
-  - 3 → `"other"` (passthrough)
-  - 3 → `"unknown"` (passthrough)
-  - 2 → `"CC0"`
-  - 1 → `"apache-2.0"` (passthrough — SPDX-style misses the `apache 2.0` rule)
-  - 1 → `"The dataset is made av…"` (truncated; prose form bypasses CC BY rule)
-
-Verified by `scripts/verify-license.mjs`.
-
-## Notes for pipeline implementer
-
-- Reproduce all 8 rules in order (first match wins). Do not reorder.
-- The `apache-2.0` SPDX-style-lowercase form intentionally falls through to passthrough — the existing rule only matches prose forms (`"apache license 2"` or `"apache 2"` with a space). Don't broaden the rule.
-- The truncation produces `${license.slice(0, 22)}…`. That's 22 chars plus a single Unicode ellipsis (`…`, U+2026). Don't substitute three dots (`...`).
-- Truncation triggers at `length > 24`, not `length > 22`. A 24-char string passes through; a 25-char string truncates. Preserve this asymmetry.
-- The empty + `"Not specified"` → `""` short-circuit uses falsy semantics in JS (handles `null`/`undefined`/`""` together). Pipeline-side equivalent should treat all three input shapes as the same.
-- Field name suggestion: `benchmark_card.ethical_and_legal_considerations.license_short` — keeps the existing free-text `data_licensing` available for detail views.
-
-Verification: run `scripts/verify-license.mjs` against pipeline-emitted `license_short` once it ships. Goal: zero divergence across the 85 production benchmark cards.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/license-normalization.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits `license_short` matching this spec across all benchmark cards
-- [ ] TS deleted; `components/eval-card.tsx:38-48` AND `app/evals/page.tsx:24-41` both read `card?.ethical_and_legal_considerations?.license_short` directly. `licenseBadgeClass` stays in both files (UI presentation).
-
-## Future product decision (deferred)
-
-The current rule set has known coverage gaps that produce ugly truncation:
-- Free-form CC BY descriptions (e.g. "The dataset is made available under a CC BY license.") aren't recognized as CC BY → truncate to "The dataset is made av…"
-- "Open Data Commons Attribution License" (ODC-By) isn't recognized → truncate to "Open Data Commons Attr…"
-- SPDX-style lowercase identifiers (`apache-2.0`, `mit`, `cc-by-4.0`) aren't recognized as their canonical short forms
-
-If the team later decides users would benefit from broader recognition (e.g. SPDX identifier mapping, looser CC matching), the rule can be expanded in pipeline. That's explicitly out of scope for this refactor.

notes/transformations/04-dataset-url-synthesis.md

@@ -1,173 +0,0 @@
-# Dataset URL synthesis
-
-Drafted 2026-04-28. Migration item #20 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency, not fixing data. TS-as-is is the canonical spec. The fallback chain is functionally complete; pipeline just needs to do the resolution once and emit the result.
-
-## Rule (as TS implements it today)
-
-`components/eval-card.tsx:83-86` resolves `datasetUrl` from `summary.source_data` via a 3-branch nullish-coalescing chain (NOT truthiness). The literal expression:
-
-```ts
-const datasetUrl =
-  sourceData?.dataset_url ??
-  (Array.isArray(sourceData?.url) ? sourceData?.url?.[0] : sourceData?.url) ??
-  (sourceData?.hf_repo ? `https://huggingface.co/datasets/${sourceData.hf_repo}` : undefined)
-```
-
-Reading order:
-
-1. `source_data.dataset_url` — used if not `null`/`undefined`. Empty string `""` is RETURNED (not nullish).
-2. `source_data.url` — if array, take `url[0]` (no truthiness check on the element); if string, use directly. Falls through only if the resolved value is `null`/`undefined`.
-3. `https://huggingface.co/datasets/${source_data.hf_repo}` — only if `hf_repo` is truthy (this branch uses a ternary, not `??`, so `""` falls through to `undefined`).
-4. Else → `undefined`.
-
-The constructed HF URL uses the literal template — no encoding, no slash normalization, no validation. Whatever the upstream `hf_repo` value is, it gets concatenated as-is.
-
-## Classification
-
-- **Default-only.** The rule fills in a value only when `dataset_url` isn't already present. Pipeline-side fix: emit `dataset_url` directly so the fallback chain becomes unnecessary; preserve any existing `dataset_url` value rather than overwriting.
-- **Cleaning → pipeline.** Derives a URL value from other fields on the same record. No aggregation or record merging. Migration target: pipeline emits `dataset_url`; TS fallback chain deletes.
-
-## Inputs and expected outputs
-
-Each row corresponds to a parameterized test case in `tests/transformations/dataset-url-synthesis.test.ts`. Pipeline must produce identical outputs for every case.
-
-### Group A — Branch firing order (first non-nullish wins)
-
-| Input `source_data` | Output | Branch |
-|---|---|---|
-| `{dataset_url: "https://example.com/x"}` | `"https://example.com/x"` | 1 |
-| `{dataset_url: "x", url: ["y"]}` | `"x"` | 1 (dataset_url short-circuits when set) |
-| `{url: ["https://a.com", "https://b.com"]}` | `"https://a.com"` | 2 (first element of array) |
-| `{url: ["only"]}` | `"only"` | 2 |
-| `{url: "https://a.com"}` | `"https://a.com"` | 2 (string form) |
-| `{hf_repo: "Mercor/ACE"}` | `"https://huggingface.co/datasets/Mercor/ACE"` | 3 (HF template) |
-| `{hf_repo: "mercor/apex-agents"}` | `"https://huggingface.co/datasets/mercor/apex-agents"` | 3 (preserves case) |
-| `{dataset_name: "x"}` | `undefined` | 4 (none of the above match) |
-| `{}` | `undefined` | 4 |
-| `null` | `undefined` | 4 (caller passes nullable; defensive) |
-| `undefined` | `undefined` | 4 |
-
-### Group B — Edge cases (`??` nullish semantics, NOT truthiness)
-
-The original expression uses `??` (nullish coalescing), so `""`, `0`, and `false` do NOT trigger fallback — only `null` and `undefined` do. The hf_repo branch internally uses `sourceData.hf_repo ? template : undefined` (a truthiness check), so empty hf_repo IS falsy.
-
-| Input `source_data` | Output | Why |
-|---|---|---|
-| `{dataset_url: "", url: ["fallback"]}` | `""` | `""` is not nullish, `??` short-circuits to it |
-| `{url: [], hf_repo: "x/y"}` | `"https://huggingface.co/datasets/x/y"` | `url[0]` is `undefined`, `??` falls through to hf_repo |
-| `{url: [""], hf_repo: "x/y"}` | `""` | `url[0]` is `""` (not nullish), `??` short-circuits to it |
-| `{url: [null], hf_repo: "x/y"}` | `"https://huggingface.co/datasets/x/y"` | `url[0]` is `null` (nullish), `??` falls through to hf_repo |
-| `{url: ["a"], hf_repo: "x/y"}` | `"a"` | url[0] truthy, short-circuits hf_repo |
-| `{hf_repo: ""}` | `undefined` | inline `hf_repo ? template : undefined` uses truthiness — `""` falls through to `undefined` |
-| `{hf_repo: "/leading-slash"}` | `"https://huggingface.co/datasets//leading-slash"` | no normalization — double slash preserved |
-
-## Current TS implementation
-
-The fallback chain exists in TWO sites, with **slightly different semantics** between them. A third related pattern (just the bare HF link) exists in a third site.
-
-### Site A — eval-card list (`??` nullish, default `undefined`)
-
-`components/eval-card.tsx:83-86`:
-
-```ts
-const datasetUrl =
-  sourceData?.dataset_url ??
-  (Array.isArray(sourceData?.url) ? sourceData?.url?.[0] : sourceData?.url) ??
-  (sourceData?.hf_repo ? `https://huggingface.co/datasets/${sourceData.hf_repo}` : undefined)
-```
-
-This is the spec described above (Group A/B in the table). Empty string `""` is RETURNED (not nullish); only `null`/`undefined` fall through.
-
-### Site B — benchmark detail page (`||` truthy, default `null`)
-
-`components/benchmark-detail.tsx:5043-5047`:
-
-```ts
-const datasetHref =
-  sourceData?.dataset_url ||
-  (Array.isArray(sourceData?.url) ? sourceData?.url?.[0] : sourceData?.url) ||
-  (sourceData?.hf_repo ? `https://huggingface.co/datasets/${sourceData.hf_repo}` : null) ||
-  null
-```
-
-**Differs from Site A on edge cases:** uses `||` (truthiness) so empty strings DO fall through; uses `null` instead of `undefined` as the final default. For `{dataset_url: ""}`, Site A returns `""` and Site B returns the next non-empty branch's value (or `null`).
-
-### Site C — direct HF link (no fallback chain)
-
-`components/benchmark-detail.tsx:5416-5420`:
-
-```ts
-{sourceData?.hf_repo && (
-  <InlineMeta label="HF Repo" value={
-    <a href={`https://huggingface.co/datasets/${sourceData.hf_repo}`} ...>
-      {sourceData.hf_repo}
-    </a>
-  } />
-)}
-```
-
-This is just the hf_repo template applied directly when `hf_repo` is truthy. It does NOT consult `dataset_url` or `url` — so even if `dataset_url` is set, this site shows the HF link separately. Different intent: this is the "HF Repo" inline-meta link, not the primary dataset link.
-
-### Summary table
-
-| Site | Path | Semantics | Default fallback | Status for migration |
-|---|---|---|---|---|
-| A | `components/eval-card.tsx:83-86` | `??` nullish | `undefined` | spec target |
-| B | `components/benchmark-detail.tsx:5043-5047` | `||` truthy | `null` | divergent from A on edge cases — pipeline-emitted `dataset_url` resolves both |
-| C | `components/benchmark-detail.tsx:5416-5420` | n/a — bare hf_repo template | n/a | UI element with different intent; out of scope |
-
-Pipeline-emitted `dataset_url` (resolved per the spec rule) serves both Sites A and B. Once it's populated, both sites just read it directly and the `??` vs `||` divergence becomes moot.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where the resolution lives | `components/eval-card.tsx` (inline, runs at every render) | not implemented | TS resolves at request time |
-| Pipeline-emitted `dataset_url` | consumed if present (branch 1) | **never populated** (verified 2026-04-28: 0/587 eval-details emit `dataset_url`) | branch 1 is currently dead — pipeline could populate to retire the fallback chain |
-| Other source_data fields | `url` (array or string), `hf_repo` consumed | both emitted | TS does the resolution work each render |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/evals/` (587 production eval-detail files):
-
-- Branch 1 (`dataset_url`): **0** files (the field is never emitted — branch is dead code today)
-- Branch 2 (`url[0]`): **564** files (96.1%)
-- Branch 3 (`url` as string): **0** files (always emitted as array)
-- Branch 4 (`hf_repo` template): **22** files (3.7%)
-- Branch 5 (`undefined`): **1** file (`cocoabench` has neither `url` nor `hf_repo`)
-
-Examples:
-- `appworld`: `{url: ["https://github.com/Exgentic/exgentic"]}` → `"https://github.com/Exgentic/exgentic"`
-- `ace`: `{hf_repo: "Mercor/ACE"}` → `"https://huggingface.co/datasets/Mercor/ACE"`
-- `cocoabench`: `{dataset_name: "CocoaBench v1.0", source_type: "other", additional_details: {…}}` → `undefined`
-
-Verified by `scripts/verify-dataset-url.mjs`.
-
-## Notes for pipeline implementer
-
-- Reproduce the 4-step fallback exactly. Don't reorder; first match wins.
-- Truthy semantics for branches 1, 4: empty string and `null`/`undefined` are falsy → fall through to next branch.
-- `url[0]` is taken **without checking truthiness**: if `url` is an array of any length, the first element is returned even if it's `null`, `""`, `0`, etc. This means an array `[null]` produces `null`, NOT `undefined`. Don't "improve" this.
-- `url` as a string short-circuits to that string — no further fallback. Even an empty string would short-circuit (passes `typeof === "string"`).
-- The HF template is `https://huggingface.co/datasets/${hf_repo}` with NO encoding, validation, or slash normalization. Whatever `hf_repo` is, it gets appended verbatim.
-- Suggested pipeline emission: add `dataset_url` field to every `source_data` object with the resolved URL. TS-side fallback chain becomes dead code (branch 1 always wins).
-
-Verification: run `scripts/verify-dataset-url.mjs` against pipeline-emitted `dataset_url` once it ships. Goal: zero divergence vs TS-as-is across 587 production eval-details.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch + edge cases (`tests/transformations/dataset-url-synthesis.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits resolved `dataset_url` on every `source_data` matching this spec
-- [ ] TS deleted; BOTH Site A (`components/eval-card.tsx:83-86`) AND Site B (`components/benchmark-detail.tsx:5043-5047`) read `sourceData?.dataset_url` directly. Site C (the bare `hf_repo` template in `components/benchmark-detail.tsx:5416-5420`) is a different UI element and stays.
-
-## Future product decision (deferred)
-
-The 1 `undefined` case (`cocoabench`) means the dataset link button will be missing/disabled for that eval. Whether pipeline should synthesize a fallback (e.g. from `additional_details.benchmark_reference_urls_json`) is a product question outside this refactor's scope.

notes/transformations/05-slug-candidates.md

@@ -1,206 +0,0 @@
-# Slug candidate generation (model + developer file lookup)
-
-Drafted 2026-04-28. Migration item #19 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. The retry logic is **load-bearing** in production (39% of model lookups and 99.9% of developer lookups depend on a non-zero retry position). Don't try to "clean up" the candidate generation — preserve it until pipeline emits canonical-by-construction filenames.
-
-## Rule (as TS implements it today)
-
-Three pure functions in `lib/model-data.ts:150-211` translate model and developer identifiers into ordered lists of candidate filenames to try when looking up the corresponding JSON in the HF cache (`models/<slug>.json`, `developers/<slug>.json`).
-
-### `pipelineSlugify(text)` — base helper
-
-Mirrors the slug rule the upstream pipeline uses:
-
-1. Strip control characters (`\x00-\x1f\x7f`).
-2. Replace any character not in `[a-zA-Z0-9._-]` with `_` (preserves dots, dashes, alnum, case).
-3. Trim leading/trailing underscores.
-4. Return `"unknown"` if the result is empty.
-
-Note: dots and dashes are preserved AS-IS; only "weird" characters become underscores. Slashes are NOT preserved — they become underscores.
-
-### `getModelDetailSlugCandidates(modelId)` — produce up to 6 candidate model slugs
-
-Inserts variants into a `Set` (so duplicates collapse) in this order:
-
-```
-withSlash       = modelId.replace(/\//g, "__")    // "openai/gpt-5.2" → "openai__gpt-5.2"
-withDots        = withSlash.replace(/\./g, "-")   // "openai__gpt-5.2" → "openai__gpt-5-2"
-candidates: pipelineSlugify(withSlash),
-            pipelineSlugify(withSlash.toLowerCase()),
-            pipelineSlugify(withDots),
-            pipelineSlugify(withDots.toLowerCase()),
-            pipelineSlugify(modelId),
-            pipelineSlugify(modelId.toLowerCase())
-return Array.from(set)
-```
-
-The `Set` collapses duplicates: e.g., for an already-lowercase input, the `.toLowerCase()` variants are no-ops and drop out, so the actual returned array is shorter.
-
-### `getDeveloperSlugCandidates(developerOrRouteId)` — up to 6 candidate developer slugs
-
-Same Set-based pattern but with different transformations:
-
-```
-underscoreSlug          = pipelineSlugify(input)
-lowercaseUnderscoreSlug = pipelineSlugify(input.toLowerCase())
-hyphenSlug              = input.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "")
-compactSlug             = input.toLowerCase().replace(/[^a-z0-9]+/g, "")
-candidates: underscoreSlug,
-            lowercaseUnderscoreSlug,
-            underscoreSlug.replace(/_/g, "-"),
-            lowercaseUnderscoreSlug.replace(/_/g, "-"),
-            hyphenSlug (if non-empty),
-            compactSlug (if non-empty)
-return Array.from(set)
-```
-
-## Classification
-
-- **Lookup transformation (default-only)**. The retry walks candidates and uses the first that resolves to an actual file. Pipeline-side fix: emit a single canonical filename per model/developer that matches `model_route_id`/`developer_route_id` exactly. Then no retry needed.
-- **Cleaning → pipeline.** The slug derivation (`pipelineSlugify`) and canonical ID emission are value transforms per record. The retry loop exists only to compensate for the pipeline not yet emitting a stable canonical ID — once it does, both the slug logic and the retry delete together. No aggregation.
-
-## Inputs and expected outputs
-
-### Group A — `pipelineSlugify`
-
-| Input | Output | Rule |
-|---|---|---|
-| `"openai__gpt-5"` | `"openai__gpt-5"` | passthrough (alnum + dash + underscore allowed) |
-| `"openai__gpt-5.2"` | `"openai__gpt-5.2"` | passthrough (dot allowed) |
-| `"openai/gpt-5"` | `"openai_gpt-5"` | slash → underscore (slash not in allowed set) |
-| `"OpenAI"` | `"OpenAI"` | passthrough (case preserved) |
-| `"x0000001"` | `"x0000001"` | passthrough |
-| `"foo bar"` | `"foo_bar"` | space → underscore |
-| `"foo!@#bar"` | `"foo___bar"` | each special char → `_` |
-| `"___foo___"` | `"foo"` | trim leading/trailing underscores |
-| `"!!!"` | `"unknown"` | empty after trim → fallback |
-| `""` | `"unknown"` | empty → fallback |
-
-### Group B — `getModelDetailSlugCandidates`
-
-| Input | Candidates returned | Why these are distinct |
-|---|---|---|
-| `"openai/gpt-5"` | `["openai__gpt-5", "openai_gpt-5"]` | already-lowercase + no dots → only slash and slash-stripped variants survive Set dedup |
-| `"openai/gpt-5.2"` | `["openai__gpt-5.2", "openai__gpt-5-2", "openai_gpt-5.2"]` | dotted form gets the with-dots variant (position 1) |
-| `"OpenAI/GPT-5"` | `["OpenAI__GPT-5", "openai__gpt-5", "OpenAI_GPT-5", "openai_gpt-5"]` | case-mixed input → both case variants survive |
-| `"anthropic/claude-3.7-sonnet"` | `["anthropic__claude-3.7-sonnet", "anthropic__claude-3-7-sonnet", "anthropic_claude-3.7-sonnet"]` | dotted version |
-| `"unknown/foo"` | `["unknown__foo", "unknown_foo"]` | already-lowercase + no dots |
-
-### Group C — `getDeveloperSlugCandidates`
-
-| Input | Candidates returned (in order, deduped) |
-|---|---|
-| `"openai"` | `["openai"]` (all variants collapse to same form) |
-| `"OpenAI"` | `["OpenAI", "openai"]` (case variants distinct) |
-| `"01-ai"` | `["01-ai", "01ai"]` (compactSlug strips the dash) |
-| `"Mistral AI"` | `["Mistral_AI", "mistral_ai", "Mistral-AI", "mistral-ai", "mistralai"]` (space → underscore + hyphen + compact variants) |
-| `"01_ai"` | `["01_ai", "01-ai", "01ai"]` (underscore-slug, dash variant, compact) |
-
-## Current TS implementation
-
-The four functions are tightly coupled — `pipelineSlugify` is the base; the others build on it.
-
-| Concern | Location | Used by |
-|---|---|---|
-| Base slugifier | `lib/model-data.ts:150-157` (`pipelineSlugify`) | the other three slug functions |
-| Developer route_id derivation (exported) | `lib/model-data.ts:159-161` (`getDeveloperRouteId`) | sets `route_id` on output objects in 7 places (see below) |
-| Model candidates | `lib/model-data.ts:167-185` (`getModelDetailSlugCandidates`) | model lookup retry |
-| Developer candidates (exported) | `lib/model-data.ts:187-211` (`getDeveloperSlugCandidates`) | developer lookup retry |
-
-### Call sites — model lookups (`getModelDetailSlugCandidates` retry)
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:1492` | `getModelSummaryById` — first attempt: try candidates of the URL-passed modelId |
-| `lib/model-data.ts:1527` | `getModelSummaryById` fallback — for each variant's raw_model_ids, try candidates |
-
-So the model lookup is THREE-stage in `getModelSummaryById`:
-1. Direct candidates from the input `modelId` (line 1492)
-2. If a card matches in `model-cards.json`, try its `model_route_id` directly (line 1516)
-3. Iterate every variant's `raw_model_ids` and try candidates of each (line 1527)
-
-### Call sites — developer lookups (`getDeveloperSlugCandidates` retry)
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:1343` | inside developer-list build; iterate candidates of `entry.developer` |
-| `lib/model-data.ts:1413` | `getDeveloperSummaryById` — try candidates of the URL-passed routeId |
-| `lib/model-data.ts:1452` | `getDeveloperSummaryById` fallback — try candidates of the matched developer's name |
-
-### Call sites — `getDeveloperRouteId` (output-side route_id derivation)
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:1320` | `getDeveloperList` — set `route_id` on each developer summary |
-| `lib/model-data.ts:1368` | (build path A) |
-| `lib/model-data.ts:1402` | `getDeveloperSummaryById` — set `route_id` on returned summary |
-| `lib/model-data.ts:1435` | (build path B) |
-| `lib/model-data.ts:1448` | comparison: `e.developer === routeId \|\| getDeveloperRouteId(e.developer) === routeId` |
-| `lib/model-data.ts:1476` | (build path C) |
-| `lib/duckdb-data.ts:307` | DuckDB backend — set `route_id` on developer list output |
-
-`getDeveloperRouteId` is the function that DERIVES `route_id` from `developer` — it's what makes the comparison at line 1448 work, and it's how the API output gets a stable `route_id` field for routing. Deleting `getDeveloperRouteId` without addressing these callers would break developer-page navigation.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| File naming (models/) | n/a (consumer side) | filenames written by pipeline; some use `route_id`, others use a dot-stripped variant | TS retries up to 6 candidates per request to find the right file |
-| File naming (developers/) | n/a (consumer side) | filenames are slug-cased developer names; `developers.json` does NOT carry `route_id` | TS derives candidates from the developer name itself |
-| Lookup overhead | up to 6 HF fetch attempts per missing-direct lookup | none (it's just emitting files) | wasted requests on cold-cache; redirected by retry logic |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/`:
-
-**Model lookups (5,830 cards):**
-- Candidate position 0 hits: **3,529** (60.5%) — `route_id` matches the file directly
-- Candidate position 1 hits: **2,297** (39.4%) — needed the dot→dash conversion (e.g. `gpt-5.2` → file `gpt-5-2`)
-- Misses (none of 6 candidates worked): **4** (0.07%)
-
-**Developer lookups (824 developers):**
-- Candidate position 0 hits: **468** (56.8%) — slugified raw input matches
-- Candidate position 1 hits: **351** (42.6%) — needed `.toLowerCase()` (developers with mixed-case names)
-- Candidate position 3 hits: **4** (0.5%) — needed underscore→dash on the lowercased slug
-- Misses: **1** (`x0000001` — no developer file under that name)
-
-The retry is doing real work: 39% of models and 43% of developers would 404 on direct lookup.
-
-Verified by `scripts/verify-slug-candidates.mjs`.
-
-## Notes for pipeline implementer
-
-The cleanest pipeline-side fix: **always emit `models/<route_id>.json` and `developers/<route_id>.json` directly**, where `route_id` is the canonical form already on each card. Then TS does a single direct lookup; the retry chain becomes dead code.
-
-If pipeline can't easily change file naming, the second-best option is to emit a **slug→file map** in `manifest.json` so TS does an O(1) lookup with no fallback.
-
-Concrete requirements for the simpler "canonical filenames" path:
-
-1. For every card in `model-cards.json`: write `models/<card.model_route_id>.json` with the contents.
-   - For dotted family_ids like `openai/gpt-5.2`, the `model_route_id` is `openai__gpt-5.2` (with dot preserved). The current cache files for these use dashes (`openai__gpt-5-2.json`). Pick one form and stick with it.
-2. For every entry in `developers.json`: ensure `route_id` is populated (currently absent) and write `developers/<route_id>.json`.
-3. The 4 model misses + 1 developer miss currently in production should be investigated separately — they represent missing files, not naming-convention issues.
-
-Don't try to reproduce the 6-candidate generation logic upstream. The point of the migration is to make it unnecessary.
-
-Verification: once pipeline ships canonical naming, every card in `model-cards.json` should resolve via `fs.existsSync('models/' + card.model_route_id + '.json')` directly. Run `scripts/verify-slug-candidates.mjs` and confirm `position 0 hits === total cards`.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/slug-candidates.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits `models/<route_id>.json` matching `model_route_id` exactly for all 5,830 cards
-- [ ] Pipeline emits `developers/<route_id>.json` matching `developer_route_id` for all 824 developers
-- [ ] Pipeline adds `route_id` field to every entry in `developers.json` (currently absent — TS derives via `getDeveloperRouteId(developer)`)
-- [ ] Pipeline-emitted `developer_route_id` matches `pipelineSlugify(developer.trim().toLowerCase())` for every developer (so the 7 `getDeveloperRouteId` call sites can read pipeline values directly without re-deriving)
-- [ ] TS deleted; callers do single direct lookup. Deletion includes ALL FOUR functions (`pipelineSlugify`, `getDeveloperRouteId`, `getModelDetailSlugCandidates`, `getDeveloperSlugCandidates`) plus the 12 call sites enumerated above.
-
-## Future product decision (deferred)
-
-The 4 model misses and 1 developer miss represent files that don't exist. Whether those are "should exist but pipeline forgot" or "intentionally absent" is a product question. Surface to pipeline owner separately during the cleanup pass.

notes/transformations/06-developer-name-canonicalization.md

@@ -1,159 +0,0 @@
-# Developer name canonicalization
-
-Drafted 2026-04-28. Migration item #9 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. The transformation has known imperfections (random HF handles get mechanically title-cased; some users' preferred capitalization doesn't survive) but those are deferred product decisions, not bugs to fix in this migration.
-
-## Rule (as TS implements it today)
-
-`normalizeDeveloperName(name)` (`lib/model-data.ts:236-244`) applies one of three transformations in this order:
-
-1. **Map hit (case-insensitive lookup):** lowercase the name, then look up in `KNOWN_DEVELOPER_NAMES`. If present, return the mapped canonical form.
-2. **Title-case fallback:** if input is fully lowercase AND starts with `[a-z]`, return `name.charAt(0).toUpperCase() + name.slice(1)`. Only the first character is capitalized.
-3. **Passthrough:** return input unchanged.
-
-The map (`lib/model-data.ts:217-234`) has 16 entries:
-
-| Map key | Canonical |
-|---|---|
-| `openai` | OpenAI |
-| `google` | Google |
-| `anthropic` | Anthropic |
-| `meta` | Meta |
-| `microsoft` | Microsoft |
-| `mistralai` | Mistral AI |
-| `deepseek` | DeepSeek |
-| `deepseek-ai` | DeepSeek |
-| `cohere` | Cohere |
-| `nvidia` | NVIDIA |
-| `alibaba` | Alibaba |
-| `amazon` | Amazon |
-| `apple` | Apple |
-| `ibm` | IBM |
-| `xai` | xAI |
-| `x-ai` | xAI |
-
-Note that some map keys collide with their canonical form (e.g. `google` → `Google` is just case-fixing) while others apply substantive transforms (`mistralai` → `Mistral AI` adds a space; `deepseek-ai` → `DeepSeek` strips the `-ai` suffix; `xai` → `xAI` mid-word capital).
-
-## Classification
-
-- **Unconditional normalization.** The function always runs on whatever `developer` string is present — it does not check for a pre-existing canonical field. Map hits, title-case fallback, and passthrough are all branches of the same unconditional transform. Pipeline-side fix: emit `developer` in canonical form directly; no consumer should re-derive it.
-- **Cleaning → pipeline.** Pure value transform on a single field. No aggregation or record merging. Migration target: pipeline emits canonical `developer`; TS `normalizeDeveloperName` and `KNOWN_DEVELOPER_NAMES` delete.
-
-## Inputs and expected outputs
-
-Each row corresponds to a parameterized test case in `tests/transformations/developer-name-canonicalization.test.ts`.
-
-### Group A — Map hits (case-insensitive, substantive transforms)
-
-| Input | Output | Rule |
-|---|---|---|
-| `openai` | `OpenAI` | map (case fix) |
-| `OpenAI` | `OpenAI` | map (case-insensitive lookup → same canonical form) |
-| `OPENAI` | `OpenAI` | map |
-| `mistralai` | `Mistral AI` | map (space added) |
-| `MistralAI` | `Mistral AI` | map (case-insensitive) |
-| `deepseek-ai` | `DeepSeek` | map (`-ai` suffix dropped) |
-| `DeepSeek-AI` | `DeepSeek` | map (case-insensitive) |
-| `xai` | `xAI` | map (mid-word cap) |
-| `x-ai` | `xAI` | map (alias) |
-| `nvidia` | `NVIDIA` | map (uppercase) |
-| `IBM` | `IBM` | map (case-insensitive lookup → uppercase canonical) |
-
-### Group B — Title-case fallback (lowercase input, not in map)
-
-| Input | Output | Why |
-|---|---|---|
-| `jaspionjader` | `Jaspionjader` | lowercase + starts with [a-z] → title-case first char only |
-| `allenai` | `Allenai` | lowercase + not in map → first-char uppercase only |
-| `bunnycore` | `Bunnycore` | same |
-| `zelk12` | `Zelk12` | same — digits inside don't matter |
-
-### Group C — Passthrough (mixed case, not in map)
-
-| Input | Output | Why |
-|---|---|---|
-| `JayHyeon` | `JayHyeon` | already has uppercase → not lowercase → passthrough |
-| `DreadPoor` | `DreadPoor` | same |
-| `Qwen` | `Qwen` | (Qwen is NOT in the map; passes through) |
-| `prithivMLmods` | `prithivMLmods` | mixed case, passthrough as-is — no first-char capitalization (input has uppercase, so the lowercase check fails) |
-| `Quazim0t0` | `Quazim0t0` | same |
-| `01-ai` | `01-ai` | does NOT start with [a-z] (starts with digit) → fallback rule fails → passthrough. Note: 01-ai is NOT in the map. |
-| `01_ai` | `01_ai` | same |
-
-### Group D — Edge cases
-
-| Input | Output | Notes |
-|---|---|---|
-| `  google  ` | `Google` | trim happens inside `key = name.trim().toLowerCase()` BUT the title-case branch and passthrough use the ORIGINAL `name` (not trimmed). For `  google  `: key = "google" → matches map → "Google" |
-| `  jaspionjader  ` | `  jaspionjader  ` | key = "jaspionjader" → no map hit. Title-case check uses original `name` which has spaces — `"  jaspionjader  " === "  jaspionjader  ".toLowerCase()` is true, BUT `/^[a-z]/.test("  jaspionjader  ")` is FALSE (starts with space). → falls to passthrough |
-| (empty string) | (empty string) | trim makes key empty → no map hit. Title-case check: `"" === "".toLowerCase()` is true but `/^[a-z]/.test("")` is false → passthrough returns "" |
-
-The edge case shows a TS quirk: leading whitespace prevents the title-case rule from firing, so `"  jaspionjader  "` passes through unchanged. The map-lookup uses the trimmed form and works for known names regardless.
-
-## Current TS implementation
-
-| Concern | Location |
-|---|---|
-| Map | `lib/model-data.ts:217-234` (`KNOWN_DEVELOPER_NAMES`) |
-| Function (exported) | `lib/model-data.ts:236-244` (`normalizeDeveloperName`) |
-
-### Call sites (5 total)
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:387` | `hfModelCardToEvaluationCardData` — set `developer` on output card |
-| `lib/model-data.ts:1367` | developer-list build path A |
-| `lib/model-data.ts:1401` | `getDeveloperSummaryById` — set `developer` on returned summary |
-| `lib/model-data.ts:1434` | developer-list build path B |
-| `lib/duckdb-data.ts:306` | DuckDB backend — set `developer` on developer list output |
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where canonicalization runs | request time, in 5 call sites | not implemented; raw `developer` string emitted as-is | TS canonicalizes per request |
-| Output field | inline transformation of `developer` field | n/a | TS-canonicalized name appears on user-visible UI |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/`:
-
-**`developers.json` (824 entries):**
-- Map hits: **15** (1.8%) — names like `Google`, `OpenAI`, `Alibaba` (the map's main job is case-fixing on these inputs since they already arrive Title-cased)
-- Title-case fallback: **458** (55.6%) — lowercase HF handles like `jaspionjader`, `allenai`, `bunnycore` get first-char-uppercased
-- Passthrough: **351** (42.6%) — mixed-case handles like `JayHyeon`, `prithivMLmods`, `Qwen` survive unchanged
-
-**`model-cards.json` (5,830 cards):**
-- Map hits: **695** (11.9%)
-- Title-case fallback: **2,824** (48.4%)
-- Passthrough: **2,311** (39.6%)
-
-The substantive map transforms (`mistralai` → `Mistral AI`, `deepseek-ai` → `DeepSeek`) DO fire in production — the model-cards.json mapHit count being higher than developers.json (11.9% vs 1.8%) suggests more model entries use the lowercase-suffix forms (e.g., `deepseek-ai` from the HF org slug) than the developers.json which already uses canonical forms.
-
-Verified by `scripts/verify-developer-name.mjs`.
-
-## Notes for pipeline implementer
-
-- Reproduce all 16 map entries exactly (both case-fix entries like `google → Google` and substantive transforms like `mistralai → Mistral AI`).
-- Reproduce the title-case fallback exactly: only fire when the entire string equals `name.toLowerCase()` AND starts with `[a-z]`. Don't capitalize anything else (no smart casing of multi-word names, no unicode-aware uppercasing).
-- The leading-whitespace quirk (`"  jaspionjader  "` passes through unchanged because the title-case regex fails on the leading space) should be preserved as-is.
-- Suggested pipeline emission: add `canonical_developer_name` field to `developers.json` entries and to every model card. Don't overwrite the upstream `developer` field; new field for clarity.
-
-Verification: run `scripts/verify-developer-name.mjs` against pipeline output once it ships. Goal: zero divergence vs TS-as-is across both 824 developers and 5,830 model cards.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/developer-name-canonicalization.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits `canonical_developer_name` on every developer entry + model card matching this spec
-- [ ] TS deleted; replace 5 call sites (4 in lib/model-data.ts + 1 in lib/duckdb-data.ts) with direct field reads. Delete `KNOWN_DEVELOPER_NAMES` table + `normalizeDeveloperName`.
-
-## Future product decision (deferred)
-
-The title-case fallback produces stylistically-questionable output for HF user handles (`jaspionjader → Jaspionjader`). Whether the team wants to (a) expand the map to cover more cases, (b) leave random HF handles as-is, or (c) take a different approach (e.g. fetch the user's preferred display name from HF API) is out of scope for this refactor.

notes/transformations/07-timestamp-normalization.md

@@ -1,165 +0,0 @@
-# Timestamp normalization
-
-Drafted 2026-04-28. Migration item #13 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. Three different timestamp normalizers exist in production, with subtly different semantics. They produce different numeric values for the same input but happen to converge on production data (99.99% is unix-seconds-strings; the divergence only fires when comparing across formats, which production rarely does). The migration target: emit a single canonical timestamp format upstream so all three normalizers can be deleted.
-
-## Rule (as TS implements it today — three variants)
-
-Three independent functions parse string timestamps into comparable numbers:
-
-### Variant A — `lib/model-data.ts:76-81` (`normalizeEvalTimestamp`)
-
-```ts
-function normalizeEvalTimestamp(value: string) {
-  const numericTimestamp = Number(value)
-  return !Number.isNaN(numericTimestamp) && !value.includes("-")
-    ? numericTimestamp * 1000
-    : new Date(value).getTime()
-}
-```
-
-- Uses `Number()` (strict — entire string must be numeric or returns `NaN`)
-- If numeric AND no `-` in input → multiply by 1000 (treats as **unix seconds**, output in ms)
-- Else → `new Date(value).getTime()` (ISO date parsing, output in ms)
-- Returns `NaN` if neither path produces a finite number (no defensive fallback)
-
-### Variant B — `lib/hf-data.ts:1049-1061` (`toComparableTimestamp`)
-
-```ts
-function toComparableTimestamp(timestamp: string | undefined) {
-  if (!timestamp) return Number.NEGATIVE_INFINITY
-  const numericTimestamp = Number.parseFloat(timestamp)
-  if (Number.isFinite(numericTimestamp)) return numericTimestamp
-  const parsedTimestamp = new Date(timestamp).getTime()
-  return Number.isFinite(parsedTimestamp) ? parsedTimestamp : Number.NEGATIVE_INFINITY
-}
-```
-
-- Uses `Number.parseFloat()` (lenient — parses leading numeric prefix; e.g. `"2026-04-13"` → `2026`)
-- If parseFloat returns finite → return AS-IS (NO `* 1000` multiplier)
-- Else → fallback to `Date.getTime()` or `NEGATIVE_INFINITY`
-- Defensive: undefined → `NEGATIVE_INFINITY`
-
-### Variant C — `components/benchmark-detail.tsx:1418-1426` (`toComparableTimestamp`)
-
-Same as Variant B but parameter is `string` (not `string | undefined`) and there's no leading `if (!timestamp)` check. Otherwise functionally identical.
-
-## Classification
-
-This item has two halves that land in different places:
-
-- **Cleaning (value format canonicalization) → pipeline.** The pipeline currently emits `retrieved_timestamp` as a unix-seconds-string. Converting to ISO 8601 is a value-change that belongs upstream; once done, all consumers read a consistently-formatted string with no parsing quirks.
-- **Reshape (variant dedup / sort-key derivation) → DuckDB SQL.** The 3 normalizers + 8 call sites exist solely to compare timestamps in order to pick the freshest variant or sort models by recency. That's a `MAX(retrieved_timestamp)` or `ORDER BY retrieved_timestamp DESC` operation — reshape work that has no business running at request time in TS. Once timestamps are ISO 8601, SQL comparison is lexicographic and correct. With a relational parquet schema, variant dedup becomes `QUALIFY ROW_NUMBER() OVER (PARTITION BY variant_key ORDER BY retrieved_timestamp DESC) = 1` instead of three TS normalizers.
-
-The two halves delete together: pipeline emits ISO 8601 (cleaning done) → SQL replaces the comparison call sites (reshape done) → all three TS functions deleted.
-
-## Inputs and expected outputs
-
-Each table below describes ONE variant. Pipeline must produce identical outputs per variant when canonical timestamps still roundtrip through these functions; the deletion target is to remove all three.
-
-### Group A — Variant A (`normalizeEvalTimestamp`)
-
-| Input | Output | Path |
-|---|---|---|
-| `"1774096306"` | `1774096306000` | numeric, no dash → `* 1000` (unix seconds → ms) |
-| `"1774096306.427425"` | `1774096306427.4248` | numeric, no dash → `* 1000` |
-| `"2026-04-13T12:34:56Z"` | `1776083696000` | not numeric → `Date.getTime()` |
-| `"2025-01-01"` | `1735689600000` | not numeric → `Date.getTime()` |
-| `"-1774096306"` | (a Date in 1969) | numeric BUT includes `-` → falls to `Date.getTime()` of negative-number-string → unexpected |
-| `"not a date"` | `NaN` | not numeric AND `Date(...)` is invalid → returns NaN |
-| `""` | `NaN` | Number("") = 0, no dash, → 0 * 1000 = 0... actually wait, Number("") is 0, !isNaN(0) is true, includes("-") false, → 0 * 1000 = 0. So empty returns 0, not NaN. |
-| `"20240620"` | `20240620000` | numeric, no dash → `* 1000`. Treated as unix seconds (year 1970) — NOT as YYYYMMDD date |
-
-### Group B — Variant B (`toComparableTimestamp` in lib/hf-data.ts)
-
-| Input | Output | Path |
-|---|---|---|
-| `"1774096306"` | `1774096306` | parseFloat finite → return as-is (NO multiplier) |
-| `"1774096306.427425"` | `1774096306.427425` | parseFloat finite → return as-is |
-| `"2026-04-13T12:34:56Z"` | `2026` | parseFloat parses leading "2026" → finite → returns `2026` (TS quirk: ISO datetimes look like the year-as-number, NOT compared as ms-of-epoch) |
-| `"2025-01-01"` | `2025` | parseFloat → 2025 (TS quirk again) |
-| `"not a date"` | `NEGATIVE_INFINITY` | parseFloat NaN → Date NaN → fallback |
-| `""` | `NEGATIVE_INFINITY` | falsy → defensive fallback |
-| `undefined` | `NEGATIVE_INFINITY` | falsy → defensive fallback |
-| `"20240620"` | `20240620` | parseFloat finite → return as-is |
-
-### Group C — Variant C (`toComparableTimestamp` in components/benchmark-detail.tsx)
-
-Same as Variant B except `""` and `undefined` paths:
-
-| Input | Output | Path |
-|---|---|---|
-| `""` | `NEGATIVE_INFINITY` | parseFloat("") = NaN, Date("").getTime() = NaN → fallback |
-| `undefined` | (TypeError at call site, since signature is `string` not `string \| undefined`) | undefined isn't allowed; parseFloat(undefined) = NaN, but TS would flag the call |
-
-In practice the `string` signature means callers always pass strings, so the `if (!timestamp)` check is unnecessary.
-
-### Group D — Cross-variant divergence (TS quirk)
-
-For the same input, the three variants produce DIFFERENT numbers. Comparing values from different variants is unsafe — but in production each variant is used in a self-contained scope, so this divergence doesn't usually fire.
-
-| Input | Variant A | Variant B | Variant C |
-|---|---|---|---|
-| `"1774096306.427425"` | `1774096306427.4248` (ms) | `1774096306.427425` (seconds, no multiplier) | `1774096306.427425` |
-| `"2026-04-13T12:34:56Z"` | `1776083696000` (ms-of-epoch from Date) | `2026` (parseFloat extracts the year!) | `2026` |
-| Comparing the two above (a vs b) | a < b (correct: 2026 is more recent) | a > b (**incorrect**: parseFloat treats ISO as the number 2026) | a > b (**incorrect**) |
-
-**This is a real bug in Variants B and C** for cross-format comparisons. It doesn't manifest in production because 99.99% of timestamps in `.cache/hf-data/models/*.json` are unix-seconds-strings. Do NOT fix in this migration; document and let pipeline canonicalize the format upstream so the bug becomes structurally impossible.
-
-## Current TS implementation
-
-| Concern | Location | Callers |
-|---|---|---|
-| Variant A — `normalizeEvalTimestamp` | `lib/model-data.ts:76-81` | 4 sites: `lib/model-data.ts:266, 650, 945-946, 1124` (all sort/compare timestamps when picking latest or sorting model_results) |
-| Variant B — `toComparableTimestamp` | `lib/hf-data.ts:1049-1061` | 2 sites: `lib/hf-data.ts:1311-1312` (compare in flattenHierarchyNode variant-bucket reduction) |
-| Variant C — `toComparableTimestamp` | `components/benchmark-detail.tsx:1418-1426` | 2 sites: `components/benchmark-detail.tsx:1600-1601` (variant deduplication) |
-
-Total: 3 functions + 8 caller sites across 3 files.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where canonicalization runs | request time, in 3 functions | not implemented; raw `retrieved_timestamp` strings emitted | TS parses on every comparison |
-| Output format | varies per variant (ms vs seconds) | `retrieved_timestamp` is unix-seconds-string in 99.99% of rows; ISO datetime in 0.006% | mixed; TS handles each variant differently but production format consistency means it usually works |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/models/*.json`:
-
-- Total `retrieved_timestamp` values: **86,183**
-- Unix-seconds-string format (`"1774096306.427425"`): **86,178** (99.994%)
-- ISO datetime format (`"2024-10-27T00:00:00Z"`): **5** (0.006%)
-- Empty / null: **0**
-- Other: **0**
-
-Verified by `scripts/verify-timestamp.mjs`.
-
-## Notes for pipeline implementer
-
-- **Recommended canonical format: ISO 8601** (`"2026-04-13T12:34:56Z"`). Lexicographic sort works as chronological sort; `Date(...)` parsing is unambiguous; matches what AGENTS.md uses elsewhere.
-- Once pipeline emits all timestamps as ISO 8601:
-  - Variant A's `* 1000` multiplier path becomes dead (no numeric input → all paths use `Date.getTime()`)
-  - Variants B and C's `parseFloat` quirk becomes irrelevant (ISO inputs → parseFloat NaN → fall to `Date.getTime()`)
-- All three variants then become equivalent and can be replaced with a single `Date(ts).getTime()` inline (or a shared one-line helper).
-- Don't try to migrate to a different format mid-flight (e.g. ms-of-epoch as bigint); ISO matches what the rest of the system expects.
-- The 5 existing ISO-format rows in production are evidence this format already works for the cache; the rest just need to be converted upstream.
-
-Verification: once pipeline ships ISO timestamps for all 86,183 rows, run `scripts/verify-timestamp.mjs` and confirm the unixSecondsString count drops to 0.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each variant's semantics + the cross-variant divergence (`tests/transformations/timestamp-normalization.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits all `retrieved_timestamp` values as ISO 8601 across all 86,183 rows
-- [ ] TS deleted; replace 3 functions + 8 callers with a single shared `Date(ts).getTime()` (or inline). Files: `lib/model-data.ts`, `lib/hf-data.ts`, `components/benchmark-detail.tsx`.
-
-## Future product decision (deferred)
-
-The `parseFloat` bug in Variants B and C produces incorrect ordering for cross-format comparisons. We're choosing to fix-by-canonicalization-upstream rather than fix-in-place. Whether the bug should be patched in TS as a defensive measure (in case a non-ISO timestamp slips through after migration) is a separate decision.

notes/transformations/08-benchmark-display-names.md

@@ -1,329 +0,0 @@
-# Benchmark display names
-
-Drafted 2026-04-28. Migration item #8 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. The transformation has known imperfections (one of the two implementations is a substring-match contains-test that will mangle e.g. "MMLU-Pro" by replacing the entire string with the long-form for "MMLU"; the active map is hand-curated and only covers ~34 known suite/parent keys) but those are deferred product decisions, not bugs to fix in this migration.
-
-## Rule (as TS implements it today)
-
-The repo has **two** functions named `getBenchmarkDisplayName` with different semantics; the active one in production is `lib/model-data.ts` (the `lib/eval-processing.ts` copy is an unused/duplicate — see "Duplicate implementation" below).
-
-### Active implementation — `lib/model-data.ts:146-148`
-
-`getBenchmarkDisplayName(benchmark: string)` applies one of two transformations:
-
-1. **Map hit (normalized lookup):** normalize the input via `normalizeBenchmarkKeyForLookup` (lowercase; replace any run of `-`, `.`, or whitespace with a single `_`; strip leading/trailing `_`), then look up in `BENCHMARK_NAMES`. If present, return the mapped canonical form.
-2. **Tokenize fallback (`humanizeToken`, `lib/model-data.ts:90-96`):** split the *original* (un-normalized) input on `[_-]+`, drop empty parts, capitalize the first character of each part, join with a single space. Note: only the first character is uppercased — `mmlu` becomes `Mmlu`, not `MMLU`.
-
-The map (`lib/model-data.ts:109-140`) has 30 entries — all hand-maintained suite/family keys.
-
-| Map key | Canonical |
-|---|---|
-| `hfopenllm_v2` | HF Open LLM v2 |
-| `helm_lite` | HELM Lite |
-| `helm_capabilities` | HELM Capabilities |
-| `helm_classic` | HELM Classic |
-| `helm_instruct` | HELM Instruct |
-| `helm_mmlu` | HELM MMLU |
-| `reward_bench` | RewardBench |
-| `reward_bench_2` | RewardBench 2 |
-| `bfcl` | BFCL |
-| `global_mmlu_lite` | Global MMLU Lite |
-| `swe_bench` | SWE-bench |
-| `arc_agi` | ARC-AGI |
-| `tau_bench_2` | TAU-Bench 2 |
-| `ace` | ACE |
-| `apex_agents` | APEX Agents |
-| `apex_v1` | APEX v1 |
-| `appworld` | AppWorld |
-| `browsecompplus` | BrowseComp+ |
-| `livecodebenchpro` | LiveCodeBench Pro |
-| `sciarena` | SciArena |
-| `terminal_bench_2_0` | Terminal Bench 2.0 |
-| `la_leaderboard` | LA Leaderboard |
-| `theory_of_mind` | Theory of Mind |
-| `fibble_arena` | Fibble Arena |
-| `fibble1_arena` | Fibble Arena v1 |
-| `fibble2_arena` | Fibble Arena v2 |
-| `fibble3_arena` | Fibble Arena v3 |
-| `fibble4_arena` | Fibble Arena v4 |
-| `fibble5_arena` | Fibble Arena v5 |
-| `wordle_arena` | Wordle Arena |
-
-The lookup is normalization-insensitive: `"HELM Lite"`, `"helm-lite"`, `"helm.lite"`, `"  helm   lite  "` all normalize to `helm_lite` and hit the map.
-
-### Suite-display-name companion — `components/benchmark-detail.tsx:308-336`
-
-`benchmark-detail.tsx` carries its own `SUITE_DISPLAY_NAMES` table (an exact 30-entry copy of `BENCHMARK_NAMES`) plus two further override tables (`DISPLAY_TOKEN_OVERRIDES`, `DISPLAY_NAME_OVERRIDES`) and a different normalize/tokenize pipeline (`normalizeDisplayLabel`/`normalizeDisplayToken`). The suite-name path:
-
-1. `normalizeSuiteKey(key)` collapses `[-.\s]+` → `_`, strips edge `_`, then applies two regex special-cases: `/^fibble\d*_arena$/` collapses to `fibble_arena`, `/^arc_agi_v\d+/` collapses to `arc_agi`.
-2. `getSuiteDisplayName(key)` returns `SUITE_DISPLAY_NAMES[normalizedKey] ?? normalizeDisplayLabel(key)`.
-3. The fallback (`normalizeDisplayLabel`) is more sophisticated than `humanizeToken`: it splits on `/`, then on whitespace, then per-token applies `DISPLAY_TOKEN_OVERRIDES` (a 26-entry table that knows acronyms like `mmlu → MMLU`, `helm → HELM`, `gpt → GPT`).
-
-The benchmark-detail suite-name path is **not** the same function as `getBenchmarkDisplayName` — it is consumed only inside `benchmark-detail.tsx` for rendering suite headers. The model-detail path (`lib/model-data.ts → getBenchmarkDisplayName`) is the one consumed across the rest of the app (model cards, comparison index, eval rollups, DuckDB backend).
-
-### Duplicate implementation — `lib/eval-processing.ts:861-885`
-
-A second function with the same name `getBenchmarkDisplayName` lives in `lib/eval-processing.ts`. It uses a completely different rule:
-
-```
-const mapping = { 'MMLU': 'Massive Multitask Language Understanding', 'MMLU-Pro': 'MMLU Professional', ... }
-for (const [key, value] of Object.entries(mapping)) {
-  if (name.toUpperCase().includes(key.toUpperCase())) return value
-}
-return name
-```
-
-It does case-insensitive substring matching against a 10-entry mapping of long-form descriptive names. **It disagrees with the `lib/model-data.ts` version on every map hit** (e.g. for input `"MMLU"`, model-data returns `"Mmlu"` via fallback, eval-processing returns `"Massive Multitask Language Understanding"`). Reachability:
-
-- `lib/eval-processing.ts:903` — `getBenchmarkDisplayName(compositeBenchmarkKey)` inside `groupEvaluationsByBenchmark`. `groupEvaluationsByBenchmark` is exported but is **not imported anywhere else in the repo** (verified by `rg "groupEvaluationsByBenchmark"` — only its own declaration appears). Functionally dead.
-
-The substring-include rule has a known soft-bug: input `"MMLU-Pro"` would return the mapping for `"MMLU"` (`"Massive Multitask Language Understanding"`) because the loop iterates in insertion order and `MMLU` comes first; the `MMLU-Pro` entry never wins. Documented as TS-as-is; do not "fix" this in the migration.
-
-## Classification
-
-- **Unconditional normalization.** The function always runs on whatever benchmark key/name is present — it does not check for a pre-existing canonical field. Map hits and tokenize fallback are both branches of the same unconditional transform. Pipeline-side fix: emit a canonical `display_name` field per benchmark; no consumer should re-derive it.
-- **Cleaning → pipeline.** Pure value transform on a single string field. No aggregation or record merging. Migration target: pipeline emits canonical display name on every benchmark/eval entry; TS map + `getBenchmarkDisplayName` + the duplicate in `eval-processing.ts` + the parallel `SUITE_DISPLAY_NAMES` table in `benchmark-detail.tsx` all delete.
-
-## Inputs and expected outputs
-
-Each row corresponds to a parameterized test case in `tests/transformations/benchmark-display-names.test.ts`.
-
-### Group A — Map hits (normalized-key lookup against `BENCHMARK_NAMES`)
-
-| Input | Output | Rule |
-|---|---|---|
-| `helm_lite` | `HELM Lite` | exact normalized match |
-| `HELM Lite` | `HELM Lite` | normalize: lower + space→`_` → `helm_lite` |
-| `helm-lite` | `HELM Lite` | normalize: dash→`_` |
-| `helm.lite` | `HELM Lite` | normalize: dot→`_` |
-| `  helm   lite  ` | `HELM Lite` | normalize: whitespace runs → `_`, trim edges |
-| `arc_agi` | `ARC-AGI` | substantive transform: returns dashed form |
-| `swe_bench` | `SWE-bench` | substantive (lowercase `bench`) |
-| `reward_bench` | `RewardBench` | substantive (concatenated, no separator) |
-| `reward_bench_2` | `RewardBench 2` | substantive |
-| `terminal_bench_2_0` | `Terminal Bench 2.0` | substantive (literal `_0` becomes `.0` in output) |
-| `mistralai` (n/a — not a benchmark) | — | not in benchmark map |
-| `hfopenllm_v2` | `HF Open LLM v2` | substantive (3-token split) |
-| `bfcl` | `BFCL` | uppercase |
-| `ace` | `ACE` | uppercase |
-| `apex_agents` | `APEX Agents` | partial uppercase |
-| `apex_v1` | `APEX v1` | partial uppercase + lowercase v |
-| `appworld` | `AppWorld` | mixed-case substantive |
-| `browsecompplus` | `BrowseComp+` | adds `+` |
-| `livecodebenchpro` | `LiveCodeBench Pro` | substantive |
-| `sciarena` | `SciArena` | substantive |
-| `la_leaderboard` | `LA Leaderboard` | partial uppercase |
-| `theory_of_mind` | `Theory of Mind` | substantive (lowercase `of`) |
-| `fibble_arena` | `Fibble Arena` | base entry |
-| `fibble1_arena` | `Fibble Arena v1` | substantive |
-| `fibble2_arena` | `Fibble Arena v2` | substantive |
-| `fibble3_arena` | `Fibble Arena v3` | substantive |
-| `fibble4_arena` | `Fibble Arena v4` | substantive |
-| `fibble5_arena` | `Fibble Arena v5` | substantive |
-| `wordle_arena` | `Wordle Arena` | substantive |
-| `global_mmlu_lite` | `Global MMLU Lite` | partial uppercase |
-| `helm_capabilities` | `HELM Capabilities` | partial uppercase |
-| `helm_classic` | `HELM Classic` | partial uppercase |
-| `helm_instruct` | `HELM Instruct` | partial uppercase |
-| `helm_mmlu` | `HELM MMLU` | full uppercase |
-| `tau_bench_2` | `TAU-Bench 2` | substantive (`TAU-Bench`, dashed) |
-
-### Group B — Tokenize fallback (`humanizeToken`)
-
-Inputs that miss the map go through `humanizeToken(originalInput)`: split on `[_-]+`, drop empty parts, uppercase the first char of each part, join with `" "`.
-
-| Input | Output | Why |
-|---|---|---|
-| `bbh` | `Bbh` | single token, only first char uppercased — NOT `BBH` |
-| `gpqa` | `Gpqa` | only first char uppercased — NOT `GPQA` |
-| `mmlu` | `Mmlu` | NOT `MMLU` (this is what's served when only the model-data path runs) |
-| `gsm8k` | `Gsm8k` | digits inside don't capitalize differently |
-| `MATH` | `MATH` | already uppercase, untouched (only first-char of each token is *set* — but `M` is already upper) |
-| `MMLU-PRO` | `MMLU PRO` | split on dash; each token already starts upper; rest of token preserved as-is |
-| `MMLU` | `MMLU` | passthrough — already starts uppercase, fallback's `charAt(0).toUpperCase()` is a no-op on `M`, slice preserves `MLU` |
-| `helm air bench` | `HELM Lite` (NO!) — wait | normalizes to `helm_air_bench`, not in map → tokenize fallback uses ORIGINAL `helm air bench` → split on `[_-]+` is single token `helm air bench` → `Helm air bench` |
-| `Helm air bench` | `Helm air bench` | NOT in map (only `helm_air_bench` would be — and isn't); fallback splits on `[_-]+` only, so the spaces survive and only the first char gets uppercased |
-| `humaneval` | `Humaneval` | not in map |
-| `truthfulqa` | `Truthfulqa` | not in map |
-| `BBQ` | `BBQ` | not in map; split on `[_-]+` is single `BBQ`; first char already upper, rest preserved |
-| `swe-bench-verified` | `Swe Bench Verified` | not in map; split on `-` → 3 tokens, each first-cap |
-| `swe_bench_verified_mini` | `Swe Bench Verified Mini` | split on `_` → 4 tokens |
-| `multi_swe_bench` | `Multi Swe Bench` | split on `_` → 3 tokens |
-| `helm_air_bench` | `Helm Air Bench` | not in map (only the *suite* keys above are); split on `_` → 3 tokens |
-| `helm_safety` | `Helm Safety` | not in map |
-| `swe_bench_verified` | `Swe Bench Verified` | not in map (only `swe_bench` is) |
-| `cocoabench` | `Cocoabench` | single token |
-| `llm_stats` | `Llm Stats` | not in map |
-| `artificial_analysis_llms` | `Artificial Analysis Llms` | not in map |
-
-The **systematic quirk**: `humanizeToken` only uppercases the first character of each token. It does NOT consult the same acronym table that the map encodes (so `mmlu → Mmlu`, `bbh → Bbh`, `gpqa → Gpqa`). This is why suites like `helm_lite` need an explicit map entry — without one, fallback would produce `Helm Lite` (already pretty close), but for `mmlu` the fallback produces the visibly-wrong `Mmlu`. The companion table in `benchmark-detail.tsx` (`DISPLAY_TOKEN_OVERRIDES`) DOES know the acronyms — but that table is consumed by a different code path.
-
-### Group C — Edge cases
-
-| Input | Output | Notes |
-|---|---|---|
-| `""` (empty) | `""` | normalize → `""`, no map hit; humanizeToken splits empty → empty array → `[].join(" ")` → `""` |
-| `"_"` | `""` | normalize → `""` (edge `_` stripped), no map hit; humanizeToken splits `_` → `[""]` → filter empty → `[].join(" ")` → `""` |
-| `"___helm___lite___"` | `HELM Lite` | normalize collapses runs of `_` and trims → `helm_lite` → map hit |
-| `"HELM-LITE"` | `HELM Lite` | normalize → `helm_lite` → map hit |
-| `"helm  lite"` (2 spaces) | `HELM Lite` | normalize collapses whitespace runs → `helm_lite` |
-| `"a"` | `A` | not in map; humanizeToken → `["a"]` → `["A"]` → `"A"` |
-| `"a-b"` | `A B` | split → `["a","b"]` → `["A","B"]` → `"A B"` |
-
-### Group D — Duplicate `getBenchmarkDisplayName` in `lib/eval-processing.ts` (functionally dead, document for completeness)
-
-The substring-include rule (10-entry mapping). Documented inputs:
-
-| Input | Output | Rule branch |
-|---|---|---|
-| `null` | `Unknown Benchmark` | guard at top of function |
-| `undefined` | `Unknown Benchmark` | guard |
-| `""` | `Unknown Benchmark` | guard (`!name`) |
-| `MMLU` | `Massive Multitask Language Understanding` | substring match on `MMLU` |
-| `mmlu` | `Massive Multitask Language Understanding` | case-insensitive substring |
-| `MMLU-Pro` | `Massive Multitask Language Understanding` | iteration order: `MMLU` matches first; `MMLU-Pro` never reached |
-| `GSM8K` | `Grade School Math 8K` | match |
-| `HumanEval` | `Human Eval (Code)` | match |
-| `MBPP` | `Mostly Basic Python Problems` | match |
-| `HellaSwag` | `HellaSwag (Commonsense)` | match |
-| `ARC` | `AI2 Reasoning Challenge` | match |
-| `TruthfulQA` | `TruthfulQA` | match (key === value) |
-| `BBH` | `Big-Bench Hard` | match |
-| `MATH` | `MATH Dataset` | match |
-| `helm_lite` | `helm_lite` | no match → passthrough |
-| `MMLU Lite something` | `Massive Multitask Language Understanding` | substring match still fires anywhere in the name |
-
-Note that this function is **not the active path** in production; it lives inside `groupEvaluationsByBenchmark` which is unreferenced. Tests cover it for completeness so a pipeline implementer porting both functions sees the divergence in behaviour explicitly.
-
-## Current TS implementation
-
-| Concern | Location |
-|---|---|
-| Active map (`BENCHMARK_NAMES`, 30 entries) | `lib/model-data.ts:109-140` |
-| Active key-normalizer (`normalizeBenchmarkKeyForLookup`) | `lib/model-data.ts:142-144` |
-| Active tokenize fallback (`humanizeToken`) | `lib/model-data.ts:90-96` |
-| Active function (`getBenchmarkDisplayName`, exported) | `lib/model-data.ts:146-148` |
-| Suite-name companion map (`SUITE_DISPLAY_NAMES`, 30 entries; copy of `BENCHMARK_NAMES`) | `components/benchmark-detail.tsx:101-132` |
-| Suite-name token overrides (`DISPLAY_TOKEN_OVERRIDES`, 26 entries) | `components/benchmark-detail.tsx:134-162` |
-| Suite-name overrides (`DISPLAY_NAME_OVERRIDES`) | `components/benchmark-detail.tsx:164-173` |
-| Suite key normalizer (with fibble/arc-agi regex special-cases) | `components/benchmark-detail.tsx:308-313` |
-| Suite display-name lookup | `components/benchmark-detail.tsx:333-336` |
-| **Duplicate** function (functionally dead) | `lib/eval-processing.ts:861-885` |
-
-### Call sites of the active `getBenchmarkDisplayName` (15 total)
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:274` | `top_scores` rollup — set `benchmark` display name on score entry |
-| `lib/model-data.ts:410` | `benchmark_names` array on developer summary |
-| `lib/model-data.ts:459` | `benchmarkDisplayName` for hierarchy entries |
-| `lib/model-data.ts:485` | `latest_source_name` on category aggregation |
-| `lib/model-data.ts:778` | `composite_benchmark_name` on category-mode aggregation |
-| `lib/model-data.ts:785` | `latest_source_name` (same record) |
-| `lib/model-data.ts:821` | `composite_benchmark_name` on benchmark-mode aggregation |
-| `lib/model-data.ts:828` | `latest_source_name` (same record) |
-| `lib/model-data.ts:903` | `suiteDisplayName` for suite aggregation |
-| `lib/model-data.ts:1056` | `suiteDisplayName` (second aggregator) |
-| `lib/model-data.ts:1362` | model-card rollup A |
-| `lib/model-data.ts:1396` | model-card rollup B |
-| `lib/model-data.ts:1429` | model-card rollup C |
-| `lib/model-data.ts:1470` | model-card rollup D |
-| `lib/duckdb-data.ts:301` | DuckDB backend — set `benchmark` on per-model rollup |
-
-`normalizeBenchmarkKeyForLookup` itself is also called separately at `lib/model-data.ts:1572` and `:1579` to derive suite-key matches (independent of display-name derivation).
-
-### Call sites of `SUITE_DISPLAY_NAMES`/`normalizeDisplayLabel` (renderer-only)
-
-`components/benchmark-detail.tsx` consumes `normalizeDisplayLabel` at ~30 sites for in-render labels (model name, organization, dataset name, source name, run label, subtask label, etc.). The suite-display-name path (`getSuiteDisplayName`) is only called inside this file. None of these are used outside the benchmark detail page; they are presentation-layer helpers that operate on already-emitted strings.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where display name is derived | request time, in 15+ call sites | pipeline emits `benchmark_parent_name`, `benchmark_family_name`, `display_name`, `canonical_display_name` on each eval entry; raw `benchmark` is also present | TS re-derives display name from the key field, ignoring the pipeline's already-canonical `*_name` fields |
-| Key field consumed | `benchmark_parent_key` / `benchmark_family_key` / `benchmark` (mostly key-shaped strings like `helm_lite`) | n/a — pipeline emits both keys and names | TS map hit yields canonical name; non-mapped keys fall through to mechanical title-case |
-| Acronym handling | `BENCHMARK_NAMES` map: 30 hand-curated entries; everything else gets `humanizeToken` (only first char per token uppercased) | `display_name` / `canonical_display_name` already encode the canonical capitalization (e.g. `BBH`, `GPQA`, `MMLU`) | TS produces user-visible `Mmlu` / `Bbh` / `Gpqa` for unmapped acronym keys; pipeline's `display_name` would have correct casing |
-| Suite/family rollup labels | `getSuiteDisplayName` in `benchmark-detail.tsx` does its own thing (DISPLAY_TOKEN_OVERRIDES knows acronyms); active `getBenchmarkDisplayName` in `model-data.ts` does NOT consult those overrides | n/a | The two TS paths can produce *different* display names for the same key — e.g. for `mmlu`, `getBenchmarkDisplayName` returns `Mmlu` but the suite path returns `MMLU` |
-
-### Concrete worked examples (audit numbers)
-
-Audited 2026-04-28 against `.cache/hf-data/eval-list.json` (587 evals) and `.cache/hf-data/model-cards-lite.json` (5,830 cards) by `scripts/verify-benchmark-display-names.mjs`.
-
-**Distinct benchmark-key strings in production (eval-list.json):**
-
-| Field | Distinct values | mapHit (distinct) | fallback (distinct) | mapHit calls / 587 | fallback calls / 587 |
-|---|---|---|---|---|---|
-| `benchmark` | 544 | 15 (2.8%) | 529 (97.2%) | 20 | 567 |
-| `benchmark_parent_key` | 34 | 25 (73.5%) | 9 (26.5%) | 71 | 516 |
-| `benchmark_family_key` | 34 | 24 (70.6%) | 10 (29.4%) | 65 | 522 |
-| `benchmark_parent_name` | 544 | 15 (2.8%) | 529 (97.2%) | 20 | 567 |
-
-**Distinct benchmark fields on model-cards-lite.json (5,830 cards):**
-
-| Field | Distinct | mapHit | fallback |
-|---|---|---|---|
-| `card.benchmark_names[]` | 377 | 14 (3.7%) | 363 (96.3%) |
-| `card.top_benchmark_scores[].benchmarkKey` | 339 | 17 (5.0%) | 322 (95.0%) |
-| `card.top_benchmark_scores[].benchmark` | 301 | 13 (4.3%) | 288 (95.7%) |
-
-**Notable fallback outputs (visibly-wrong style produced by `humanizeToken`):**
-
-| Input | TS-computed |
-|---|---|
-| `BBH` | `BBH` (no-op — already first-cap) |
-| `MMLU-PRO` | `MMLU PRO` (loses the dash) |
-| `artificial_analysis_llms` | `Artificial Analysis Llms` (`LLMs` → `Llms`) |
-| `helm_air_bench` | `Helm Air Bench` (`HELM` → `Helm`) |
-| `helm_safety` | `Helm Safety` |
-| `swe_bench_verified` | `Swe Bench Verified` (`SWE` → `Swe`) |
-| `swe_bench_verified_mini` | `Swe Bench Verified Mini` |
-| `multi_swe_bench` | `Multi Swe Bench` |
-| `llm_stats` | `Llm Stats` |
-| `hfopenllm` (family key) | `Hfopenllm` |
-| `ARC-AGI v2` (from `benchmark_names[]`) | `ARC AGI v2` (loses the dash) |
-| `BrowseComp-Plus` | `BrowseComp Plus` (loses the dash) |
-
-**TS vs pipeline-emitted display fields (587 evals):**
-
-| Comparison | Agree | Disagree | Notes |
-|---|---|---|---|
-| TS(`benchmark_parent_key`) == pipeline `benchmark_parent_name` | 9 | 578 | Pipeline's `*_name` is the per-eval display name (e.g. `BBH`), not the suite roll-up name. They aren't meant to match — TS path produces the suite name (`HF Open LLM v2`), pipeline produces the leaf eval name (`BBH`). |
-| TS(`benchmark_family_key`) == pipeline `benchmark_family_name` | 8 | 579 | Same dynamic. |
-| TS(`benchmark`) == pipeline `display_name` | 86 | 501 | Disagreements include `MMLU-PRO` → `MMLU PRO` and the "key vs leaf-eval display" mismatch as above (e.g. `Artificial Analysis LLM API` vs `artificial_analysis.median_output_tokens_per_second`). |
-| TS(`benchmark`) == pipeline `canonical_display_name` | 86 | 501 | Same as `display_name`. |
-
-**Divergence summary:**
-- Only ~3% of distinct `benchmark` strings hit the map; ~74% of distinct suite keys (`benchmark_parent_key`) do.
-- For the ~97% of `benchmark` strings that miss, the active TS function passes them through `humanizeToken` which mangles acronyms (`MMLU-PRO` → `MMLU PRO`). For inputs that are already nicely cased (most pipeline-emitted `benchmark` strings are), the fallback can be a strict regression vs the input.
-- The pipeline's `display_name` and `canonical_display_name` already provide leaf-eval display names; the TS function is doing suite-key → suite-display-name work that pipeline does NOT yet emit (no `parent_display_name` field). The `benchmark_parent_name` field exists but holds the *leaf eval name* picked from one child, not the suite display name.
-- The duplicate `getBenchmarkDisplayName` in `eval-processing.ts` is unreachable — `groupEvaluationsByBenchmark` has zero importers (verified by ripgrep).
-
-Run `scripts/verify-benchmark-display-names.mjs` for the live numbers.
-
-## Notes for pipeline implementer
-
-- **Prefer to expose pre-computed `display_name` / `canonical_display_name` on every benchmark/eval entry rather than asking consumers to map keys → names.** Pipeline already does this for ~all entries (verify field coverage with the audit script). The TS layer is essentially defending against the historical case where consumers received only a snake_case key.
-- If the pipeline keeps emitting both keys and names, the migration target is: TS callers read `entry.display_name` (or `benchmark_parent_name`, etc.) directly; the `BENCHMARK_NAMES` map + `getBenchmarkDisplayName` + `humanizeToken` (active) + `SUITE_DISPLAY_NAMES`/`normalizeSuiteKey`/`getSuiteDisplayName` (companion) all delete.
-- The 30 entries in `BENCHMARK_NAMES` (and the parallel 30 in `SUITE_DISPLAY_NAMES`) encode product decisions about how to render the suite-level rollups. If the pipeline does not yet emit a *suite-level* display name (separate from per-eval `display_name`), it should — exactly the 30 entries above are the acceptance set.
-- The two regex special-cases in `benchmark-detail.tsx` (`/^fibble\d*_arena$/ → fibble_arena`, `/^arc_agi_v\d+/ → arc_agi`) are normalization-layer rules — pipeline should fold versioned variants of these suites into the same canonical key OR the consumer must continue to apply the collapse. Document the chosen approach.
-- The duplicate `getBenchmarkDisplayName` in `lib/eval-processing.ts` should be deleted along with `groupEvaluationsByBenchmark`. It has no live callers.
-- The `benchmark-detail.tsx` file's many `normalizeDisplayLabel` call sites (model name, org name, dataset name, run label, etc.) are a separate concern — they are presentation-only normalization on already-emitted strings, not a key→name lookup. Whether to migrate them upstream is a separate item.
-
-Verification: run `scripts/verify-benchmark-display-names.mjs` against pipeline output once it ships. Goal: zero divergence vs TS-as-is across every distinct `benchmark` / `benchmark_parent_key` / `benchmark_family_key` value in the 587-eval cache.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/benchmark-display-names.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits `display_name` / `canonical_display_name` (already does on per-eval) PLUS suite-level display name covering the 30 `BENCHMARK_NAMES` entries on every benchmark/eval entry
-- [ ] TS deleted; replace 15 active call sites + 30+ `normalizeDisplayLabel` sites in `benchmark-detail.tsx` with direct field reads. Delete `BENCHMARK_NAMES`, `getBenchmarkDisplayName` (model-data.ts), `humanizeToken`, `normalizeBenchmarkKeyForLookup`, `SUITE_DISPLAY_NAMES`, `DISPLAY_TOKEN_OVERRIDES`, `DISPLAY_NAME_OVERRIDES`, `normalizeSuiteKey`, `getSuiteDisplayName`, `normalizeDisplayLabel`, `normalizeDisplayToken`, plus the duplicate `getBenchmarkDisplayName` + `groupEvaluationsByBenchmark` in `eval-processing.ts`.
-
-## Future product decision (deferred)
-
-`BENCHMARK_NAMES` is hand-curated and only covers 30 suite/parent keys; many leaf benchmarks fall through to a fallback that mangles acronyms (`mmlu → Mmlu`, `bbh → Bbh`). Whether to (a) expand the map to cover the long tail, (b) ship pipeline-emitted `display_name` everywhere and delete the map entirely, or (c) take a different approach (HTML-style override file, attribute on the source eval, etc.) is out of scope for this refactor. Document-don't-improve.

notes/transformations/09-metric-display-name-expansion.md

@@ -1,282 +0,0 @@
-# Metric display name expansion
-
-Drafted 2026-04-28. Migration item #10 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. Originally both functions in this spec were claimed to be "defensive scaffolding firing 0 times against current data." That claim was **partially wrong** and is corrected below (verified 2026-04-28).
-
-This spec covers two related functions with the same product intent (expand a non-informative metric name by prefixing the benchmark), but they have very different statuses in the current codebase:
-
-1. **`getEvaluationDisplayName`** (`lib/eval-processing.ts:70-86`) — **DEAD CODE via orphaned caller chain.** Its callers (`createEvaluationCard` line 537, `groupEvaluationsByBenchmark` line 893) are exported from `lib/eval-processing.ts` but never invoked from `app/`, `components/`, or any other `lib/` file. The function never runs in production. (The earlier audit's "0 fires" was correct in result but incorrect in reasoning — it audited `evaluations_by_category` from cache files, which is pipeline-pre-flattened. The function actually consumes the *post-`flattenModelEvaluations`* shape, where bare-generic names ARE present.)
-2. **`prefersBenchmarkName`** (`lib/model-data.ts:462-470`) — **active code path that fires 0 times against current data.** It runs inside `hfEvalEntryToListItem`, which is called from `lib/model-data.ts:1261, 1301` and `lib/duckdb-data.ts:171` — all live read paths. None of the 4 heuristic patterns match any of the 587 eval-list entries in production.
-
-This split matters because the migration recommendations are different for each. See "Recommended migration path" below.
-
-## Rule (as TS implements it today)
-
-### Rule 1 — `getEvaluationDisplayName(evaluation, result)`
-
-Computes the display string for one `evaluation_result` row.
-
-```
-benchmarkName = getBenchmarkName(evaluation, result)
-metricName    = result.evaluation_name.trim()
-
-if metricName === benchmarkName:           return metricName       // already redundant; show once
-if GENERIC_EVALUATION_NAMES.has(metricName.toLowerCase()):
-                                            return `${benchmarkName} - ${metricName}`
-otherwise:                                  return metricName
-```
-
-`GENERIC_EVALUATION_NAMES` is a 6-entry lowercase-keyed set (`lib/eval-processing.ts:27-34`):
-
-| Key |
-|---|
-| `score` |
-| `accuracy` |
-| `mean win rate` |
-| `exact match` |
-| `f1` |
-| `pass@1` |
-
-`getBenchmarkName` (`lib/eval-processing.ts:49-68`) resolves the benchmark string with this precedence:
-
-1. `result.source_data.dataset_name` (when source_data is an object, not an array)
-2. `evaluation.benchmark`
-3. `evaluation.source_data.dataset_name` (when source_data is an object)
-4. `result.evaluation_name`
-5. `evaluation.evaluation_id`
-
-### Rule 2 — `prefersBenchmarkName` (inline, `lib/model-data.ts:459-470`)
-
-Decides whether to substitute the benchmark display name for an eval-list entry's display string.
-
-```
-benchmarkDisplayName = getBenchmarkDisplayName(entry.benchmark_parent_name || entry.benchmark || "")
-rawDisplayName       = entry.evaluation_name || entry.display_name || entry.benchmark_leaf_name || entry.eval_summary_id
-normalized           = rawDisplayName.trim().toLowerCase()
-
-prefersBenchmarkName = Boolean(benchmarkDisplayName) && (
-  normalized.startsWith("accuracy on ") ||
-  normalized.startsWith("score on ")    ||
-  normalized.includes("for scorer")     ||
-  normalized.includes("model_graded")
-)
-
-evaluation_name on output = prefersBenchmarkName ? benchmarkDisplayName : rawDisplayName
-```
-
-Note the asymmetry: the first two checks are `startsWith`, the second two are `includes`. This is faithful to the TS code (not "fixed" here).
-
-## Classification
-
-- **Unconditional normalization.** Both rules always run on whatever metric/eval string is present; neither defers to a pre-existing canonical field. Pipeline-side fix: emit the final `display_name` already in expanded form (or leave it as-is for the cases where neither rule fires — i.e. all 86,183 production rows today). No consumer should re-derive.
-- **Cleaning → pipeline.** Pure value transform on a single field per record. No aggregation, no joining, no record merging. Migration target: pipeline emits `display_name` already in the form TS would produce; TS deletes both helpers and inlines a direct field read.
-
-## Inputs and expected outputs
-
-Each row corresponds to a parameterized test case in `tests/transformations/metric-display-name-expansion.test.ts`.
-
-### Group A — `getEvaluationDisplayName`: generic name expansion
-
-Input: `(evaluation, result)` synthesized so `getBenchmarkName` resolves to the value in the "benchmark" column.
-
-| benchmark | result.evaluation_name | Output | Why |
-|---|---|---|---|
-| `MMLU` | `Accuracy` | `MMLU - Accuracy` | metric `accuracy` is generic → prefix benchmark |
-| `GSM8K` | `accuracy` | `GSM8K - accuracy` | lowercased generic still triggers; output keeps original casing of metric |
-| `MATH` | `EXACT MATCH` | `MATH - EXACT MATCH` | uppercase generic still triggers (set check is `.toLowerCase()`) |
-| `RewardBench` | `Mean Win Rate` | `RewardBench - Mean Win Rate` | "mean win rate" is in the set |
-| `HumanEval` | `pass@1` | `HumanEval - pass@1` | symbol-bearing generic still in the set |
-| `SuperGLUE` | `f1` | `SuperGLUE - f1` | shortest generic |
-| `OpenBookQA` | `Score` | `OpenBookQA - Score` | "score" is generic |
-
-### Group B — `getEvaluationDisplayName`: passthrough (non-generic)
-
-| benchmark | result.evaluation_name | Output | Why |
-|---|---|---|---|
-| `MMLU` | `MMLU` | `MMLU` | metricName === benchmarkName → return as-is (early return; expansion never considered) |
-| `MMLU` | `BLEU` | `BLEU` | not in generic set → passthrough |
-| `RewardBench` | `Chat Hard` | `Chat Hard` | not generic, distinct from benchmark → passthrough |
-| `MMLU` | `accuracy_strict` | `accuracy_strict` | substring of "accuracy" but not equal → not in set → passthrough |
-| `MMLU` | `Accuracy ` (trailing space) | `MMLU - Accuracy` | `.trim()` on metricName before set lookup → matches |
-| `MMLU` | `   accuracy   ` | `MMLU - accuracy` | trim happens to metricName |
-
-### Group C — `getEvaluationDisplayName`: `getBenchmarkName` precedence
-
-These exercise the resolution chain that feeds the rule.
-
-| Setup | Resolved benchmark | Why |
-|---|---|---|
-| `result.source_data = { dataset_name: "RewardBench" }`, `evaluation.benchmark = "reward-bench"` | `RewardBench` | result.source_data.dataset_name wins (precedence #1) |
-| `result.source_data = ["url1", "url2"]` (array), `evaluation.benchmark = "reward-bench"` | `reward-bench` | array source_data is skipped → falls to `evaluation.benchmark` |
-| `result.source_data = undefined`, `evaluation.benchmark = "reward-bench"` | `reward-bench` | evaluation.benchmark (precedence #2) |
-| `result.source_data = undefined`, `evaluation.benchmark = ""`, `evaluation.source_data = { dataset_name: "MMLU" }` | `MMLU` | evaluation.source_data.dataset_name (precedence #3) — note empty string is falsy |
-| All sources empty, `result.evaluation_name = "Foo"` | `Foo` | precedence #4 |
-| All empty, `evaluation.evaluation_id = "id-123"` | `id-123` | precedence #5 (final fallback) |
-
-### Group D — `prefersBenchmarkName`: heuristic matches
-
-For each, the eval-list entry has `benchmark_parent_name = "MMLU"` (so `benchmarkDisplayName` resolves to a non-empty string).
-
-| `evaluation_name` (input) | Output `evaluation_name` | Why |
-|---|---|---|
-| `accuracy on subset_humanities` | `MMLU` | `startsWith("accuracy on ")` |
-| `Accuracy On SubsetHumanities` | `MMLU` | normalized to lowercase before startsWith |
-| `score on test_set` | `MMLU` | `startsWith("score on ")` |
-| `xyz for scorer judge_v2` | `MMLU` | `includes("for scorer")` (anywhere in string) |
-| `for scorer xyz at start` | `MMLU` | `includes("for scorer")` matches at start too |
-| `something model_graded thing` | `MMLU` | `includes("model_graded")` (underscore, not space) |
-| `model_graded` | `MMLU` | substring match works on the whole string |
-
-### Group E — `prefersBenchmarkName`: passthrough
-
-| `evaluation_name` | Output | Why |
-|---|---|---|
-| `MMLU - Accuracy` | `MMLU - Accuracy` | does not start with "accuracy on " (has prefix); no other token matches |
-| `Accuracy` | `Accuracy` | bare "accuracy" doesn't satisfy `startsWith("accuracy on ")` (no " on ") |
-| `accuracy onset` | `accuracy onset` | "accuracy on" with no trailing space; the rule literal is `"accuracy on "` (note trailing space) — but "accuracy onset".startsWith("accuracy on ") is **false** because position 11 is "s" not " ". Good — no match. |
-| `score onyx` | `score onyx` | `startsWith("score on ")` requires literal trailing space — "onyx" fails |
-| `Model Graded Eval` | `Model Graded Eval` | `model_graded` (underscore) does not match "Model Graded" (space) after lowercasing → "model graded eval" does not contain "model_graded" |
-| `accuracy_for_scorer` | `accuracy_for_scorer` | `for scorer` (with space) does not match `for_scorer` after lowercasing — "accuracy_for_scorer" does not contain "for scorer" |
-| `Scorer based eval` | `Scorer based eval` | "scorer based eval" does not contain "for scorer" |
-| `BBH` | `BBH` | none of the four conditions match |
-
-### Group F — `prefersBenchmarkName`: `benchmarkDisplayName` empty short-circuits
-
-| Setup | Output | Why |
-|---|---|---|
-| `benchmark_parent_name = ""`, `benchmark = ""`, `evaluation_name = "accuracy on x"` | `accuracy on x` (raw) | `Boolean(benchmarkDisplayName)` is false → `prefersBenchmarkName = false` → falls back to raw |
-
-### Group G — `prefersBenchmarkName`: `rawDisplayName` precedence
-
-Order: `entry.evaluation_name` → `entry.display_name` → `entry.benchmark_leaf_name` → `entry.eval_summary_id`.
-
-| Setup | rawDisplayName | Why |
-|---|---|---|
-| `evaluation_name = "score on x"` | `score on x` | first non-falsy field |
-| `evaluation_name = ""`, `display_name = "MMLU"` | `MMLU` | empty string is falsy → falls through |
-| All empty except `eval_summary_id = "id_xyz"` | `id_xyz` | final fallback |
-
-## Current TS implementation
-
-| Concern | Location |
-|---|---|
-| Generic-names set | `lib/eval-processing.ts:27-34` (`GENERIC_EVALUATION_NAMES`) |
-| Benchmark-name resolver | `lib/eval-processing.ts:49-68` (`getBenchmarkName`) |
-| Per-result expansion | `lib/eval-processing.ts:70-86` (`getEvaluationDisplayName`) |
-| Eval-list-entry heuristic | `lib/model-data.ts:459-470` (`prefersBenchmarkName`, inline; assigns to `evaluation_name` field) |
-| Benchmark display name (used by Rule 2) | `lib/model-data.ts:146-148` (`getBenchmarkDisplayName`) |
-
-### Call sites
-
-`getEvaluationDisplayName` (2 call sites, both internal to `lib/eval-processing.ts`):
-
-| Location | Context |
-|---|---|
-| `lib/eval-processing.ts:631` | `processModelEvaluations` — populates `allScores[].benchmark` for per-model score aggregation |
-| `lib/eval-processing.ts:900` | `groupEvaluationsByBenchmark` — populates `BenchmarkEvalSummary.evaluation_name` keyed by `eval_summary_id` |
-
-`prefersBenchmarkName` (1 call site, declared inline):
-
-| Location | Context |
-|---|---|
-| `lib/model-data.ts:462-470` | `hfEvalEntryToListItem` — sets `evaluation_name` on `BenchmarkEvalListItem` for the browse-evals list page |
-
-`GENERIC_EVALUATION_NAMES`: only consumed by `getEvaluationDisplayName` itself.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where expansion runs | request time, in 3 call sites total | not implemented as a transform; pipeline emits `display_name` / `evaluation_name` already in their final form | TS expansion logic exists but never fires against current pipeline output |
-| Generic-name detection | runtime check against 6-entry set | n/a — no metric in production has a bare-generic `evaluation_name` | no observable difference today |
-| Heuristic prefix detection | runtime regex/substring on 4 patterns | n/a — no eval-list entry has the patterns today | no observable difference today |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/`:
-
-**`getEvaluationDisplayName` against the 5,830 model files (86,183 total `(evaluation, result)` pairs):**
-- `metric === benchmark` early-return: **30,968 (35.9%)** — most rows hit this; `evaluation_name` is already identical to the resolved benchmark name, so the function just returns it
-- Generic-name expansion fires: **0 (0.0%)** — zero rows have a metric whose lowercased name is in `GENERIC_EVALUATION_NAMES`
-- Passthrough (non-generic, distinct from benchmark): **55,215 (64.1%)**
-
-Distribution of generic names hit in production: **empty.** The 6-entry set is dead code against current data.
-
-**`prefersBenchmarkName` against the 587 eval-list entries:**
-- `accuracy on …` matches: **0**
-- `score on …` matches: **0**
-- `for scorer` matches: **0**
-- `model_graded` matches: **0**
-- Total entries where heuristic fires: **0 (0.0%)**
-
-Both transformations are **defensive scaffolding** — preserved for shapes the pipeline used to or could produce, but the current corpus produces neither generic bare-metric names nor heuristic-matching display strings.
-
-Verified by `scripts/verify-metric-display-name.mjs`.
-
-## Verified state (2026-04-28)
-
-**For `getEvaluationDisplayName`:**
-- Caller chain trace: called from `createEvaluationCard` (`lib/eval-processing.ts:631`) and `groupEvaluationsByBenchmark` (`lib/eval-processing.ts:900`). Both functions are exported from `lib/eval-processing.ts` but **not called from any file in `app/`, `components/`, `scripts/`, or other `lib/`** — verified by `grep -r`. The full chain `processEvaluationsToCards → createEvaluationCard → getEvaluationDisplayName` and `processEvaluationsToBenchmarkSummaries → groupEvaluationsByBenchmark → getEvaluationDisplayName` runs only inside the module's exports; no consumer triggers it.
-- Data check: pipeline DOES emit bare-generic `metric_name` in 38,140 of 82,781 metrics in `hierarchy_by_category` (~46%). `flattenModelEvaluations` in `lib/hf-data.ts:1275` propagates this to `evaluation_name` on flattened result rows. So *if* the function were called, the expansion path WOULD fire — on 39,831 of 86,183 result rows. But nothing calls it.
-- The earlier audit script (`scripts/verify-metric-display-name.mjs`) walked `models/<id>.json`'s `evaluations_by_category` (pipeline-pre-flattened, specific names) — that path doesn't go through `flattenModelEvaluations`, so it correctly reported "0 fires" for that traversal, but it didn't capture that the function would fire on the `hierarchy_by_category` traversal that `flattenModelEvaluations` actually performs.
-
-**For `prefersBenchmarkName`:**
-- Caller chain trace: lives inline at `lib/model-data.ts:462-470` inside `hfEvalEntryToListItem`. That function IS actively called: `lib/model-data.ts:1261, 1301` and `lib/duckdb-data.ts:171`. Live read path on browse-evals pages.
-- Data check: across all 587 eval-list entries in `.cache/hf-data/eval-list.json`, **none of the 4 heuristic patterns match** — verified directly with a one-liner script (count = 0). The active path runs on every request but never finds a match.
-
-## Recommended migration path
-
-Two separate decisions, one per function:
-
-### `getEvaluationDisplayName` (and the orphaned subsystem) — delete locally, no pipeline involvement
-
-The function and its caller chain are dead code. Safe to delete without any pipeline coordination:
-
-- `getEvaluationDisplayName` (`lib/eval-processing.ts:70-86`)
-- `GENERIC_EVALUATION_NAMES` (`lib/eval-processing.ts:27-34`)
-- `getEvaluationSummaryId` (`lib/eval-processing.ts:88-94`) — calls `getBenchmarkName`, only used by orphaned chain
-- `createEvaluationCard` (`lib/eval-processing.ts:537+`)
-- `processEvaluationsToCards` (`lib/eval-processing.ts:815`)
-- `processEvaluationsToBenchmarkSummaries` (`lib/eval-processing.ts:1000`)
-- `groupEvaluationsByBenchmark` (`lib/eval-processing.ts:893`)
-- `loadEvaluations` (`lib/eval-processing.ts:788`) — only called by the two `processEvaluations*` orphans
-- `getCategoryStats` (`lib/eval-processing.ts:747`) — verify via grep before deleting; may also be orphaned
-
-Plus the imports of `createEvaluationCard` and `groupEvaluationsByBenchmark` in `lib/model-data.ts:18, 20` (unused imports).
-
-**No contract test about bare-generic metric names** — pipeline emits them today, the contract would fail. The function was never preventing user-visible bugs because nothing called it. The active UI rendering path uses `metric.display_name` (which IS pipeline-pre-expanded as `"ACE / Score"`, `"RewardBench / Accuracy"`, etc.) — that's the field consumers actually read.
-
-**Verify-script update:** `scripts/verify-metric-display-name.mjs` is no longer meaningful for this function once it's deleted. Either delete the script or rewrite it to audit the `hierarchy_by_category` traversal (so the spec stays honest about what the function would do if revived).
-
-`getBenchmarkName` (`lib/eval-processing.ts:49-68`) has separate consumers and stays.
-
-### `prefersBenchmarkName` — delete locally + add contract test
-
-This one IS in an active path, fires 0 times, and matches the "implicit safety net → explicit contract test" pattern cleanly:
-
-- Delete the inline 9-line block at `lib/model-data.ts:462-470` (replace with direct read of the resolved display string).
-- Add a Tier A contract test in `tests/pipeline-contract.test.ts`:
-  - Assertion: no eval-list entry's display string (`evaluation_name || display_name || benchmark_leaf_name || eval_summary_id`) starts with `"accuracy on "` or `"score on "` (case-insensitive).
-  - Assertion: no eval-list entry's display string contains `"for scorer"` or `"model_graded"`.
-- If pipeline ever regresses, the contract test fails loudly with the specific eval_summary_id.
-
-Pipeline owner is told: "this 4-pattern absence is currently true; if you ever start emitting display strings in those shapes, please coordinate so the contract test is updated alongside the data change."
-
-## Migration checklist
-
-- [x] Spec written (corrected 2026-04-28)
-- [x] Tests cover each rule branch (`tests/transformations/metric-display-name-expansion.test.ts`) — note these test the function in isolation; they do not assert that the function is reached from any user-visible path.
-- [ ] Verify-script disposition decided (delete `scripts/verify-metric-display-name.mjs` or rewrite to audit the `hierarchy_by_category` traversal that reflects what the function would do if called)
-- [ ] Delete the orphaned subsystem from `lib/eval-processing.ts`: `getEvaluationDisplayName`, `GENERIC_EVALUATION_NAMES`, `getEvaluationSummaryId` (verify orphan status), `createEvaluationCard`, `processEvaluationsToCards`, `processEvaluationsToBenchmarkSummaries`, `groupEvaluationsByBenchmark`, `loadEvaluations`, `getCategoryStats` (verify orphan status). Plus unused imports of `createEvaluationCard` and `groupEvaluationsByBenchmark` in `lib/model-data.ts:18, 20`. Keep `getBenchmarkName` — separate consumers.
-- [ ] Delete `prefersBenchmarkName` block at `lib/model-data.ts:462-470` and replace with direct read of resolved display string.
-- [ ] Add Tier A contract test in `tests/pipeline-contract.test.ts`: no eval-list display string matches the 4 heuristic patterns (`startsWith("accuracy on ")`, `startsWith("score on ")`, `includes("for scorer")`, `includes("model_graded")`).
-- [ ] Notify pipeline owner: 4-pattern absence is currently true; coordinate before changing eval-list display string emission.
-
-## Future product decision (deferred)
-
-The defensive scaffolding only ever mattered for upstream data shapes that the pipeline no longer produces. If product wants to expand the generic-name set (e.g., add `"recall"`, `"precision"`, `"bleu"`) or the heuristic patterns (e.g., add `"judged by"`, `"with prompt"`), that's a separate decision; this spec just locks in TS-as-is.

notes/transformations/10-params-parsing.md

@@ -1,376 +0,0 @@
-# Params billions parsing
-
-Drafted 2026-04-28. Migration item #12 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. Five separate parameter-count parsers exist across the app, with subtly different unit grammars, fallback chains, and anchoring. They produce the same answer for the most common production inputs (clean `"7B"` / `"34.389"` style strings) but diverge sharply on edge cases. The migration target: emit a single canonical `params_billions` (numeric, billions) upstream so all five parsers can be deleted.
-
-## Rule (as TS implements it today — five variants)
-
-Five independent code paths convert a free-form parameter-count token into a billions-of-parameters number.
-
-### Variant A — `lib/model-data.ts:312-354` (`parseParamsBillions`)
-
-```ts
-function parseParamsBillions(value: unknown): number | null {
-  if (typeof value === "number") {
-    return Number.isFinite(value) && value > 0 ? value : null
-  }
-  if (typeof value !== "string") return null
-
-  const normalized = value.trim().toLowerCase()
-  if (!normalized) return null
-
-  const compact = normalized.replace(/,/g, "")
-  const tokenMatch = compact.match(/(\d+(?:\.\d+)?)\s*(trillion|tn|t|billion|bn|b|million|mn|m|thousand|k)\b/)
-  if (tokenMatch) {
-    const amount = Number.parseFloat(tokenMatch[1])
-    if (!Number.isFinite(amount) || amount <= 0) return null
-    const unit = tokenMatch[2]
-    if (unit === "trillion" || unit === "tn" || unit === "t") return amount * 1000
-    if (unit === "billion" || unit === "bn" || unit === "b") return amount
-    if (unit === "million" || unit === "mn" || unit === "m") return amount / 1000
-    if (unit === "thousand" || unit === "k") return amount / 1_000_000
-  }
-
-  const numeric = Number.parseFloat(compact)
-  return Number.isFinite(numeric) && numeric > 0 ? numeric : null
-}
-```
-
-- Polymorphic input (`unknown`); accepts `number` directly (positive only).
-- For strings: lowercases, strips commas, then scans for `<number><unit>` where unit ∈ {trillion, tn, t, billion, bn, b, million, mn, m, thousand, k}.
-- Unit table converts to billions; `t` → ×1000, `b` → as-is, `m` → ÷1000, `k` → ÷1_000_000.
-- Falls back to `parseFloat` of the whole string (assumed to be billions). **Positive-only**: rejects 0 and negatives at both branches.
-
-Used by: `lib/model-data.ts:409` (`parseParamsBillions(entry.params_billions)` in `hfModelCardToEvaluationCardData`). Sole caller. Input is `entry.params_billions` from `model-cards.json`, which in production is always `number | null` (see audit) — so only the `typeof value === "number"` branch ever fires.
-
-### Variant B — `components/eval-detail.tsx:81-119` (`parseParamsBillionsFromText`)
-
-```ts
-function parseParamsBillionsFromText(value: string | null | undefined) {
-  if (!value) return null
-  const normalized = value.trim().toLowerCase()
-  if (!normalized) return null
-
-  const compact = normalized.replace(/,/g, "")
-  const tokenMatch = compact.match(/(\d+(?:\.\d+)?)\s*(trillion|tn|t|billion|bn|b|million|mn|m|thousand|k)\b/)
-  if (tokenMatch) {
-    const amount = Number.parseFloat(tokenMatch[1])
-    if (!Number.isFinite(amount)) return null   // ← NO `<= 0` check (differs from A)
-    /* same unit table as Variant A */
-  }
-  const numeric = Number.parseFloat(compact)
-  return Number.isFinite(numeric) ? numeric : null   // ← NO `> 0` check (differs from A)
-}
-```
-
-- Same regex + unit table as A.
-- **Differs from A** on two checks: A rejects `amount <= 0` and `numeric <= 0`; B accepts `0`, negatives, and any finite number. So `"0B"` → `0` here, `null` in Variant A. The `"-5"` parseFloat fallback returns `-5` in B but `null` in A. (For `"-5B"` both return `5`, since the regex's `\d+` matches the `5` substring and the leading minus is silently dropped.)
-- Used by `getParamsBillionsFromModelInfo` (Variant D) for `additional_details.params_billions` (string in production, see audit) and `model_info.parameter_count` strings.
-
-### Variant C — `components/eval-detail.tsx:121-155` (`parseParamsBillionsFromText`'s sibling, `parseParamsBillionsFromModelName`)
-
-```ts
-function parseParamsBillionsFromModelName(modelName: string | null | undefined) {
-  if (!modelName) return null
-  const sizeTokens = Array.from(modelName.matchAll(/\b(\d+(?:\.\d+)?)\s*([tmbk])\b/gi))
-  if (sizeTokens.length === 0) return null
-
-  const lastToken = sizeTokens[sizeTokens.length - 1]
-  const numericValue = Number.parseFloat(lastToken[1])
-  if (!Number.isFinite(numericValue)) return null
-
-  const unit = lastToken[2].toLowerCase()
-  if (unit === "t") return numericValue * 1000
-  if (unit === "b") return numericValue
-  if (unit === "m") return numericValue / 1000
-  if (unit === "k") return numericValue / 1_000_000
-  return null
-}
-```
-
-- Word-boundary match (`\b...\b`) on **single-letter unit only** (`t|m|b|k`, case-insensitive).
-- Picks the **last** matching token in the string (e.g. `"Llama-3-70B-Instruct-8K"` → matches `70B` and `8K` → returns last (`8K` = 0.000008B)). This is **a known TS quirk**: model names containing context-window suffixes (`8K`, `32K`, `128K`) cause the parser to return the context-window size instead of the parameter count.
-- Used by `getParamsBillionsFromModelInfo` (Variant D) as final fallback when `additional_details.params_billions` and `parameter_count` are both absent/unparseable.
-
-### Variant D — `components/eval-detail.tsx:157-184` (`getParamsBillionsFromModelInfo`)
-
-Composite orchestrator (not a parser itself). Order:
-
-1. `additional_details.params_billions` ?? `additional_details.parameter_count` ?? `additional_details.num_parameters` ?? `additional_details.params`
-   - if `number` → return as-is (no validity check; could be negative or non-finite)
-   - if `string` → `parseParamsBillionsFromText` (Variant B)
-2. else if `modelInfo.parameter_count` is `string` → `parseParamsBillionsFromText` (Variant B)
-3. else → `parseParamsBillionsFromModelName(modelInfo.name)` (Variant C)
-
-Used at: `components/eval-detail.tsx:350, 1253, 1368` (paramsBillions cell in eval-detail tables, leaderboard sort filtering, "any model has params" header check).
-
-### Variant E — `components/model-compare-dialog.tsx:44-60` (`parseParamsBillionsFromModelName`)
-
-```ts
-function parseParamsBillionsFromModelName(modelName: string | null | undefined) {
-  if (!modelName) return null
-  const sizeTokens = Array.from(modelName.matchAll(/\b(\d+(?:\.\d+)?)\s*([bm])\b/gi))
-  if (sizeTokens.length === 0) return null
-
-  const lastToken = sizeTokens[sizeTokens.length - 1]
-  const numericValue = Number(lastToken[1])
-  if (!Number.isFinite(numericValue)) return null
-
-  const unit = lastToken[2].toLowerCase()
-  if (unit === "b") return numericValue
-  if (unit === "m") return numericValue / 1000
-  return null
-}
-```
-
-- Same shape as Variant C, but unit set is **only `b|m`** (no `t`, no `k`) and uses `Number()` instead of `parseFloat`.
-- Used by `formatParamsBillions(value, modelName)` only when the explicit numeric `value` is null/NaN — so it's the fallback parser for the compare-dialog header label.
-
-### Variant F — `app/evals/[id]/page.tsx:434-437` (inline regex)
-
-```ts
-const sizeMatch = (data.name + " " + id).match(/\b(\d+(?:\.\d+)?)\s*[bB]\b/)
-if (sizeMatch) sizeB = parseFloat(sizeMatch[1])
-```
-
-- One-shot regex against the **concatenation of `name + " " + id`** (not just name).
-- Unit set: **only `b|B`**. No multi-unit support, no fallback.
-- `match()` (not `matchAll()`) → returns **first** match (Variants C and E pick the **last**). For a name like `"Llama-3-8B-70B-Instruct"` Variant F returns `8`, Variant C returns `70`.
-- Used at the matrix-leaderboard sizeB filter (`m.sizeB`, lines 472-473) for the params-range slider.
-
-## Classification
-
-- **Cleaning / standardization → pipeline.** Pure value transform on a single field per row. The product decision being encoded — "express parameter count in billions" — is a per-record canonicalization that belongs upstream. Pipeline-side fix: emit a single numeric `params_billions` (in billions) on every model record; consumers stop parsing.
-- **Unconditional normalization.** Each variant runs unconditionally over its source fields; none defer to a pre-existing canonical numeric (because none exists at the per-result level today). Pipeline-side fix is to emit the canonical value before consumers see the row, not to gate normalization on its absence.
-
-(One nuance: Variant D's *fallback chain* — try `additional_details.params_billions`, then `parameter_count`, then `name` — is itself a small bit of reshape logic. After pipeline emits a canonical numeric, the chain collapses to a single field read.)
-
-## Inputs and expected outputs
-
-Each table below describes ONE variant.
-
-### Group A — Variant A (`parseParamsBillions`, lib/model-data.ts)
-
-| Input | Output | Path |
-|---|---|---|
-| `7` (number) | `7` | number, finite, > 0 → return as-is |
-| `0` (number) | `null` | number, > 0 fails → null |
-| `-3` (number) | `null` | number, > 0 fails → null |
-| `NaN` | `null` | number, finite fails → null |
-| `null` / `undefined` / `[]` | `null` | not number, not string → null |
-| `"7B"` | `7` | regex matches → unit `b` → 7 |
-| `"7b"` | `7` | lowercased; same |
-| `"70B params"` | `70` | regex matches at start, `b` → 70 |
-| `"1.5B"` | `1.5` | float supported |
-| `"405b"` | `405` | lowercased |
-| `"7 billion"` | `7` | full word `billion` → 7 |
-| `"7bn"` | `7` | `bn` alias |
-| `"1.2T"` | `1200` | `t` → ×1000 |
-| `"2 trillion"` | `2000` | `trillion` → ×1000 |
-| `"2T params"` | `2000` | regex stops at `t\b`; trailing text ignored |
-| `"560M"` | `0.56` | `m` → ÷1000 |
-| `"560 million"` | `0.56` | `million` → ÷1000 |
-| `"1000K"` | `0.001` | `k` → ÷1_000_000 |
-| `"1,500B"` | `1500` | comma stripped |
-| `"34.389"` | `34.389` | no unit token → parseFloat fallback |
-| `"7  B"` (double space) | `7` | regex `\s*` matches |
-| `"abc"` | `null` | no match, parseFloat NaN → null |
-| `""` | `null` | trim → "" → early return |
-| `"   "` | `null` | trim → "" → early return |
-| `"0B"` | `null` | regex matches, amount=0, `<= 0` reject → null |
-| `"3.5tn"` | `3500` | `tn` alias |
-| `"7Banana"` | `7` | regex `b\b` fails (no boundary after `b`); falls to `parseFloat("7banana")` = `7` → returns 7. **TS quirk: trailing junk allowed in parseFloat fallback** |
-| `"-5B"` | `5` | regex `\d+` doesn't include `-`, but matches the `5b` substring → amount=5; `>0` passes → returns `5`. **TS quirk: leading minus is silently dropped** |
-
-### Group B — Variant B (`parseParamsBillionsFromText`, eval-detail.tsx)
-
-Same as A except:
-
-| Input | A | B | Why |
-|---|---|---|---|
-| `"0B"` | `null` | `0` | B has no `<= 0` reject |
-| `"-5"` (string) | `null` | `-5` | B has no `> 0` reject on parseFloat fallback |
-| `"-5B"` (string) | `5` | `5` | both: regex matches the `5b` substring → amount=5 (TS quirk: leading minus dropped silently) |
-| `"NaN"` (string) | `null` | `null` | parseFloat("nan") = NaN, `isFinite` false → null in both |
-| number input | passes-through (with `>0` check) | n/a (B rejects non-strings) | A is polymorphic; B is string-only |
-
-All other rows in Group A apply to B identically (string-input rows only).
-
-### Group C — Variant C (`parseParamsBillionsFromModelName`, eval-detail.tsx)
-
-| Input | Output | Path |
-|---|---|---|
-| `"Llama-3-70B-Instruct"` | `70` | matchAll finds `70B`, last token, `b` → 70 |
-| `"Llama-3-8B-Instruct-8K"` | `0.000008` | matchAll finds `8B` and `8K`; **last** token is `8K` → ÷1_000_000 → 0.000008. **TS quirk** |
-| `"Llama-3-70B-Instruct-32K"` | `0.000032` | last token is `32K` → 0.000032. **TS quirk: context-window beats param count** |
-| `"Mixtral-8x7B"` | `null` | regex needs `\b` before the `\d`; `8x7b` has no boundary between `x` and `7`, so no token matches. **TS quirk: MoE-style names parse to null** |
-| `"Phi-3.5-mini-3.8B"` | `3.8` | matches `3.8B` |
-| `"560M"` | `0.56` | `m` token → ÷1000 |
-| `"GPT-4"` | `null` | no `\b\d+[tmbk]\b` token |
-| `"Yi-1.5-34B-32K"` | `0.000032` | last token `32K` (context window!) → 0.000032 |
-| `"Qwen2-7B-Instruct"` | `7` | last token `7B` |
-| `"7 billion"` | `null` | regex requires single-letter unit; `billion` has no `\bb\b` since it's word-internal |
-| `"1.2T"` | `1200` | `t` → ×1000 |
-| `""` / `null` / `undefined` | `null` | early return |
-
-### Group D (1) — Variant D (`getParamsBillionsFromModelInfo`, orchestrator)
-
-Behavior depends on which field is populated:
-
-| modelInfo state | Result |
-|---|---|
-| `additional_details.params_billions` is `number` 7 | `7` (returned as-is, no validation) |
-| `additional_details.params_billions` is `number` -3 | `-3` (no validity check; passed through) |
-| `additional_details.params_billions` is `string "7.242"` | `7.242` (Variant B parseFloat fallback) |
-| `additional_details.params_billions` is `string "7B"` | `7` (Variant B regex) |
-| `additional_details.params_billions` absent, `additional_details.parameter_count` is `"34.389"` | `34.389` (Variant B) |
-| `additional_details` absent, `modelInfo.parameter_count` is `"7B"` | `7` (Variant B) |
-| All `additional_details.*` and `parameter_count` absent, `modelInfo.name` is `"Llama-3-70B-Instruct"` | `70` (Variant C) |
-| All absent, `modelInfo.name` is `"Llama-3-8B-Instruct-8K"` | `0.000008` (Variant C TS quirk) |
-
-### Group E — Variant E (`parseParamsBillionsFromModelName`, model-compare-dialog.tsx)
-
-Same as Variant C but rejects `t` and `k`:
-
-| Input | C | E | Why |
-|---|---|---|---|
-| `"Llama-3-70B-Instruct"` | `70` | `70` | both — `b` token |
-| `"Llama-3-8B-8K"` | `0.000008` | `8` | E ignores `K` → last `b|m` token is `8B` |
-| `"Yi-1.5-34B-32K"` | `0.000032` | `34` | E correctly returns 34 (TS quirk: E is *more correct* on names with context-window suffixes!) |
-| `"1.2T"` | `1200` | `null` | E doesn't accept `t` |
-| `"Mixtral-8x7B"` | `null` | `null` | both — `8x7b` has no `\b` before the digit |
-| `"560M"` | `0.56` | `0.56` | both |
-| `"7Banana"` | `null` | `null` | both — regex requires `\b` boundary |
-
-### Group F — Variant F (`(name + " " + id).match(/\b(\d+(?:\.\d+)?)\s*[bB]\b/)`, app/evals/[id]/page.tsx)
-
-| Input (`name + " " + id`) | Output | Path |
-|---|---|---|
-| `"Llama-3-70B meta/llama-3-70b"` | `70` | first `70B` matches |
-| `"Llama-3-70B-Instruct-8K meta/llama-3-70b-instruct"` | `70` | first match wins; `8K` not a `b\|B` so ignored entirely |
-| `"Yi-1.5-34B-32K 01-ai/yi-1-5-34b-32k"` | `34` | first match `34B` |
-| `"GPT-4 openai/gpt-4"` | `null` | no `b\|B` token |
-| `"Mixtral-8x7B mistralai/mixtral-8x7b"` | `null` | `8x` blocks word-boundary; no `\b\d` anchor; no match |
-| `"560M openai/foo-560m"` | `null` | F only accepts `b\|B` |
-| `"1.5B-instruct meta/foo-1-5b"` | `1.5` | first `1.5B` |
-| `"" + " " + ""` | `null` | empty → no match |
-
-**Cross-variant ordering quirk:** Variants C and E pick the *last* size-token; Variant F picks the *first*. For ambiguous names the answer can differ; in practice production names tend to have one numeric+unit token so this rarely matters.
-
-### Group G — Cross-variant divergence
-
-For the same input, the variants produce different outputs. In production, format consistency keeps disagreement narrow but real:
-
-For inputs that aren't model names (free-text params strings), A and B are the relevant variants:
-
-| Input | A | B |
-|---|---|---|
-| `"7B"` | `7` | `7` |
-| `"7"` (string) | `7` (parseFloat fallback) | `7` |
-| `7` (number) | `7` | n/a (B rejects non-strings) |
-| `0` (number) | `null` (>0 reject) | n/a |
-| `NaN` (number) | `null` | n/a |
-| `"0B"` | `null` (rejects amount ≤0) | `0` |
-| `"-5"` (string) | `null` (>0 reject on parseFloat fallback) | `-5` |
-| `"-5B"` (string) | `5` | `5` (regex matches `5b`; minus dropped — both variants) |
-| `"7Banana"` | `7` | `7` (parseFloat lenient — both variants) |
-
-For inputs that ARE model names (the C/E/F input domain), all five variants can be applied:
-
-| Input | A | B | C | E | F (`name + " " + id`) |
-|---|---|---|---|---|---|
-| `"Llama-3-70B-Instruct"` | `70` (first regex match) | `70` | `70` (last token = `70B`) | `70` | `70` (first match) |
-| `"Llama-3-70B-Instruct-8K"` | `70` (`match()` returns first; first `(\d+)(unit)` is `70b`) | `70` (same regex as A) | `0.000008` (C's `matchAll` → last token = `8K`, ÷1_000_000 → 0.000008) | `8` (E ignores `K`; last `b\|m` token = `8B`) | `70` (F is `[bB]` only; first match = `70B`) |
-| `"Yi-1.5-34B-32K"` | `34` | `34` | `0.000032` (last token `32K`) | `34` | `34` |
-| `"Mixtral-8x7B"` | `7` (A's regex has no leading `\b` — `\d+` can match anywhere, including right after `x` → matches `7b` → 7) | `7` (same as A) | `null` (C's regex starts with `\b\d` — no `\b` between `x` and `7` → no match) | `null` | `null` |
-| `"560M"` | `0.56` | `0.56` | `0.56` | `0.56` | `null` (F is B-only) |
-| `"1.2T"` | `1200` | `1200` | `1200` | `null` (E is `b\|m`-only) | `null` (F is B-only) |
-| `"7 billion"` | `7` | `7` | `null` (C requires single-letter unit only; `billion` doesn't match the `[tmbk]` class) | `null` | `null` |
-
-**This is not a bug to fix in the migration.** It's evidence that the parsers were written with subtly different assumptions about input shape (model-name vs free-text vs trusted-numeric). The pipeline-canonical fix collapses all five into a single field read.
-
-## Current TS implementation
-
-| Variant | Function | Location | Callers | Source field |
-|---|---|---|---|---|
-| A | `parseParamsBillions` | `lib/model-data.ts:312-354` | 1 site: `lib/model-data.ts:409` | `entry.params_billions` from `model-cards.json` (number\|null in prod) |
-| B | `parseParamsBillionsFromText` | `components/eval-detail.tsx:81-119` | 2 sites inside Variant D: `eval-detail.tsx:170, 177` | strings from `additional_details.params_billions / parameter_count / num_parameters / params` and `modelInfo.parameter_count` |
-| C | `parseParamsBillionsFromModelName` | `components/eval-detail.tsx:121-155` | 1 site inside Variant D: `eval-detail.tsx:183` | `modelInfo.name` |
-| D | `getParamsBillionsFromModelInfo` (orchestrator) | `components/eval-detail.tsx:157-184` | 3 sites: `eval-detail.tsx:350, 1253, 1368` | `ModelResultForBenchmark["model_info"]` (built per-result by `lib/hf-data.ts:1133` `buildModelInfoForVariant`) |
-| E | `parseParamsBillionsFromModelName` (compare-dialog) | `components/model-compare-dialog.tsx:44-60` | 1 site: `model-compare-dialog.tsx:64` (`formatParamsBillions` fallback) | model display name in compare dialog |
-| F | inline regex | `app/evals/[id]/page.tsx:434-437` | 1 site (inline use): lines 472-473 (sizeB filter for slider) | `data.name + " " + id` from per-row matrix-leaderboard model entries |
-
-Total: 5 distinct parsers + 1 orchestrator + 8 caller sites across 4 files.
-
-## Pipeline status — divergences
-
-### Side-by-side comparison table
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Where canonicalization runs | request time, in 5 functions | `model-cards.json.params_billions` is already a clean number for 87% of cards (5072/5830); per-result `additional_details.params_billions` is a *string* in ~58% of model files | TS parses on every render |
-| Output format | varies per variant: number-of-billions; some return 0/negative, some null on edge | model-card level: clean float in billions; per-result: string requiring downstream parse | mostly converges in production but the per-result string parsing is unnecessary work |
-| Coverage | Variants C/E/F regex-fallback fires when no `additional_details` data exists | model-cards: 13% (758/5830) have `params_billions=null` (no data, irrespective of parser) | "Not reported" appears for 13% of cards regardless of parser correctness |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/` (5,830 model-cards, 5,830 model files, **86,183 model_result rows**):
-
-- **`model-cards.json` (top-level, drives Variant A)**: 5,830 entries.
-  - `params_billions` is `number`: 5,072 (87.0%) — Variant A returns positive value
-  - `params_billions` is `null`: 758 (13.0%) — Variant A returns null
-  - `params_billions` is string: **0** → Variant A's string-parsing branches are entirely dead code in production
-- **Per-row `model_info.additional_details.params_billions` (drives Variants B → D fallback)**:
-  - undefined: 58,822 (68.3%) — D falls through to model-name fallback
-  - string: 27,361 (31.7%) — virtually all `cleanDecimal` shape (e.g. `"7.242"`, `"34.389"`); 18 rows are `"-1.0"` (negative sentinel — Variant B accepts it as `-1`, A would reject)
-  - number: 0
-- **Variant D resolution path counts** (out of 86,183 rows):
-  - `addPbString` (additional_details.params_billions string → B): 27,361 (31.7%)
-  - `modelNameFallback` (Variant C from name): 18,174 (21.1%)
-  - **`noResolution`** (D returns null): 40,648 (47.2%) — these rows display "Not reported"
-- **Model name format distribution** (drives Variants C/E/F):
-  - `hasBOnly` (single B-token, no context-window): 36,286 (42.1%) — happy path; all parsers converge
-  - `hasBAndContextWindow` (e.g. `Yi-1.5-34B-32K`): **526 (0.61%)** — these hit Variant C's TS quirk
-  - `hasMOnly` (e.g. `d-SmolLM2-360M`): 417 (0.48%)
-  - `hasMoEPattern` (e.g. `WizardLM-2-8x22B`): 1,173 (1.36%) — A/B parse via no-leading-`\b` regex; C/E/F return null
-  - `noUnitToken` (e.g. `Yi Large Preview`, `GPT-4`): 47,781 (55.4%) — no parser matches
-  - `hasBAndT`: 0
-- **Cross-variant agreement on names** (A/B/C/E/F applied to the model name string):
-  - All five converge: 81,442 (94.50%)
-  - **Variant C TS-quirk hit (context-window beats param count)**: **472 rows (0.55%)** — only C is wrong; A/B/E/F all return correct param count
-  - F-only-missing (F returns null because no B-token but others find m/k/t): 3,001 (3.48%)
-  - Other disagreement: 1,268 (1.47%)
-
-**Top quirk in production**: Variant C returns the context-window size (`32K → 0.000032B`, `16K → 0.000016B`, `8K → 0.000008B`) instead of the parameter count for **472 model_result rows** with names like `Yi-1.5-34B-32K`, `Yi-1.5-34B-Chat-16K`. Variant D's fallback chain only reaches Variant C (the model-name parser) when `additional_details.params_billions` is missing, so the user-visible impact is bounded — but those 472 rows render with a `~0B` parameter count on the eval-detail leaderboard, well below the params-range filter floor.
-
-Verified by `scripts/verify-params-parsing.mjs`.
-
-## Notes for pipeline implementer
-
-- **Recommended canonical field: `params_billions: float | null`** at the *per-result* level (pipeline emits it on every `model_result.model_info`, not just at the top-level model card).
-- Eliminate the multi-field fallback chain in Variant D by emitting the resolved value in one canonical place. Existing fields (`additional_details.params_billions`, `additional_details.parameter_count`, `additional_details.num_parameters`, `additional_details.params`, `modelInfo.parameter_count`) can stay for compatibility but consumers stop reading them.
-- The model-name regex fallback (Variants C/E/F) is the *only* path that fires when `additional_details` is missing. Pipeline should attempt to parse from name **once** upstream (with whatever quirks it chooses; default to the Variant E semantics — `b|m` only — to avoid the context-window false-positive) and emit the result. Document the upstream parser's choice clearly so this spec can be retired.
-- The "params_billions in millions vs billions" unit is implicit; recommend keeping the field name `params_billions` to avoid breaking changes, and storing the value in **billions** as today.
-- Once pipeline emits per-result `params_billions`:
-  - Variant D collapses to a single field read.
-  - Variants A/B/C/E/F all become deletable.
-  - Variant F's `name + " " + id` regex disappears with the rest.
-
-Verification: once pipeline ships per-result `params_billions`, run `scripts/verify-params-parsing.mjs` and confirm the regex-fallback ("name-derived") row count drops to 0.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each variant's semantics + cross-variant divergence (`tests/transformations/params-parsing.test.ts`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline emits per-result `params_billions` numeric (in billions) across the full corpus
-- [ ] TS deleted; replace 5 functions + orchestrator + 8 callers with a single field read. Files: `lib/model-data.ts`, `components/eval-detail.tsx`, `components/model-compare-dialog.tsx`, `app/evals/[id]/page.tsx`.
-
-## Future product decision (deferred)
-
-The Variant C "context-window suffix beats parameter count" quirk (`"Llama-3-8B-8K"` → 0.000008B) is a real bug. We're choosing to fix-by-canonicalization-upstream rather than fix-in-place. Whether the pipeline parser should match Variant C, Variant E (which avoids the quirk by ignoring `K`/`T`), or implement a smarter "prefer the larger token" heuristic is a separate design decision for the pipeline owner.
-
-The Variant A `<= 0` rejection (treats `"0B"` as missing data) versus Variant B passthrough (`"0B"` → `0`) is another deferred decision. Production never emits `0`-valued params, so this also doesn't manifest today.

notes/transformations/11-benchmark-card-attachment.md

@@ -1,235 +0,0 @@
-# Benchmark-card attachment (per-eval lookup join)
-
-Drafted 2026-04-28. Migration item #17 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec. The retry-loop iteration over candidate names is **load-bearing** in production: 85% of evals (499/587) reach this code path because the pipeline does not inline `benchmark_card` for them. Don't try to "fix" the candidate-name derivation or the dedup-on-first-name-collision behavior in `getMap()` — preserve them until pipeline always inlines `benchmark_card` upstream.
-
-## Rule (as TS implements it today)
-
-For each `BenchmarkEvalSummary` (detail page) or `BenchmarkEvalListItem` (list page), if `benchmark_card` is not already populated on the record, derive an ordered list of candidate names from the eval and try each one against a deduped `Map<string, BenchmarkCard>`. The first match wins; first hit attaches the card via spread (`{ ...summary, benchmark_card: card }`); no match leaves the record unchanged (passthrough).
-
-The lookup is composed from three pieces:
-
-1. **Map build** (`lib/benchmark-metadata.ts:11-28`, `readPipelineBenchmarkCards`)
-   - Source: `benchmark-metadata.json` (Record<string, BenchmarkCard>) — currently 85 cards in production.
-   - For each card with `card.benchmark_details.name`, generate `candidateBenchmarkKeys(name)` and insert each key into a `Map`.
-   - **First-write-wins** dedup: `if (!map.has(key)) map.set(key, card)`. If two cards normalize to the same key, the first one inserted (i.e. the first one returned by `Object.values()`) takes the slot.
-   - Cached for the lifetime of the process via `cachedMapPromise`.
-
-2. **Per-name candidate generation** (`lib/benchmark-metadata-utils.ts:23-33`, `candidateBenchmarkKeys`)
-   - Input: a free-text benchmark name.
-   - Produces an array of up to 4 lookup keys via a `Set` (so duplicates collapse), in this order:
-     1. `base = normalizeBenchmarkKey(name)` — the canonical form (see below).
-     2. `base.replace(/-/g, " ")` — dashes → spaces.
-     3. `base.replace(/ /g, "-")` — spaces → dashes.
-     4. `base.replace(/[^a-z0-9]/g, "")` — strip everything to alnum.
-
-3. **`normalizeBenchmarkKey`** (`lib/benchmark-metadata-utils.ts:10-18`) — the base normalizer:
-   - Returns `""` for falsy input (does NOT fall through to `"unknown"` like `pipelineSlugify` does).
-   - Strip a leading `<alnum_underscore_token>` followed by optional space and a `/` (e.g. `"hfopenllm_v2/mmlu"` → `"mmlu"`).
-   - `.toLowerCase()`.
-   - Collapse runs of `_` or `-` to a single space.
-   - Collapse whitespace to single space.
-   - `.trim()`.
-
-4. **The retry loop over per-record candidate names** (`lib/model-data.ts:863-872`, `attachBenchmarkCardToSummary`)
-   - Builds a list of three candidate names from the summary (in this exact order):
-     1. `summary.evaluation_name`
-     2. `summary.composite_benchmark_name`
-     3. `summary.composite_benchmark_key`
-   - For each, calls `getBenchmarkCard(candidate)` (which itself runs `candidateBenchmarkKeys` on the name and tries each key against the map).
-   - First hit wins. No `.filter(Boolean)` here, so empty-string candidates still hit `getBenchmarkCard` (which then returns `null` because `normalizeBenchmarkKey("")` returns `""`).
-
-   The list-item variant (`lib/duckdb-data.ts:133-156`, `attachBenchmarkCardsToEvalListItems` and `lib/model-data.ts:1264-1282` inline in `getEvalListData`) is identical except:
-   - Order is `[evaluation_name, composite_benchmark_key, composite_benchmark_name]` (key BEFORE name — the inverse of the summary version).
-   - Both wrap with `.filter(Boolean)` to drop empty/undefined candidates before the loop.
-
-   This three-vs-three asymmetry between the summary path and list path is **TS-as-spec**: do not "harmonize" it. Same record can resolve to different cards via the two paths if the second and third candidates point at different benchmarks (in production this difference is benign — see "Divergences detected").
-
-## Classification
-
-- **Default-only** (do NOT overwrite when value present). Both attach functions guard with `if (summary.benchmark_card) return summary` / `if (item.benchmark_card) return item`. Pipeline-side fix: emit `benchmark_card` inline for every eval, then this branch is dead.
-- **Cleaning → pipeline.** This is a per-record lookup join (eval × benchmark_card). The map build, candidate-name generation, and three-attempt retry exist only to reconcile the eval's free-text name against the benchmark-metadata file. Once the pipeline writes `benchmark_card: <BenchmarkCard>` directly into every `eval-list.json` / `eval-detail.json` record, all four pieces (map build, candidateBenchmarkKeys, normalizeBenchmarkKey, the two attach functions) delete together. No aggregation; no derived view.
-
-## Inputs and expected outputs
-
-### Group A — `normalizeBenchmarkKey`
-
-| Input | Output | Rule branch |
-|---|---|---|
-| `"MMLU"` | `"mmlu"` | lowercase only |
-| `"BIG-Bench Hard (BBH)"` | `"big bench hard (bbh)"` | dash → space |
-| `"hfopenllm_v2/mmlu"` | `"mmlu"` | composite prefix stripped |
-| `"hfopenllm_v2 / mmlu"` | `"mmlu"` | composite prefix stripped (with space before `/`) |
-| `"GPQA / Diamond"` | `"diamond"` | composite prefix stripped — regex `/^[a-z0-9_]+ ?\//i` allows one optional space between the leading token and the `/` |
-| `""` | `""` | falsy short-circuit (does NOT fall back to "unknown") |
-| `"  MMLU  "` | `"mmlu"` | trim |
-| `"foo___bar"` | `"foo bar"` | underscore run collapsed to single space |
-| `"foo - - bar"` | `"foo bar"` | dash + space runs collapse |
-
-### Group B — `candidateBenchmarkKeys`
-
-| Input | Candidates returned (deduped, in order) | Why |
-|---|---|---|
-| `"MMLU"` | `["mmlu"]` | base only — no dashes, no spaces, alnum-only |
-| `"BIG-Bench Hard (BBH)"` | `["big bench hard (bbh)", "big-bench-hard-(bbh)", "bigbenchhardbbh"]` | base; spaces→dashes; alnum-only. Dashes→spaces collides with base. |
-| `"GSM-8K"` | `["gsm 8k", "gsm-8k", "gsm8k"]` | base; spaces→dashes; alnum-only |
-| `"gsm 8k"` | `["gsm 8k", "gsm-8k", "gsm8k"]` | identical result; base/dashes-form collision |
-| `"hfopenllm_v2/mmlu"` | `["mmlu"]` | composite prefix stripped first → no dashes/spaces |
-| `""` | `[""]` | base is `""`; the four variants all collapse to `""` |
-
-### Group C — `getBenchmarkCard` (per-name lookup against the deduped map)
-
-Given a map built from cards `{ "mmlu": cardA, "big bench hard (bbh)": cardB, "gsm 8k": cardC }`:
-
-| Input | Resolution | Notes |
-|---|---|---|
-| `"MMLU"` | cardA | base candidate `"mmlu"` hits |
-| `"mmlu_categories/mmlu_pro"` | cardA only if `"mmlu pro"` not in map; otherwise `null`/no hit unless the prefix-stripped form `"mmlu_pro"` (after `^[a-z0-9_]+ ?\//` strips `mmlu_categories/`) → `"mmlu pro"` was indexed. Demonstrates: composite prefix stripping can either help (find a generic card for the leaf) or miss (if the leaf has its own card the map didn't index under that exact spelling). |
-| `"BBH"` | `null` if cardB indexed under full title only | Reverse-lookup limitation — see "Divergences detected" |
-| `""` | `null` | `normalizeBenchmarkKey("")` short-circuits to `""`; all four candidates are `""`; `map.get("")` → undefined |
-
-### Group D — `attachBenchmarkCardToSummary` retry order
-
-For a summary with `evaluation_name="bbh/category_x"`, `composite_benchmark_name="BIG-Bench Hard (BBH)"`, `composite_benchmark_key="bbh"`:
-
-| Position | Candidate name | Lookup behaviour |
-|---|---|---|
-| 0 | `"bbh/category_x"` | `normalizeBenchmarkKey` strips `bbh/` → `"category_x"` → `"category x"` — likely no match |
-| 1 | `"BIG-Bench Hard (BBH)"` | `candidateBenchmarkKeys` produces `"big bench hard (bbh)"` → likely match |
-| 2 | `"bbh"` | base `"bbh"` → only matches if cards indexed the abbreviation |
-
-If position 1 hits, the loop short-circuits and the spread happens. If all three miss, `summary` returned unchanged.
-
-### Group E — pre-attached benchmark_card (default-only guard)
-
-| Input | Output |
-|---|---|
-| `summary.benchmark_card = <existing card>` | returned as-is, no map lookup |
-| `summary.benchmark_card = null` | falsy → falls through to retry loop |
-| `summary.benchmark_card = undefined` | falsy → falls through to retry loop |
-
-## Current TS implementation
-
-| Concern | Location |
-|---|---|
-| Map build + caching | `lib/benchmark-metadata.ts:11-36` (`readPipelineBenchmarkCards`, `getMap`, `cachedMapPromise`) |
-| Per-name lookup | `lib/benchmark-metadata.ts:38-49` (`getBenchmarkCard`) |
-| Reverse map flatten (used by `/api/benchmark-metadata` route) | `lib/benchmark-metadata.ts:51-66` (`getAllBenchmarkCards`) |
-| Candidate-key generation | `lib/benchmark-metadata-utils.ts:23-33` (`candidateBenchmarkKeys`) |
-| Base normalizer | `lib/benchmark-metadata-utils.ts:10-18` (`normalizeBenchmarkKey`) |
-| Summary attach (3-candidate retry) | `lib/model-data.ts:860-875` (`attachBenchmarkCardToSummary`) |
-| List-item attach (3-candidate retry, key before name) | `lib/duckdb-data.ts:133-156` (`attachBenchmarkCardsToEvalListItems`) |
-| List-item attach (inline, key before name, JSON backend) | `lib/model-data.ts:1264-1282` (inside `getEvalListData`) |
-
-### Call sites
-
-| Location | What it does |
-|---|---|
-| `lib/model-data.ts:870` | `attachBenchmarkCardToSummary` core (3-candidate retry over [evaluation_name, composite_benchmark_name, composite_benchmark_key]) |
-| `lib/model-data.ts:1272` | `getEvalListData` inline 3-candidate retry over [evaluation_name, composite_benchmark_key, composite_benchmark_name] (note the swapped 2nd/3rd) |
-| `lib/model-data.ts:1562` | aggregate eval path — calls `attachBenchmarkCardToSummary` per sub-eval before passing into `aggregateBenchmarkSummaries` |
-| `lib/model-data.ts:1595` | synthetic-matrix eval path — single attach call |
-| `lib/model-data.ts:1602` | direct eval lookup — single attach call |
-| `lib/duckdb-data.ts:147` | DuckDB list-item retry (in `attachBenchmarkCardsToEvalListItems`) |
-| `lib/duckdb-data.ts:179` | DuckDB single-eval path — calls `attachBenchmarkCardToSummary` |
-| `lib/duckdb-data.ts:218` | DuckDB list path — calls `attachBenchmarkCardsToEvalListItems` |
-| `app/api/benchmark-metadata/route.ts:5` | `/api/benchmark-metadata` route — exposes `getAllBenchmarkCards()` (the reverse flatten); not part of the per-eval attach but lives in the same module |
-
-Total: 7 attach call sites across 2 files (3 in `model-data.ts` + 1 inline + 3 in `duckdb-data.ts` + the inline list loop). All would delete together once pipeline always inlines.
-
-## Pipeline status — divergences
-
-Audited 2026-04-28 against `.cache/hf-data/` (587 evals, 85 benchmark cards in `benchmark-metadata.json`).
-
-### Coverage of inline `benchmark_card` (audited 2026-04-28)
-
-- **88 / 587 evals (15.0%)** already have `benchmark_card` populated inline by the pipeline (in both `eval-list.json` and the per-eval JSONs under `evals/`).
-- **499 / 587 evals (85.0%)** fall through to the runtime retry loop.
-- **`eval-list.json` and per-eval `evals/*.json` agree** on which records carry inline cards (88 each — same set). The pipeline inlines symmetrically across both files.
-
-### Retry-loop position distribution (of the 499 lookups)
-
-| Position | Summary path (name, name, key) | List path (name, key, name) |
-|---|---|---|
-| 0 (1st candidate hits) | 10 (2.0%) | 10 (2.0%) |
-| 1 (2nd candidate hits) | 0 | 0 |
-| 2 (3rd candidate hits) | 0 | 0 |
-| -1 (no candidate hits) | 489 (98.0%) | 489 (98.0%) |
-
-**The retry tail is dead code in production today.** Of the 499 lookups, the 1st candidate either hits (10) or no candidate ever hits (489). The 2nd and 3rd candidate retries never resolve anything. This is consistent with the data: the 489 misses are evaluations like `"artificial_analysis.median_output_tokens_per_second"` where no card exists in `benchmark-metadata.json` at all — not a lookup failure, just absence.
-
-That said: the retry IS the only thing that adds the 88 inline + 10 retry-position-0 = 98 cards to the runtime view. The retry positions 1 and 2 are kept TS-as-spec — they may be load-bearing in past or future data, and don't cost anything to preserve until pipeline always inlines.
-
-### Asymmetric retry order between summary and list paths
-
-### Asymmetric retry order between summary and list paths
-
-| Path | Order | Source |
-|---|---|---|
-| Summary attach | `[evaluation_name, composite_benchmark_name, composite_benchmark_key]` | `lib/model-data.ts:863-867` |
-| List-item attach (both backends) | `[evaluation_name, composite_benchmark_key, composite_benchmark_name]` | `lib/duckdb-data.ts:140-144`, `lib/model-data.ts:1270` |
-
-User-visible effect: an eval where `composite_benchmark_key` resolves to a different card than `composite_benchmark_name` would attach the wrong (or different) card depending on whether you reached it through the list page or the detail page. **In production today: 0 disagreements** (audited 2026-04-28). The asymmetry is theoretically observable but has no current impact — usually because the 1st candidate (`evaluation_name`) hits before either path reaches its swapped 2nd/3rd positions.
-
-Reproduce-don't-improve: the pipeline-side fix is to inline `benchmark_card` so both call sites resolve to the same value. Do NOT pick one of the two retry orders and propagate it.
-
-### Map-build first-write-wins collisions
-
-`readPipelineBenchmarkCards` indexes each card under `candidateBenchmarkKeys(card.benchmark_details.name)` and uses `if (!map.has(key)) map.set(key, card)`. If two cards have names that normalize to the same key, the **second** card is silently dropped from the map under that key (it may still be reachable via another candidate key it generates uniquely, but the colliding key permanently points at the first card encountered).
-
-Order is `Object.values(cards)` — i.e. the JSON insertion order from `benchmark-metadata.json`. **Production count: 4 collisions** (audited 2026-04-28) involving 2 distinct duplicated names:
-
-- `"Holistic Evaluation of Language Models (HELM)"` — `helm_capabilities` keeps the slot; `helm_instruct` is silently dropped under all colliding keys. The two cards have **different content** (overview lengths 466 vs 514). Anyone looking up "HELM" by name gets `helm_capabilities`; the `helm_instruct` card is unreachable via the runtime lookup unless the eval references the key `helm_instruct` directly (which would route through `composite_benchmark_key`, then `normalizeBenchmarkKey("helm_instruct")` → `"helm instruct"` → no match in the map either).
-- `"LiveCodeBench"` — `livecodebenchpro` keeps the slot; `livecodebench_pro` is dropped. The two cards have **identical content** (same `overview` string). Benign.
-
-The HELM case is a real data-correctness divergence: TS silently picks one of two distinct cards under the same name. Reproduce-don't-improve: the pipeline-side fix is to disambiguate the names in `benchmark-metadata.json` (or have evals reference cards by stable id rather than name), then inline the chosen card on each eval. Don't add disambiguation logic in TS.
-
-Reproduce-don't-improve: pipeline should emit per-eval `benchmark_card` directly so this map-build path becomes dead code. No need to teach pipeline to detect collisions.
-
-### Reverse-lookup limitation
-
-A card's `benchmark_details.name` is the only string indexed. If an eval references a benchmark by a different name (abbreviation, alternate casing, missing parens), the lookup misses unless one of the four `candidateBenchmarkKeys` variants happens to collide with a variant of the card's name.
-
-Example: a card named `"BIG-Bench Hard (BBH)"` is indexed under `"big bench hard (bbh)"`, `"big-bench-hard-(bbh)"`, `"bigbenchhardbbh"`. An eval named `"BBH"` produces candidates `["bbh"]` only — miss. The audit script counts how many evals fall into this category.
-
-Reproduce-don't-improve: this is the entire reason for migration item #17. Pipeline knows the card-to-eval mapping at build time; it shouldn't push a fuzzy-match problem to the runtime.
-
-**Orphaned cards (audited 2026-04-28): 29 / 83 distinct cards** in `benchmark-metadata.json` are not reached by any eval through any lookup path — examples include `"arc_agi_v1_public_eval"`, `"arc_agi_v2_semi_private"`, the `bfcl_*` family of 7 cards, `"IMDB"`, `"NarrativeQA"`, `"NaturalQuestions (open-book)"`, `"RAFT"`. These cards are either (a) for benchmarks not yet evaluated in the corpus, or (b) cards whose `benchmark_details.name` doesn't normalize to anything any eval looks up. Pipeline owner should triage. (Not a TS bug — TS faithfully looks up; the cards are never asked for.)
-
-## Notes for pipeline implementer
-
-The cleanest fix: **emit `benchmark_card: <BenchmarkCard>` inline on every eval** — both `eval-list.json` entries and every `evals/<id>.json` detail file. The pipeline already does this for 15% of evals; extend coverage to 100%.
-
-Acceptance criteria:
-
-1. Every record in `eval-list.json` has `benchmark_card` populated when a card exists for that benchmark; `null` only when no card exists in `benchmark-metadata.json` for the benchmark.
-2. Every `evals/<id>.json` file has matching `benchmark_card` for the same eval (same value as the list entry — they should not disagree).
-3. The DuckDB-emitted parquet (consumed via `lib/duckdb-data.ts`) carries `benchmark_card` in the same column for every row.
-
-Once pipeline meets criteria, all of the following delete:
-
-- `lib/benchmark-metadata.ts` entirely (after migrating `/api/benchmark-metadata` to read from `benchmark-metadata.json` directly or from a pipeline-emitted reverse map).
-- `lib/benchmark-metadata-utils.ts:candidateBenchmarkKeys`, `normalizeBenchmarkKey` — but the file may need to stay for the `/api/benchmark-metadata` route's `lookupBenchmarkCard` if any client component depends on it. Audit before deleting.
-- `lib/model-data.ts:attachBenchmarkCardToSummary` (function + 3 call sites).
-- `lib/model-data.ts` inline retry inside `getEvalListData` (lines 1264-1282).
-- `lib/duckdb-data.ts:attachBenchmarkCardsToEvalListItems` (function + 1 call site) + `attachBenchmarkCardToSummary` import + the wrapping call in `toEvalSummary`.
-
-Don't try to reproduce the 4-candidate `candidateBenchmarkKeys` derivation upstream. The point of the migration is to make it unnecessary.
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/benchmark-card-attachment.test.ts`)
-- [x] Audit script produced (`scripts/verify-benchmark-card-attachment.mjs`)
-- [ ] Filed with pipeline owner (link)
-- [ ] Pipeline emits `benchmark_card` inline on 100% of `eval-list.json` entries (currently 15%)
-- [ ] Pipeline emits `benchmark_card` inline on 100% of `evals/*.json` files (currently 15%)
-- [ ] DuckDB parquet carries `benchmark_card` column in `evalList` and `evalDetails` outputs
-- [ ] TS code deleted; callers read pipeline field directly (7 call sites + 1 module)
-
-## Future product decision (deferred)
-
-The `getAllBenchmarkCards()` reverse-flatten (used by `/api/benchmark-metadata` route) consumes the same map but produces a `Record<normalizedKey, card>` with `seen.has(card)` dedup, keyed by `normalizeBenchmarkKey(card.benchmark_details.name)` (which may not match what the map keys point at if there were collisions). Whether the route should keep using the map-derived form or read from `benchmark-metadata.json` directly is a separate decision for the cleanup pass — surface to pipeline owner.

notes/transformations/12-instance-level-data.md

@@ -1,258 +0,0 @@
-# Per-instance JSONL normalization
-
-Drafted 2026-04-28. Migration item #7 in `notes/migration-plan.md`.
-
-## Framing reminder
-
-We are refactoring for UI efficiency. TS-as-is is the canonical spec for behaviour. This spec covers a per-record value transform that extracts UI-friendly strings (input/response/correctness/etc.) from rich per-sample objects emitted by the pipeline. Audited 2026-04-28 against the full corpus, the parser's many fallback branches mostly do not fire — pipeline emits a single canonical shape — but the parser preserves them as defensive scaffolding for older or different harness output formats.
-
-**Architecture choice (cleaning-only, no SQL involvement):** the inline `instance_examples` preview (≤5 samples per result) gets normalized in pipeline; the per-result `source_url` pointing to a full JSONL dump (typically 50 samples) stays as an on-demand UI fetch, NOT pre-ingested into a parquet table. This avoids speculative pipeline ingestion work and matches the actual product need (lazy-load all samples on user click, not cross-corpus sample querying). The orphaned `fetchInstanceLevelData` (`lib/hf-data.ts:890-917`) gets re-wired to a future "show all samples" UI feature rather than deleted.
-
-## Rule (as TS implements it today)
-
-### `parseInstanceLevelData` (`lib/hf-data.ts:933-1043`)
-
-Pure function: takes a JSON object, walks `instance_examples[]`, returns `SampleResult[]`.
-
-Top-level guard:
-- If `data` is null/non-object → return `[]`
-- If `data.instance_examples` is array → use it
-- Else if `data` itself is array → use it
-- Else → `[]`
-
-Per-example field extraction (each is a fallback chain):
-
-**`input` (string)** — first non-empty wins:
-1. `raw.input` is string → use as-is
-2. `raw.input.raw` is set → `String(raw.input.raw)`
-3. `raw.prompt` → use as-is
-4. `raw.question` → use as-is
-5. `raw.doc.question` → use as-is
-6. `raw.doc` exists → `JSON.stringify(raw.doc).slice(0, 500)`
-7. (none) → empty string
-
-**`ground_truth` (string | undefined)** — first non-null wins:
-1. `raw.input.reference` is array → `array.join(", ")`; else → `String(...)`
-2. `raw.ground_truth` → `String(...)`
-3. `raw.target` → `String(...)`
-4. `raw.gold` → `String(...)`
-5. `raw.doc.answer` → `String(...)`
-6. (none) → `undefined`
-
-**`response` (string)** — first non-empty wins:
-1. `raw.output` is set → string-as-is or `JSON.stringify(...)`
-2. `raw.response` → use as-is
-3. `raw.model_output` → use as-is
-4. `raw.answer_attribution` is non-empty array → take last element's `extracted_value` (or empty string)
-5. `raw.messages` is non-empty array → reverse + find last assistant message → string content or stringified
-6. `raw.filtered_resps[0][0]` → use
-7. `raw.resps[0][0]` → use
-8. (none) → empty string
-
-**`is_correct` (boolean | undefined)** — first defined wins:
-1. `raw.evaluation.is_correct` (boolean)
-2. `raw.is_correct` (boolean)
-3. `raw.metrics.exact_match === 1` → true; `=== 0` → false; else undefined
-4. (none) → `undefined`
-
-**`metadata` (object | undefined)** — merged from (in order):
-- `raw.evaluation` (if object)
-- `raw.performance` (if object)
-- `raw.metadata` (if object)
-- `raw.metrics` (if object)
-- If merged object is empty → `undefined`
-
-**`sample_id` (string)** — first non-null wins:
-1. `raw.sample_id` → as-is
-2. `raw.doc_id` → as-is
-3. `raw.id` → as-is
-4. (none) → `String(arrayIndex)` (positional fallback)
-
-**`choices` (any | undefined)** — first non-null wins:
-1. `raw.choices`
-2. `raw.doc.choices`
-3. (none) → `undefined`
-
-If a row's first-pass map returns `null` (i.e. `raw` was null or non-object), it's filtered out via `.filter(s => s !== null)`.
-
-### `fetchInstanceLevelData` (`lib/hf-data.ts:890-917`) — currently orphaned
-
-Takes a `(url, limit?)`. Fetches the URL via `fetch()`. Splits text on newlines (filters empty lines). For each line up to `limit` (or all if no limit), tries `JSON.parse`; skips malformed lines. Wraps the parsed array as `{ instance_examples: parsed }` and passes to `parseInstanceLevelData`. Returns `SampleResult[]` or `[]` on any error.
-
-**Active call sites: zero.** Verified by grep across `app/`, `components/`, `scripts/`, other `lib/` files. Only mention is its own declaration. Git log shows one commit ("Refresh eval cards UI and backend data flow") in its history.
-
-The function is intended for "load more / show all samples" UI feature — pipeline ships a `source_url` per row pointing to the full JSONL (typically 50 samples), but the inline `instance_examples` preview only carries 5. The fetcher would let UI request the full set on user demand. **This UI feature has never shipped.**
-
-## Classification
-
-- **Unconditional normalization.** The function always runs on whatever shape is provided; never gates on a pre-existing canonical field. Pipeline-side fix: emit a canonical per-sample shape so the multi-field fallback chains become unnecessary.
-- **Cleaning → pipeline.** Pure value transform per sample. No aggregation, no reshape, no cross-record operations. Migration target: pipeline emits canonical per-sample shape on both the inline preview AND the URL JSONL files; TS parser shrinks to direct field reads or deletes entirely.
-- **NOT reshape.** Per the architecture choice in the framing note above, samples stay as on-demand fetches via `source_url`; no parquet `instance_samples` table, no SQL queries over samples. (If a future product feature wants cross-model sample search/filter/comparison, that's a separate reshape spec.)
-
-## Inputs and expected outputs
-
-### Group A — Pipeline-canonical shape (the only shape that fires in production today)
-
-Input shape (from cache `result.instance_level_data.instance_examples[i]`):
-
-```jsonc
-{
-  "schema_version": "...",
-  "evaluation_id": "...",
-  "model_id": "...",
-  "evaluation_name": "...",
-  "sample_id": "...",
-  "sample_hash": "...",            // sometimes present
-  "interaction_type": "multi_turn",
-  "input": { "raw": "..." },        // ALWAYS object with .raw in production
-  "output": "..." | { ... },        // sometimes
-  "messages": [{ role: "...", content: "..." }, ...],  // typically present
-  "answer_attribution": [..., { "extracted_value": "..." }],  // typically present
-  "evaluation": { "is_correct": true|false, ... },  // ALWAYS present
-  "performance": { ... },
-  "metadata": { ... },
-  "token_usage": { ... },
-  "error": null | "...",
-  "hierarchy": [...]
-}
-```
-
-Expected output (`SampleResult`):
-
-| Output field | Source path that fires | Notes |
-|---|---|---|
-| `sample_id` | `raw.sample_id` (100% of production) | always present |
-| `input` | `raw.input.raw` (100%) | branch #2 in the chain |
-| `ground_truth` | `raw.input.reference` (100%) | branch #1 |
-| `response` | `raw.answer_attribution` (97.31%) OR `raw.messages` (2.49%) OR `raw.output` (0.20%) | branches #4, #5, #1 |
-| `is_correct` | `raw.evaluation.is_correct` (100%) | branch #1 |
-| `choices` | `undefined` (100%) | no branch fires; field is unset in production |
-| `metadata` | merged from `raw.evaluation`, `raw.performance`, `raw.metadata`, `raw.metrics` (always at least 2 of 4 present) | merged object |
-
-### Group B — Defensive fallback branches (zero firing rate in current production)
-
-| Branch | Output field | Production hits | Origin (presumed) |
-|---|---|---|---|
-| `raw.input` (string) | input | 0 | older harness shapes |
-| `raw.prompt` | input | 0 | lm-eval-harness |
-| `raw.question` | input | 0 | other harnesses |
-| `raw.doc.question` | input | 0 | HELM-style |
-| `raw.doc` (JSON.stringify) | input | 0 | last-resort |
-| `raw.ground_truth` | ground_truth | 0 | older shapes |
-| `raw.target` | ground_truth | 0 | classification benchmarks |
-| `raw.gold` | ground_truth | 0 | older lm-eval |
-| `raw.doc.answer` | ground_truth | 0 | HELM-style |
-| `raw.response` | response | 0 | older shapes |
-| `raw.model_output` | response | 0 | older shapes |
-| `raw.filtered_resps[0][0]` | response | 0 | lm-eval-harness format |
-| `raw.resps[0][0]` | response | 0 | lm-eval-harness format |
-| `raw.is_correct` | is_correct | 0 | flat shape |
-| `raw.metrics.exact_match` | is_correct | 0 | metric-based correctness |
-| `raw.doc_id` | sample_id | 0 | HELM-style |
-| `raw.id` | sample_id | 0 | generic |
-| index fallback | sample_id | 0 | last-resort |
-| `raw.choices` | choices | 0 | multiple-choice |
-| `raw.doc.choices` | choices | 0 | HELM multiple-choice |
-
-These branches exist for shapes the pipeline currently does not emit. **Preserve verbatim** until pipeline-side guarantees the canonical shape across all data sources.
-
-### Group C — `fetchInstanceLevelData` JSONL parsing edge cases
-
-| Input | Behavior |
-|---|---|
-| URL returns `!res.ok` (404, 500, etc.) | returns `[]` (no throw) |
-| URL throws (network error) | logs warning to console, returns `[]` |
-| Empty body | splits to `[]`, returns `[]` |
-| Body with empty lines | `.filter(line => line.trim())` strips them |
-| Body with malformed JSON line | swallowed in inner try-catch; line skipped, processing continues |
-| `limit=0` or `limit=undefined` | parses ALL lines |
-| `limit > lines.length` | parses all lines (capped via `Math.min`) |
-
-## Current TS implementation
-
-| Concern | Location | Notes |
-|---|---|---|
-| `parseInstanceLevelData` | `lib/hf-data.ts:933-1043` | The parser; ~110 lines |
-| `fetchInstanceLevelData` | `lib/hf-data.ts:890-917` | URL fetcher; orphaned (zero callers) |
-| Active call site | `lib/hf-data.ts:1273` (inside `flattenHierarchyNode`) | `parseInstanceLevelData(result.instance_level_data)` → `inlineSamples` |
-| Internal call site | `lib/hf-data.ts:912` | inside `fetchInstanceLevelData` itself, recursive call to the parser |
-| `SampleResult` type | `lib/benchmark-schema.ts:135` | Output shape definition |
-| Output field | `BenchmarkEvaluation.detailed_evaluation_results_per_samples` | `lib/benchmark-schema.ts:33` |
-
-### Caller chain for `parseInstanceLevelData`
-
-`getModelSummaryById` (lib/model-data.ts:1490+) → `flattenModelEvaluations` → `flattenHierarchyNode` (lib/hf-data.ts:1273) → `parseInstanceLevelData(result.instance_level_data)` → set as `inlineSamples` on each variant bucket → propagated to `BenchmarkEvaluation.detailed_evaluation_results_per_samples`.
-
-UI consumers (read `data.detailed_evaluation_results_per_samples`):
-- `components/benchmark-detail.tsx:3869` — random sample for preview block
-- `components/benchmark-detail.tsx:4174-4216` — sample preview UI in benchmark detail
-- `components/benchmark-detail.tsx:4982` — variant-level sample availability check
-- `components/benchmark-detail.tsx:5284-5286` — variant sample picker
-- `components/benchmark-detail.tsx:5569-5608` — sample preview list with INSTANCE_PREVIEW_LIMIT and "see all" expansion
-
-### Caller chain for `fetchInstanceLevelData`
-
-None. Function is exported and unreached. Preserved with the intent that a future "show all samples" UI consumer wires up to it.
-
-## Pipeline status
-
-### Side-by-side comparison
-
-| Aspect | TS (this spec) | Pipeline today | Result for users |
-|---|---|---|---|
-| Inline preview shape | parser handles many variants | emits ONE canonical shape (`input.raw`, `evaluation.is_correct`, etc.) | parser's fallback branches almost all dead |
-| URL JSONL shape | same parser handles | emits IDENTICAL canonical shape (verified by sampling one URL on 2026-04-28) | parser would work the same on URL data |
-| Inline preview size | parser doesn't care | always exactly 5 samples per `instance_examples` array | UI capped at 5 today |
-| Total samples per row | n/a | typically 50 (per `instance_count` field), one outlier 18 | only 10% accessible to UI today |
-| URL-fetch use case | `fetchInstanceLevelData` exists | `source_url` always emitted | dead code on TS side; no UI consumer |
-
-### Concrete worked example with quantified scope
-
-Audited 2026-04-28 against `.cache/hf-data/`. Verified by `scripts/verify-instance-level-data.mjs`.
-
-**Prevalence:**
-- Total model files: 5,830
-- Files with any `instance_level_data`: **55 (0.94%)**
-- Total `(metric × model_result)` rows: 86,183
-- Result rows with `instance_level_data`: **712 (0.83%)**
-- Total inline preview examples (sum of `instance_examples.length`): **3,532** (always ≤5 per row)
-- Total full samples (sum of `instance_count`): **66,057** (full set available via `source_url`, not loaded today; ~19× larger than what UI currently shows)
-
-**ild-level shape uniformity (712/712 rows):**
-- Top-level keys are always exactly `{interaction_type, instance_count, source_url, instance_examples}`
-- `interaction_type` is always `"multi_turn"` (no single_turn samples in cache)
-
-**Per-example branch firing rates (3,532 examples):**
-- `input`: `input.raw` 100%
-- `ground_truth`: `input.reference` 100%
-- `response`: `answer_attribution` 97.31%, `messages` 2.49%, `output` 0.20%
-- `is_correct`: `evaluation.is_correct` 100%
-- `sample_id`: `sample_id` 100%
-- `choices`: nothing (always undefined)
-
-The 7-branch input chain, 5-branch ground_truth chain, 4-branch is_correct chain, 4-branch sample_id chain, 2-branch choices chain are **defensive scaffolding** for shapes the pipeline does not currently emit. The 7-branch response chain has 3 active sub-branches.
-
-**URL JSONL shape verification:** sampled one source_url (`anthropic__anthropic-claude-3-7-sonnet/swe_bench_verified_mini_...`); first line had identical 18 keys to the inline `instance_examples[0]`, with `input.raw` and `evaluation.is_correct` in expected paths. Pipeline emits the same canonical shape on both inline and URL paths.
-
-## Notes for pipeline implementer
-
-- The pipeline already emits a canonical per-sample shape consistently. **No structural change needed to current emission.** The migration is to make this shape an explicit guarantee, not to change what's being emitted.
-- Suggested guarantee: every `instance_examples[i]` (inline AND in JSONL at `source_url`) has at minimum `{sample_id, input.raw, input.reference?, evaluation.is_correct, answer_attribution? || messages?, metadata?}`.
-- Once that guarantee is documented and verified, the TS parser shrinks dramatically: extract `raw.input.raw`, `raw.input.reference`, `raw.evaluation.is_correct`, `raw.sample_id` as direct field reads. The response field still needs the 3-branch fallback (answer_attribution → messages → output) until pipeline emits a single normalized `response` field.
-- **Do NOT pre-ingest the URL JSONL into pipeline parquet** (per the architecture choice). The runtime UI fetches `source_url` on demand; this is the orphaned `fetchInstanceLevelData`'s intended use. The benefit of pre-ingestion (cross-corpus SQL queries over samples) is speculative; defer until a product feature demands it.
-- The "shape uniformity" finding (712/712 rows have identical ild-level keys; all are `multi_turn`) suggests the pipeline already enforces the canonical shape. Worth documenting in the pipeline contract test (`tests/pipeline-contract.test.ts`).
-
-## Migration checklist
-
-- [x] Spec written
-- [x] Tests cover each rule branch (`tests/transformations/instance-level-data.test.ts`)
-- [x] Audit script (`scripts/verify-instance-level-data.mjs`)
-- [ ] Filed with pipeline owner with the spec + tests + audit script as acceptance criterion
-- [ ] Pipeline contract: explicit guarantee of canonical per-sample shape (Tier A test asserting `every instance_example has input.raw, sample_id, evaluation.is_correct`)
-- [ ] TS deleted: `parseInstanceLevelData` shrinks to direct field reads (~10 lines instead of 110), or fully deleted if pipeline emits already-flat normalized records. `fetchInstanceLevelData` stays orphaned-but-preserved for the future "show all samples" UI feature, OR is wired up if that feature ships.
-
-## Future product decisions (deferred)
-
-- **"Show all samples" UI feature** — would un-orphan `fetchInstanceLevelData` and let users see the full 50-sample set instead of just the 5-sample preview. Lazy fetch on user click. This is the concrete capability the URL-fetch architecture supports; the spec assumes it's a product roadmap item, not committed scope.
-- **Cross-model sample querying / search / filter** — would require `instance_samples.parquet` and SQL queries (the alternative architecture I initially proposed). Out of scope for this spec; revisit if/when product asks.
-- **Single-turn samples** — pipeline currently only emits `interaction_type: multi_turn`. If pipeline starts emitting single_turn shapes that exercise dormant parser branches (e.g. flat `input` strings, `prompt`/`question` fields), the spec's "100% canonical shape" claim breaks and the parser fallbacks become live again.

notes/transformations/README.md

@@ -1,89 +0,0 @@
-# Transformations registry
-
-Canonical specs for data transformations the Next.js app currently performs that should ultimately live in the upstream pipeline (`eval_cards_backend_pipeline`).
-
-## Framing
-
-The TS code in `lib/` does meaningful data transformation work — token canonicalization, variant grouping, source-metadata defaults, category inference, score normalization, etc. — most of which the Python pipeline does not yet do (or does differently). For this migration:
-
-- **TS is the current source of truth** for what these transformations should produce. The TS rules have been refined over time against real evaluation data; they encode product decisions.
-- **Each transformation is a candidate to move upstream.** The pipeline owns data shape and content; doing the transformation there means every downstream consumer (not just this Next.js app) gets the canonical form.
-- **The migration path** for each item: write a spec here → write executable tests sourced from the spec → hand the spec + tests to the pipeline owner → verify pipeline output matches across the full corpus → delete the TS implementation.
-- **Pre-processing in this repo (e.g. a one-shot Python script run at data-ingestion time) is allowed but not required.** Default is "transformation stays in TS until pipeline catches up." Front-load only when the TS implementation is so brittle or expensive that on-the-fly is intolerable.
-
-## One thing to watch: defaults vs unconditional normalization
-
-For each rule, classify in the spec how it interacts with pre-existing data:
-
-- **Default-only (don't overwrite when pipeline already emits a value)**: e.g. "if `source_metadata.evaluator_relationship` is missing, default to `other`". Pipeline-side fix is to emit the default upstream rather than letting consumers fill in.
-- **Unconditional normalization (always overwrite)**: e.g. "lowercase the `v` in version tokens regardless of upstream input". Pipeline-side fix is to apply the rule before emitting; downstream consumers should not need to re-derive.
-
-Mis-classifying these is the failure mode that bit us in Phase 3 (treating a normalization rule as if it were a default, ending up overwriting category data). Each spec must call out which kind it is.
-
-## Where it belongs: cleaning vs reshape
-
-A second classification each spec should call out — *what kind of work* is this transformation?
-
-- **Cleaning / standardization** (changes a value): license shorthand, developer name canonicalization, identity tokens, timestamp format, category labels. These belong in the pipeline; the per-item workflow above is built for them. Default-vs-normalization (the section above) is the sub-question.
-- **Reshape / dedup / aggregate** (computes a derived view): variant dedup with "freshest wins", per-category counts, top-scores ranking, hierarchy flattening. These belong in **DuckDB SQL** — either materialized into pipeline parquet (the answer is the same for every consumer; emit it pre-computed) or expressed as a query at request time (consumers slice differently; let SQL do the work). The TS implementation here is scaffolding.
-
-For the reshape class the migration target shape is itself a design choice — capture the *operation* (e.g. "max `retrieved_timestamp` wins per variant key, take its `source_metadata` along") and flag it for the parquet-schema / SQL conversation rather than mechanically translating the TS code line-by-line. See `notes/migration-plan.md` § "Data direction" for the full framing.
-
-## Index
-
-| # | Transformation | Spec | Tests | Pipeline status | Migration item |
-|---|---|---|---|---|---|
-| 01 | Model identity canonicalization | [01-identity-canonicalization.md](01-identity-canonicalization.md) | [tests/transformations/identity-canonicalization.test.ts](../../tests/transformations/identity-canonicalization.test.ts) | partial (model_family_id ✅, model_family_name ❌ on 1,260 cards) | #1 |
-| 02 | Setup-alias variant merging | [02-setup-alias-merging.md](02-setup-alias-merging.md) | [tests/transformations/setup-alias-merging.test.ts](../../tests/transformations/setup-alias-merging.test.ts) | not started (cache file shows pre-merge state; runtime normalizer is what merges) | #2 |
-| 03 | License string normalization | [03-license-normalization.md](03-license-normalization.md) | [tests/transformations/license-normalization.test.ts](../../tests/transformations/license-normalization.test.ts) | not implemented; pipeline emits free-text `data_licensing` only | #18 |
-| 04 | Dataset URL synthesis | [04-dataset-url-synthesis.md](04-dataset-url-synthesis.md) | [tests/transformations/dataset-url-synthesis.test.ts](../../tests/transformations/dataset-url-synthesis.test.ts) | not implemented; `dataset_url` field never populated in prod (564/587 use `url[0]`, 22/587 use `hf_repo` template) | #20 |
-| 05 | Slug candidate generation (file lookup) | [05-slug-candidates.md](05-slug-candidates.md) | [tests/transformations/slug-candidates.test.ts](../../tests/transformations/slug-candidates.test.ts) | not implemented; 39% of model lookups + 43% of developer lookups need a non-zero retry position in production | #19 |
-| 06 | Developer name canonicalization | [06-developer-name-canonicalization.md](06-developer-name-canonicalization.md) | [tests/transformations/developer-name-canonicalization.test.ts](../../tests/transformations/developer-name-canonicalization.test.ts) | not implemented; pipeline emits raw `developer` string. TS map covers 1.8% of devs / 11.9% of cards; title-case fallback fires on 55.6% of devs / 48.4% of cards | #9 |
-| 07 | Timestamp normalization | [07-timestamp-normalization.md](07-timestamp-normalization.md) | [tests/transformations/timestamp-normalization.test.ts](../../tests/transformations/timestamp-normalization.test.ts) | not implemented; production is 99.99% unix-seconds-strings (86,178/86,183) + 5 ISO datetime. Three different TS variants exist with subtly different semantics — pipeline canonicalization to ISO 8601 collapses them | #13 |
-| 08 | Benchmark display names | [08-benchmark-display-names.md](08-benchmark-display-names.md) | [tests/transformations/benchmark-display-names.test.ts](../../tests/transformations/benchmark-display-names.test.ts) | not implemented; 30-entry hand-curated map covers ~74% of distinct suite keys but only ~3% of distinct `benchmark` values; ~97% fall through to a `humanizeToken` fallback that mangles acronyms (`MMLU-PRO` → `MMLU PRO`, `helm_air_bench` → `Helm Air Bench`). A second functionally-dead duplicate exists in `lib/eval-processing.ts` with substring-match semantics that disagree with the active path. | #8 |
-| 09 | Metric display name expansion | [09-metric-display-name-expansion.md](09-metric-display-name-expansion.md) | [tests/transformations/metric-display-name-expansion.test.ts](../../tests/transformations/metric-display-name-expansion.test.ts) | defensive scaffolding; both rules fire 0 times against current corpus (0/86,183 result rows for generic-name expansion; 0/587 eval-list entries for prefersBenchmarkName heuristic) | #10 |
-| 10 | Params billions parsing | [10-params-parsing.md](10-params-parsing.md) | [tests/transformations/params-parsing.test.ts](../../tests/transformations/params-parsing.test.ts) | partial; `model-cards.json.params_billions` is clean number for 87% of cards (5072/5830); per-row `additional_details.params_billions` is string for 31.7% of rows (27,361/86,183), 47.2% (40,648) have no resolved value at all. Five different TS parsers diverge on edge cases — Variant C's "context-window beats param count" quirk fires on 472 rows (0.55%, names like `Yi-1.5-34B-32K`) | #12 |
-| 11 | Benchmark-card attachment (per-eval lookup join) | [11-benchmark-card-attachment.md](11-benchmark-card-attachment.md) | [tests/transformations/benchmark-card-attachment.test.ts](../../tests/transformations/benchmark-card-attachment.test.ts) | partial; pipeline inlines `benchmark_card` on 88/587 evals (15%), other 499 fall through to runtime retry. Of those, 10 hit at position 0 and 489 miss entirely (most lack any matching card in `benchmark-metadata.json`). Map-build first-write-wins silently drops the `helm_instruct` card (different content from kept `helm_capabilities`, both named "HELM"); 29/83 cards orphaned | #17 |
-| 12 | Per-instance JSONL normalization | [12-instance-level-data.md](12-instance-level-data.md) | [tests/transformations/instance-level-data.test.ts](../../tests/transformations/instance-level-data.test.ts) | pipeline emits canonical shape; parser's defensive fallback branches mostly dead (input/ground_truth/is_correct/sample_id all 100% via the canonical path; response splits 97.31% answer_attribution / 2.49% messages / 0.20% output). 712/86,183 result rows (0.83%) have inline samples; full sets (~35k samples) sit behind `source_url` and are accessible only via the orphaned `fetchInstanceLevelData` (no current UI consumer). | #7 |
-
-(More entries land as we work each migration item. See `notes/migration-plan.md` for the full backlog.)
-
-## File format
-
-Each spec follows the same structure so pipeline owner can read them uniformly. Template:
-
-```markdown
-# <Transformation name>
-
-## Rule
-[Plain-English description of what the transformation does]
-
-## Classification
-- [ ] Default-only (do not overwrite when value present)
-- [ ] Unconditional normalization (always apply)
-[explanation]
-
-- [ ] Cleaning / standardization → pipeline (changes a value's content; use per-item workflow)
-- [ ] Reshape / dedup / aggregate → DuckDB SQL (computes a derived view; capture operation, flag for parquet-schema/SQL conversation)
-[explanation — if both halves apply, explain the split]
-
-## Inputs and expected outputs
-[Table: input | expected output | notes / which rule branch this hits]
-
-## Current TS implementation
-- [file:line references]
-- [key helpers/constants]
-
-## Pipeline status
-[Per-rule status against full live cache: matches / disagrees / not implemented]
-
-## Divergences detected
-[Concrete examples of pipeline-vs-TS disagreement, with row counts]
-
-## Migration checklist
-- [ ] Spec written
-- [ ] Tests cover each rule branch
-- [ ] Filed with pipeline owner (link)
-- [ ] Pipeline emits matching values across full corpus
-- [ ] TS code deleted; callers read pipeline fields directly
-```

notes/transformations/reshape-design.md

@@ -1,247 +0,0 @@
-# Reshape-class operations: parquet schema + SQL plan
-
-Drafted 2026-04-28. Synthesis of 5 per-item operation catalogs in `notes/transformations/reshape/` plus the reshape halves of #2 (setup-alias merging) and #13 (timestamp normalization). Reads against the principle in `notes/migration-plan.md` § "Data direction".
-
-This doc is **a discussion artifact for the pipeline owner**, not a unilateral architecture commitment. It proposes a parquet schema delta, sketches the SQL each reshape becomes, recommends materialize-vs-query-time per item, and orders the dependency graph. Pipeline owner has authority to push back on any of it.
-
-## Inventory
-
-| Item | Operation | Per-item doc | Recommendation |
-|---|---|---|---|
-| #2 (reshape half) | Variant bucket reduction (`GROUP BY variant_key, MAX(retrieved_timestamp)`) | `02-setup-alias-merging.md` § "Dual class" | Pipeline emits already-deduped rows in `model_results` parquet (materialize-by-emission) |
-| #3 | Hierarchy flatten + family summary | `reshape/03-hierarchy-flatten.md` | Materialize Steps 1+2 (flat `model_results`); query-time Step 3 (family rollup, category bucketing) |
-| #5 | Composite eval rollup (`/evals/aggregate__<suite>`) | `reshape/05-composite-eval-rollup.md` | Materialize per-(suite, model) rollup table |
-| #6 | Matrix leaderboard synthesis (`/evals/matrix__<suite>`) | `reshape/06-matrix-leaderboard.md` | Materialize wide matrix; query-time row filter |
-| #13 (reshape half) | Timestamp comparison / dedup | `07-timestamp-normalization.md` § "Classification" | SQL inline (`MAX(retrieved_timestamp)`, `ROW_NUMBER()`); no materialization needed |
-| #14 | Score summary stats (per-eval aggregations) | `reshape/14-score-summary-stats.md` | Materialize 9 columns into `eval-list.json` extension |
-| #16 | Per-category benchmark counts (`COUNT(DISTINCT benchmark) GROUP BY model, category`) | `reshape/16-per-category-counts.md` | Materialize `category_stats` per model (current TS is *known wrong*) |
-
-## Required parquet schema delta
-
-The current schema (`scripts/pipeline.py:write_experimental_parquet_table`) is 11 typed metadata columns + `payload_json VARCHAR`. SQL can route on metadata columns but cannot see inside the blob. **Every reshape item above is bottlenecked on the same schema delta: promote nested per-result fields to a relational table.**
-
-### Proposed: `model_results.parquet` (one row per metric × model_result, post-#2 dedup)
-
-The unifying schema across #3, #5, #6, #14, #16 needs roughly the same columns. Designing it once unlocks all of them.
-
-```
--- Routing / partition keys
-model_family_id           VARCHAR  -- joins to model_summaries.parquet
-model_route_id            VARCHAR  -- joins to model_summaries.parquet
-developer                 VARCHAR  -- already a metadata column on model_summaries
-
--- Eval / benchmark identity
-eval_summary_id           VARCHAR  -- already a metadata column
-benchmark                 VARCHAR
-benchmark_family_key      VARCHAR  -- already a metadata column on eval_summaries
-benchmark_family_name     VARCHAR
-benchmark_parent_key      VARCHAR
-benchmark_parent_name     VARCHAR
-benchmark_leaf_key        VARCHAR
-benchmark_leaf_name       VARCHAR
-display_name              VARCHAR  -- post-inheritance from buildFlattenHierarchyContext
-canonical_display_name    VARCHAR
-benchmark_display_name    VARCHAR  -- post #8 cleaning; the user-facing label
-slice_key                 VARCHAR
-slice_name                VARCHAR
-category_key              VARCHAR  -- raw "agentic"/"reasoning"/etc.; mapped form derived via SQL CASE or pipeline-emit (see open question 7); #11 affects accuracy
-
--- Metric identity
-metric_summary_id         VARCHAR
-metric_key                VARCHAR
-metric_name               VARCHAR
-metric_display_name       VARCHAR  -- post #10 cleaning
-metric_canonical_display_name VARCHAR
-metric_unit               VARCHAR
-min_score                 DOUBLE   -- from metric_config; needed by #5, #14
-max_score                 DOUBLE   -- from metric_config; needed by #5, #14
-lower_is_better           BOOLEAN  -- from metric_config; needed by #5, #6, #14
-evaluation_description    VARCHAR  -- from metric_config; needed by #5
-
--- Per-result fields
-evaluation_id             VARCHAR  -- result-level
-raw_model_id              VARCHAR
-model_id                  VARCHAR
-model_result_route_id     VARCHAR
-model_name                VARCHAR
-score                     DOUBLE
-retrieved_timestamp       VARCHAR  -- ISO 8601 once #13 cleaning ships; today unix-seconds-string
-sample_size               BIGINT
-has_generation_config     BOOLEAN  -- presence flag for #14
-detailed_evaluation_results VARCHAR
-instance_level_data       JSON
-source_data               JSON     -- inherited; could be promoted further if hot
-
--- Source metadata (promoted from struct to flat columns; needed by #5, #6, #14)
-source_metadata.evaluator_relationship VARCHAR  -- enum: first_party / third_party / other
-source_metadata.source_type           VARCHAR
-source_metadata.source_name           VARCHAR
-source_metadata.source_organization_name VARCHAR
-
--- Variant identity (depends on #2 cleaning)
-variant_key               VARCHAR  -- pre-resolved per #2 setup-alias normalization
-variant_label             VARCHAR
-
--- Cleaning-driven derived columns (depend on cleaning items)
-benchmark_card            JSON     -- depends on migration #17 inlining (specced as `notes/transformations/11-benchmark-card-attachment.md`)
-```
-
-### Sidecar tables
-
-```
--- composite_membership.parquet — one row per (suite_key, sub_eval_summary_id), drives #5
-suite_key            VARCHAR
-eval_summary_id      VARCHAR
-suite_display_name   VARCHAR  -- from eval-hierarchy.json family display_name; #8 affects this
-```
-
-### What stays in `payload_json`
-
-After this delta, `payload_json` still carries the nested original shape for debugging / migration parity. Once parity is verified per item, fields consumed only via the relational columns can be dropped from the blob.
-
-### Pipeline-side family-membership filter
-
-Important: pipeline should apply the `belongsToModelFamily` filter at emission time so every row in `model_results.parquet` is already correctly assigned to its `model_family_id`. SQL then runs `WHERE model_family_id = ?` and the TS family-membership logic disappears. (Currently the filter happens in TS — `lib/hf-data.ts:1110-1131`. Lifting it upstream eliminates the ~6-line filter from the SQL replacement and avoids the "raw_model_ids set ∪ variant.raw_model_ids ∪ model_info.id ∪ model_family_id" CTE.)
-
-## Per-item SQL sketches
-
-Each operation's full SQL is in its catalog file. This section gives one-paragraph summaries to read alongside the inventory.
-
-### #3 hierarchy flatten — Steps 1+2 materialized, Step 3 query-time
-Tree walk → flat list with variant bucket reduction = **what `model_results.parquet` IS**. Pipeline does it once at build. Consumer Step 3 (family summary, category bucketing) becomes a ~10-line SQL query against `model_results` per request. Removes `flattenHierarchyNode` (~150 lines), `createModelFamilySummary` (~80 lines), `createModelSummary` (~65 lines), `getAggregatedVariantDescriptor`, `sortVariants`, `buildVariantLookup`, `resolveVariantMeta`, `belongsToModelFamily`, `buildModelInfoForVariant`. See `reshape/03-hierarchy-flatten.md` for the full SQL.
-
-### #5 composite eval rollup — fully materialize
-Pipeline pre-computes per-(suite, model) rows + suite-level summary into `composite_eval_rollup.parquet`. Eliminates the 2-21 (outlier 471) `fetchHFEvalDetail` calls per page view. Removes `aggregateBenchmarkSummaries` (168 lines) and the fan-out loop in `getEvalSummaryById`. The rollup SQL itself (per-eval normalize → per-model average → suite stats) is in `reshape/05-composite-eval-rollup.md`.
-
-### #6 matrix leaderboard — materialize wide matrix, query-time row filter
-Pipeline pre-computes per-(suite, model_id) → values map + reconciled metadata using DuckDB `PIVOT` semantics. Eliminates the per-cell walk (median 6, max 471 sub-evals × ~91 models = ~10⁴ tuples per request for `llm_stats`). Row-filter knobs (developer, source_type, top-K) stay query-time SQL — emerging UI need. Removes `buildSingleMetricSuiteMatrixSummary` (~165 lines).
-
-### #13 timestamp comparison — SQL inline, no separate materialization
-Once cleaning #13 lands ISO 8601 timestamps, the comparisons embedded in #3, #5, #6, #14 collapse to `MAX(retrieved_timestamp)` and `ROW_NUMBER() OVER (... ORDER BY retrieved_timestamp DESC)`. The 3 TS normalizers + 8 callers all delete as a unit. No separate parquet artifact.
-
-### #14 score summary stats — materialize 9 columns into eval-list extension
-Pipeline emits all 9 aggregated columns (`models_count`, `avg_score`, `avg_score_norm`, `best_model`, `worst_model`, `evaluator_names`, `source_types`, `latest_source_name`, `third_party_ratio`, `missing_generation_config_count`) per eval. Pattern matches today's existing `eval-list.json` materialization of `models_count` + `top_score`. SQL is textbook `GROUP BY eval_summary_id` + arithmetic + set aggregations. Removes the finalisation loop in `groupEvaluationsByBenchmark` (~45 lines) and the aggregation block in `hfEvalDetailToSummary` (~45 lines).
-
-### #16 per-category counts — materialize `category_stats` per model
-Smallest, clearest reshape. Pipeline emits `category_stats: Record<Category, number>` per model via `COUNT(DISTINCT benchmark_family_key) GROUP BY model_route_id, category`. Replaces the fake `Math.floor(total / categories.length)` distribution in `lib/model-data.ts:369-379` AND the real-but-different distinct-count in `lib/eval-processing.ts:653-666`. Same model gets the same answer everywhere after the migration. **Blocked on #11 category accuracy** — without it, 84% of evals collapse into one `other` bucket.
-
-## Materialize vs query-time — rationale per item
-
-| Item | Recommendation | Why |
-|---|---|---|
-| #2 reshape | Materialize via #3's `model_results` | Bucket reduction is invariant; every consumer needs the same dedup |
-| #3 Steps 1+2 | Materialize | Tree walk is expensive and invariant; every model-detail page needs it |
-| #3 Step 3 | Query-time | Consumer-shape varies (per-variant for detail, per-benchmark for eval-detail); SQL lets each consumer slice without paying for others |
-| #5 | Materialize | Identical answer per consumer; only 13 multi-eval families today; eliminates 2-471 detail fetches per page view |
-| #6 wide matrix | Materialize | Column shape + per-cell winners are deterministic |
-| #6 row filter | Query-time | Forthcoming UI: developer / source-type / top-K filters are consumer-driven |
-| #13 reshape | SQL inline | No artifact needed; comparisons embed in #3/#5/#6/#14 queries |
-| #14 | Materialize | 9 columns × ~587 evals; per-eval scope; no consumer slices these per-category |
-| #16 | Materialize | Trivial size (≤9 categories × ~5,830 models); identical for every consumer |
-
-The pattern: **default to materialize for invariant work**; reserve query-time SQL for consumer-driven slicing (row filters, top-N where N varies, custom faceting).
-
-## Dependency order
-
-```
-Cleaning items (must land FIRST — they emit canonical values that reshape SQL reads)
-  #2  setup-alias merging cleaning half  → variant_key column
-  #13 timestamp normalization cleaning   → ISO 8601 retrieved_timestamp
-  #11 category accuracy improvement      → meaningful category column (84% currently "other")
-  #8  benchmark display names            → benchmark_display_name column
-  #10 metric display name expansion      → metric_display_name column
-
-Schema delta (pipeline-owner conversation; ~one PR in pipeline repo)
-  Promote nested fields to relational `model_results.parquet`
-  Add sidecar `composite_membership.parquet`
-  Apply pipeline-side family-membership filter at emission time
-
-Reshape items (in roughly ascending complexity; all unblocked once schema lands)
-  #16 per-category counts        — smallest; gated on #11 (category accuracy)
-  #14 score summary stats        — gated on #2, #13; #4 already shipped
-  #3  hierarchy flatten          — gated on #2, #13
-  #5  composite eval rollup      — gated on #13, #8, #17 (benchmark-card); benefits from #14 landing first
-  #6  matrix leaderboard         — gated on #13, #8, #17 (benchmark-card), #1 (identity canonicalization); benefits from #3 landing first
-```
-
-The schema delta is the load-bearing pipeline-owner conversation. Once `model_results.parquet` exists in the right shape, **the 6 reshape items can be implemented in parallel**, each as a single SQL query in `lib/duckdb-data.ts` replacing the current `JSON.parse(payload_json) → TS adapter` chain.
-
-## Cross-cutting TS-as-spec quirks for pipeline-owner attention
-
-These are decisions the pipeline owner will face when implementing the schema + emission. Not "fix these" — "these are choices to make".
-
-1. **`>=` vs `>` tie-break in bucket reduction.** TS uses `>=` (last-iteration-order wins on timestamp tie) in #3, #6, #14. SQL `ROW_NUMBER() OVER (... ORDER BY retrieved_timestamp DESC)` is implementation-defined on ties unless an explicit secondary key is added. Recommendation: tie-break on `evaluation_id DESC` for stability. Document the divergence; tie-collisions are rare in production (timestamps are floats with microsecond precision).
-
-2. **Variant identity computed twice in #3.** `lib/hf-data.ts resolveVariantMeta` (Step 1b) and `lib/eval-processing.ts getAggregatedVariantDescriptor` (Step 3) re-derive variant identity from different inputs. They can disagree (e.g. `"20240620-thinking"` vs `"2024-06-20"`). Should reconcile to a single canonical `variant_key` once #2 cleaning lands. Likely produces subtle off-by-one variant counts today.
-
-3. **`evaluator_names` always `[]` on the active path.** `hfEvalDetailToSummary` initializes to `[]` and never populates. The eval-card UI's "Evaluators" pill shows 0 for every eval today — latent bug. Pipeline can fix-by-canonicalization (emit the sorted DISTINCT set) and accept that the pill will start showing real numbers. Flag as deferred product decision.
-
-4. **`models_count` divergence.** TS recomputes it post-#2-dedup; pipeline emits it pre-#2-dedup. Disagrees for any model with merged variants (notably anything with `additional_details.mode` ∈ {prompt/fc/thinking}). Resolves naturally when #2 lands and pipeline's emitted value matches TS's recomputed.
-
-5. **Sort direction in #5 from FIRST sub-eval's `lower_is_better`.** Fragile when a suite mixes higher-is-better and lower-is-better metrics. Order of `summaries[0]` is whatever `eval-hierarchy.json family.eval_summary_ids` lists first. Pipeline must preserve order or replicate the choice. Recommend: pick `lower_is_better=false` if any sub-eval is higher-is-better.
-
-6. **Two implementations of `category_stats` produce different answers** (#16). Path A (grid) is fake distribution; Path B (detail page) is real `COUNT(DISTINCT benchmark)`. Same model, two pages, two answers. Materialization eliminates both implementations.
-
-7. **Cell tie-break in #6 is "last in iteration order".** Whether SQL `ROW_NUMBER() ORDER BY retrieved_timestamp DESC` matches TS depends on whether pipeline emits `metric.model_results[]` in retrieved_timestamp-DESC order. **Verify this assumption before flipping the SQL on.** If pipeline order is non-deterministic, the SQL is a "freshest wins" *upgrade* over TS's "iteration order wins" — likely fine, but call out in the migration commit.
-
-8. **Score normalization in #5 happens BEFORE averaging.** `normalize(score)` per sub-eval (using each sub-eval's own min/max), then arithmetic mean across sub-evals. The "obvious" alternative (average raw scores then normalize) produces different numbers when sub-evals have different score ranges. SQL must do `AVG(per_eval.normalized_score)`, not `AVG(suite_components.normalized_score)`.
-
-9. **Suite-level `avg_score` in #5 is avg-of-per-model-avgs, not avg-of-all-component-scores.** Diverges when sub-eval coverage is unbalanced. SQL `AVG(per_model.avg_normalized_score)`, not `AVG(suite_components.normalized_score)`.
-
-10. **Empty-metric short-circuit in #14 puts a benchmark display name into `latest_source_name`.** `lib/model-data.ts:785` sets `latest_source_name = getBenchmarkDisplayName(benchmarkKey)` when an eval has zero metrics — a *display name* string in a *source name* field. Almost certainly a placeholder bug. Pipeline can fix-by-canonicalization (emit `null` and chase down any consumer that breaks); flag as deferred product decision.
-
-11. **Two GROUP BY entry points in #14 with different score-source semantics.** `groupEvaluationsByBenchmark` iterates `eval_.evaluation_results` (multi-metric possible); `hfEvalDetailToSummary` iterates `metric.model_results` of *only* the first metric. Active read path for eval-detail pages is `hfEvalDetailToSummary` — pipeline emission should match its single-primary-metric semantics, since that's what users see today.
-
-## Open questions for pipeline owner
-
-1. **How relational should parquet go?** Promote nested fields to typed columns (this doc's recommendation) or stay closer to current `payload_json` blob and rely on DuckDB's JSON functions? Going relational unlocks ~10× more SQL work but is a bigger schema change.
-2. **Where does `composite_membership` live?** Sidecar parquet table (this doc's recommendation) or stays nested in `eval-hierarchy.json`?
-3. **Score-normalization ownership.** Pipeline pre-normalizes `score_norm` as a column, OR DuckDB does at query time using `min_score`/`max_score`/`lower_is_better` columns? This doc assumes the latter (per-row columns); pre-normalizing is also viable.
-4. **Pipeline emission order of `metric.model_results[]`.** Is it deterministic? Sorted by anything? Affects whether SQL's `ROW_NUMBER() ORDER BY retrieved_timestamp DESC` matches TS's "last-wins-iteration" semantics for #6 cell tie-breaks.
-5. **Pipeline-side family-membership filter.** This doc recommends pipeline filters at emission time (so SQL just `WHERE model_family_id = ?`). Alternative: SQL replicates the `belongsToModelFamily` set logic. Pipeline-side is much cleaner.
-6. **Variant identity reconciliation (#3 quirk #2).** Double-pass should collapse to single canonical `variant_key` once #2 cleaning lands. Pipeline owner picks the canonical rule.
-7. **`category_stats` vs `category` column for #16.** Pipeline emits `category_stats: Record<Category, number>` directly on each model card (consumer-shape), OR pipeline emits per-row `category` and DuckDB pivots at query time? Consumer-shape is simpler; query-time gives flexibility for free.
-
-## Migration sequencing
-
-### Phase 1 — cleaning items land (in pipeline)
-Pipeline emits canonical values per the existing per-item workflow (`notes/migration-plan.md` § "Per-item workflow"). Status as of 2026-04-28:
-
-- **Specced + ready for handoff:** #2 (variant_key), #13 (ISO 8601), #8 (benchmark_display_name), #10 (metric_display_name). Plus #17 benchmark-card inlining (specced as `11-benchmark-card-attachment.md`) and #1 identity canonicalization. See per-item specs in `notes/transformations/`.
-- **Blocked, not part of Phase 1:** #11 category accuracy is gated on pipeline-side classification work — pipeline currently emits `category: "other"` on 84% of evals. #16 reshape is gated on #11; if #11 doesn't land in time, #16 ships against the current "84% other" data and shows the limitation, OR ports `inferCategoryFromBenchmark` upstream as a workaround.
-
-**No reshape SQL touches yet** in this phase.
-
-### Phase 2 — schema delta lands (in pipeline)
-Pipeline emits `model_results.parquet` (the unifying relational table) alongside the existing `payload_json` blob. Both columns live in parquet during the parity window — TS continues to read `payload_json`, the new SQL reads relational columns. This is a **single PR in `eval_cards_backend_pipeline`** for the schema, plus the family-membership filter.
-
-### Phase 3 — DuckDB queries replace TS reshape, one item at a time
-Per item (in any order, since they're independent once schema lands):
-1. Write SQL query in `lib/duckdb-data.ts` (alongside existing `JSON.parse(payload_json) → TS adapter` path).
-2. Reshape-class snapshot test becomes the **TS-vs-SQL parity gate** (per `notes/testing-strategy.md` § "Reshape-class items: testing addendum"). Snapshot is committed; SQL output is computed at test time; equality is the gate.
-3. Once parity holds across the full corpus (verified via `pnpm audit-adapters --diff`), delete the TS implementation. Callers switch to read the SQL/materialized output.
-
-Recommended order within Phase 3 (smallest unblocked first to build confidence):
-1. **#14 score summary stats** — clean Phase-1 dependencies (#2, #13). Fixes the 0-evaluators latent bug, materializes 9 columns. Best first item.
-2. **#16 per-category counts** — smallest reshape mechanically. *Caveat:* gated on #11 category accuracy (currently blocked on pipeline-side classification work — pipeline emits `category: "other"` on 84% of evals). Either ship against the "84% other" data and show the limitation, OR port `inferCategoryFromBenchmark` upstream as part of Phase 1.
-3. **#3 hierarchy flatten** — the foundational reshape; Steps 1+2 are the unifying `model_results` use, Step 3 is the consumer surface. Clean Phase-1 dependencies.
-4. **#5 composite eval rollup** — eliminates the 2-471 detail-fetch fan-out. Best after #14 ships (so per-eval stats are materialized first).
-5. **#6 matrix leaderboard** — biggest behavior change (PIVOT); benefits from #3 + #5 patterns being in place.
-
-### Phase 4 — cleanup
-Once all reshape items are SQL/materialized: drop unused fields from `payload_json` blobs (or leave for debugging). Delete the runtime `flattenModelEvaluations` → `createModelFamilySummary` chain in `lib/duckdb-data.ts toModelSummary`. Update `notes/transformations/README.md` index to reflect completed items.
-
-## Cross-references
-
-- `notes/migration-plan.md` § "Data direction" — the cleaning vs reshape principle this doc operates under
-- `notes/transformations/README.md` § "Where it belongs" — per-spec classification framework
-- `notes/testing-strategy.md` § "Reshape-class items: testing addendum" — how reshape items get tested (TS-vs-SQL parity)
-- `notes/transformations/02-setup-alias-merging.md` — reshape half of #2 (cited above)
-- `notes/transformations/07-timestamp-normalization.md` — reshape half of #13 (cited above)
-- `notes/transformations/reshape/{03,05,06,14,15,16}-*.md` — full per-item operation catalogs
-
-## Open items for follow-up
-
-1. **`PIPELINE_CATEGORY_MAP` location decision.** TS-side map vs pipeline-emit-mapped vs SQL `CASE`. Coupled to #11.
-2. **Audit script for TS-vs-SQL parity per reshape item** — pattern to be established with #14 (the first reshape to ship), then templated for #16, #3, #5, #6.
-3. **Pipeline-owner review.** This doc is a proposal; nothing is committed until pipeline owner has weighed in on the schema delta and the open questions above.

notes/transformations/reshape/03-hierarchy-flatten.md

@@ -1,321 +0,0 @@
-# Hierarchy flatten + family summary — reshape operation
-
-Drafted 2026-04-28. Migration item #3 in `notes/migration-plan.md`. First entry in the reshape catalog (companion to nothing in `notes/transformations/` today — those are cleaning specs and follow a different shape). Will be referenced from the synthesis at `notes/transformations/reshape-design.md` when it lands.
-
-## Framing reminder
-
-This is a **reshape**, not a cleaning item. The migration target is *not* "pipeline emits these exact `BenchmarkEvaluation[]` objects." It's "pipeline emits relational rows that DuckDB can flatten + group + dedup with a query." The TS implementation is the operational spec; the SQL replacement preserves behavior, not implementation.
-
-Per `notes/migration-plan.md` § "Data direction": cleaning belongs in pipeline emission, reshape belongs in DuckDB SQL. This item is the reference reshape. It is also the canonical example for the schema-relationality conversation — the existing parquet schema (one `payload_json VARCHAR` blob per record) cannot do this work; pipeline must promote the nested fields to typed columns first.
-
-## Operation (in SQL terms)
-
-### Inputs
-- **Source rows:** model-summary records in `model_summaries.parquet` (one per `model_family_id`). Today: each row is `(record_type, model_route_id, model_family_id, …, payload_json)` and `payload_json` carries the entire `HFModelDetail` blob — `hierarchy_by_category`, `variants[]`, `raw_model_ids`, `model_info`.
-- **Nested fields the operation reads (currently inside `payload_json`):**
-  - `hierarchy_by_category: Record<categoryKey, HFModelHierarchyNode[]>` — the tree. Walk recursively via `subtasks`.
-  - For each leaf `metric` on each node: `metric.model_results[]` — submission rows with `score`, `retrieved_timestamp`, `source_metadata`, `raw_model_id`, `model_id`, `model_route_id`, `model_name`, `developer`, `evaluation_id`.
-  - Tree context fields propagated down: `eval_summary_id`, `benchmark`, `benchmark_family_key/name`, `benchmark_parent_key/name`, `benchmark_leaf_key/name`, `display_name`, `canonical_display_name`, `category` (the category key from `hierarchy_by_category` map, mapped via `PIPELINE_CATEGORY_MAP`).
-  - Metric-level fields: `metric_summary_id`, `metric_key`, `metric_name`, `display_name`, `canonical_display_name`, `metric_config`, `slice_key`, `slice_name`.
-  - For variant identity: `detail.variants[]` (`variant_key`, `variant_label`, `raw_model_ids[]`).
-  - For ownership filtering: `detail.raw_model_ids[]` ∪ `variant.raw_model_ids[]` ∪ `model_info.id` ∪ `model_family_id`.
-
-### Step 1 — flatten the tree (one row per metric × model_result)
-Recursively walk `hierarchy_by_category` (root nodes per category, then `node.subtasks[]`), inheriting context (eval_summary_id, benchmark, family/parent/leaf names, source_data) down through subtask levels. For each leaf-metric-with-results, emit one row per `model_result` filtered to those that belong to this model family (see Step 1a).
-
-Conceptual emitted shape per row:
-```
-(model_family_id, eval_summary_id, category_key,
- benchmark, benchmark_family_key, benchmark_parent_key, benchmark_leaf_key,
- benchmark_family_name, benchmark_parent_name, benchmark_leaf_name,
- display_name, canonical_display_name,
- slice_key, slice_name, source_data,
- metric_summary_id, metric_key, metric_name, metric_config,
- evaluation_id, raw_model_id, model_id, model_route_id, model_name, developer,
- score, retrieved_timestamp, source_metadata,
- detailed_evaluation_results, instance_level_data,
- variant_key  /* resolved per Step 1b */)
-```
-
-#### Step 1a — `belongsToModelFamily` filter
-A `model_result` belongs to this model family iff any of:
-- `normalize(result.model_route_id) == normalize(detail.model_route_id)`, OR
-- `normalize(result.raw_model_id) ∈ rawModelIds`, OR
-- `normalize(result.model_id) ∈ rawModelIds`
-
-where `rawModelIds = detail.raw_model_ids ∪ flat(variants[].raw_model_ids) ∪ detail.model_info.id ∪ detail.model_family_id`, all lowercased + trimmed. (See `lib/hf-data.ts:1110-1131` and `:1386-1395`.)
-
-#### Step 1b — `resolveVariantMeta` lookup
-Given a `model_result`, derive its `variant_key`:
-1. Build a lookup `variantLookup: Map<normalized_raw_model_id, variant_key>` from `detail.variants[*].raw_model_ids`.
-2. Try `result.raw_model_id` then `result.model_id` against the lookup.
-3. If no match AND `detail.variants.length === 1` → use that single variant's key/label.
-4. Else fallback to first non-empty candidate id, or `detail.model_info.variant_key`, or literal `"default"`.
-
-This is the same `variant_key` produced by item #2 (setup-alias merging) with normalization already applied — so post-#2 the result of this lookup should equal the canonicalized `setup_alias_key` the pipeline emits.
-
-### Step 2 — group by `(eval_summary_id, metric_summary_id, variant_key)` within each model
-
-Each leaf-metric × variant-bucket becomes ONE `BenchmarkEvaluation`. Inside a bucket multiple `model_results` are merged:
-- `evaluation_results[]` ← append every result's score record (concatenate, no dedup).
-- `latestTimestamp` ← `MAX(retrieved_timestamp)` across the bucket.
-- `source_metadata` ← the `source_metadata` of the row whose `retrieved_timestamp == latestTimestamp` (first one wins on ties because `>=`, see TS-quirk #1 below).
-- `inlineSamples` ← first non-empty `parseInstanceLevelData(result.instance_level_data)` encountered; subsequent rows do not overwrite if `existing.inlineSamples` already has values.
-
-The output `evaluation_id` for the merged record is `${metric.metric_summary_id}__${variantKey}`.
-
-### Step 3 — group by `family-id` and reshape into `ModelEvaluationSummary`
-
-Take the flat `BenchmarkEvaluation[]` and:
-1. **Re-bucket by variant** (in `createModelFamilySummary`, `lib/eval-processing.ts:453-532`) using `getAggregatedVariantDescriptor(eval.model_info)`. Note: this is a SECOND variant resolution that re-derives variant identity from `model_info` rather than using the `variant_key` set in Step 1b. The two should agree post-#2; today they can disagree (see TS-quirk #2).
-2. **Per variant:** call `createModelSummary` (`lib/eval-processing.ts:280-344`) which:
-   - Buckets evaluations by `category` (the `BenchmarkEvaluation.category` field set in Step 1, falling back to `inferCategoryFromBenchmark(result.evaluation_name)` when missing).
-   - Computes `total_evaluations = SUM(evaluations[i].evaluation_results.length)`.
-   - Computes `last_updated = MAX(retrieved_timestamp)` rendered as ISO string.
-   - Lists `categories_covered = DISTINCT(category)`.
-3. **Sort variants** by `version_date DESC, total_evaluations DESC, variant_label ASC` (`sortVariants`, `lib/eval-processing.ts:436-451`).
-4. **Family-level rollup:** call `createModelSummary(allEvaluations)` — same shape but at the family level — then overlay `model_family_id`, `model_route_id`, `model_family_name` from `getCanonicalModelIdentity(evaluations[0].model_info)`, and `raw_model_ids = SORTED DISTINCT(eval.model_info.id)`.
-
-### Output
-- Intermediate: `BenchmarkEvaluation[]` (the flat-list output of `flattenModelEvaluations`).
-- Final: `ModelEvaluationSummary` (the family-rollup output of `createModelFamilySummary`).
-
-Both are presentation shapes consumed by `lib/model-data.ts` getters and the model-detail pages.
-
-## Current TS implementation
-
-| Concern | Location | Notes |
-|---|---|---|
-| Tree walk + per-metric flatten + variant bucket reduction | `lib/hf-data.ts:1228-1378` (`flattenHierarchyNode`) | Recursive; inherits context via `buildFlattenHierarchyContext` |
-| Public entry (per-model) | `lib/hf-data.ts:1384-1414` (`flattenModelEvaluations`) | Iterates `hierarchy_by_category` keys; maps category via `PIPELINE_CATEGORY_MAP` (`lib/hf-data.ts:1425-1435`) |
-| Variant lookup builder | `lib/hf-data.ts:1063-1079` (`buildVariantLookup`) | |
-| Per-result variant resolver | `lib/hf-data.ts:1081-1108` (`resolveVariantMeta`) | 4-tier fallback |
-| Family-membership filter | `lib/hf-data.ts:1110-1131` (`belongsToModelFamily`) | |
-| Per-variant model_info synthesizer | `lib/hf-data.ts:1133-1155` (`buildModelInfoForVariant`) | |
-| Inline-samples parser | `lib/hf-data.ts:933` (`parseInstanceLevelData`) | Tolerant JSON/array parser |
-| Hierarchy context builder | `lib/hf-data.ts:1175-1226` (`FlattenHierarchyContext` + `buildFlattenHierarchyContext`) | Inheritance-with-defaults for tree context |
-| Bucket-merge timestamp comparator | `lib/hf-data.ts:1049-1061` (`toComparableTimestamp`) | Variant B from spec #07 — has the `parseFloat` quirk |
-| Family rollup | `lib/eval-processing.ts:453-532` (`createModelFamilySummary`) | |
-| Per-summary aggregation | `lib/eval-processing.ts:280-344` (`createModelSummary`) | Buckets by category, sums totals, max-timestamp |
-| Variant descriptor (re-derived) | `lib/eval-processing.ts:360-434` (`getAggregatedVariantDescriptor`) | Re-runs setup-alias logic, this time from `model_info` |
-| Variant sort | `lib/eval-processing.ts:436-451` (`sortVariants`) | |
-
-### Caller sites
-- `lib/model-data.ts:1495, 1497, 1518, 1520, 1530, 1532` — `getModelSummaryById` and slug-retry siblings.
-- `lib/duckdb-data.ts:189-194` (`toModelSummary`) — DuckDB read path currently re-runs both adapters against the `payload_json` blob to match JSON-path output exactly. **This is the call site the SQL replacement targets.**
-- `lib/eval-processing.ts:825` — batch path in `processEvaluationsToCards` (developer-aggregate flow).
-
-## Required parquet columns
-
-The current parquet schema (per `notes/migration-plan.md` § "Data direction"): 11 typed metadata columns + `payload_json VARCHAR`. SQL can route on the metadata columns but cannot see inside the blob. To do this operation in SQL, pipeline needs to promote the nested fields to a separate relational table.
-
-### Proposed: `model_results.parquet` (one row per metric × model_result, post-flatten)
-
-```
-model_family_id            VARCHAR  -- routing key (matches model_summaries.parquet)
-model_route_id             VARCHAR  -- routing key (matches model_summaries.parquet)
-eval_summary_id            VARCHAR  -- from hierarchy node
-category_key               VARCHAR  -- raw pipeline key ("agentic", "reasoning", ...) — TS maps via PIPELINE_CATEGORY_MAP at read time
-benchmark                  VARCHAR
-benchmark_family_key       VARCHAR
-benchmark_family_name      VARCHAR
-benchmark_parent_key       VARCHAR
-benchmark_parent_name      VARCHAR
-benchmark_leaf_key         VARCHAR
-benchmark_leaf_name        VARCHAR
-display_name               VARCHAR  -- post-inheritance from buildFlattenHierarchyContext
-canonical_display_name     VARCHAR  -- post-inheritance
-slice_key                  VARCHAR
-slice_name                 VARCHAR
-source_data                JSON     -- inherited; either struct or string[]
-metric_summary_id          VARCHAR
-metric_key                 VARCHAR
-metric_name                VARCHAR
-metric_display_name        VARCHAR
-metric_canonical_display_name VARCHAR
-metric_config              JSON
-evaluation_id              VARCHAR  -- result-level
-raw_model_id               VARCHAR
-model_id                   VARCHAR
-model_result_route_id      VARCHAR
-model_name                 VARCHAR
-developer                  VARCHAR
-score                      DOUBLE
-retrieved_timestamp        VARCHAR  -- ISO 8601 once #13 ships; today unix-seconds-string
-source_metadata            JSON
-detailed_evaluation_results VARCHAR
-instance_level_data        JSON
-variant_key                VARCHAR  -- pre-resolved per Step 1b (depends on #2 emitting it)
-```
-
-### Cross-item dependencies for the schema
-- **#2 setup-alias merging** — `variant_key` must be the *normalized* key (post-`isSetupAliasQualifier` rules). Without this, Step 1b stays in TS or in a CTE that re-derives.
-- **#13 timestamp normalization** — `retrieved_timestamp` must be a single canonical format (recommended: ISO 8601 string, lexicographic sort = chronological sort). Until then SQL needs `epoch_ms(CAST(retrieved_timestamp AS DOUBLE) * 1000)` plus a fallback for ISO strings — workable but the TS quirks (Variant B's `parseFloat` bug per spec #07) become unobservable only once the format is unified.
-- **#4 source-metadata** (DONE) — pipeline already emits `source_metadata` on every row; the typed `source_metadata JSON` column is straightforward.
-- **Category map** — TS currently maps the raw `category_key` ("agentic" → "Agentic", "coding" → "General", etc.) via `PIPELINE_CATEGORY_MAP` at read time. For this reshape we recommend leaving the raw key in parquet and applying the case mapping in SQL via a small CASE expression (or keeping it client-side until #11 lands).
-- **Family-membership filter** — Step 1a needs `raw_model_ids` set per family. Easiest path: pipeline filters at emission time so every row in `model_results.parquet` is already guaranteed to belong to its `model_family_id`. Then SQL just `WHERE model_family_id = ?` and the TS `belongsToModelFamily` logic disappears.
-
-### Independent of schema decisions
-- **The variant bucket reduction itself** is pure SQL once timestamps are comparable; doesn't need any new schema beyond the row-per-result table.
-- **`createModelSummary`'s category bucketing + totals** is also pure SQL once a row-per-result table exists.
-
-## Sketch SQL query
-
-Assumes the proposed `model_results.parquet` schema, ISO-8601 timestamps, and pipeline-side family-membership filtering.
-
-### Step 1+2 — flat list, with variant-bucket dedup applied (one row per `(eval_summary_id, metric_summary_id, variant_key)`)
-
-```sql
-WITH
-results AS (
-  SELECT *
-  FROM read_parquet('model_results.parquet')
-  WHERE model_family_id = ?  -- single-model query
-),
--- Within a (eval_summary_id, metric_summary_id, variant_key) bucket, pick the
--- row whose retrieved_timestamp is freshest. Aggregate the rest as arrays.
-freshest AS (
-  SELECT *
-  FROM results
-  QUALIFY ROW_NUMBER() OVER (
-    PARTITION BY eval_summary_id, metric_summary_id, variant_key
-    ORDER BY retrieved_timestamp DESC, evaluation_id DESC  -- tie-break stable
-  ) = 1
-),
-bucket_results AS (
-  SELECT
-    eval_summary_id,
-    metric_summary_id,
-    variant_key,
-    list({
-      evaluation_name: metric_name,
-      display_name: metric_display_name,
-      canonical_display_name: metric_canonical_display_name,
-      metric_summary_id: metric_summary_id,
-      metric_key: metric_key,
-      evaluation_timestamp: retrieved_timestamp,
-      source_data: source_data,
-      metric_config: metric_config,
-      score_details: { score: score },
-      detailed_evaluation_results_url: detailed_evaluation_results
-    } ORDER BY retrieved_timestamp) AS evaluation_results,
-    -- first non-empty inline-samples in insertion order
-    list_filter(list(instance_level_data ORDER BY retrieved_timestamp), x -> x IS NOT NULL)[1]
-      AS inline_samples
-  FROM results
-  GROUP BY eval_summary_id, metric_summary_id, variant_key
-)
-SELECT
-  '0.2.2' AS schema_version,
-  f.eval_summary_id,
-  f.metric_summary_id || '__' || f.variant_key AS evaluation_id,
-  f.retrieved_timestamp,
-  f.benchmark,
-  f.display_name,
-  f.canonical_display_name,
-  f.category_key AS category,  -- map to CategoryType in app layer or via CASE here
-  f.benchmark_family_key, f.benchmark_family_name,
-  f.benchmark_parent_key, f.benchmark_parent_name,
-  f.benchmark_leaf_key,   f.benchmark_leaf_name,
-  f.slice_key, f.slice_name,
-  f.source_data,
-  f.source_metadata,
-  -- model_info synthesized per buildModelInfoForVariant
-  struct_pack(
-    id := COALESCE(f.raw_model_id, f.model_id),
-    name := f.model_name,
-    developer := f.developer,
-    model_version := CASE WHEN f.variant_key <> 'default' THEN f.variant_key ELSE NULL END
-  ) AS model_info,
-  br.evaluation_results,
-  br.inline_samples AS detailed_evaluation_results_per_samples
-FROM freshest f
-JOIN bucket_results br USING (eval_summary_id, metric_summary_id, variant_key);
-```
-
-### Step 3 — family summary (rolling up the flat list)
-
-```sql
-WITH flat AS ( /* the query above */ ),
-per_variant AS (
-  SELECT
-    variant_key,
-    MAX(retrieved_timestamp) AS last_updated,
-    SUM(len(evaluation_results)) AS total_evaluations,
-    array_agg(DISTINCT category) AS categories_covered,
-    -- evaluations grouped by category as a struct of arrays
-    map_from_entries(
-      array_agg(struct_pack(category := category, eval := flat))
-        OVER (PARTITION BY variant_key)  -- pseudocode; real shape uses GROUP BY + LIST_AGG
-    ) AS evaluations_by_category
-  FROM flat
-  GROUP BY variant_key
-),
-family AS (
-  SELECT
-    MAX(retrieved_timestamp) AS last_updated,
-    SUM(len(evaluation_results)) AS total_evaluations
-  FROM flat
-)
-SELECT * FROM per_variant, family;
-```
-
-(The `evaluations_by_category` map is awkward in SQL — most consumers will end up assembling it client-side from the flat-list query. That's fine; the operationally important reshape is Steps 1+2.)
-
-## Materialize vs query-time
-
-- **Materialize upstream if:** the answer is identical for every consumer. The flat `model_results` table absolutely should be materialized — every consumer of model-detail pages needs it, and re-walking the tree on every request is exactly what we're trying to leave behind.
-- **Query-time SQL if:** consumers slice differently. Step 3 (family summary) is consumer-shape — different pages want different slices (per-category, per-variant, per-benchmark). Run it as a query.
-
-**Recommendation: materialize Steps 1+2 (the flat post-bucket-reduction table) into `model_results.parquet`; compute Step 3 (family summary, category bucketing) at query time.**
-
-Rationale: the tree-walk + variant-bucket reduction is invariant work that every model-detail page needs identically — pre-computing it removes the recursive walk from the request hot path. The category bucketing and variant rollup, by contrast, are presentation-shape transforms that vary per page (model-detail wants per-variant, eval-detail wants per-benchmark, etc.) — keeping them in SQL lets each consumer slice without paying for the others' shapes.
-
-A nice property: this split also lets `eval_summaries.parquet` and `developer_summaries.parquet` reuse the same `model_results.parquet` rows — they're currently re-walking the same tree from their own angles.
-
-## TS-as-spec quirks
-
-The reshape SQL must preserve these or explicitly defer them as product decisions. Don't unilaterally fix.
-
-1. **`>=` not `>` in the bucket-merge timestamp comparison.** `lib/hf-data.ts:1310-1313`: when a later result ties an earlier result's timestamp exactly, the LATER one wins (overwrites `latestTimestamp` and `sourceMetadata`). With `ROW_NUMBER() … ORDER BY retrieved_timestamp DESC` SQL gets implementation-defined tie-breaking unless you add an explicit secondary sort. Recommendation: tie-break on `evaluation_id DESC` (or another stable secondary key) and document that it differs from TS's "later in iteration order wins" for cross-bucket ties. Quantify on actual data before shipping; if collisions are rare, the divergence is acceptable.
-
-2. **Variant identity is computed twice with potentially different rules.** Step 1b (`resolveVariantMeta` in `lib/hf-data.ts`) uses `variants[].raw_model_ids` lookup — strict id match. Step 3 (`getAggregatedVariantDescriptor` in `lib/eval-processing.ts:394-434`) re-derives from `model_info.additional_details.mode` + `getCanonicalModelIdentity`. These can disagree: e.g. a result whose Step-1b `variant_key` is `"20240620-thinking"` becomes Step-3 `variant_key: "2024-06-20"` (setup-alias merging collapses the qualifier). The SQL replacement should produce a single canonical `variant_key` per result post-#2 and not re-bucket later. Today's TS double-bucketing is a likely source of subtle off-by-one variant counts; capture it as observed behavior and decide the desired single-pass semantics with the pipeline owner before reshaping.
-
-3. **Inline-samples merge is "first non-empty wins, no overwrite"**, NOT "freshest wins." If the first result emitted into a bucket has no `instance_level_data` and a later result does, that later one's samples are kept. If both have samples, the FIRST one's are kept regardless of timestamp. This is a separate code path from the timestamp-keyed metadata merge. SQL replacement preserves this with `list_filter(list(instance_level_data ORDER BY <iteration order>), x -> x IS NOT NULL)[1]` — but iteration order in TS is `metric.model_results[]` array order from the parquet, which SQL needs an explicit sort to mimic. This is fragile; flag for synthesis.
-
-4. **Category in Step 1 is the per-node `category_key` from `hierarchy_by_category` keys, not from each `model_result`.** Two results landing in different category branches but for the same `(metric_summary_id, variant_key)` pair would produce two separate `BenchmarkEvaluation` entries, not one merged. This is structural to how the tree walk works; SQL needs to include `category_key` (or `eval_summary_id`, which is per-leaf) in the GROUP BY.
-
-5. **`PIPELINE_CATEGORY_MAP` lossy collapses.** `coding`, `instruction_following`, `language_understanding` all map to `"General"`. This is intentional (per the comment at `lib/hf-data.ts:1419-1424`) until #11 (category accuracy) ships. The SQL should preserve the same mapping at read time, or push the map into a typed column emitted by pipeline.
-
-## Cross-item dependencies
-
-### Cleaning items that must land first
-- **#2 setup-alias merging** — provides the canonical `variant_key` Step 1b currently re-derives. Without it, the SQL needs a CTE that re-runs the alias-qualifier rules in DuckDB (workable but ugly).
-- **#13 timestamp normalization** — provides comparable `retrieved_timestamp` so `MAX()` / `ROW_NUMBER() ORDER BY` are correct. Today's mixed-format strings sort lexicographically wrong (`"1774096306"` < `"2024-..."`).
-- **#4 source-metadata** — DONE. Step 2's "freshest source_metadata wins" needs every row to carry it; this is already true.
-
-### Reshape items this feeds
-- **#5 composite eval rollup** (`aggregateBenchmarkSummaries`) — also walks model evaluations per benchmark; would consume the flat `model_results` table directly rather than re-walking trees.
-- **#6 matrix leaderboard synthesis** — same.
-- **#14 score summary stats** — finalizes the per-benchmark grouping; consumes the flat table.
-
-### Pipeline-side prerequisites
-- Pipeline must emit a relational `model_results.parquet` (or equivalent table) with the columns above. This is the schema-relationality conversation flagged in `notes/migration-plan.md`.
-- Pipeline-side family-membership filter (so the SQL doesn't have to replicate `belongsToModelFamily`).
-- Decide where `PIPELINE_CATEGORY_MAP` lives (pipeline emits mapped string vs SQL CASE vs TS-side map).
-
-## Migration checklist
-
-- [x] Operation cataloged
-- [ ] Schema delta proposed (in synthesis doc `notes/transformations/reshape-design.md`)
-- [ ] SQL query reviewed by pipeline owner
-- [ ] Pipeline emits `model_results.parquet` with the relational columns
-- [ ] DuckDB query implemented in `lib/duckdb-data.ts` (replaces `toModelSummary`'s `flattenModelEvaluations(payload) → createModelFamilySummary(...)` chain at line 189-194)
-- [ ] Parity test (TS vs SQL output) passes — see `notes/testing-strategy.md` § "Reshape-class items: testing addendum" for the snapshot-as-parity-gate pattern. Snapshot lives at `tests/adapters/flatten-model-evaluations.test.ts` (per the deferred plan in testing-strategy.md § "Test-additions deferred to specific migration items") and `tests/adapters/create-model-family-summary.test.ts`.
-- [ ] TS code deleted: `flattenHierarchyNode`, `flattenModelEvaluations`, `buildFlattenHierarchyContext`, `buildVariantLookup`, `resolveVariantMeta`, `belongsToModelFamily`, `buildModelInfoForVariant` in `lib/hf-data.ts`; `createModelFamilySummary`, `createModelSummary`, `getAggregatedVariantDescriptor`, `sortVariants` in `lib/eval-processing.ts`. Callers in `lib/model-data.ts:1495-1532` and `lib/eval-processing.ts:825` switch to the SQL-backed reader.
-
-## Future product decisions (deferred)
-
-- Whether the double-pass variant resolution (Step 1b vs Step 3) should be reconciled to a single `variant_key`. Likely yes once #2 lands; flag for the synthesis.
-- Whether `>=` (TS today) or `>` (more conventional "first wins ties") is the desired bucket-merge tie-break. Current TS quirk is observable but rare in production.
-- Whether `evaluations_by_category` should be re-shaped client-side from a flat-list query or carried as a nested column. The latter is awkward in SQL/Parquet; recommendation is the former.

notes/transformations/reshape/05-composite-eval-rollup.md

@@ -1,230 +0,0 @@
-# Composite eval rollup
-
-Drafted 2026-04-28. Migration item #5 in `notes/migration-plan.md`. Reshape-class operation catalog (not a rule-table replication). See `notes/transformations/README.md` § "Where it belongs: cleaning vs reshape" and `notes/migration-plan.md` § "Data direction" for the framing this spec follows.
-
-## Framing reminder
-
-This is a **reshape** item, not a cleaning item. The output captures the *operation* (group-by keys, aggregation, ordering) so the parquet-schema + SQL conversation can happen. We are not line-by-line porting `aggregateBenchmarkSummaries` into Python; we're describing what the SQL needs to compute. TS-as-is is the spec for behaviour — including the score-normalization-then-average choice — but the implementation target is SQL or pre-materialized parquet, not a TS-shaped Python function.
-
-## Migration item
-
-- **Item:** #5 — composite eval rollup (`/evals/aggregate__<suite_key>` route).
-- **TS implementation:** `lib/model-data.ts:877-1044` (`aggregateBenchmarkSummaries`, ~168 lines).
-- **Trigger / call site:** `lib/model-data.ts:1543-1568` (`getEvalSummaryById`, the `evalId.startsWith("aggregate__")` branch). Fired by the route handler at `app/evals/[id]/page.tsx` whenever the URL is `/evals/aggregate__<suite_key>`.
-- **Per-request cost (today):** for an aggregate URL, TS calls `fetchHFEvalDetail(eval_summary_id)` once per sub-eval (lines 1558-1563 do `Promise.all(matchingEvals.map(...))`). Typical composite is **2-21 sub-evals** (per `eval-hierarchy.json`: `reward_bench` 2, `livecodebenchpro` 3, `fibble_arena` 5, `helm_capabilities` 6, `helm_safety` 6, `multi_swe_bench` 6, `helm_lite` 10, `helm_classic` 15, `artificial_analysis_llms` 21). One outlier family `llm_stats` has 471 sub-evals. Each sub-eval detail file is the fully-expanded per-model results blob. This happens at request time, on every page view, with no caching beyond the underlying `fetchHFEvalDetail` LRU.
-- **Aggregate URL → suite_key:** strip the `aggregate__` prefix; the remainder is matched against `eval-hierarchy.json` family keys via `e.benchmark.toLowerCase().replace(/[-.\s]+/g, "_").replace(/^_+|_+$/g, "")` to find sub-evals (`lib/model-data.ts:1550-1552`).
-
-## Operation in SQL terms
-
-For a given `suite_key` (composite), produce one row per `(suite_key, model_id)` containing:
-
-1. The per-model average of normalized sub-eval scores (`AVG(normalize(score, min_score, max_score))` across the suite's sub-evals).
-2. Latest evaluation timestamp + source metadata across that model's components (winner of `MAX(retrieved_timestamp)` is "latest component"; its `source_metadata`, `source_data`, `result.*` are inherited).
-3. A pre-rolled `aggregate_components[]` list of the per-sub-eval contributions (raw score, normalized score, source attribution) used by `eval-detail.tsx` to render the per-row drill-down.
-
-Then, suite-level rollups on top of those per-model rows:
-
-4. `models_count = COUNT(DISTINCT model_id)`.
-5. `avg_score = AVG(per_model_avg_normalized_score)` (avg-of-avgs; see TS quirk #2 below).
-6. `best_model` / `worst_model` = first / last after sorting by `avg_score_normalized` (DESC if `lower_is_better=false`, ASC if true). The sort key is the lower-cased single-metric direction taken from the **first sub-eval's** `metric_config.lower_is_better` (TS quirk #4).
-7. `evaluator_names = sorted DISTINCT UNION` of every sub-eval's `evaluator_names`.
-8. `source_types = sorted DISTINCT UNION` of every sub-eval's `source_types`.
-9. `third_party_ratio = SUM(third_party result rows) / SUM(all underlying result rows)` across all sub-evals' `model_results` arrays (computed pre-rollup, over raw rows — not over the rolled-up models).
-10. `missing_generation_config_count = SUM(...)` across sub-evals.
-11. `latest_source_name`: when there's exactly one sub-eval, copy its name; when there are multiple, the literal string `"Multiple sources"` (TS quirk #5).
-12. A `metric_config` synthesized from the first sub-eval's metric_config but with `min_score=0`, `max_score=1`, `unit="normalized average"`, and `evaluation_description = "Average normalized score across <list of sub-eval names sorted A-Z>"` when more than one source.
-
-Steps 4-12 are scalar/vector reductions over the result of steps 1-3.
-
-## Required parquet columns
-
-To do this in SQL, the pipeline needs (a) a relational result table and (b) a relational composite-membership table:
-
-### Table A: `result_rows` (one row per (eval_summary_id, model_id, variant, retrieved_timestamp))
-
-Existing logical fields, promoted from `payload_json`:
-
-| Column | Type | Source |
-|---|---|---|
-| `eval_summary_id` | VARCHAR | already in metadata column |
-| `model_id` | VARCHAR | currently nested in `model_results[].model_info.id` |
-| `score` | DOUBLE | currently nested in `model_results[].score` |
-| `retrieved_timestamp` | TIMESTAMP (ISO) | nested; see reshape spec #07 for canonicalization |
-| `evaluator_relationship` | VARCHAR | nested in `model_results[].source_metadata` |
-| `source_name` | VARCHAR | nested in `model_results[].source_metadata` |
-| `source_type` | VARCHAR | nested in `model_results[].source_metadata` |
-| `source_organization_name` | VARCHAR | nested in `model_results[].source_metadata` |
-| `sample_size` | BIGINT | nested in `model_results[].score_details.sample_size` |
-| `missing_generation_config` | BOOLEAN | implied — currently surfaced as a count field on the eval summary |
-
-### Table B: `eval_metric_config` (one row per eval_summary_id)
-
-| Column | Type | Source |
-|---|---|---|
-| `eval_summary_id` | VARCHAR | key |
-| `metric_min_score` | DOUBLE | currently nested in `metric_config.min_score` |
-| `metric_max_score` | DOUBLE | currently nested in `metric_config.max_score` |
-| `lower_is_better` | BOOLEAN | currently nested in `metric_config.lower_is_better` |
-| `evaluation_name` | VARCHAR | already on summary; needed for the sort and join |
-| `category` | VARCHAR | already in metadata column |
-
-### Table C: `composite_membership` (one row per (suite_key, sub_eval_summary_id))
-
-| Column | Type | Source |
-|---|---|---|
-| `suite_key` | VARCHAR | family `key` from `eval-hierarchy.json` (e.g. `helm_lite`) |
-| `eval_summary_id` | VARCHAR | each entry of family.eval_summary_ids |
-| `suite_display_name` | VARCHAR | family `display_name` (currently TS overrides via `getBenchmarkDisplayName(suite_key)` lookup; see TS quirk #6) |
-
-This table is the new structural artifact the pipeline owes us. Today the pipeline has the data in `eval-hierarchy.json` but it's nested JSON; promoting to a relational table is what unlocks the `GROUP BY` below.
-
-## Sketch SQL query
-
-Two scenarios as the prompt asked: (a) sub-eval rows live relationally, (b) composite membership lives relationally. Both apply here.
-
-```sql
--- Per-model component-level rows for the requested composite
-WITH suite_components AS (
-  SELECT
-    cm.suite_key,
-    cm.suite_display_name,
-    r.eval_summary_id,
-    emc.evaluation_name AS sub_eval_name,
-    r.model_id,
-    r.score,
-    r.retrieved_timestamp,
-    r.source_name,
-    r.source_type,
-    r.source_organization_name,
-    r.evaluator_relationship,
-    r.sample_size,
-    -- TS quirk #1: normalize FIRST, average LATER
-    CASE
-      WHEN (emc.metric_max_score - emc.metric_min_score) > 0
-      THEN (r.score - emc.metric_min_score) / (emc.metric_max_score - emc.metric_min_score)
-      ELSE r.score
-    END AS normalized_score,
-    emc.lower_is_better
-  FROM composite_membership cm
-  JOIN result_rows r USING (eval_summary_id)
-  JOIN eval_metric_config emc USING (eval_summary_id)
-  WHERE cm.suite_key = ?  -- the param from /evals/aggregate__<suite_key>
-),
-
--- Per-(suite, model) rollup
-per_model AS (
-  SELECT
-    suite_key,
-    model_id,
-    AVG(normalized_score) AS avg_normalized_score,
-    SUM(COALESCE(sample_size, 0)) AS total_sample_size,
-    -- "Latest component" wins for source_metadata + result fields
-    arg_max(STRUCT_PACK(
-      source_name, source_type, source_organization_name,
-      evaluator_relationship
-    ), retrieved_timestamp) AS latest_source_metadata,
-    MAX(retrieved_timestamp) AS evaluation_timestamp,
-    -- aggregate_components[] for drill-down rendering
-    list(STRUCT_PACK(
-      eval_summary_id,
-      composite_benchmark_name := sub_eval_name,
-      score, normalized_score, retrieved_timestamp,
-      source_name, source_type, source_organization_name, evaluator_relationship
-    ) ORDER BY sub_eval_name) AS aggregate_components
-  FROM suite_components
-  GROUP BY suite_key, model_id
-),
-
--- Suite-level rollup on top
-suite_summary AS (
-  SELECT
-    suite_key,
-    COUNT(DISTINCT model_id) AS models_count,
-    AVG(avg_normalized_score) AS avg_score,  -- TS quirk #2: avg-of-avgs
-    SUM(third_party_count) AS total_third_party,
-    SUM(underlying_count) AS total_underlying
-  FROM (
-    SELECT
-      suite_key, model_id, avg_normalized_score,
-      COUNT(*) FILTER (WHERE evaluator_relationship = 'third_party') AS third_party_count,
-      COUNT(*) AS underlying_count
-    FROM suite_components
-    GROUP BY suite_key, model_id, avg_normalized_score
-  )
-  GROUP BY suite_key
-)
-
--- Final shape: per-model rows ordered by score (direction depends on lower_is_better
--- of the FIRST sub-eval, see TS quirk #4)
-SELECT
-  pm.*,
-  ss.models_count,
-  ss.avg_score,
-  ss.total_third_party::DOUBLE / NULLIF(ss.total_underlying, 0) AS third_party_ratio
-FROM per_model pm
-JOIN suite_summary ss USING (suite_key)
-ORDER BY pm.avg_normalized_score DESC;  -- flip to ASC if first sub-eval is lower_is_better
-```
-
-The `evaluator_names` and `source_types` unions, the `aggregate_sources[]` list, and the `missing_generation_config_count` are scalar reductions on top — straightforward and omitted from the sketch.
-
-## Materialize vs query-time
-
-**Recommendation: materialize.** Pipeline emits one row per `(suite_key, model_id)` into a new parquet table (`composite_eval_rollup` or equivalent) plus the suite-level rollup as a sibling table. Runtime DuckDB just does `SELECT * WHERE suite_key = ?` — no JOIN, no AVG, no normalization at request time.
-
-Rationale:
-
-- **The answer is identical for every consumer.** No user-driven slicing on top of the composite (no per-category filtering, no per-developer cut). The aggregate page renders the same table to everyone who hits the same suite URL.
-- **Composite count is small.** ~13 multi-eval families today, growing slowly; cheap to recompute on every pipeline run.
-- **Sub-eval fan-out is the single biggest per-request cost on this route.** Today's TS path issues 2-21 (occasionally 471) `fetchHFEvalDetail` calls per page view. Materialization eliminates them entirely.
-- **Score-normalization choice is a product decision.** Baking it into the parquet means the choice is committed once at pipeline build time. Anyone consuming the column gets the canonical answer; no consumer needs to remember "normalize before averaging."
-
-Honest tradeoff:
-
-- **Pipeline-side recompute on every run.** Pipeline already does a full rebuild (see `migration-plan.md` "Cross-repo coordination"), so this is "another job in the existing batch," not "a new orchestration burden."
-- **Schema growth.** Adds two new tables (per-model rollup + suite-level rollup) plus a new `aggregate_components[]` STRUCT column. Worth it given the alternative is JSON-blob extraction at every request.
-- **Query-time would be viable** if we wanted to support arbitrary user-specified composites (e.g. "build me an aggregate of `mmlu_pro` + `gpqa` + `humaneval`"). We don't have that product feature today and there are no signals we will. If we add it, then `suite_components` CTE above runs query-time over the relational `result_rows`/`eval_metric_config` tables; that's still cheaper than the current TS fan-out.
-
-## TS-as-spec quirks
-
-These are TS choices the pipeline must reproduce. Don't "fix" them — capture, ship, talk later.
-
-1. **Score normalization happens BEFORE averaging.** `lib/model-data.ts:937-941`: `components.map(... normalizeSummaryScore(summary, modelResult.score)) ... reduce(...) / length`. This is min-max normalization per sub-eval (using each sub-eval's own `metric_config.min_score` and `metric_config.max_score`), then arithmetic mean across sub-evals. The "obvious" alternative — average raw scores then normalize once — would produce different numbers when sub-evals have different score ranges (which is the whole point of normalizing). TS's order is correct for cross-metric aggregation; the SQL must do the same. Captured in the sketch CTE: `normalize` lives in `suite_components`, `AVG` lives in `per_model`.
-
-2. **Suite-level `avg_score` is avg-of-per-model-avgs, not avg-of-all-component-scores.** `lib/model-data.ts:990-991`: `aggregatedModelResults.reduce((sum, r) => sum + r.score, 0) / aggregatedModelResults.length`. With unbalanced sub-eval coverage (some models present in only some sub-evals), the two formulations diverge. TS picks the per-model-mean grouping; SQL must do `AVG(per_model.avg_normalized_score)`, NOT `AVG(suite_components.normalized_score)`.
-
-3. **"Latest component wins" for the per-model `evaluation_timestamp` and `source_metadata`.** `lib/model-data.ts:943-947`: sort components DESC by `normalizeEvalTimestamp(evaluation_timestamp)`, take the first. The aggregate row's `result.*`, `source_metadata`, `source_data` all inherit from that single latest sub-eval. So a model that has 6 sub-evals rolled up shows the source metadata of whichever sub-eval was most recent, not a synthesized view. (Note the dependency on reshape spec #07 for timestamp normalization — see "Cross-item dependencies".)
-
-4. **Sort direction comes from the FIRST sub-eval's `lower_is_better`.** `lib/model-data.ts:987-988`: `const lowerIsBetter = first.metric_config.lower_is_better`. If the suite mixes higher-is-better and lower-is-better metrics (rare today but possible — pipeline doesn't enforce homogeneity), TS picks whichever direction `summaries[0]` happens to use. The order of `summaries` is whatever `getEvalSummaryById` produces from `Promise.all(matchingEvals.map(...))`, which is the order of `eval-hierarchy.json` family.eval_summary_ids. Pipeline must preserve that order or replicate the choice (e.g. "pick `lower_is_better=false` if any sub-eval is higher-is-better").
-
-5. **`latest_source_name` is the literal string `"Multiple sources"` when len > 1.** `lib/model-data.ts:1021-1022`. Single-sub-eval composites get the real name; multi-sub-eval composites get a fixed sentinel. Don't try to be smart and concatenate names — the consumer (`eval-detail.tsx:540-541`) already does that separately from the `aggregate_sources[]` array.
-
-6. **Suite display name comes from the `BENCHMARK_NAMES` lookup, NOT from `eval-hierarchy.json`'s `display_name`.** `lib/model-data.ts:903`: `getBenchmarkDisplayName(aggregationKey)` falls through to the `humanizeToken` fallback if not in the hand-curated map. This is migration item #8 (benchmark display names) — capture as a dependency, but in the rollup itself the display name is derived from the `suite_key` not joined from the hierarchy.
-
-7. **Within-composite sub-eval ordering is alphabetical by `composite_benchmark_name`.** Both `aggregateSources` (line 900) and per-model `aggregate_components` (line 962) are `.sort((a, b) => a.composite_benchmark_name.localeCompare(b.composite_benchmark_name))`. Stable English-locale sort. SQL `ORDER BY sub_eval_name` reproduces this.
-
-8. **`composite_benchmark_name` for each component uses the SUB-EVAL's own `evaluation_name`, not the parent suite name.** `lib/model-data.ts:894`: `composite_benchmark_name: summary.evaluation_name`. The field name is misleading; in the per-component context it means "the sub-eval's own display name" (this is what the drill-down UI in `eval-detail.tsx:1046+` renders).
-
-9. **`metric_config` for the aggregate is synthesized.** `lib/model-data.ts:924-933`: takes `first.metric_config` (i.e. first sub-eval's), then forces `min_score=0`, `max_score=1`, `unit="normalized average"`, and rewrites `evaluation_description` to `"Average normalized score across <comma-separated sorted sub-eval names>"` when more than one source. The `lower_is_better` and `score_type` are inherited from the first sub-eval verbatim — so if the composite mixes `binary` and `continuous` sub-evals, the aggregate is reported as whatever the first one is.
-
-## Cross-item dependencies
-
-- **#7 timestamp normalization** — the "latest component wins" logic (TS quirk #3) calls `normalizeEvalTimestamp` (Variant A from the timestamp spec). Once timestamps are ISO 8601 upstream and the reshape half lives in SQL, `MAX(retrieved_timestamp)` and `arg_max(..., retrieved_timestamp)` replace the TS sort. Order of operations: timestamp canonicalization should land first (or co-land), so the SQL's lexicographic `MAX` is correct.
-- **#8 benchmark display names** — `getBenchmarkDisplayName(suite_key)` provides the suite's user-facing label (TS quirk #6). When #8 ships and the pipeline emits canonical display names, the rollup table can drop `suite_display_name` and join against the canonical table.
-- **#3 hierarchy flatten** — the family→sub-eval mapping (`composite_membership` table above) is what `eval-hierarchy.json` already encodes as nested JSON. #3 will likely promote the hierarchy to a relational table; this rollup spec depends on that promotion (or a sidecar table built specifically for composites).
-- **#11 benchmark-card attachment** — TS calls `attachBenchmarkCardToSummary(hfEvalDetailToSummary(detail))` per sub-eval before passing to `aggregateBenchmarkSummaries` (line 1562). The `benchmark_card` of `summaries[0]` becomes the aggregate's `benchmark_card`. Once benchmark cards are inlined upstream by #11, this attachment step disappears.
-- **Composite of all four**: only when 7+8+3 (and ideally 11) are landed can the whole rollup move to materialized parquet cleanly. Without 7 the latest-component logic is brittle; without 3 there's no clean way to drive the rollup loop on the pipeline side; without 8 the suite display name has to be a TS lookup post-hoc.
-
-## Migration checklist
-
-- [x] Spec written (operation captured in SQL terms; TS quirks documented)
-- [ ] Pipeline-schema conversation: decide whether to (a) promote `result_rows`/`eval_metric_config` to relational columns and `composite_membership` to a sidecar table, then materialize the rollup, or (b) ship a single pre-computed `composite_eval_rollup` parquet that bakes in TS's choices as columns. Recommendation: (b) for the rollup itself, with (a) as a parallel deliverable so other reshape items (matrix, top-scores, summary stats) can share the relational base.
-- [ ] Pipeline emits the materialized rollup; verify per-model `avg_normalized_score`, suite `avg_score`, `latest_source_metadata`, sort order, `aggregate_components[]` for at least the 13 multi-eval families.
-- [ ] Update `getEvalSummaryById` to read the rollup directly when `evalId.startsWith("aggregate__")` instead of fanning out `fetchHFEvalDetail` calls.
-- [ ] Delete `aggregateBenchmarkSummaries` (`lib/model-data.ts:877-1044`) and the `Promise.all` fan-out in `getEvalSummaryById` (`lib/model-data.ts:1543-1568`).
-
-## Future product decisions (deferred)
-
-- Whether to support **user-defined composites** (build an aggregate from arbitrary sub-evals at request time). Today the answer is no; if it becomes yes, the relational `result_rows`/`eval_metric_config` path becomes the load-bearing one and the materialized rollup becomes the cached fast-path for the curated 13.
-- Whether to expose **non-normalized averages** alongside the normalized one. TS hides the raw average; consumers asking "what's the actual MMLU score?" have to look at individual sub-evals. A pre-materialized rollup makes both columns equally cheap to surface.
-- Whether the **`avg_score = avg-of-per-model-avgs`** choice (TS quirk #2) is the right one when sub-eval coverage is unbalanced. Don't fix in this migration.

notes/transformations/reshape/06-matrix-leaderboard.md

@@ -1,238 +0,0 @@
-# Matrix leaderboard synthesis
-
-Drafted 2026-04-28. Migration item #6 in `notes/migration-plan.md`. **Reshape-class** — per `notes/migration-plan.md` § "Data direction", the destination is DuckDB SQL (either materialized into parquet or computed at query time), not a pipeline value-emit.
-
-## Framing reminder
-
-We are **refactoring for UI efficiency**, not fixing correctness. TS-as-is is the canonical spec — including its filters and tie-breakers. The deliverable here is an **operation catalog** (GROUP BY keys, aggregations, joins, PIVOT shape) rather than a rule-by-rule replication, so the pipeline-schema / SQL conversation can happen against a precise target.
-
-## Migration item
-
-- **Item:** #6 matrix leaderboard synthesis
-- **TS location:** `lib/model-data.ts:1046-1210` (`buildSingleMetricSuiteMatrixSummary`, ~165 lines), called from `lib/model-data.ts:1570-1596` inside `getEvalSummaryById`
-- **Trigger:** request to `/evals/matrix__<suite_key>` (URL-side prefix `matrix__` is matched at line 1570; `suite_key` is the `benchmark_parent_key` / `benchmark_family_key` / `benchmark` of a group of sub-evals)
-- **Output:** a synthetic `BenchmarkEvalSummary` whose `leaderboard_metrics` are columns (subtasks) and `leaderboard_rows` are rows (models × score per subtask)
-
-## Classification
-
-- **Reshape / dedup / aggregate / pivot.** Not a value cleanup — it's a long→wide pivot over many sub-eval `model_results` rows.
-- **Both materialize and query-time are viable, with a split** (see "Materialize vs query-time" below).
-
-## Operation in SQL terms
-
-Conceptually the operation is: **for each suite, PIVOT every (sub-eval, model) score into a model × subtask matrix, taking the most-recent submission per cell.**
-
-In one sentence: `PIVOT scores ON subtask USING max(score) FILTER (most-recent submission per (model, subtask)) GROUP BY model`.
-
-Pre-pivot input shape (one row per submission of a model on a sub-eval / metric):
-
-```
-suite_key, subtask_key, subtask_name, metric_summary_id, metric_key, metric_name,
-model_id, model_name, developer, model_route_id,
-score, retrieved_timestamp, source_metadata, source_data
-```
-
-The TS implementation is a manual pivot built from JSON: it walks each sub-eval detail, takes its sole metric's `model_results[]`, and folds each row into `rowStates: Map<modelId, {values: {[columnKey]: score}, ...}>` keyed by `model_id`. Dedup-on-`(model_id, columnKey)` happens implicitly because each later assignment overwrites the previous; the per-row "winning timestamp" is the **highest seen** across all column writes (see TS-as-spec quirk #2 below).
-
-### Filtering rules applied before the pivot
-
-1. **Eligibility filter on sub-evals (column gate):** keep `detail` only if `detail.metrics.length === 1` AND `extractDetailSubtasks(detail).length === 0`. In SQL: keep sub-evals that have exactly one root metric and zero subtasks of their own. Suites that contain a multi-metric or multi-subtask sub-eval **silently drop that sub-eval as a column** (no diagnostic).
-2. **Suite-eligibility floor:** if fewer than 2 sub-evals survive the filter, return null (no matrix). Same floor on `matchingEvals` at the call site (line 1585: `if (matchingEvals.length < 2) return null`).
-3. **Metrics-count floor:** after the loop, `if (leaderboardMetrics.length < 2) return null` (line 1158).
-4. **Summary-score exclusion:** at the call site (line 1575), `entry.is_summary_score === true` rows are excluded from the candidate set entirely.
-
-### Column derivation
-
-```
-column_key = "subtask:" || subtask_key || ":" || metric_token
-```
-where `subtask_key = detail.benchmark_leaf_key || slugify(detail.eval_summary_id)` and `metric_token = metric.metric_summary_id || metric.metric_key || slugify(metric.display_name)`. Column order is **alphabetical by `benchmark_leaf_name || eval_summary_id`** (line 1059-1061), not by the column_key itself.
-
-### Row derivation
-
-Rows are keyed by `model_id = modelResult.model_id || modelResult.model_name`. Models with neither id nor name are dropped silently (line 1118-1120).
-
-### Cell value
-
-`row.values[columnKey] = modelResult.score ?? null`. Last write wins per `(model_id, columnKey)` pair — but in TS the loop visits each `(detail, model_result)` tuple exactly once, and `columnKey` is unique per `detail`, so "last write" only fires when a single sub-eval contains multiple `model_results` rows for the same `model_id` (i.e. multiple submissions of the same model to the same sub-eval). When that happens, **the last one in iteration order wins**, with no explicit tie-break — see TS-as-spec quirk #1.
-
-### Per-row timestamp & source-metadata reconciliation
-
-For each row, three fields (`evaluation_timestamp`, `source_metadata`, `source_data`) are reconciled across all the cells written to that row. The rule: whichever cell write had the **highest `normalizeEvalTimestamp`** wins these fields (line 1149-1154). If timestamps tie, the first-seen cell keeps them (the comparison is `>=`, but the first write happens in the no-existing branch on line 1127-1142 which initializes them; later writes only overwrite if strictly greater than or equal).
-
-## Required parquet columns (input to the SQL pivot)
-
-The pivot needs one row per `(eval_summary_id, model_id, retrieved_timestamp)` with these fields exposed as typed columns (today they're nested in `payload_json`):
-
-| Column | Source in TS | Used for |
-|---|---|---|
-| `eval_summary_id` | `detail.eval_summary_id` | filter to suite, also slug fallback |
-| `benchmark_parent_key` | `entry.benchmark_parent_key` | suite routing (matched against `suite_key` from URL) |
-| `benchmark_family_key` | `entry.benchmark_family_key` | suite routing fallback |
-| `benchmark` | `entry.benchmark` | suite routing fallback |
-| `is_summary_score` | `entry.is_summary_score` | exclude rollup rows |
-| `benchmark_leaf_key` | `detail.benchmark_leaf_key` | column key |
-| `benchmark_leaf_name` | `detail.benchmark_leaf_name` | column display + sort key |
-| `metric_count_in_detail` | derived: `len(metrics)` | column-eligibility filter |
-| `subtask_count_in_detail` | derived: `len(extractDetailSubtasks(detail))` | column-eligibility filter |
-| `metric_summary_id` | `metric.metric_summary_id` | column key + metric_config |
-| `metric_key` | `metric.metric_key` | column key fallback |
-| `metric_name` | `metric.metric_name` | display |
-| `metric_display_name` | `metric.display_name` | column key fallback (slugified) |
-| `lower_is_better` | `metric.lower_is_better` | column metadata |
-| `unit` | `metric.unit` | column metadata |
-| `evaluation_description` | `metric.evaluation_description` | suite metric_config |
-| `min_score`, `max_score`, `score_type` | `metric.metric_config.*` | suite metric_config |
-| `model_id` | `result.model_id` | row key |
-| `model_name` | `result.model_name` | row display + row key fallback |
-| `model_route_id` | `result.model_route_id` | model linkout |
-| `developer` | `result.developer` | row display |
-| `score` | `result.score` | cell value |
-| `retrieved_timestamp` | `result.retrieved_timestamp` | per-row reconciliation tie-breaker |
-| `source_metadata` (struct) | `result.source_metadata` | per-row reconciliation winner |
-| `source_data` (struct) | `detail.source_data` | per-row reconciliation winner |
-| `benchmark_card` (struct) | `detail.benchmark_card` | first-seen → suite-level field |
-
-The `(detail, metric, model_result)` triple is what TS already iterates — promoting these to flat parquet rows is the schema change.
-
-## Sketch SQL query
-
-DuckDB syntax. Three CTEs: (1) filter to eligible sub-evals, (2) pick winning submission per `(model, subtask)`, (3) PIVOT.
-
-```sql
-WITH suite_rows AS (
-  -- Pre-pivot: one row per (sub-eval, model, submission)
-  -- Suite-eligibility + column-eligibility filters live here
-  SELECT
-    eval_summary_id,
-    benchmark_leaf_key,
-    benchmark_leaf_name,
-    metric_summary_id,
-    metric_key,
-    metric_name,
-    metric_display_name,
-    lower_is_better,
-    unit,
-    model_id,
-    model_name,
-    developer,
-    model_route_id,
-    score,
-    retrieved_timestamp,
-    source_metadata,
-    source_data,
-    -- Synthetic column key matching TS's "subtask:<subtask_key>:<metric_token>"
-    'subtask:' ||
-      coalesce(benchmark_leaf_key, slugify(eval_summary_id)) || ':' ||
-      coalesce(metric_summary_id, metric_key, slugify(metric_display_name))
-      AS column_key
-  FROM eval_results_flat
-  WHERE
-    -- Suite routing — matches TS's normalizeBenchmarkKeyForLookup
-    normalize_bench_key(coalesce(benchmark_parent_key, benchmark_family_key, benchmark)) = ?
-    AND NOT is_summary_score
-    -- Column eligibility: single root metric, no subtasks (TS line 1058)
-    AND metric_count_in_detail = 1
-    AND subtask_count_in_detail = 0
-),
-ranked AS (
-  -- Pick the winning submission per (model, subtask) cell.
-  -- TS uses "last write wins" because each (detail × model) pair is visited once
-  -- in iteration order; multiple submissions to the same sub-eval by the same model
-  -- collapse to whichever appears last in model_results[]. We approximate that with
-  -- ROW_NUMBER ordered by retrieved_timestamp DESC; see TS-as-spec quirk #1.
-  SELECT *,
-    ROW_NUMBER() OVER (
-      PARTITION BY model_id, column_key
-      ORDER BY retrieved_timestamp DESC
-    ) AS rn
-  FROM suite_rows
-  WHERE model_id IS NOT NULL OR model_name IS NOT NULL
-),
-winners AS (
-  SELECT * FROM ranked WHERE rn = 1
-)
--- The pivot. DuckDB PIVOT syntax:
-PIVOT winners
-  ON column_key
-  USING max(score)
-  GROUP BY model_id, model_name, developer, model_route_id;
-```
-
-Then a second pass over `winners` derives the per-row reconciled `evaluation_timestamp / source_metadata / source_data` (highest-timestamp cell wins) and the `metrics_present` count (count of non-null cells per row).
-
-For the suite-level fields (`leaderboard_metrics` array, `metric_config`, `benchmark_card`, `evaluation_name`), a separate aggregation over `winners` collects the distinct columns (`SELECT DISTINCT column_key, benchmark_leaf_name, metric_name, …`) sorted by `benchmark_leaf_name`, plus a `MIN()` or first-row pick for `benchmark_card` and `metric_config`.
-
-The "single row per model with a values map" output shape is still ergonomic to assemble TS-side from the PIVOT result; the heavy lifting (filter, dedup, pivot) is in SQL.
-
-## Materialize vs query-time
-
-Two natural splits, both can ship:
-
-- **Materialize the column shape and per-cell winners.** A suite's column set is fixed by its sub-eval inventory; the per-cell winner across submissions is deterministic given the data. Both can be written into parquet at pipeline build time as a derived `matrix_<suite_key>` table (or one wide `eval_matrix` table partitioned by suite). This eliminates request-time work for the dominant case.
-- **Compute the row filter at query time.** Today TS doesn't filter rows at all (every model with a `model_id` shows up). Forthcoming UI work may want consumer-driven row filters: "only show models with `models_count > N` evaluations", "only third-party submissions", "filter by developer", "top-K by mean score". Those are query-time concerns and want SQL — `WHERE` / `LIMIT` against the materialized matrix.
-
-**Recommendation:** materialize the wide-form `(suite_key, model_id) → values map + reconciled metadata` table; expose row-filter / sort knobs as query-time SQL parameters. Recompute the materialized table on every pipeline build (cheap relative to the rest of the build); recompute on demand if a single benchmark family is added.
-
-The column-eligibility filter (`metric_count = 1 AND subtask_count = 0`) is a build-time concern — it never varies per request. The row-filter is the only thing that should remain query-time.
-
-## TS-as-spec quirks
-
-These are deliberate behaviours of the current TS that the SQL replacement must preserve until a separate product call says otherwise.
-
-1. **Cell tie-break on multiple submissions of the same model to the same sub-eval is "last in iteration order wins", with no explicit ordering of `model_results[]`.** TS line 1117 iterates `metric.model_results ?? []` directly; the order is whatever the pipeline emitted. There is no `MAX(score)` or "freshest wins" sort applied — the loop just overwrites `row.values[columnKey]` on each pass. In SQL this maps cleanly to `ROW_NUMBER() OVER (PARTITION BY model_id, column_key ORDER BY retrieved_timestamp DESC) = 1` only if the pipeline already emits `model_results[]` in retrieved_timestamp-DESC order. **Verify this assumption against the pipeline's actual emission order before flipping the SQL on.** If pipeline order is non-deterministic, the SQL will produce different cell values than TS for cells with multiple submissions — the migration must either (a) accept the divergence as a "freshest wins" upgrade, or (b) reproduce pipeline's exact array order, which is hostile.
-
-2. **Per-row timestamp reconciliation uses `>=` not `>`.** Line 1149: `if (nextTimestamp >= existing._timestampValue)`. Combined with quirk #1, this means for a model with multiple cells at identical timestamps, the **last-written cell's** source_metadata / source_data win. The SQL replacement should pick the source_metadata of the row with the highest `MAX(retrieved_timestamp)` across all of the model's cells; ties resolve to whatever DuckDB picks (non-deterministic, but identical-timestamp ties are rare in practice).
-
-3. **Silent column drops.** Sub-evals failing the `metric_count === 1 && subtask_count === 0` filter are excluded with no diagnostic. In production this means HELM Lite (10 sub-evals) might silently surface fewer columns than expected if any sub-eval has nested metrics. The pipeline owner should know this is "by design" until the product team weighs in.
-
-4. **`column_key` collisions on multi-metric same-subtask are impossible by construction.** Because the eligibility filter forces `metric_count === 1` per sub-eval, each `(subtask_key, metric_token)` column is unique. If the filter is loosened later, the column_key derivation (`subtask:K:M`) is collision-safe.
-
-5. **No row filtering applied.** Every model id encountered is rendered as a row (subject to having a `model_id` or `model_name`). Row-count for popular suites: `helm_lite` → 91 distinct models in the dominant sub-eval alone. This is fine for now; capture as a known query-time-extension point.
-
-6. **Score of 0 vs null distinction.** `modelResult.score ?? null` — only `undefined` becomes null; numeric 0 is preserved. SQL `MAX(score)` over a single row preserves the 0; over multiple rows with one being NULL, it returns the non-null. Equivalent in this case because the dedup happens before the pivot.
-
-7. **Timestamp normalization uses Variant A (`normalizeEvalTimestamp` from `lib/model-data.ts:76-81`),** which has its own quirks documented in `notes/transformations/07-timestamp-normalization.md` (negative-string fallback, empty-string returns 0, etc.). This is shared with the composite rollup (#5) and other model-data call sites — once #13 ships pipeline-emitted ISO 8601 timestamps, the timestamp comparison in this query becomes a plain `MAX(retrieved_timestamp)` (lexicographic on ISO strings).
-
-8. **Suite-level metric_config takes the first eligible sub-eval's config** (line 1083-1085: `if (!metricConfig) metricConfig = ...`). If sub-evals disagree on `min_score / max_score / lower_is_better`, **the alphabetically-first sub-eval's values are used for the whole suite**. Document this; SQL's equivalent is `FIRST(metric_config ORDER BY benchmark_leaf_name)`.
-
-9. **`benchmark_card` takes the first sub-eval that has one** (line 1087-1089: `if (!benchmarkCard && detail.benchmark_card) benchmarkCard = ...`). Same pattern as #8.
-
-10. **Cross-row `sharedMetricName`** is set only if every sub-eval reports the same `metric_name` (line 1162). When metrics differ across sub-evals, the suite-level `evaluation_description` falls back to the first sub-eval's metric description, but the suite is still rendered. SQL: `MIN(metric_name) FILTER (WHERE metric_name IS NOT NULL)` only when `COUNT(DISTINCT metric_name) = 1`, else null.
-
-## Cross-item dependencies
-
-- **#13 timestamp normalization** — the `MAX(retrieved_timestamp)` per `(model, subtask)` only works correctly once timestamps are in a comparable canonical form. Until pipeline emits ISO 8601, this query depends on Variant A's seconds-vs-ms quirks. Ship #13 first.
-- **#5 composite rollup** — sibling reshape that walks the same `details[]` set. The two should share the parquet schema (a flat `eval_results_flat` row table). Spec'd separately but coordinate.
-- **#11 benchmark-card attachment** — the matrix synthesis result is wrapped in `attachBenchmarkCardToSummary()` after build (line 1595). That join is a separate item; the matrix spec assumes it runs as-is.
-- **#1 identity canonicalization** — the matrix uses `model_id` as the row key. If pipeline-side identity canonicalization changes any model ids, matrix rows that previously merged will split (or vice versa). Migration order: ship #1 first, then re-run snapshot.
-- **#8 benchmark display names** — the suite-level `evaluation_name = getBenchmarkDisplayName(suiteKey)` depends on the BENCHMARK_NAMES map. Once #8 ships pipeline-emitted display names, replace inline.
-
-## Scope (informs "happens on every page view" framing)
-
-Audited 2026-04-28 against `.cache/hf-data/eval-list-lite.json`:
-
-- **587** total eval-list rows; **34** distinct parent buckets (`benchmark_parent_key || benchmark_family_key || benchmark`).
-- **13** matrix-eligible buckets (≥2 sub-evals after `is_summary_score` exclusion). Each one fires `buildSingleMetricSuiteMatrixSummary` exactly once per `/evals/matrix__<suite_key>` request.
-- Sub-eval count per matrix request:
-  - **median 6**, **mean 43.5**, **max 471** (`llm_stats`), **min 2** (`reward_bench`).
-  - Other notable suites: `artificial_analysis_llms` (21), `helm_classic` (15), `helm_lite` (10), `swe_polybench` (8), `helm_instruct` (7), `hfopenllm_v2` / `helm_safety` / `helm_capabilities` / `multi_swe_bench` (6).
-- Each sub-eval detail file carries one metric with up to ~91 `model_results` (sampled `helm_lite_*`).
-
-**Per-request work today:** for `helm_lite`, one matrix render fetches 10 detail files (up to ~10 KB each from cache) and walks ~10 × ~91 = ~910 (detail, model_result) tuples in TS. For `llm_stats`, that's 471 detail fetches and tens of thousands of tuples — every page view. SQL materialization eliminates this entirely; query-time SQL with row filters bounds it.
-
-## Migration checklist
-
-- [x] Spec written (TS-as-is, including quirks)
-- [ ] Snapshot test for `buildSingleMetricSuiteMatrixSummary` against curated detail set (Tier B; gate for SQL replacement — see `notes/testing-strategy.md` § "Reshape-class items: testing addendum")
-- [ ] Pipeline-schema decision: promote `(eval_summary_id, model_id, metric, retrieved_timestamp)` to typed parquet columns for the matrix input set
-- [ ] SQL implementation in `lib/duckdb-data.ts` (or pipeline-side materialized view) reproducing the pivot
-- [ ] Parity gate: TS-vs-SQL diff zero across all 13 matrix-eligible suites
-- [ ] TS deleted: `buildSingleMetricSuiteMatrixSummary` (lines 1046-1210), call site collapsed to `getEvalSummaryById` reading the materialized matrix or invoking the SQL
-
-## Future product decision (deferred)
-
-- Whether silent column drops (TS-as-spec quirk #3) should surface a UI affordance ("3 sub-evals omitted because they have nested metrics").
-- Whether row filtering should be exposed as a UI knob (developer / source_type / models_count threshold) — quirk #5.
-- Whether suite-level `metric_config` should be a hard error when sub-evals disagree, rather than alphabetically-first-wins (quirk #8).
-- Whether to replace "last-write-wins on duplicate submissions" with explicit "freshest-timestamp wins" (quirk #1) — likely a no-op in production but a real semantic upgrade.

notes/transformations/reshape/14-score-summary-stats.md

@@ -1,194 +0,0 @@
-# Score summary stats (per-eval aggregations)
-
-Drafted 2026-04-28. Migration item #14 in `notes/migration-plan.md`. Reshape-class.
-
-## Framing reminder
-
-Refactoring for UI efficiency, not fixing data correctness. TS-as-is is the canonical spec — its quirks (in particular, *re-deriving* `models_count` and effectively-`top_score` even though pipeline already emits them per eval; ignoring the metric primary-vs-leaderboard distinction; using `metric_config.{min,max}_score` defaults of 0/1 to "normalize" already-0-to-1 scores into themselves) must be reproduced when the operation moves to SQL or be explicitly accepted as divergences.
-
-This is a textbook reshape: a `GROUP BY eval_summary_id` over the underlying `model_results` rows with arithmetic + set aggregations. It depends on item #2 (variant bucket reduction) being landed first — the per-eval rows the GROUP BY runs over are *already-deduped* model_result rows, not raw submissions.
-
-## Operation in SQL terms
-
-Input: one row per `(eval_summary_id, model_result)` from the post-#2-dedup `metric.model_results[]` array (one row per `(model_id × variant_key × evaluation_metric)` after the variant bucket reduction). Each row carries `score`, `evaluation_timestamp` / `retrieved_timestamp`, `source_metadata.{source_type, source_organization_name, evaluator_relationship, source_name}`, `generation_config` presence flag.
-
-Output: one row per `eval_summary_id` with the following aggregated columns:
-
-| Column | Aggregation | Notes |
-|---|---|---|
-| `models_count` | `COUNT(*)` | over the deduped model_results, NOT distinct model_id |
-| `avg_score` | `AVG(score)` | raw score, no min/max scaling |
-| `avg_score_norm` | `(AVG(score) - min_score) / (max_score - min_score)` | from per-eval `metric_config`; default `min=0`, `max=1`, → `range=1` so score already-in-[0,1] is unchanged |
-| `best_model` | `(model_name, score)` of `MIN(score)` if `lower_is_better` else `MAX(score)` | tie-break: input order (TS uses stable sort) |
-| `worst_model` | mirror of best_model | |
-| `evaluator_names` | `array_agg(DISTINCT source_organization_name)` | TS preserves *insertion order* (no sort); see TS-as-spec quirks |
-| `source_types` | `array_agg(DISTINCT source_type ORDER BY source_type)` | locale-compare sort |
-| `latest_source_name` | `arg_max(source_name, comparable_timestamp)` over result rows | with `>=` tiebreak (not `>`) → last-wins on ties; see #13 timestamp normalization caveat |
-| `third_party_ratio` | `COUNT(*) FILTER (WHERE evaluator_relationship = 'third_party') / COUNT(*)` | denominator is `models_count` |
-| `missing_generation_config_count` | `COUNT(*) FILTER (WHERE generation_config IS NULL)` | absent/null/empty all count as missing in TS |
-
-The grouping happens **per `eval_summary_id`** (which corresponds to one `(benchmark_leaf_key, primary_metric)` slice). Composite-benchmark rollups across multiple eval_summary_ids are item **#5** (`aggregateBenchmarkSummaries`), out of scope here.
-
-## Current TS implementation
-
-| Concern | Location | Notes |
-|---|---|---|
-| Multi-source-of-truth groupby finalisation | `lib/eval-processing.ts:946-991` (`groupEvaluationsByBenchmark`) | runs over `BenchmarkEvaluation[]` loaded from caches; legacy path. Builds the GROUP BY by accumulating into an object, then iterates and computes the aggregations in a separate `for` loop. |
-| HF-detail-derived single-eval finalisation | `lib/model-data.ts:751-853` (`hfEvalDetailToSummary`) | runs over a single `HFEvalDetail` (one `eval_summary_id`); reads `metric.model_results` of the `primaryMetric` (= `allMetrics[0]`). This is the *active* path used by `app/evals/[id]/page.tsx` via `getEvalSummaryById` (line 1601, 1562). |
-| HF-detail empty-metric short-circuit | `lib/model-data.ts:772-804` | when no primary metric, returns a zero-stats summary with `models_count=0`, `avg_score=0`, `best/worst_model=null`, `latest_source_name = getBenchmarkDisplayName(benchmarkKey)` — note this is the **benchmark display name**, not a real source name. TS-as-spec quirk #1. |
-| Score normalization helper | `lib/model-data.ts:83-88` (`normalizeSummaryScore`) | `min=0`, `max=1`, `range=1` defaults — so `avg_score_norm = avg_score` for the 0-1 score case. |
-| Per-row timestamp normalization (inline) | `lib/model-data.ts:76-81` (`normalizeEvalTimestamp`) and inline duplicate at `lib/eval-processing.ts:961-967` | Variant A from `notes/transformations/07-timestamp-normalization.md`; latest-wins comparison uses `>=` (last-wins on tie). |
-| Per-row score timestamp source field | `lib/eval-processing.ts:933` uses `result.evaluation_timestamp`; `lib/model-data.ts:717` uses `mr.retrieved_timestamp ?? ""` then sets `evaluation_timestamp` from it | The two paths read different upstream fields under the hood; pipeline must ensure both resolve to the same canonical timestamp. |
-
-Confirmed line numbers (verified 2026-04-28):
-- `groupEvaluationsByBenchmark` body: `lib/eval-processing.ts:893-994`
-- Finalisation loop: `lib/eval-processing.ts:946-991`
-- `hfEvalDetailToSummary` body: `lib/model-data.ts:751-854`
-- `normalizeEvalTimestamp`: `lib/model-data.ts:76-81`
-- `normalizeSummaryScore`: `lib/model-data.ts:83-88`
-
-## Required parquet columns
-
-For SQL to do this work directly without pulling `payload_json`, parquet needs (per deduped model_result row, post-#2):
-
-- `eval_summary_id` (already a metadata column)
-- `model_id` / `model_route_id` / `model_name` (currently nested in `metrics[].model_results[].*` inside payload)
-- `score` (currently nested)
-- `retrieved_timestamp` (currently nested; ISO-canonicalized after item #13)
-- `source_metadata.source_type` (currently nested)
-- `source_metadata.source_organization_name` (currently nested)
-- `source_metadata.evaluator_relationship` (currently nested)
-- `source_metadata.source_name` (currently nested)
-- `generation_config` presence boolean — `has_generation_config` (currently nested under `model_results[].generation_config`)
-
-Per-eval scalars (already available somewhere, but need to be on the result row or joinable):
-
-- `metric_config.min_score` / `max_score` / `lower_is_better` (per `(eval_summary_id, primary_metric)` — pipeline currently inlines under `metrics[]` in `eval-detail.json`)
-
-Today's parquet schema has `eval_summary_id`, `models_count`, plus `payload_json`. Doing this in SQL would require a relational promotion of `metric.model_results[]` (one row per result, joined to per-eval `metric_config`). That's the open design question flagged in `notes/migration-plan.md` § "Data direction".
-
-## Sketch SQL query
-
-```sql
--- Per-eval summary stats over deduped model_results.
--- Assumes a relational table `eval_results` with one row per
--- (eval_summary_id, model_result) post-item-#2 variant dedup,
--- and per-eval metric_config columns inlined (or joined from a sibling table).
-SELECT
-  eval_summary_id,
-  COUNT(*)                                              AS models_count,
-  AVG(score)                                            AS avg_score,
-  CASE WHEN (max_score - min_score) > 0
-       THEN (AVG(score) - min_score) / (max_score - min_score)
-       ELSE 0
-  END                                                   AS avg_score_norm,
-  -- best / worst: argmax/argmin over score (direction depends on lower_is_better)
-  CASE WHEN lower_is_better
-       THEN STRUCT_PACK(name := arg_min(model_name, score), score := MIN(score))
-       ELSE STRUCT_PACK(name := arg_max(model_name, score), score := MAX(score))
-  END                                                   AS best_model,
-  CASE WHEN lower_is_better
-       THEN STRUCT_PACK(name := arg_max(model_name, score), score := MAX(score))
-       ELSE STRUCT_PACK(name := arg_min(model_name, score), score := MIN(score))
-  END                                                   AS worst_model,
-  -- evaluator names: TS preserves insertion order, NOT sorted (quirk)
-  list(DISTINCT source_organization_name)               AS evaluator_names,
-  -- source_types: TS sorts by localeCompare
-  list_sort(list(DISTINCT source_type))                 AS source_types,
-  -- latest source_name: arg_max with >= tie-break (last-wins)
-  arg_max(source_name, retrieved_timestamp)             AS latest_source_name,
-  -- third_party_ratio: filtered count over total
-  COUNT(*) FILTER (WHERE evaluator_relationship = 'third_party')::DOUBLE
-    / NULLIF(COUNT(*), 0)                               AS third_party_ratio,
-  COUNT(*) FILTER (WHERE generation_config IS NULL)     AS missing_generation_config_count
-FROM eval_results
-GROUP BY eval_summary_id, min_score, max_score, lower_is_better;
-```
-
-Notes on the sketch:
-- `arg_max(source_name, retrieved_timestamp)` is DuckDB-native; the `>=` tie-break in TS would need an explicit `ORDER BY retrieved_timestamp DESC, row_order DESC LIMIT 1` if exact byte-parity is required (see TS-as-spec quirks).
-- `list(DISTINCT …)` in DuckDB returns insertion order of distinct values in the partition — should match TS `Array.from(new Set([...]))` for `evaluator_names`.
-- The `STRUCT_PACK` for best/worst is illustrative; in practice we'd materialize `best_model_name` and `best_model_score` as separate columns.
-- `score` and `min_score`/`max_score` are read from the same row group; if `metric_config` is sibling-joined, `GROUP BY` keys must include those columns (or use a window).
-
-## Materialize vs query-time
-
-**Recommendation: materialize per-eval at pipeline emission time.** Pipeline already emits `eval-list.json` with `models_count` and `top_score` per eval, which proves materialization is the established pattern. Extending that to the full set above (`avg_score`, `avg_score_norm`, `evaluator_names`, `source_types`, `latest_source_name`, `third_party_ratio`, `missing_generation_config_count`, `best_model`, `worst_model`) is the same operation, just more outputs.
-
-Why materialize:
-
-- **No consumer slices these by category, by model-developer, or by anything else within an eval.** Audit (2026-04-28) of `lib/`, `app/`, `components/`: every read is per-eval scalar (`summary.avg_score`, `summary.third_party_ratio`, `summary.evaluator_names.length`, etc.). The sole "category breakdown" use (`benchmark-detail.tsx:5727`) is over models within a benchmark, computed independently — not over these summary stats.
-- **The aggregation is deterministic and consumer-invariant** — every consumer would compute the same numbers, which is exactly the materialize-when-the-answer-is-the-same heuristic from `notes/migration-plan.md` § "Data direction".
-- **Per-eval scope is small** (one row per eval, ~587 evals in production today); blob size is trivial.
-- **It removes the "TS recomputes what pipeline already provides" divergence** flagged below — instead of expanding the divergence by adding 7 more recomputed fields, we collapse the existing one by lifting all 9 to pipeline.
-
-When query-time SQL would be the call instead: if a future consumer wants e.g. "third_party_ratio for this eval *restricted to a developer subset*", that's a query-time aggregation. Today no such consumer exists. Re-evaluate when one shows up.
-
-The composite-benchmark rollup (item #5 `aggregateBenchmarkSummaries`) and matrix synthesis (item #6) are query-time-shaped because they slice across eval_summary_ids in different ways per request. Item #14 is the pre-aggregate that those build on.
-
-## TS-as-spec quirks
-
-These are TS behaviors the spec preserves. Pipeline must reproduce or each is an explicitly-accepted divergence.
-
-### 1. TS recomputes `models_count` and an effective `top_score` even though pipeline already provides them per eval
-
-Pipeline's `eval-list.json` emits per-eval `models_count` (e.g. 4492) and `top_score` (e.g. 0.8269), verified 2026-04-28 against `.cache/hf-data/eval-list.json`. TS ignores both:
-
-- `models_count` is recomputed at `lib/eval-processing.ts:948` as `summary.model_results.length`, and at `lib/model-data.ts:825` as `modelResults.length`.
-- `top_score` is not stored, but `best_model.score` is computed at `lib/eval-processing.ts:984-989` and `lib/model-data.ts:831-836` from the sorted-by-score model_results — which should be the same value as `top_score` *for the primary metric* assuming neither side filters differently.
-
-**Do they disagree today?** Almost certainly yes for some rows. Pipeline's `models_count` counts pre-#2-dedup rows (raw model_result entries on the eval); TS's recomputed value counts post-#2-dedup rows (after `normalizeSingleModelCardEntry` collapses thinking-budget variants etc.). For evals with thinking-budget submissions (e.g. `openai/gpt-5.2`), pipeline's count will be higher than TS's. This is the same root cause as the #2 spec's "pipeline emits 7 variants, TS shows 2". Surface as a known divergence; **do not fix in this spec — it is properly resolved by landing #2 first**, after which pipeline's emitted `models_count` will match what TS recomputes (and the recomputation can be dropped).
-
-For `top_score` vs TS-derived `best_model.score`: similar story. Pipeline's `top_score` is `MAX(score)` over pre-#2-dedup rows; TS's is over post-dedup rows. Different denominators, potentially different maxima (though MAX is more robust to dedup than COUNT or AVG).
-
-### 2. The empty-metric short-circuit returns a zero-stats summary with `latest_source_name = getBenchmarkDisplayName(benchmarkKey)`
-
-`lib/model-data.ts:785` sets `latest_source_name` to a *benchmark display name string* (e.g. "MMLU Professional") — not a source name — when there are no metrics. Downstream UI then renders this as if it were a source label. This is almost certainly a placeholder bug, but it's TS behavior today and any consumer that special-cases `latest_source_name` matching a benchmark name is reading this. Pipeline should reproduce by emitting `null`-or-display-name in the same condition, OR by accepting this as a fix-by-canonicalization (recommended: emit `null` and chase down any consumer that breaks, since this only fires when the eval has zero metrics — a degenerate case).
-
-### 3. Two GROUP BY entry points, different score-source semantics
-
-`groupEvaluationsByBenchmark` (eval-processing.ts:893) iterates `eval_.evaluation_results`; `hfEvalDetailToSummary` (model-data.ts:751) iterates `metric.model_results` of the *first* metric (`allMetrics[0]`). These are different shapes feeding the same aggregation, and they apply different filters: the eval-processing path takes *every* `evaluation_result` (potentially multi-metric); the model-data path takes only one metric's results. The active read path for eval-detail pages is `hfEvalDetailToSummary`. Pipeline emission should match `hfEvalDetailToSummary`'s semantics (single primary metric per eval_summary_id) — that's what users actually see today.
-
-### 4. `evaluator_names` is insertion-ordered, not sorted
-
-`lib/eval-processing.ts:940-942` pushes into a deduped array as it iterates; `hfEvalDetailToSummary` initializes to `[]` (line 826) and never populates it at all. Two contradicting behaviors in the same codebase — for the active path (`hfEvalDetailToSummary`), `evaluator_names` is **always empty**. The eval-card UI (`components/eval-card.tsx:162`) reads `summary.evaluator_names.length`, so eval-detail pages today always render "0 evaluators". This is a latent bug, but it's TS behavior. Pipeline should decide whether to (a) reproduce the empty-array behavior to preserve UI byte-parity, or (b) fix-by-canonicalization and emit the sorted DISTINCT set, accepting that the "Evaluators" pill will start showing real numbers. Recommended: (b), and call it out in the migration commit.
-
-### 5. Latest-source uses `>=` tie-break (last-wins)
-
-`lib/eval-processing.ts:968` uses `timestamp >= latestTimestamp`, so on a timestamp tie the *later iteration order* wins. SQL `arg_max` is implementation-defined on ties. If exact byte-parity matters across the migration, the SQL needs an explicit secondary sort. Otherwise accept as a divergence on the rare tied-timestamp case.
-
-### 6. `avg_score_norm` defaults make it a no-op for 0-1 scores
-
-When `metric_config.min_score` and `max_score` are absent (the common case), defaults are `0` and `1` → `range = 1` → `avg_score_norm = avg_score`. The "normalization" only does work when the metric explicitly carries non-default min/max. Reproducing in SQL is the `CASE WHEN range > 0` branch in the sketch above; matches TS's `range > 0 ? … : 0` (note: TS returns 0 not score when range == 0; minor edge-case divergence with `normalizeSummaryScore` which returns `score`).
-
-## Cross-item dependencies
-
-**Hard dependencies (this item cannot land cleanly without them):**
-
-- **#2 setup-alias variant merging (reshape half).** The aggregations are over deduped variants. If pipeline emits stats over raw rows, TS-recomputed stats over deduped rows will continue to diverge for any model with merged variants. Land #2's reshape half first; then the per-eval row set that #14's GROUP BY runs over is the same set TS uses today.
-- **#13 timestamp normalization.** `latest_source_name` requires a comparable timestamp. Today TS uses `normalizeEvalTimestamp` (Variant A from #13's spec) inline; once pipeline emits ISO 8601, SQL `arg_max(source_name, retrieved_timestamp)` works lexicographically with no parsing.
-
-**Already-shipped dependencies:**
-
-- **#4 source-metadata synthesis fallback.** Done. Pipeline now emits `source_metadata.{source_type, source_organization_name, evaluator_relationship, source_name}` on every model_result row, so the `evaluator_names`, `source_types`, `third_party_ratio`, and `latest_source_name` aggregations have non-null inputs to read. Without #4 these aggregations would silently drop rows.
-
-**Soft dependencies (independent but adjacent):**
-
-- **#5 composite eval rollup.** Builds on per-eval summaries. Once #14 lands as materialized per-eval stats, #5's `aggregateBenchmarkSummaries` becomes a query-time roll-up over already-aggregated rows (cheaper) instead of a re-aggregation from raw model_results.
-- **#6 matrix leaderboard synthesis.** Same pattern — reads per-eval summaries when slicing across a multi-metric suite.
-
-## Migration checklist
-
-- [x] Spec written (TS-as-is, including quirks)
-- [ ] Pipeline-side schema decision: relational promotion of `metric.model_results[]` to parquet rows, OR materialize the 9 aggregated columns into `eval-list.json` directly. Recommended: materialize (per "Materialize vs query-time" section).
-- [ ] Pipeline emits the 9 columns per eval_summary_id matching this spec across the full corpus
-- [ ] Verify against `.cache/hf-data/eval-list.json` extended shape; today only `models_count` and `top_score` are present
-- [ ] Audit script (none yet — could mirror `scripts/verify-timestamp.mjs` against the recomputed-vs-emitted values for the 9 columns)
-- [ ] TS deleted: `lib/eval-processing.ts:946-991` finalisation loop; `lib/model-data.ts:806-853` aggregation block; `lib/model-data.ts:83-88` (`normalizeSummaryScore`). Callers read pipeline-emitted fields directly.
-- [ ] Snapshot test (`tests/transformations/score-summary-stats.test.ts` — currently absent; reshape-class snapshots double as TS-vs-SQL parity gates per `notes/testing-strategy.md` § "Reshape-class items: testing addendum")
-
-## Future product decisions (deferred)
-
-- Whether the empty-metric `latest_source_name = display_name` placeholder (TS quirk #2) should be `null` or kept as a backstop string.
-- Whether `evaluator_names` should be empty (TS active-path behavior) or populated with the sorted DISTINCT set (TS legacy-path behavior).
-- Whether `models_count` should mean "distinct model_id count" or "deduped result count" — TS today uses the latter; users may expect the former. Resolves naturally once #2 lands.

notes/transformations/reshape/16-per-category-counts.md

@@ -1,192 +0,0 @@
-# Per-category benchmark counts
-
-Drafted 2026-04-28. Migration item **#16** in `notes/migration-plan.md`. **Reshape class** — the operation lives in DuckDB SQL (materialized into the parquet artifact). Use this spec as the canonical "obvious SQL" example: it is the smallest, clearest reshape item in the migration.
-
-## Framing reminder
-
-We are refactoring for UI efficiency, not data correctness. TS-as-is is the canonical spec for *behavior preservation* — except this is the rare reshape item where the current TS is **demonstrably wrong** (it ships a fake distribution; see TS-as-spec quirks below). The migration replaces wrong-with-correct, so the snapshot delta on switch-over is *expected* and the snapshot itself cannot be the gate. The gate is: "real per-category counts populate the same `category_stats: Record<Category, number>` shape; UI renders without divergence." See `notes/testing-strategy.md` § "Reshape-class items: testing addendum".
-
-## Migration item
-
-| Field | Value |
-|---|---|
-| Item | #16 |
-| Class | Reshape |
-| Operation | per-(model, category) DISTINCT-benchmark count |
-| Materialize? | Yes — small fixed shape, identical for every consumer |
-| Target column | `category_stats: Record<CategoryType, number>` on the model card payload (already the consumer-facing field) |
-| TS files affected | `lib/model-data.ts:369-379` (the wrong fake), `lib/eval-processing.ts:653-666` (the correct heavy-data path) |
-| Pipeline file expected | `scripts/pipeline.py` model-card writer; populate `category_stats` in `model-cards.json` summary entries |
-
-## Operation in SQL terms
-
-Group every model_result row by `(model_route_id, category)` and count distinct `benchmark_family_key`. The result is the per-model, per-category benchmark-coverage count consumed by the model card UI.
-
-This is one statement. It has no edge cases beyond null/empty handling and the choice of grouping key (route-id vs family-id) — both already settled by the existing parquet schema.
-
-## Current TS implementation
-
-There are **two implementations** of `category_stats` in TS today, depending on which data path supplies the model card:
-
-### Path A — `lib/model-data.ts:360-380` (`hfModelCardToEvaluationCardData`)
-
-Input: `HFModelCardEntry` (the lightweight `model-cards.json` summary; `entry.categories_covered: string[]`, `entry.total_evaluations: number`, no per-category breakdown).
-
-Logic at lines 369-379 (verbatim):
-
-```ts
-// Distribute total evaluations across categories proportionally
-const categoryStats: Record<string, number> = {}
-const perCat = categories.length > 0
-  ? Math.max(1, Math.floor(entry.total_evaluations / categories.length))
-  : 0
-let remaining = entry.total_evaluations
-for (let i = 0; i < categories.length; i++) {
-  const count = i === categories.length - 1 ? remaining : Math.min(perCat, remaining)
-  categoryStats[categories[i]] = count
-  remaining -= count
-}
-```
-
-This is the **fake distribution**. It does not look at any per-category data because the input doesn't carry any. It just slices `total_evaluations` evenly across `categories.length`, with the last category taking the rounding remainder.
-
-Callers of `hfModelCardToEvaluationCardData` (the listing/grid pages where this fake fires):
-- `lib/model-data.ts:1235, 1242` — model index sorting
-- `lib/model-data.ts:1386, 1416, 1455` — developer detail pages, comparison pages
-- `lib/duckdb-data.ts:163` — DuckDB shadow read parity path
-
-### Path B — `lib/eval-processing.ts:653-666` (`createModelFamilySummary` → `categoryStats`)
-
-Input: full `BenchmarkEvaluation[]` (per-eval-detail data, with `evaluations_by_category` populated).
-
-Logic at lines 653-666 (verbatim):
-
-```ts
-// Calculate category stats (count of unique benchmarks per category)
-const categoryStats: Record<CategoryType, number> = {} as any
-
-for (const category of summary.categories_covered) {
-  const evals = summary.evaluations_by_category[category] || []
-  const categoryBenchmarks = new Set<string>()
-
-  for (const eval_ of evals) {
-    for (const result of eval_.evaluation_results) {
-      categoryBenchmarks.add(getBenchmarkName(eval_, result))
-    }
-  }
-  categoryStats[category] = categoryBenchmarks.size
-}
-```
-
-This is the **real distribution**: COUNT(DISTINCT benchmark) per category, computed from the heavy nested data. Used on model-detail pages where the full payload is loaded.
-
-Callers of `createModelFamilySummary`:
-- `lib/model-data.ts:1497, 1520, 1532` — model-detail pages
-- `lib/duckdb-data.ts:194` — DuckDB shadow read for model detail
-
-The two paths produce **different `category_stats` for the same model** today. The grid/index UI sees the fake distribution; the detail page sees the real one. No code reconciles them.
-
-## Required parquet columns
-
-The pipeline already emits the columns needed (per the schema documented in `notes/migration-plan.md` § "Data direction" — `record_type`, `model_route_id`, `model_family_id`, `eval_summary_id`, `developer_route_id`, `developer`, `category`, `benchmark_family_key`, `models_count`, `total_evaluations`, `last_updated`, `payload_json`). For this aggregation:
-
-- `model_route_id` — group key (per-model)
-- `category` — group key (per-category)
-- `benchmark_family_key` — distinct-count target
-
-No schema change needed for the SQL to run. Confirm against `scripts/pipeline.py:write_experimental_parquet_table` in the pipeline repo when handing off — the column list above is from documentation, not direct inspection.
-
-## Sketch SQL query
-
-```sql
-SELECT
-  model_route_id,
-  category,
-  COUNT(DISTINCT benchmark_family_key) AS benchmark_count
-FROM model_results
-WHERE record_type = 'model_result'  -- if needed; depends on whether row is pre-filtered
-  AND benchmark_family_key IS NOT NULL
-GROUP BY model_route_id, category
-```
-
-To materialize as the `category_stats: Record<CategoryType, number>` shape that the consumer expects, pivot per model:
-
-```sql
-SELECT
-  model_route_id,
-  MAP_FROM_ENTRIES(
-    LIST({k: category, v: COUNT(DISTINCT benchmark_family_key)})
-  ) AS category_stats
-FROM model_results
-WHERE benchmark_family_key IS NOT NULL
-GROUP BY model_route_id
-```
-
-(DuckDB syntax for assembling a `MAP(category VARCHAR, count BIGINT)`. The pipeline writer can also assemble the dict in Python after running the basic GROUP BY — equivalent.)
-
-Verification: against the live cache, `total_evaluations` for a given model should approximately equal `SUM(benchmark_count)` across its categories — *approximately* because today's TS fake guarantees the sum, but the real count is "distinct benchmarks per category" which can sum to less (a benchmark in two categories gets counted once per category) or more than `total_evaluations` (which is row-count, not distinct-benchmark-count). This sum-divergence is the load-bearing finding for "the fake was always wrong, not just imprecise."
-
-## Materialize vs query-time
-
-**Materialize.** This is the unambiguous case for materialization:
-
-- The aggregation is small (one row per model × number-of-categories ≤ 9, so ≤ ~50k cells across the full corpus of 5,830 models).
-- The answer is identical for every consumer — no slicing, no filtering. Both the grid and the detail page want the same `Record<Category, number>`.
-- The consumer shape is already known and stable: `category_stats: Record<CategoryType, number>` on the model card.
-- Materializing into `model-cards.json` (summary path) eliminates Path A's fake entirely; the detail path can keep its own computation initially, but eventually both reads from the materialized field.
-
-Recommended landing: pipeline emits `category_stats` as a dict on each model card summary entry. Both TS Path A and Path B then read the field directly; both implementations get deleted.
-
-## TS-as-spec quirks (Path A — the fake)
-
-These are the precise behaviors today. Document them — even though they're wrong — because they may have shaped UI design:
-
-1. **Always-equal counts.** With `Math.floor(total / categories.length)`, every category gets an identical `perCat` value (modulo rounding). For a model with `total_evaluations = 12` and `categories = ["Reasoning", "Coding", "Math", "Knowledge"]`, every category gets `3`. Real data would not look like this.
-
-2. **`Math.max(1, …)` floor.** When `total_evaluations < categories.length`, `Math.floor` would yield `0`, but `Math.max(1, …)` forces at least `1`. So a model with `total_evaluations = 2` and `categories.length = 4` ends up with the first 2 categories at `1` each and the remainder distributed via `Math.min(perCat, remaining)` — practically: category 0 → 1, category 1 → 1, category 2 → 0 (because remaining = 0 and `Math.min(1, 0) = 0`), category 3 → 0 (last category takes `remaining = 0`). So you get `{ cat0: 1, cat1: 1, cat2: 0, cat3: 0 }` for total=2, categories=4. **Not always-equal** in this branch — a quirk worth flagging.
-
-3. **Last-category-takes-remainder.** The terminal category collects `remaining` instead of `perCat`, so it can be larger than the others when `total % categories.length != 0`. For `total = 13, categories.length = 4`, you get `{ 3, 3, 3, 4 }`. The last category in the array (alphabetical or insertion order from `mapHFCategories`) silently looks more covered than the others.
-
-4. **Zero-categories edge.** `categories.length === 0` → `perCat = 0`, loop never runs, `categoryStats = {}`. The consumer gets an empty record. UI must handle. (`benchmark-evaluation-card.tsx:248-257` filters `count > 0` before rendering, so empty is safe; verified.)
-
-5. **Zero-total-evaluations edge.** `total_evaluations = 0`, `categories.length > 0` → `perCat = Math.max(1, 0) = 1`. Then `remaining = 0`, loop iterates: `i = 0`, `count = Math.min(1, 0) = 0`, `categoryStats[cat0] = 0`, `remaining = 0`. Subsequent iterations same. Last iteration: `count = remaining = 0`. Final: `{ cat0: 0, cat1: 0, … }`. The `Math.max(1, …)` doesn't bite here because `Math.min(perCat, remaining=0) = 0`. UI sees all-zeros, filters them out via `count > 0` check. Safe.
-
-6. **No relation to actual benchmark distribution.** A model could have all 50 of its evaluations in `Knowledge` and 0 in `Reasoning`, but if the categories list is `["Knowledge", "Reasoning"]` (because some other model in the same category exposure was tagged that way, or because the upstream `categories_covered` was a union), the fake shows `25` and `25`. There is no signal in the input that lets the fake do better.
-
-7. **Path A vs Path B disagreement.** Same model, two pages, two answers. Today's grid shows fake; today's detail shows real. The pipeline materialization eliminates this.
-
-### TS-as-spec quirks (Path B — the real one)
-
-Path B is essentially correct (COUNT(DISTINCT benchmark) per category) but uses `getBenchmarkName(eval_, result)` as the distinct key rather than `benchmark_family_key`. Two divergences:
-
-- The display-name path (#8 in the migration catalog) is messy — different evals can map to the same display name even with different family keys. The SQL `COUNT(DISTINCT benchmark_family_key)` is the right key going forward; expect small count differences vs Path B today on models where a benchmark family has multiple display-named members. Document as expected when running parity.
-- Path B counts at the (eval, result) level — one eval can carry multiple results, each with its own `benchmark`. The SQL on `model_results` rows is at the same granularity (one row per result), so the COUNT(DISTINCT benchmark_family_key) GROUP BY (model_route_id, category) will line up.
-
-## UI components to audit when going from fake-equal to real-uneven counts
-
-1. **`components/benchmark-evaluation-card.tsx:248-257`** — `categoryCoverage` derived value. Filters `count > 0` (safe under both fake and real), sorts by `count desc, category asc`. Today the sort is effectively a tie-break on category name because counts are always equal under fake; once counts are real, the sort starts ordering by actual coverage. **Visual change: bar/chip ordering shifts.** Worth a screenshot diff.
-
-2. **`components/benchmark-evaluation-card.tsx:240-247`** — `topDomains` (above the category coverage block). Independent of `category_stats`, but the visual proximity means if categoryCoverage starts looking lopsided, design might want to revisit ordering of the two adjacent blocks.
-
-3. **Card grid alignment.** If grid cells display category counts as side-by-side bars (need to grep templates), today every model's bars are equal width within itself; under real counts they'll be uneven. CSS that assumes "always-equal" widths might overflow or look awkward.
-
-4. **Filter UI / faceted browse (if any).** Searching by "models with > N benchmarks in Reasoning" today returns weird answers (e.g. a 12-eval model with 4 categories registers as 3 in Reasoning even if it has 0 actual reasoning benchmarks). After the migration, this becomes meaningful — and any test fixtures that assumed "all models register in all their listed categories" need updating.
-
-No widespread breakage is expected; the consumer shape is unchanged. The risk is cosmetic (sort order, bar widths) and statistical (filters that were lying now tell truth).
-
-## Cross-item dependencies
-
-- **#11 category inference.** The SQL groups by the pipeline's `category` column. Today 84% of evals emit `category: "other"` (per `notes/migration-plan.md`), which would collapse most distinct-benchmark counts into one giant `other` bucket — defeating the point of the per-category breakdown. **#16 is blocked on #11 producing useful category labels.** Or alternatively: the pipeline-side aggregation must run after applying TS's regex category inference, by porting `inferCategoryFromBenchmark` upstream first.
-- **#8 benchmark display names.** Affects the *labels* shown next to the count, not the count itself. Independent.
-- **#3 hierarchy flatten.** The SQL groups by `model_route_id`, which is the post-flatten family identity. If hierarchy flattening changes the route-id assignment, recompute. Should be stable for now.
-
-## Migration checklist
-
-- [x] Spec written
-- [ ] Tests cover the materialized-output shape (`Record<CategoryType, number>` with non-negative integers, sum vs total_evaluations divergence accepted)
-- [ ] Confirm parquet schema columns (`model_route_id`, `category`, `benchmark_family_key`) against `scripts/pipeline.py:write_experimental_parquet_table` in pipeline repo
-- [ ] Filed with pipeline owner — recommend materializing `category_stats` into `model-cards.json` summary entries (smallest, clearest reshape) and **flag dependency on #11 category accuracy**
-- [ ] Pipeline emits `category_stats` populated for all 5,830 model card entries
-- [ ] Adapter snapshot regenerated; review the diff (it WILL be large because the fake is replaced) — the snapshot is *not* the gate, the materialized-shape contract is
-- [ ] TS deleted: remove the fake at `lib/model-data.ts:369-379`, remove the categoryStats block at `lib/eval-processing.ts:653-666`, both paths read `entry.category_stats` from the pipeline field
-- [ ] UI screenshot diff on `components/benchmark-evaluation-card.tsx` to confirm the now-uneven bars render acceptably; design tweak if not

notes/ts-to-pipeline-migration.md

@@ -1,285 +0,0 @@
-# Move TypeScript data processing into the dataset pipeline
-
-Working notes — drafted 2026-04-26, updated 2026-04-27. Cleaning/reshape annotations added 2026-04-28.
-
-> **Framing update (2026-04-28).** This doc was written assuming every item moves to pipeline emission. The Data direction principle (see `notes/migration-plan.md` § "Data direction: cleaning upstream, reshape in SQL") refines that: **cleaning** items (value transforms on a single record) belong in pipeline emission and follow the per-item workflow in `notes/migration-plan.md`. **Reshape** items (dedup, aggregation, groupby, sort+select, hierarchy flatten) belong in DuckDB SQL — either materialized into pipeline parquet or computed at query time. Each item below is now annotated. For the active per-item migration view, see `notes/transformations/` and `notes/migration-plan.md`.
-
-## Status
-
-- ✅ **#4 source-metadata synthesis fallback** — deleted 2026-04-27. Pipeline (commit 9090cc5) now carries `source_metadata` on every hierarchy `model_result` row; verified 86 183/86 183 production rows + 43 859/43 859 eval-detail rows. Removed `getCanonicalSourceMetadata` (lib/hf-data.ts), `buildSourceMetadataIndex` (lib/hf-data.ts), and three duplicate inline fallbacks in lib/model-data.ts (`buildBenchmarkLeaderboardMatrix`, `toModelResultsForMetric`, `buildSingleMetricSuiteMatrixSummary`). Added a runtime `assertSourceMetadata(result, context)` guard at each read site so a future pipeline regression fails loud (with model + eval IDs) instead of silently emitting `undefined` into the UI which dereferences `.evaluator_relationship` unguarded. The 1st/3rd-party badge collapse bug is now structurally impossible.
-  - **Intended behaviour change:** every model_result on eval pages now shows the real first/third-party badge instead of the previously-hardcoded "Other". 86 183/86 183 rows reclassified (3 119 first_party, 83 056 third_party, 8 still other). This was the explicit goal of item #4 ("known correctness gap").
-- ⚠️ **#11 category-inference fallback (mostly reverted)** — narrowed 2026-04-27. Pipeline emits `category` on every eval-detail and every hierarchy node, BUT 84% of eval-details (496/587) carry `category: "other"` — the pipeline currently does NOT replicate the regex's classification work. Replacing the regex fallback would silently General-ify ~6 700 model rows including 4 known Safety evals (`helm_safety`, `helm_safety_simplesafetytests`, `helm_classic_truthfulqa`, `llm_stats_truthfulqa`) and 12+ hierarchy `(catKey, displayName)` pairs covering 1 500+ Safety-classified model rows (RewardBench safety, AIRBench subtasks, etc.).
-  - **Done safely:** added `coding`, `instruction_following`, `language_understanding` to `PIPELINE_CATEGORY_MAP` (all → "General", matching prior `inferCategoryFromBenchmark` returns). Removed `?? inferCategoryFromBenchmark(c)` fallback in `mapHFCategories`, replaced with `?? "General"` — provably equivalent for all 9 currently-emitted pipeline keys.
-  - **Reverted (left for follow-up):** the `inferCategoryFromBenchmark` calls in `hfEvalDetailToSummary` (lib/model-data.ts:760, :803), `groupEvaluationsByBenchmark` (lib/eval-processing.ts:906), and `buildSingleMetricSuiteMatrixSummary` (lib/model-data.ts:1170) — keeping the regex inference until the pipeline emits accurate `category` for currently-`other` benchmarks.
-  - **Pre-existing inconsistency, not addressed:** `hfEvalEntryToListItem` (lib/model-data.ts:436) reads pipeline category, while `hfEvalDetailToSummary` reads regex. Same eval can show different categories in eval-list vs eval-detail. Was already in HEAD; dropping it requires the pipeline-side fix above first.
-
-### Open follow-ups for the pipeline side
-
-1. **Reclassify `category: "other"` evals.** 84% of eval-details get the catch-all. Either (a) train a classifier on the regex's intent, (b) tighten the pipeline's mapping rules, or (c) emit two fields (raw + ui_category). Until this lands, the regex inference must stay.
-2. **Mark known Safety benchmarks correctly.** TruthfulQA, SimpleSafetyTests, the 6 AIRBench 2024 subtasks, RewardBench safety, etc., all currently emit non-Safety categories.
-3. **Consider warn-once logging in `mapHFCategories`** when a new pipeline key arrives that isn't in the map. Today they silently default to "General".
-
-## Parity setup
-
-`scripts/compare-data-backends.mjs` is the regression net for this migration. Run:
-
-```bash
-# JSON-backed dev server, reading the pipeline output directly (offline)
-HF_DATA_LOCAL_DIR=/Users/jchim/projects/eval_cards_backend_pipeline/output \
-  HF_DATA_OFFLINE=1 PORT=3001 pnpm dev
-
-# DuckDB-backed dev server, same data source
-DATA_BACKEND=duckdb \
-  LOCAL_PIPELINE_OUTPUT=/Users/jchim/projects/eval_cards_backend_pipeline/output \
-  HF_DATA_LOCAL_DIR=/Users/jchim/projects/eval_cards_backend_pipeline/output \
-  HF_DATA_OFFLINE=1 PORT=3002 pnpm dev
-
-# Compare (PARITY_FAIL_FAST=0 prints every divergence path)
-PARITY_FAIL_FAST=0 node scripts/compare-data-backends.mjs \
-  --json-base http://localhost:3001 --duckdb-base http://localhost:3002
-```
-
-`HF_DATA_LOCAL_DIR` overrides `lib/hf-data.ts`'s default cache path to the sibling pipeline `output/`; `HF_DATA_OFFLINE=1` blocks remote fetches so background refreshes can't poison parity. Both env vars added 2026-04-27 specifically to enable this loop.
-
-The harness covers 8 endpoints × 1 ID each, and only what's in the local pipeline output (3 evals, 28 models). It is *not* exhaustive — production has 587 evals / 5830 models — but it has already caught the model-summary pass-through bug (lib/duckdb-data.ts `toModelSummary` was returning the raw pipeline payload with lowercase category keys). When working on a new migration item, run parity before AND after the change.
-
-
-
-The Eval Cards Next.js app does not own its data: a Python pipeline publishes JSON
-artifacts to `evaleval/card_backend` on Hugging Face, `scripts/cache-hf-data.mjs`
-clones them at build time, and the `lib/` server modules adapt them at request
-time. A meaningful chunk of "shape‑fixing" still happens in TypeScript on every
-request or build. This note inventories those places and proposes which should
-move into the pipeline (dataset creation step) so the frontend becomes a pure
-read of canonical JSON.
-
-## Architecture recap
-
-- `scripts/cache-hf-data.mjs` — clones `evaleval/card_backend` and writes
-  `.cache/hf-data/{manifest, model-cards, eval-list, developers, benchmark-metadata,
-  eval-hierarchy, comparison-index}.json` plus per-detail directories
-  `models/`, `evals/`, `developers/`. Also re-normalizes some files in JS after
-  download.
-- `lib/hf-data.ts` — read layer over the cache + remote, plus converters that
-  flatten the pipeline's hierarchy back into `BenchmarkEvaluation[]`.
-- `lib/model-data.ts` — adapts raw HF artifacts into the app's domain types
-  (`BenchmarkEvalSummary`, `EvaluationCardData`, `ModelEvaluationSummary`).
-- `lib/eval-processing.ts` — re-aggregates per-model evaluations into family /
-  variant summaries.
-- `app/api/*` — thin route wrappers around the getters above.
-
-The pipeline already produces hierarchy, leaderboards, composite groupings, and
-benchmark cards. The items below are what the frontend still has to do that
-the pipeline could just emit instead.
-
-## Concrete migration candidates
-
-### 1. Family / variant identity parsing — *cleaning*
-- Where: `lib/model-family.ts` (`getCanonicalModelIdentity`), duplicated in
-  `scripts/cache-hf-data.mjs:141-303` (`normalizeHandle`,
-  `getCanonicalFamilyInfo`, `getNormalizedVariantMeta`,
-  `normalizeCachedModelCardFile`).
-- What it does: parses `model_info.id` (e.g. `anthropic/claude-3-5-sonnet-20240620`)
-  to derive `familyId`, `familySlug`, `versionDate`, `versionQualifier`,
-  `variantKey`, `variantLabel`. The cache script *rewrites* `model-cards.json`
-  in place after download so the runtime sees a canonicalized version.
-- Move: pipeline should emit canonical `family_id`, `family_slug`,
-  `version_date`, `variant_key`, `variant_label` directly so neither the build
-  script nor `lib/model-family.ts` has to re‑derive them.
-
-### 2. Setup-alias merging ("prompt" / "fc" / "thinking" variants) — *dual: cleaning + reshape*
-- Where: `lib/eval-processing.ts:371-434`
-  (`getSetupAliasMode`, `getAggregatedVariantDescriptor`); duplicated in
-  `scripts/cache-hf-data.mjs:199-246` (`getNormalizedVariantMeta`,
-  `isSetupAliasQualifier`).
-- What it does: looks at `model_info.additional_details.mode` to decide whether
-  two model rows are the same release with different prompting setups
-  ("prompt" / "fc" / "function calling" / "thinking…") and merges them under
-  one variant key.
-- Move:
-  - **Cleaning half** — pipeline emits a canonical `variant_key` (or `setup_alias_key`) per submission row.
-  - **Reshape half** — once the key is upstream, the bucket reduction (multiple rows with the same `variant_key` → single variant entry, `MAX(retrieved_timestamp)`, merged evaluation results) becomes SQL `GROUP BY variant_key` rather than a TS reduce loop.
-
-### 3. Hierarchy → flat `BenchmarkEvaluation[]` rebuild — *reshape*
-- Where: `lib/hf-data.ts:1236-1436`
-  (`flattenModelEvaluations`, `flattenHierarchyNode`,
-  `buildSourceMetadataIndex`).
-- What it does: walks the pipeline's `hierarchy_by_category` tree, attaches
-  `source_metadata` from a separate `evaluations_by_category` index (because
-  hierarchy rows don't carry it — see comment at `hf-data.ts:1410-1416`),
-  reconciles timestamps across variants, attaches inline samples, and re‑emits
-  a flat array so `createModelFamilySummary()` can group it again.
-- Move:
-  - Denormalize `source_metadata` onto every hierarchy leaf so the side index
-    isn't needed (this half is cleaning, already done as part of #4).
-  - The flatten + variant-bucket reduction itself is reshape: emit a relational parquet schema (one row per `(eval_summary_id, variant_key, retrieved_timestamp, source_metadata, …)`) so the DuckDB backend can `SELECT … QUALIFY ROW_NUMBER() OVER (PARTITION BY variant_key ORDER BY retrieved_timestamp DESC) = 1` instead of TS walking the hierarchy and reducing variants. Decision pending: relational parquet vs pre-flattened JSON payload.
-
-### 4. Source-metadata synthesis fallback — *cleaning (DONE)*
-- Where: `lib/hf-data.ts:1049-1064` (`getCanonicalSourceMetadata`); copies in
-  `lib/model-data.ts:736-741` (`toModelResultsForMetric`),
-  `lib/model-data.ts:1114-1119` (`buildSingleMetricSuiteMatrixSummary`).
-- What it does: when the artifact omits source metadata, hardcodes
-  `source_type: "documentation"`, `evaluator_relationship: "other"`. The
-  result: any code path that goes through these silently collapses 1st / 3rd-party
-  badges to "Other".
-- Move: source metadata should always be present on the artifact — the
-  fallback is a known correctness gap.
-
-### 5. Composite / aggregate eval construction — *reshape*
-- Where: `lib/model-data.ts:874-1041` (`aggregateBenchmarkSummaries`).
-- What it does: for any URL of the form `/evals/aggregate__<suite_key>`,
-  fetches every sub-eval detail individually, normalizes scores, computes
-  per-model averages, sorts, builds aggregate components, etc. ~170 lines.
-- Move: per-model averaging across sub-evals + sort + composite assembly is reshape (groupby + aggregate). Two valid landing spots: (a) materialize as a first-class eval-detail file at pipeline emission time (`eval-hierarchy.json` already knows the composites), or (b) compute at query time in DuckDB SQL once eval rows are relational. Materialization is simpler if every consumer wants the same composite shape; query-time SQL gives consumer-driven slicing for free.
-
-### 6. Synthetic single-metric matrix leaderboard — *reshape*
-- Where: `lib/model-data.ts:1043-1214`
-  (`buildSingleMetricSuiteMatrixSummary`).
-- What it does: for `/evals/matrix__<suite_key>`, fetches every sub-eval,
-  builds a model × subtask matrix, picks columns / rows, deduplicates, and
-  reconciles per-row timestamps. ~170 lines.
-- Move: pivot from long-form rows to wide model × subtask matrix is a SQL `PIVOT` (or grouped `MAX(score) FILTER (WHERE subtask = …)`) once eval rows are relational. Same materialize-vs-query-time choice as #5; lean toward materialization since the matrix shape is identical for every consumer.
-
-### 7. Instance-level JSONL parsing — *cleaning*
-- Where: `lib/hf-data.ts:919-1029` (`parseInstanceLevelData`,
-  `fetchInstanceLevelData`).
-- What it does: ~110 lines of heuristics probing for `input.raw`, `prompt`,
-  `question`, `doc.question`, `doc`, `output`, `model_output`,
-  `messages[].content`, `filtered_resps[0][0]`, `resps[0][0]`,
-  `answer_attribution`, `evaluation.is_correct`, `metrics.exact_match`, etc.
-- Why: shape detection across lm-eval-harness, HELM, Inspect, and other
-  harness outputs.
-- Move: pipeline should normalize each instance example to one canonical
-  `{sample_id, input, ground_truth, response, is_correct, metadata}` shape so
-  the frontend just renders.
-
-### 8. Display-name lookup tables — *cleaning*
-- Where:
-  - `lib/model-data.ts:93-124` (`BENCHMARK_NAMES`, `getBenchmarkDisplayName`)
-  - `components/benchmark-detail.tsx:101-173` (`SUITE_DISPLAY_NAMES`,
-    `DISPLAY_TOKEN_OVERRIDES`, `DISPLAY_NAME_OVERRIDES`)
-  - `lib/eval-processing.ts:861-885` (`getBenchmarkDisplayName`)
-- What it does: hand-maintained maps that translate keys like `helm_lite` →
-  "HELM Lite", `arc_agi` → "ARC-AGI", etc.
-- Move: schema already has `canonical_display_name`; pipeline should populate
-  it once and the frontend should drop the maps.
-
-### 9. Developer name canonicalization — *cleaning*
-- Where: `lib/model-data.ts:201-228` (`KNOWN_DEVELOPER_NAMES`,
-  `normalizeDeveloperName`).
-- What it does: `openai` → "OpenAI", `mistralai` → "Mistral AI",
-  `deepseek-ai` → "DeepSeek", etc.
-- Move: belongs in the pipeline's developer table.
-
-### 10. Generic metric-name expansion — *cleaning*
-- Where: `lib/eval-processing.ts:27-34, 70-86` (`GENERIC_EVALUATION_NAMES`,
-  `getEvaluationDisplayName`); mirrored heuristically in
-  `lib/model-data.ts:444-454` (`prefersBenchmarkName` detection of
-  "accuracy on…", "score on…", "for scorer…", "model_graded").
-- What it does: expands "Accuracy" → "MMLU - Accuracy" when the metric name
-  is generic.
-- Move: pipeline should emit a `display_name` that's already the right
-  thing.
-
-### 11. Category inference fallback — *cleaning (partial)*
-- Where: `lib/benchmark-schema.ts:182-206` (`inferCategoryFromBenchmark`),
-  used in `lib/eval-processing.ts:300-307`,
-  `lib/model-data.ts:776, 819, 1194`. Plus `PIPELINE_CATEGORY_MAP` /
-  `mapHFCategories` in `lib/hf-data.ts:1453-1469`.
-- What it does: regex fallback when the pipeline omits `category`.
-- Move: pipeline should emit `category` for every row so neither fallback nor
-  `mapHFCategories` is necessary.
-
-### 12. Parameter-count parsing from free text and model names — *cleaning*
-- Where:
-  - `lib/model-data.ts:296-338` (`parseParamsBillions`)
-  - `components/eval-detail.tsx:81-184`
-    (`parseParamsBillionsFromText`, `parseParamsBillionsFromModelName`,
-    `getParamsBillionsFromModelInfo`)
-  - `app/evals/[id]/page.tsx:434-437` (regex `\b(\d+(?:\.\d+)?)\s*[bB]\b`
-    against `name + " " + id`)
-- What it does: parses "70B", "1.5B", "405b", "7 billion", "1.2T" etc. The
-  matrix leaderboard even regex-extracts size from the model display name.
-- Move: pipeline should emit a normalized numeric `params_billions` so the
-  frontend doesn't need parsers in three places.
-
-### 13. Timestamp normalization — *dual: cleaning + reshape*
-- Where: `lib/eval-processing.ts:322-330, 572-577, 962-968`,
-  `lib/model-data.ts:60-65`, `components/eval-detail.tsx:218-238`,
-  `lib/hf-data.ts:1035-1047` (`toComparableTimestamp`).
-- What it does: timestamps arrive as either unix-seconds-as-string
-  (`"1774096306.427425"`) or ISO strings, and the same `Number(ts)` /
-  `new Date(ts)` branching is reimplemented at every read site.
-- Move:
-  - **Cleaning** — pipeline emits ISO-8601 strings everywhere (currently 99.99% unix-seconds-strings + 5 ISO).
-  - **Reshape** — the comparison call sites (8 of them across `lib/model-data.ts`, `lib/hf-data.ts`, `components/benchmark-detail.tsx`) all exist to pick the freshest variant or sort by recency. Once timestamps are ISO 8601, `MAX(retrieved_timestamp)` and `ORDER BY retrieved_timestamp DESC` work as SQL — three TS normalizers + 8 callers delete together. See `notes/transformations/07-timestamp-normalization.md`.
-
-### 14. Score normalization and summary stats — *reshape*
-- Where: `lib/eval-processing.ts:946-991` (`groupEvaluationsByBenchmark`
-  finalisation), `lib/model-data.ts:67-72, 803-851` (`hfEvalDetailToSummary`).
-- What it does: recomputes `models_count`, `avg_score`, `avg_score_norm`,
-  `best_model`, `worst_model`, `evaluator_names`, `source_types`,
-  `latest_source_name`, `third_party_ratio`,
-  `missing_generation_config_count` even though `eval-list.json` already
-  carries `top_score` / `models_count`.
-- Move: aggregation over `model_results` (count, avg, min/max, distinct counts, ratios) is reshape — pure SQL groupby. Materialize at pipeline emission OR compute at query time once `model_results` are relational rows in parquet. Either way the TS finalisation deletes.
-
-### 16. Per-category benchmark counts on model cards — *reshape*
-- Where: `lib/model-data.ts:354-363` (proportional distribution in
-  `hfModelCardToEvaluationCardData`).
-- What it does: because `model-cards.json` doesn't carry `category_stats`, TS
-  does `Math.floor(total / categories.length)` to *fake* a per-category split.
-- Move: per-(model, category) benchmark count is `SELECT model, category, COUNT(DISTINCT benchmark) GROUP BY model, category` — pure SQL groupby. Materialize as `category_stats` on the model card OR compute at query time. The TS `Math.floor(total / categories.length)` placeholder ships incorrect numbers; the SQL version is the actual answer.
-
-### 17. Benchmark-card attachment at request time — *cleaning*
-- Where: `lib/benchmark-metadata.ts:38-49` (`getBenchmarkCard`,
-  `getMap`), `lib/model-data.ts:857-872` (`attachBenchmarkCardToSummary`).
-- What it does: iterates 3 candidate names per eval and looks each up in a
-  deduped `Map<string, BenchmarkCard>`.
-- Move: eval-detail files already sometimes carry `benchmark_card` inline —
-  the pipeline should always inline it (or always reference by stable key) so
-  the runtime lookup table can be deleted.
-
-### 18. License canonicalization — *cleaning*
-- Where: `components/eval-card.tsx:22-48` (`LICENSE_COLORS`,
-  `licenseBadgeClass`, `shortenLicense`).
-- What it does: "Creative Commons Attribution 4.0" → "CC BY 4.0",
-  "Apache License 2.0" → "Apache 2.0", "Creative Commons Zero" → "CC0", etc.
-- Move: pipeline should expose a normalized SPDX-style license identifier
-  alongside the long string.
-
-### 19. Slug candidate generation for HF lookups — *cleaning*
-- Where: `lib/model-data.ts:151-194` (`getModelDetailSlugCandidates`,
-  `getDeveloperSlugCandidates`); used by `lib/model-data.ts:1467-1509`
-  (`getModelSummaryById`).
-- What it does: produces up to 6 spelling variants of a slug (`gpt-3.5` ↔
-  `gpt-3-5`, `__` vs `/`, `_` vs `-`) and `getModelSummaryById` retries each
-  candidate against the dataset.
-- Move: the manifest could explicitly map every `model_family_id` /
-  `route_id` → file path so retry loops disappear.
-
-### 20. Dataset URL synthesis — *cleaning*
-- Where: `components/eval-card.tsx:81-89`.
-- What it does: computes
-  `dataset_url ?? url[0] ?? https://huggingface.co/datasets/${hf_repo}` to
-  derive a clickable link.
-- Move: pipeline already has `hf_repo` and could just emit the resolved URL
-  once.
-
-## Suggested priorities
-
-If triaging by payoff:
-
-1. **#3 + #4 — hierarchy flatten + source-metadata index.** Eliminates the
-   largest in-process function in the codebase and a known correctness bug
-   ("Other" badge collapse).
-2. **#5 + #6 — composites & matrix synthesis.** Replaces ~340 lines of
-   request-time TS reconstruction with pre-built artifacts.
-3. **#7 — instance JSONL parser.** Biggest brittleness surface; one
-   normalization step in the pipeline removes a heuristic.
-4. **#1 + #2 — identity parsing & setup-alias merging.** Removes
-   triple-maintained logic across `lib/model-family.ts`,
-   `lib/eval-processing.ts`, and `scripts/cache-hf-data.mjs`.
-5. **#8–#13 — display names, developer names, params parsing, timestamps,
-   category fallback.** Small individually, but together they are scattered
-   "polish" code that should live next to the data.