notes/transformations/README.md

Transformations registry

Canonical specs for data transformations the Next.js app currently performs that should ultimately live in the upstream pipeline (eval_cards_backend_pipeline).

Framing

The TS code in lib/ does meaningful data transformation work — token canonicalization, variant grouping, source-metadata defaults, category inference, score normalization, etc. — most of which the Python pipeline does not yet do (or does differently). For this migration:

• TS is the current source of truth for what these transformations should produce. The TS rules have been refined over time against real evaluation data; they encode product decisions.
• Each transformation is a candidate to move upstream. The pipeline owns data shape and content; doing the transformation there means every downstream consumer (not just this Next.js app) gets the canonical form.
• The migration path for each item: write a spec here → write executable tests sourced from the spec → hand the spec + tests to the pipeline owner → verify pipeline output matches across the full corpus → delete the TS implementation.
• Pre-processing in this repo (e.g. a one-shot Python script run at data-ingestion time) is allowed but not required. Default is "transformation stays in TS until pipeline catches up." Front-load only when the TS implementation is so brittle or expensive that on-the-fly is intolerable.

One thing to watch: defaults vs unconditional normalization

For each rule, classify in the spec how it interacts with pre-existing data:

• Default-only (don't overwrite when pipeline already emits a value): e.g. "if source_metadata.evaluator_relationship is missing, default to other". Pipeline-side fix is to emit the default upstream rather than letting consumers fill in.
• Unconditional normalization (always overwrite): e.g. "lowercase the v in version tokens regardless of upstream input". Pipeline-side fix is to apply the rule before emitting; downstream consumers should not need to re-derive.

Mis-classifying these is the failure mode that bit us in Phase 3 (treating a normalization rule as if it were a default, ending up overwriting category data). Each spec must call out which kind it is.

Where it belongs: cleaning vs reshape

A second classification each spec should call out — what kind of work is this transformation?

• Cleaning / standardization (changes a value): license shorthand, developer name canonicalization, identity tokens, timestamp format, category labels. These belong in the pipeline; the per-item workflow above is built for them. Default-vs-normalization (the section above) is the sub-question.
• Reshape / dedup / aggregate (computes a derived view): variant dedup with "freshest wins", per-category counts, top-scores ranking, hierarchy flattening. These belong in DuckDB SQL — either materialized into pipeline parquet (the answer is the same for every consumer; emit it pre-computed) or expressed as a query at request time (consumers slice differently; let SQL do the work). The TS implementation here is scaffolding.

For the reshape class the migration target shape is itself a design choice — capture the operation (e.g. "max retrieved_timestamp wins per variant key, take its source_metadata along") and flag it for the parquet-schema / SQL conversation rather than mechanically translating the TS code line-by-line. See notes/migration-plan.md § "Data direction" for the full framing.

Index

#	Transformation	Spec	Tests	Pipeline status	Migration item
01	Model identity canonicalization	01-identity-canonicalization.md	tests/transformations/identity-canonicalization.test.ts	partial (model_family_id ✅, model_family_name ❌ on 1,260 cards)	#1
02	Setup-alias variant merging	02-setup-alias-merging.md	tests/transformations/setup-alias-merging.test.ts	not started (cache file shows pre-merge state; runtime normalizer is what merges)	#2
03	License string normalization	03-license-normalization.md	tests/transformations/license-normalization.test.ts	not implemented; pipeline emits free-text data_licensing only	#18
04	Dataset URL synthesis	04-dataset-url-synthesis.md	tests/transformations/dataset-url-synthesis.test.ts	not implemented; dataset_url field never populated in prod (564/587 use url[0], 22/587 use hf_repo template)	#20
05	Slug candidate generation (file lookup)	05-slug-candidates.md	tests/transformations/slug-candidates.test.ts	not implemented; 39% of model lookups + 43% of developer lookups need a non-zero retry position in production	#19
06	Developer name canonicalization	06-developer-name-canonicalization.md	tests/transformations/developer-name-canonicalization.test.ts	not implemented; pipeline emits raw developer string. TS map covers 1.8% of devs / 11.9% of cards; title-case fallback fires on 55.6% of devs / 48.4% of cards	#9
07	Timestamp normalization	07-timestamp-normalization.md	tests/transformations/timestamp-normalization.test.ts	not implemented; production is 99.99% unix-seconds-strings (86,178/86,183) + 5 ISO datetime. Three different TS variants exist with subtly different semantics — pipeline canonicalization to ISO 8601 collapses them	#13
08	Benchmark display names	08-benchmark-display-names.md	tests/transformations/benchmark-display-names.test.ts	not implemented; 30-entry hand-curated map covers ~74% of distinct suite keys but only ~3% of distinct benchmark values; ~97% fall through to a humanizeToken fallback that mangles acronyms (MMLU-PRO → MMLU PRO, helm_air_bench → Helm Air Bench). A second functionally-dead duplicate exists in lib/eval-processing.ts with substring-match semantics that disagree with the active path.	#8
09	Metric display name expansion	09-metric-display-name-expansion.md	tests/transformations/metric-display-name-expansion.test.ts	defensive scaffolding; both rules fire 0 times against current corpus (0/86,183 result rows for generic-name expansion; 0/587 eval-list entries for prefersBenchmarkName heuristic)	#10
10	Params billions parsing	10-params-parsing.md	tests/transformations/params-parsing.test.ts	partial; model-cards.json.params_billions is clean number for 87% of cards (5072/5830); per-row additional_details.params_billions is string for 31.7% of rows (27,361/86,183), 47.2% (40,648) have no resolved value at all. Five different TS parsers diverge on edge cases — Variant C's "context-window beats param count" quirk fires on 472 rows (0.55%, names like Yi-1.5-34B-32K)	#12
11	Benchmark-card attachment (per-eval lookup join)	11-benchmark-card-attachment.md	tests/transformations/benchmark-card-attachment.test.ts	partial; pipeline inlines benchmark_card on 88/587 evals (15%), other 499 fall through to runtime retry. Of those, 10 hit at position 0 and 489 miss entirely (most lack any matching card in benchmark-metadata.json). Map-build first-write-wins silently drops the helm_instruct card (different content from kept helm_capabilities, both named "HELM"); 29/83 cards orphaned	#17
12	Per-instance JSONL normalization	12-instance-level-data.md	tests/transformations/instance-level-data.test.ts	pipeline emits canonical shape; parser's defensive fallback branches mostly dead (input/ground_truth/is_correct/sample_id all 100% via the canonical path; response splits 97.31% answer_attribution / 2.49% messages / 0.20% output). 712/86,183 result rows (0.83%) have inline samples; full sets (~35k samples) sit behind source_url and are accessible only via the orphaned fetchInstanceLevelData (no current UI consumer).	#7

(More entries land as we work each migration item. See notes/migration-plan.md for the full backlog.)

File format

Each spec follows the same structure so pipeline owner can read them uniformly. Template:

# <Transformation name>

## Rule
[Plain-English description of what the transformation does]

## Classification
- [ ] Default-only (do not overwrite when value present)
- [ ] Unconditional normalization (always apply)
[explanation]

- [ ] Cleaning / standardization → pipeline (changes a value's content; use per-item workflow)
- [ ] Reshape / dedup / aggregate → DuckDB SQL (computes a derived view; capture operation, flag for parquet-schema/SQL conversation)
[explanation — if both halves apply, explain the split]

## Inputs and expected outputs
[Table: input | expected output | notes / which rule branch this hits]

## Current TS implementation
- [file:line references]
- [key helpers/constants]

## Pipeline status
[Per-rule status against full live cache: matches / disagrees / not implemented]

## Divergences detected
[Concrete examples of pipeline-vs-TS disagreement, with row counts]

## Migration checklist
- [ ] Spec written
- [ ] Tests cover each rule branch
- [ ] Filed with pipeline owner (link)
- [ ] Pipeline emits matching values across full corpus
- [ ] TS code deleted; callers read pipeline fields directly