Document root GNN architecture and migration

AGENTS.md

@@ -2,10 +2,21 @@
 
 ## Goal
 
-Rewrite the ML system in `legacy/` into a clean implementation in `src/`.
+Rewrite the main ML system in `legacy/root_gnn_dgl/` into a clean
+implementation in `src/`. The sibling `legacy/physicsnemo/` tree is a prior
+rewrite attempt: it may provide design inspiration, but it is not the target
+behavioral reference.
 
-Do not modify `legacy/` unless explicitly asked. It is the behavioral
-reference implementation.
+Do not modify `legacy/` unless explicitly asked. `legacy/root_gnn_dgl/` is the
+behavioral reference implementation for this project.
+
+## Reverse-engineering scope
+
+The primary executable pipeline is `legacy/root_gnn_dgl/scripts/` plus
+`legacy/root_gnn_dgl/root_gnn_base/`, `models/`, and the YAML configurations.
+Documentation and parity work should cite those files and symbols directly.
+Treat `legacy/physicsnemo/` as out of scope except when explicitly comparing
+its abstractions as rewrite inspiration.
 
 ## Priorities
 
@@ -63,4 +74,4 @@ For non-trivial work:
 6. Run them.
 7. Summarize changes and remaining migration work.
 
-Do not rewrite the entire repository in one pass.
+Do not rewrite the entire repository in one pass.

docs/architecture.md

@@ -1,17 +1,213 @@
-# Target architecture
-
-The new implementation is organized by responsibility:
-
-- `src/gnn4colliders/`: importable production package
-- `configs/`: declarative experiment and data configuration
-- `scripts/`: thin command-line entry points
-- `tests/unit/`: focused component tests
-- `tests/integration/`: end-to-end component integration tests
-- `tests/parity/`: comparisons between the new implementation and `legacy/`
-- `notebooks/`: exploratory analysis only
-- `data/fixtures/`: small deterministic test inputs
-
-Dataset loading, preprocessing, model definition, training, evaluation, and
-serialization should remain separate modules. Randomness must be explicit and
-seedable, and configuration should be passed into application code rather than
-hardcoded in training logic.
+# Reverse-engineered architecture: `root_gnn_dgl`
+
+This document covers the target system in `legacy/root_gnn_dgl/`. The sibling
+`legacy/physicsnemo/` tree is a prior rewrite attempt and is not a behavioral
+target.
+
+## 1. High-level system description
+
+`root_gnn_dgl` is a ROOT-to-DGL graph classification system. YAML selects
+dataset, model, loss, and finish-function classes by import path. The dataset
+reads ROOT trees, converts collider objects to fully connected DGL graphs, and
+saves graph chunks. Training loads those chunks, applies fold selection and
+optional pre-batching/padding, trains a graph network, writes one PyTorch
+checkpoint per epoch, and reports weighted loss, accuracy, and ROC AUC.
+
+The primary model is `models.GCN.Edge_Network` ([`GCN.py:182-251`](../legacy/root_gnn_dgl/models/GCN.py)).
+It encodes node, edge, and global features, repeats edge -> node -> global
+message passing `n_proc_steps` times, decodes the global state, and applies
+`classify`. Fine-tuning uses `models.GCN.Transferred_Learning_Finetuning`
+([`GCN.py:884-997`](../legacy/root_gnn_dgl/models/GCN.py)), which loads a
+pretrained `Edge_Network`, removes its final classifier, and applies a new one.
+The active configs use output size 12 for multiclass pretraining and output
+size 1 for binary tasks ([`configs/stats_100K/pretraining_multiclass.yaml:1-45`](../legacy/root_gnn_dgl/configs/stats_100K/pretraining_multiclass.yaml),
+[`configs/stats_100K/finetuning_ttH_CP_even_vs_odd.yaml:1-45`](../legacy/root_gnn_dgl/configs/stats_100K/finetuning_ttH_CP_even_vs_odd.yaml)).
+
+### Entry points and flows
+
+- `scripts/training_script.py:main` and its CLI parser load YAML, create
+  loaders, construct the model, and call `train`; `--evaluate` calls
+  `evaluate` ([`training_script.py:638-843`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+- `scripts/prep_data.py:main` creates configured graph caches
+  ([`prep_data.py:68-110`](../legacy/root_gnn_dgl/scripts/prep_data.py)).
+- `scripts/inference.py:main` reconstructs an unlazy dataset, loads one or
+  more checkpoints, and writes `.npz` or ROOT scores
+  ([`inference.py:163-387`](../legacy/root_gnn_dgl/scripts/inference.py)).
+- `scripts/export_onnx.py:main` exports an ONNX-friendly model
+  ([`export_onnx.py:979-1035`](../legacy/root_gnn_dgl/scripts/export_onnx.py)).
+- `selections.py:main`, `check_dataset_files.py:main`, and
+  `plot_config_distributions.py:main` are diagnostic entry points. `run_demo.sh`
+  sequences pretraining, binary training, fine-tuning, and inference
+  ([`run_demo.sh:3-59`](../legacy/root_gnn_dgl/run_demo.sh)).
+
+The training flow is:
+
+```text
+YAML -> load_config/buildFromConfig -> RootDataset/LazyDataset
+     -> ROOT/Awkward -> DGL graph + labels/tracking/globals -> .bin cache
+     -> fold_selection -> prebatch/padding -> GraphDataLoader
+     -> Edge_Network or transfer model -> weighted loss/metrics
+     -> model_epoch_N.pt, logs, evaluation/inference output
+```
+
+`training_script.train` is the lifecycle implementation
+([`training_script.py:143-614`](../legacy/root_gnn_dgl/scripts/training_script.py));
+distributed paths use NCCL/DDP ([`training_script.py:616-839`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Inference uses `CustomPreBatchedDataset`, applies a configured finish function,
+and collects `scores`, `labels`, and `tracking_info`
+([`inference.py:20-76`](../legacy/root_gnn_dgl/scripts/inference.py),
+[`inference.py:223-325`](../legacy/root_gnn_dgl/scripts/inference.py)).
+
+## 2. Dependency and data-flow map
+
+```text
+ROOT files (raw_dir/file_names, tree_name)
+  -> RootDataset.process (uproot/Awkward; dataset.py:221-320)
+  -> node_features_from_tree (dataset.py:15-50)
+  -> full_connected_graph (dataset.py:52-59)
+  -> EdgeDataset.make_graph: [deta, dphi, dR] (dataset.py:471-482)
+  -> DGL .bin cache / LazyDataset / PreBatchedDataset
+  -> GraphDataLoader -> models.GCN -> loss/metrics -> outputs
+```
+
+`RootDataset` provides `process`, `save`, `load`, `__getitem__`, and `__len__`
+([`dataset.py:160-469`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py));
+`LazyDataset` loads one chunk through a ring buffer
+([`dataset.py:525-578`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py));
+`PreBatchedDataset.process` selects, shuffles, batches, pads, and caches
+([`batched_dataset.py:34-146`](../legacy/root_gnn_dgl/root_gnn_base/batched_dataset.py)).
+
+`load_config` uses PyYAML `FullLoader` and shallow `include` merging, while
+`buildFromConfig` dynamically imports `module`, resolves `class`, merges extra
+keys into `args`, converts list-valued weights to tensors, and injects runtime
+arguments ([`utils.py:10-43`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)).
+This reflection shape is a de facto interface for configured components.
+
+### Data and preprocessing
+
+The active node schema is seven columns: `pt`, `eta`, `phi`, `energy`, `btag`,
+`charge`, and `node_type`. `CALC_E` is `pt*cosh(eta)`, constants are broadcast
+per object type, `NODE_TYPE` is an integer type code, and feature scales are
+applied columnwise ([`dataset.py:15-50`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)).
+`full_connected_graph` makes directed all-pairs edges; `EdgeDataset` requests
+no self-loops and stores `[deta, dphi, dR]`
+([`dataset.py:52-59`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py),
+[`dataset.py:471-482`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)).
+
+Selections are strings evaluated with builtins disabled or
+`(variable, cut, operator)` triples (`check_selection`, `selection_mask`;
+[`dataset.py:75-145`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)). Fold
+selection uses `tracking[:,0] % n_folds`; tracking column 0 is fold and column 1
+is weight ([`utils.py:121-143`](../legacy/root_gnn_dgl/root_gnn_base/utils.py),
+[`dataset.py:176-182`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)).
+`hash_partition` and a seeded Torch generator control pre-batch order
+([`batched_dataset.py:27-99`](../legacy/root_gnn_dgl/root_gnn_base/batched_dataset.py)).
+Padding modes are `NONE`, `STEPS`, `FIXED`, and `NODE`; `FIXED` is hardcoded to
+16,000 nodes and 104,000 edges ([`batched_dataset.py:100-125`](../legacy/root_gnn_dgl/root_gnn_base/batched_dataset.py)).
+
+### Model, losses, and metrics
+
+`Make_MLP` builds linear/ReLU/dropout blocks followed by LayerNorm
+([`GCN.py:18-35`](../legacy/root_gnn_dgl/models/GCN.py)). Each `Edge_Network`
+step encodes inputs, copies source/destination states to edges, updates edges,
+sums edge messages into nodes, updates nodes, then mean-pools nodes/edges to
+update globals ([`GCN.py:195-249`](../legacy/root_gnn_dgl/models/GCN.py)). It
+returns logits `[graphs, out_size]` without sigmoid/softmax.
+
+The default objective is elementwise `BCEWithLogitsLoss`, multiplied by
+`tracking[:,1]`, averaged separately per unique label, then averaged across
+labels ([`training_script.py:143-185`](../legacy/root_gnn_dgl/scripts/training_script.py),
+[`training_script.py:320-359`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+`--abs` makes weights positive. Binary metrics use sigmoid threshold 0.5 and
+weighted ROC AUC; multiclass metrics use argmax and one-vs-rest ROC AUC
+([`training_script.py:438-510`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Additional configurable losses/finishers live in `models/loss.py`
+([`loss.py:6-310`](../legacy/root_gnn_dgl/models/loss.py)).
+
+### Checkpoints and outputs
+
+Training writes `Training_Directory/model_epoch_<epoch>.pt` containing `epoch`,
+`model_state_dict`, `optimizer_state_dict`, and serialized `early_stop`
+([`training_script.py:565-604`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Keys strip `module.` and compiled models save the underlying `_orig_mod` state;
+`get_last_epoch`, `get_specific_epoch`, and `get_best_epoch` load the files
+([`utils.py:145-248`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)).
+`evaluate` writes `evaluation_<epoch>.npz`; inference writes `.npz` fields
+`scores`, `labels`, `tracking_info`, or adds score branches and `selection_pass`
+to a cloned ROOT tree ([`training_script.py:57-140`](../legacy/root_gnn_dgl/scripts/training_script.py),
+[`inference.py:328-385`](../legacy/root_gnn_dgl/scripts/inference.py)).
+
+## Randomness, external services, and coupling
+
+The CLI exposes `--seed`, but `main` passes it to model construction rather
+than globally seeding Python, NumPy, or Torch
+([`training_script.py:638-753`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Pre-batching has explicit seeds, but `AugmentedDataset` mutates the process-wide
+NumPy seed ([`dataset.py:716-827`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)),
+clustering uses unseeded `torch.randperm`/`randint` (`loss.py:259-295`), and
+model reset/fine-tuning hardcodes `torch.manual_seed(2)`
+([`GCN.py:58-65`](../legacy/root_gnn_dgl/models/GCN.py),
+[`GCN.py:900-915`](../legacy/root_gnn_dgl/models/GCN.py)). CUDA kernels, DDP,
+and DataLoader behavior are not made deterministic.
+
+Implicit coupling includes repository-relative `sys.path` insertion
+([`training_script.py:14-20`](../legacy/root_gnn_dgl/scripts/training_script.py)),
+dynamic imports, mutable default lists/dicts, global `FEATURE_DTYPE`
+([`dataset.py:13`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)),
+in-place `tracking_info` mutation ([`dataset.py:176-181`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)),
+and in-place DGL graph mutation during forward.
+
+The environment assumes Python 3.8, PyTorch 2.0.1, CUDA 11.8, DGL 1.1.1,
+ROOT, Awkward, Uproot, PyYAML, and scikit-learn
+([`setup/environment.yml:1-10`](../legacy/root_gnn_dgl/setup/environment.yml),
+[`setup/environment.yml:240-295`](../legacy/root_gnn_dgl/setup/environment.yml)).
+Standard configs assume `/global/cfs/` and `/pscratch/` paths, CUDA/NCCL,
+Slurm, and optionally Podman-HPC. `setup/download_data.sh` downloads the
+external Hugging Face dataset `HWresearch/Delphes`
+([`download_data.sh:13-67`](../legacy/root_gnn_dgl/setup/download_data.sh)).
+
+## Apparent unused or secondary code
+
+Not selected by the standard stats/Delphes configs, or only reachable from
+optional workflows, are `GCN_global`, `GCN_global_2way`, most transfer variants,
+attention models, `MultiModel`, and `Clustering`
+([`GCN.py:122-1933`](../legacy/root_gnn_dgl/models/GCN.py)); `UprootDataset`,
+`tHbbEdgeDataset`, `AugmentedDataset`, and photon-ID paths
+([`uproot_dataset.py:10-31`](../legacy/root_gnn_dgl/root_gnn_base/uproot_dataset.py),
+[`dataset.py:484-827`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py),
+[`photon_ID_dataset.py:1-33`](../legacy/root_gnn_dgl/root_gnn_base/photon_ID_dataset.py));
+optional loss and similarity utilities; and the no-op
+`root_gnn_base.utils.graph_augmentation` ([`utils.py:393-395`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)).
+The main path evaluates `test_loaders`; validation loaders are only assembled
+when a config has a validation fold ([`training_script.py:682-747`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+
+## 3. De facto interfaces to preserve
+
+1. YAML `module`, `class`, `args`, plus runtime `sample_graph` and
+   `sample_global` injection.
+2. Dataset items `(DGLGraph, label, tracking, global_features)`.
+3. `ndata['features']`, `edata['features']`, seven node columns, and three edge
+   columns in `[deta, dphi, dR]` order.
+4. Tracking column 0 fold and column 1 weight semantics.
+5. `model(graph, global_feats)`, logits shape `[batch, out_size]`, and
+   `representation` where used.
+6. Weighted per-label loss, metric thresholds, checkpoint keys/prefix cleanup,
+   epoch filenames, and `.npz`/ROOT output fields.
+
+## 4. Ambiguous behavior
+
+- Historical edge order/self-loop expectations; empty and padding graph inputs.
+- Whether negative weights are meaningful or should always be absolute.
+- Whether “validation” is intended to differ from the active test-loader path.
+- Shape semantics of multi-label finishers and experimental transfer classes.
+- Whether chunk IDs must match historical `np.array_split` boundaries.
+- Required behavior for missing branches and dynamic selection expressions.
+
+## 5. Recommended rewrite boundaries
+
+Separate typed configuration; ROOT/Awkward I/O; selections/folds/features/
+edges; DGL cache/lazy loading/batching; active models and checkpoint adapters;
+objectives/metrics; training lifecycle; and inference/ONNX applications.
+Establish parity for the active `LazyDataset -> PreBatchedDataset ->
+Edge_Network` binary/multiclass path first. Add experimental classes only when
+a config or consumer proves they are required.

docs/migration.md

@@ -1,16 +1,138 @@
-# Migration plan
+# Incremental migration plan: `root_gnn_dgl`
 
-`legacy/` is the frozen behavioral reference. Do not modify it during the
-rewrite. Migrate one coherent capability at a time into `src/gnn4colliders`.
+The target is `legacy/root_gnn_dgl/`. `legacy/physicsnemo/` is a prior rewrite
+attempt and may inspire abstractions, but it is not a parity target. Neither
+legacy tree should be modified during migration.
 
-For each migrated capability:
+Each phase should add focused unit tests, a deterministic fixture in
+`data/fixtures/`, and parity tests under `tests/parity/` before moving upward.
+Record intentional differences and checkpoint consequences here.
 
-1. Identify the legacy inputs, outputs, shapes, dtypes, and metric semantics.
-2. Define a small typed interface in the new package.
-3. Add unit tests for the new implementation.
-4. Add parity tests comparing representative deterministic fixtures with the
-   legacy behavior.
-5. Record any intentional incompatibility and checkpoint implications here.
+## Phase 0 — freeze observations and fixtures
 
-The initial migration order is dataset loading and preprocessing, graph/model
-components, training and evaluation, then checkpoint and command-line support.
+Capture a small representative ROOT-equivalent fixture containing the seven
+active node features, three edge features, labels, fold values, weights, and
+globals. Record outputs of `node_features_from_tree`, `full_connected_graph`,
+`EdgeDataset.make_graph`, and `fold_selection`
+([`dataset.py:15-59`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py),
+[`dataset.py:471-482`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py),
+[`utils.py:121-143`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)). Preserve
+one `.bin`, one `model_epoch_N.pt`, one evaluation `.npz`, and one inference
+`.npz` fixture if available.
+
+## Phase 1 — configuration boundary
+
+Implement a typed configuration layer that reads `Training`, `Model`,
+optional `Loss`, and `Datasets`. Initially retain a compatibility adapter for
+`module`/`class`/`args` and runtime injection of `sample_graph` and
+`sample_global`, matching `buildFromConfig`
+([`utils.py:10-43`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)). Keep
+dynamic imports isolated at this boundary rather than spreading reflection
+through new code.
+
+## Phase 2 — pure preprocessing parity
+
+Port and test, in isolation:
+
+- branch-to-node conversion, `CALC_E`, `NODE_TYPE`, constants, scaling, empty
+  objects, and dtypes (`node_features_from_tree`,
+  [`dataset.py:15-50`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py));
+- string/tuple selections and cutflow (`check_selection`, `selection_mask`,
+  `compute_cutflow`, [`dataset.py:75-158`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py));
+- fold masks and cache suffixes (`fold_selection`, `fold_selection_name`,
+  [`utils.py:121-143`](../legacy/root_gnn_dgl/root_gnn_base/utils.py));
+- deterministic chunk partitioning (`hash_partition`,
+  [`batched_dataset.py:27-31`](../legacy/root_gnn_dgl/root_gnn_base/batched_dataset.py)).
+
+This is the highest-value parity layer: model parity is invalid if graph inputs
+differ.
+
+## Phase 3 — graph construction and cache format
+
+Implement graph construction with tests for node/edge counts, directed edge
+ordering, self-loop policy, `[deta, dphi, dR]` order, metadata, and empty graphs.
+Preserve the dataset item contract `(graph, label, tracking, global_features)`
+from `RootDataset.__getitem__`
+([`dataset.py:465-469`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py)).
+
+Then implement DGL `.bin` serialization, lazy chunk loading, pre-batching, and
+padding. Compare against `RootDataset.save/load`, `LazyDataset`, and
+`PreBatchedDataset` ([`dataset.py:396-469`](../legacy/root_gnn_dgl/root_gnn_base/dataset.py),
+[`batched_dataset.py:129-174`](../legacy/root_gnn_dgl/root_gnn_base/batched_dataset.py)).
+Treat `NONE`, `STEPS`, `FIXED`, and `NODE` as explicit features; do not hide
+the hardcoded fixed padding sizes.
+
+## Phase 4 — active model parity
+
+Port `models.GCN.Edge_Network` first. Preserve constructor parameters,
+`forward(graph, global_feats)`, feature keys, processor order, MLP LayerNorm
+placement, and logits shape. Compare intermediate and final tensors on fixed
+graphs using the legacy architecture
+([`GCN.py:18-35`](../legacy/root_gnn_dgl/models/GCN.py),
+[`GCN.py:182-251`](../legacy/root_gnn_dgl/models/GCN.py)).
+
+Next port `Transferred_Learning_Finetuning`, including pretrained
+`model_state_dict` loading, removal of the final classifier, and new classifier
+initialization ([`GCN.py:884-997`](../legacy/root_gnn_dgl/models/GCN.py)). Test
+both frozen and unfrozen modes. Defer other model classes until an active
+config or consumer proves they are needed.
+
+## Phase 5 — objectives and metrics
+
+Implement the default objective exactly: elementwise configured loss,
+tracking-column weights, per-unique-label normalization, and averaging across
+labels ([`training_script.py:320-359`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Add parity cases for positive, zero, and negative weights and binary versus
+multiclass shapes.
+
+Port metric behavior from
+[`training_script.py:438-510`](../legacy/root_gnn_dgl/scripts/training_script.py):
+sigmoid threshold 0.5, argmax, weight masking, weighted ROC AUC, one-vs-rest
+multiclass AUC, and NaN behavior when AUC is undefined. Add `models/loss.py`
+classes only with dedicated tests; do not substitute their reductions.
+
+## Phase 6 — checkpoint and lifecycle
+
+Create a checkpoint adapter preserving `model_epoch_<epoch>.pt` and keys
+`epoch`, `model_state_dict`, `optimizer_state_dict`, and `early_stop`
+([`training_script.py:565-604`](../legacy/root_gnn_dgl/scripts/training_script.py)).
+Support legacy DDP/compiled prefixes (`module.` and `_orig_mod.`) as exercised
+by checkpoint lookup and inference
+([`utils.py:145-248`](../legacy/root_gnn_dgl/root_gnn_base/utils.py),
+[`inference.py:274-290`](../legacy/root_gnn_dgl/scripts/inference.py)). Port
+`EarlyStop` state and log parsing separately
+([`utils.py:325-390`](../legacy/root_gnn_dgl/root_gnn_base/utils.py)). Verify
+resume, restart, early termination, and `.npz` fields before distributed work.
+
+## Phase 7 — CLI, inference, and export
+
+Build thin new applications around tested library interfaces in this order:
+
+1. preprocessing/cache generation (`scripts/prep_data.py`);
+2. training/evaluation (`scripts/training_script.py`);
+3. inference to `.npz` and ROOT (`scripts/inference.py`);
+4. ONNX export after PyTorch parity (`scripts/export_onnx.py`).
+
+Use subprocess integration tests with tiny fixtures. Preserve CLI options only
+where they serve an active workflow; document removed diagnostic/cluster-only
+options.
+
+## Phase 8 — reproducibility and deployment
+
+Introduce one explicit seed policy covering Python, NumPy, Torch, sampling,
+augmentation, and clustering. Add deterministic-mode tests and document GPU
+nondeterminism. Isolate Slurm/NCCL, Podman-HPC, ROOT, and Hugging Face data
+download integrations behind adapters only after local behavior is stable.
+
+## Checkpoint compatibility checklist
+
+- [ ] Load the checked-in multiclass pretrained checkpoint.
+- [ ] Load a legacy fine-tuning checkpoint after prefix normalization.
+- [ ] Resume optimizer and early-stop state.
+- [ ] Produce equivalent logits on a deterministic graph fixture.
+- [ ] Produce equivalent `.npz` score, label, and tracking fields.
+- [ ] Preserve ROOT scalar/vector score branch conventions if ROOT output stays.
+
+Known risks are documented in [`architecture.md`](architecture.md): edge order,
+self-loops, weight semantics, validation/test naming, padding, dynamic
+selection evaluation, reproducibility, and the experimental model/loss surface.