josephsoo commited on
Commit
66ea684
·
1 Parent(s): 77d097c

Refine public benchmark presentation

Browse files
Files changed (3) hide show
  1. README.md +7 -51
  2. app.py +207 -679
  3. assets/styles.css +150 -163
README.md CHANGED
@@ -7,48 +7,15 @@ short_description: Interactive BEND-BCI neural decoder benchmark
7
 
8
  # BEND-BCI Interactive Benchmark
9
 
10
- This Space is the interactive companion to BEND-BCI (Benchmarking Neural
11
- Decoders for Brain-Computer Interfaces). It presents 23 evaluated methods on
12
- five primary tasks.
 
13
 
14
- The application keeps the benchmark's scientific organization explicit:
 
15
 
16
- - decoder-selection measurements: held-out task prediction, robustness to
17
- added count noise, computational cost, and cross-recording latent
18
- consistency;
19
- - diagnostic assays: feature attribution and trial-level data valuation.
20
-
21
- Interactive matrices use within-dataset percentile ranks, with raw values and
22
- task-specific metrics retained on hover. Unavailable analyses remain visible
23
- as coverage limits and are not imputed.
24
-
25
- ## Data and figure provenance
26
-
27
- The bundled tables in `data/` are synchronized from the canonical
28
- `paper/results/*.csv` exports in the
29
- [main BEND-BCI repository](https://github.com/TangLab-UBC/behavior_benchmarking).
30
- Each source table retains artifact paths for provenance.
31
-
32
- The interactive sections follow the current generated manuscript figures:
33
-
34
- - Figure 2: prediction, robustness, and computational cost;
35
- - Figure 3: cross-recording latent consistency;
36
- - Figure 4: feature-attribution validation;
37
- - Figure 5: Data Shapley trial valuation, retraining case studies, and the
38
- held-out RNN historical-selection trajectory example.
39
-
40
- Latent display coordinates are exported after the same per-session whitening
41
- and matched-landmark display alignment used for Figure 3. The reported
42
- consistency value is symmetric alignment R²; it does not measure decoder
43
- transfer or identify a unique latent coordinate system.
44
-
45
- Feature and trial values remain signed. Feature-attribution validation is
46
- dataset-specific: Allen Neuropixels uses Spearman correlation with measured
47
- orientation selectivity, while the other primary tasks use their predefined
48
- ROC-AUC validation assays. Trial valuation reports corrupted-trial detection
49
- ROC-AUC using negative signed trial value as the detection score.
50
-
51
- ## Local validation and launch
52
 
53
  ```bash
54
  python -m venv .venv
@@ -56,14 +23,3 @@ python -m venv .venv
56
  .venv/bin/python validate_data.py
57
  .venv/bin/python app.py
58
  ```
59
-
60
- From a checkout nested inside the main benchmark repository, exact equality
61
- with the manuscript-facing tables can also be checked with:
62
-
63
- ```bash
64
- .venv/bin/python validate_data.py --canonical-root ..
65
- ```
66
-
67
- The complete artifacts, dataset preparation, benchmark execution, figure
68
- builders, and result-export workflow live in the main repository. The Space
69
- contains lightweight summaries intended for interactive inspection.
 
7
 
8
  # BEND-BCI Interactive Benchmark
9
 
10
+ BEND-BCI compares 23 neural decoding methods across predictive performance,
11
+ robustness, computational cost and cross-recording latent consistency. This
12
+ Space provides interactive results together with feature-attribution and
13
+ trial-valuation analyses.
14
 
15
+ Code, benchmark outputs and dataset preparation are available in the
16
+ [BEND-BCI repository](https://github.com/TangLab-UBC/behavior_benchmarking).
17
 
18
+ ## Run locally
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
  ```bash
21
  python -m venv .venv
 
23
  .venv/bin/python validate_data.py
24
  .venv/bin/python app.py
25
  ```
 
 
 
 
 
 
 
 
 
 
 
app.py CHANGED
@@ -1,6 +1,5 @@
1
  from __future__ import annotations
2
 
3
- import json
4
  import re
5
  from pathlib import Path
6
  from typing import Iterable, Sequence
@@ -70,18 +69,18 @@ DISPLAY_NAMES = {
70
 
71
  DATASET_LABELS = {
72
  "monkey": "Macaque center-out reaching",
73
- "allen_neuropixels": "Allen Neuropixels visual coding",
74
  "speech": "Attempted speech",
75
  "mc_pacman": "MC PacMan force decoding",
76
  "ratinabox": "RatInABox navigation",
77
  }
78
 
79
- DATASET_SHORT_LABELS = {
80
- "monkey": "Macaque reaching",
81
- "allen_neuropixels": "Allen visual coding",
82
- "speech": "Attempted speech",
83
- "mc_pacman": "MC PacMan",
84
- "ratinabox": "RatInABox",
85
  }
86
 
87
  DATASET_DESCRIPTIONS = {
@@ -226,18 +225,10 @@ RATINABOX_SCALE = [
226
  TABLE_LABELS = {
227
  "method": "Method",
228
  "workflow": "Prediction workflow",
229
- "readout": "Prediction/readout implementation",
230
  "hardware": "Primary hardware",
231
- "prediction_status": "Prediction",
232
- "robustness_status": "Robustness",
233
- "compute_status": "Computational cost",
234
- "consistency_status": "Latent consistency",
235
- "feature_status": "Feature attribution",
236
- "trial_status": "Trial valuation",
237
- "coverage_notes": "Coverage notes",
238
  "task_score": "Held-out task score",
239
  "prediction_percentile": "Within-dataset percentile",
240
- "robustness_auc": "Raw score-vs-noise AUC",
241
  "training_time_sec": "Training time (s)",
242
  "inference_time_sec": "Inference time (s)",
243
  "peak_ram_gb": "Peak RAM (GB)",
@@ -252,28 +243,9 @@ TABLE_LABELS = {
252
  "validation_target": "Validation target",
253
  "validation_metric": "Validation metric",
254
  "validation_score": "Validation score",
255
- "shap_mean_value": "Mean signed Kernel SHAP",
256
- "shap_median_value": "Median signed Kernel SHAP",
257
- "shap_min_value": "Minimum signed Kernel SHAP",
258
- "shap_max_value": "Maximum signed Kernel SHAP",
259
- "shap_fraction_positive": "Fraction positive",
260
- "shap_fraction_negative": "Fraction negative",
261
  "corrupted_trial_auc": "Corrupted-trial ROC-AUC",
262
- "iterations": "TMC permutations",
263
- "converged": "Converged",
264
- "final_error": "Final convergence error",
265
- "perturbation_fraction": "Rotated-trial fraction",
266
- "rotation_angle_deg": "Rotation angle (degrees)",
267
- "rotation_subspace_dim_spec": "Rotation subspace",
268
- "shapley_mean_value": "Mean signed Data Shapley",
269
- "shapley_median_value": "Median signed Data Shapley",
270
- "shapley_min_value": "Minimum signed Data Shapley",
271
- "shapley_max_value": "Maximum signed Data Shapley",
272
- "shapley_fraction_positive": "Fraction positive",
273
- "shapley_fraction_negative": "Fraction negative",
274
  "mixed_full": "Mixed trials",
275
  "data_shapley": "After trial-value removal",
276
- "oracle": "Oracle removal",
277
  "recovery": "Recovery ΔR²",
278
  "target_only": "Current session only",
279
  "all_sessions": "All-session pooling",
@@ -419,17 +391,6 @@ def load_historical_trajectories() -> pd.DataFrame:
419
  return frame.sort_values(["direction_index", "trial_index", "time_index"]).reset_index(drop=True)
420
 
421
 
422
- def load_release_manifest() -> dict:
423
- path = DATA_DIR / "release_manifest.json"
424
- if not path.exists():
425
- return {}
426
- try:
427
- payload = json.loads(path.read_text(encoding="utf-8"))
428
- except (OSError, json.JSONDecodeError):
429
- return {}
430
- return payload if isinstance(payload, dict) else {}
431
-
432
-
433
  prediction = load_csv("clean_prediction_summary.csv")
434
  robustness = load_csv("robustness_summary.csv")
435
  consistency = load_csv("consistency_summary.csv")
@@ -440,7 +401,6 @@ trial_retrain = load_csv("trial_shapley_retrain_summary.csv")
440
  trial_historical_trajectories = load_historical_trajectories()
441
  latent_samples = load_csv("latent_samples.csv")
442
  latent_trajectories = load_csv("latent_trajectories.csv")
443
- release_manifest = load_release_manifest()
444
 
445
 
446
  def present_rows(df: pd.DataFrame) -> pd.DataFrame:
@@ -642,7 +602,7 @@ def metric_card(label: str, value: str, detail: str | None = None, accent: str =
642
  return html.Div(children, className=classes)
643
 
644
 
645
- def source_link(filename: str, label: str = "Download source CSV") -> html.A:
646
  return html.A(
647
  label,
648
  href=f"/download/{filename}",
@@ -713,10 +673,10 @@ def figure_layout(
713
  def heatmap_layout(fig: go.Figure, *, height: int) -> go.Figure:
714
  figure_layout(fig, height=height)
715
  fig.update_layout(
716
- margin=dict(l=54, r=28, t=126, b=58),
717
  title=dict(y=0.985, yanchor="top", pad=dict(b=12)),
718
  )
719
- fig.update_xaxes(tickangle=-28)
720
  return fig
721
 
722
 
@@ -737,98 +697,11 @@ def empty_figure(message: str, *, height: int = 360) -> go.Figure:
737
  return figure_layout(fig, height=height)
738
 
739
 
740
- def availability_note(available: int, configured: int, *, unsupported: int = 0) -> str:
741
- message = f"Coverage: {available} of {configured} configured entries available"
742
- if unsupported:
743
- message += f"; {unsupported} selected method{'s' if unsupported != 1 else ''} outside this analysis"
744
- return message if message.endswith(".") else message + "."
745
-
746
-
747
- def feature_coverage_note(dataset: str, models: Sequence[str] | None) -> str:
748
- chosen = selected_models(models)
749
- configured = [model for model in chosen if model in FEATURE_ELIGIBLE]
750
- available = filter_models(active_rows(neuron_shap), chosen)
751
- available = int(available["dataset"].astype(str).eq(str(dataset)).sum())
752
- message = (
753
- f"Main Figure 4 interface: {available} of {len(configured)} configured entries "
754
- "available"
755
- )
756
- if "marble" in chosen:
757
- message += f"; full attempted grid: {available} of {len(chosen)} available. "
758
- if dataset == "allen_neuropixels":
759
- message += (
760
- "MARBLE exceeded the host-memory allocation before feature attribution."
761
- )
762
- else:
763
- message += (
764
- "MARBLE is outside the masked-input interface because masking changes its "
765
- "transductive graph construction."
766
- )
767
- return message if message.endswith(".") else message + "."
768
-
769
-
770
- def intervention_coverage_note() -> str:
771
- frame = active_rows(trial_retrain)
772
- within = frame[frame["analysis"] == "within_session_cleaning"].pivot_table(
773
- index="model", columns="condition", values="score", aggfunc="first"
774
- )
775
- historical = frame[
776
- frame["analysis"] == "cross_session_old_trial_selection"
777
- ].pivot_table(index="model", columns="condition", values="score", aggfunc="first")
778
- removal_improved = int((within["data_shapley"] > within["mixed_full"]).sum())
779
- historical_vs_current = int(
780
- (historical["oldonly_dshap_negative_removal"] > historical["target_only"]).sum()
781
- )
782
- historical_vs_pooling = int(
783
- (historical["oldonly_dshap_negative_removal"] > historical["all_sessions"]).sum()
784
- )
785
- return (
786
- f"Full coverage: {len(within)} removal and {len(historical)} historical-selection "
787
- f"entries. Removal improved {removal_improved}/{len(within)} methods "
788
- f"(mean R² {within['mixed_full'].mean():.3f}→{within['data_shapley'].mean():.3f}); "
789
- f"historical selection beat current-only training for "
790
- f"{historical_vs_current}/{len(historical)} and pooling for "
791
- f"{historical_vs_pooling}/{len(historical)} (means "
792
- f"{historical['oldonly_dshap_negative_removal'].mean():.3f} vs "
793
- f"{historical['target_only'].mean():.3f}/{historical['all_sessions'].mean():.3f}). "
794
- "The removal comparison reuses the valuation split and is a controlled diagnostic, "
795
- "not an independent generalization estimate."
796
- )
797
-
798
-
799
- def row_exists(df: pd.DataFrame, model: str, dataset: str) -> bool:
800
- if df.empty:
801
- return False
802
- return bool(
803
- ((df["model"].astype(str) == model) & (df["dataset"].astype(str) == dataset)).any()
804
- )
805
-
806
-
807
- def missing_reason(analysis: str, model: str, dataset: str) -> str:
808
- specific = {
809
- ("prediction", "marble", "allen_neuropixels"): "MARBLE graph/embedding construction exceeded the 192-GB host-memory allocation.",
810
- ("prediction", "tndm", "speech"): "TNDM training returned non-finite relevant-prior/posterior KL values.",
811
- ("prediction", "svc", "allen_neuropixels"): "Exact-kernel SVM fitting exceeded the 24-hour allocation.",
812
- ("consistency", "marble", "allen_neuropixels"): "MARBLE graph/embedding construction exceeded the host-memory allocation.",
813
- ("consistency", "tndm", "speech"): "TNDM training returned non-finite KL values before a representation was available.",
814
- ("feature", "mint", "allen_neuropixels"): "The MATLAB-backed repeated masked-input procedure did not complete in the allocation.",
815
- ("feature", "mint", "ratinabox"): "The MATLAB-backed repeated masked-input procedure did not complete in the allocation.",
816
- ("feature", "svc", "allen_neuropixels"): "Exact-kernel SVM fitting exceeded the 24-hour allocation.",
817
- ("feature", "svc", "mc_pacman"): "Repeated masked-input SVM/SVR evaluation exceeded the 24-hour allocation.",
818
- ("feature", "svc", "ratinabox"): "Repeated masked-input SVM/SVR evaluation exceeded the 24-hour allocation.",
819
- ("trial", "langevinflow_ccn", "allen_neuropixels"): "Repeated full-rank logistic coalition fits exceeded the execution allocation.",
820
- ("trial", "marble", "allen_neuropixels"): "No Stage 1 representation was available after MARBLE exceeded host memory.",
821
- ("trial", "tndm", "allen_neuropixels"): "Repeated causal-logistic coalition fits exceeded the execution allocation.",
822
- ("trial", "tndm", "mc_pacman"): "Repeated causal-linear coalition fits exceeded the execution allocation.",
823
- }
824
- return specific.get((analysis, model, dataset), "No completed summary is available in this release.")
825
-
826
-
827
  def prediction_workflow(model: str, decoder: object, status: object) -> str:
828
  if str(status).lower() != "present":
829
  return "Unavailable"
830
  decoder_name = "" if decoder is None or pd.isna(decoder) else str(decoder)
831
- direct_decoders = {
832
  "native",
833
  "dnn",
834
  "gru",
@@ -841,43 +714,18 @@ def prediction_workflow(model: str, decoder: object, status: object) -> str:
841
  "xgboost_classification",
842
  "xgboost_regression",
843
  }
844
- if decoder_name in direct_decoders:
845
- return "Predictions generated directly by trained model"
846
  # Manuscript v7 deliberately distinguishes LDNS task families: continuous
847
  # prediction uses the method recipe's ridge mapping on reconstructed rates
848
  # (alpha = 1e-6), while classification uses the standard logistic readout.
849
  if decoder_name in {"ridge", "logistic", "ldns_rate_sklearn_logistic"}:
850
- return "Shared ridge/logistic readout"
851
  if decoder_name in {"knn", "ole", "ldns_rate_sklearn_ridge"}:
852
- return "Method-specific task mapping"
853
  raise ValueError(f"Unrecognized prediction decoder for {model}: {decoder_name!r}")
854
 
855
 
856
- def readout_label(decoder: object) -> str:
857
- if decoder is None or pd.isna(decoder):
858
- return "Unavailable"
859
- labels = {
860
- "native": "Model-native output",
861
- "ridge": "Ridge regression",
862
- "logistic": "Logistic regression",
863
- "knn": "k-nearest neighbors",
864
- "ole": "Ordinary least squares",
865
- "mint_pipeline": "MINT prediction pipeline",
866
- "neds_e2e": "NEDS end-to-end head",
867
- "dnn": "DNN task head",
868
- "gru": "GRU task head",
869
- "lstm": "LSTM task head",
870
- "rnn": "RNN task head",
871
- "svc": "Support-vector classifier",
872
- "svr": "Support-vector regression",
873
- "xgboost_classification": "XGBoost classifier",
874
- "xgboost_regression": "XGBoost regressor",
875
- "ldns_rate_sklearn_ridge": "Ridge on LDNS reconstructed rates",
876
- "ldns_rate_sklearn_logistic": "Logistic regression on LDNS reconstructed rates",
877
- }
878
- return labels.get(str(decoder), str(decoder).replace("_", " "))
879
-
880
-
881
  def prediction_percentiles() -> pd.DataFrame:
882
  frame = present_rows(prediction)[["model", "dataset", "score"]].copy()
883
  frame["score"] = pd.to_numeric(frame["score"], errors="coerce")
@@ -931,7 +779,6 @@ def overview_frame(dataset: str, models: Sequence[str] | None) -> pd.DataFrame:
931
  lambda row: prediction_workflow(row["model"], row.get("decoder"), row.get("prediction_status_raw")),
932
  axis=1,
933
  )
934
- frame["prediction_status"] = np.where(frame["task_score"].notna(), "Available", "Unavailable")
935
  frame.loc[frame["model"].isin(CPU_ONLY_MODELS), "peak_vram_gb"] = np.nan
936
  return round_numeric(frame)
937
 
@@ -947,7 +794,7 @@ def overview_cards(dataset: str, models: Sequence[str] | None) -> list[html.Div]
947
  cards = [
948
  metric_card(
949
  "Dataset",
950
- DATASET_SHORT_LABELS.get(dataset, dataset),
951
  DATASET_DESCRIPTIONS.get(dataset),
952
  "prediction",
953
  ),
@@ -970,14 +817,6 @@ def overview_cards(dataset: str, models: Sequence[str] | None) -> list[html.Div]
970
  "prediction",
971
  )
972
  )
973
- cards.append(
974
- metric_card(
975
- "Prediction coverage",
976
- f"{len(available)}/{len(frame)}",
977
- "Completed method–dataset entries in the current comparison.",
978
- "prediction",
979
- )
980
- )
981
  return cards
982
 
983
 
@@ -1001,7 +840,7 @@ def prediction_ranking_figure(dataset: str, models: Sequence[str] | None) -> go.
1001
  ),
1002
  )
1003
  )
1004
- fig.update_layout(title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} held-out prediction")
1005
  fig.update_xaxes(title=metric)
1006
  fig.update_yaxes(title="", showgrid=False)
1007
  return figure_layout(fig, height=max(440, 25 * len(frame) + 145))
@@ -1045,7 +884,7 @@ def percentile_heatmap(
1045
  if eligible_models is not None:
1046
  chosen = [model for model in chosen if model in eligible_models]
1047
  if not chosen:
1048
- return empty_figure("None of the selected methods is configured for this analysis.")
1049
  frame = values.copy()
1050
  if frame.empty:
1051
  return empty_figure(empty_message)
@@ -1073,23 +912,22 @@ def percentile_heatmap(
1073
  index=ordered_models, columns=DATASETS
1074
  )
1075
  display_text = np.empty(percentile_matrix.shape, dtype=object)
1076
- customdata = np.empty((*percentile_matrix.shape, 3), dtype=object)
1077
  for row_index, model in enumerate(percentile_matrix.index):
1078
  for column_index, dataset in enumerate(percentile_matrix.columns):
1079
  percentile = percentile_matrix.iloc[row_index, column_index]
1080
  raw_value = raw_matrix.iloc[row_index, column_index]
1081
  metric = metric_matrix.iloc[row_index, column_index]
1082
  available = pd.notna(raw_value)
1083
- display_text[row_index, column_index] = "×" if not available else f"{percentile:.0f}"
1084
  customdata[row_index, column_index, 0] = (
1085
- "Unavailable" if not available else f"{float(raw_value):.4f}"
1086
  )
1087
- customdata[row_index, column_index, 1] = "Unavailable" if pd.isna(metric) else str(metric)
1088
- customdata[row_index, column_index, 2] = "Available" if available else "Unavailable"
1089
  fig = go.Figure(
1090
  go.Heatmap(
1091
  z=percentile_matrix.to_numpy(dtype=float),
1092
- x=[DATASET_SHORT_LABELS[dataset] for dataset in percentile_matrix.columns],
1093
  y=[model_label(model) for model in percentile_matrix.index],
1094
  text=display_text,
1095
  texttemplate="%{text}",
@@ -1102,8 +940,7 @@ def percentile_heatmap(
1102
  hovertemplate=(
1103
  "Method=%{y}<br>Dataset=%{x}<br>"
1104
  "Within-dataset percentile=%{z:.1f}<br>"
1105
- "Raw value=%{customdata[0]}<br>Metric=%{customdata[1]}<br>"
1106
- "Status=%{customdata[2]}<extra></extra>"
1107
  ),
1108
  hoverongaps=False,
1109
  )
@@ -1112,12 +949,12 @@ def percentile_heatmap(
1112
  if len(missing_rows):
1113
  fig.add_trace(
1114
  go.Scatter(
1115
- x=[DATASET_SHORT_LABELS[percentile_matrix.columns[index]] for index in missing_columns],
1116
  y=[model_label(percentile_matrix.index[index]) for index in missing_rows],
1117
  mode="markers",
1118
- marker=dict(symbol="x", size=9, color="#7A858E", line=dict(width=1)),
1119
  showlegend=False,
1120
- hovertemplate="Method=%{y}<br>Dataset=%{x}<br>Status=Unavailable<extra></extra>",
1121
  )
1122
  )
1123
  fig.update_layout(title=title)
@@ -1132,7 +969,7 @@ def prediction_heatmap(models: Sequence[str] | None) -> go.Figure:
1132
  return percentile_heatmap(
1133
  values,
1134
  models,
1135
- title="Prediction across tasks (within-dataset percentiles; raw scores on hover)",
1136
  colorscale=PREDICTION_SCALE,
1137
  raw_column="score",
1138
  metric_column="metric_label",
@@ -1191,29 +1028,18 @@ def robustness_figure(dataset: str, models: Sequence[str] | None) -> go.Figure:
1191
  marker=dict(size=6, symbol=index % 8),
1192
  customdata=np.repeat(row.robustness_auc, len(levels)),
1193
  hovertemplate=(
1194
- f"Method={row.method}<br>Additive count-noise λ=%{{x:.1f}}<br>"
1195
- "Task score=%{y:.4f}<br>Raw score-vs-noise AUC=%{customdata:.4f}<extra></extra>"
1196
  ),
1197
  )
1198
  )
1199
  metric = metric_name(frame["metric"].dropna().iloc[0])
1200
  fig.update_layout(
1201
- title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} robustness to noisy neural inputs",
1202
  hovermode="closest",
1203
  showlegend=len(frame) <= 12,
1204
  )
1205
- if len(frame) > 12:
1206
- fig.add_annotation(
1207
- text="Use the method comparison control to isolate curves; every curve is named on hover.",
1208
- x=0,
1209
- y=1.08,
1210
- xref="paper",
1211
- yref="paper",
1212
- showarrow=False,
1213
- xanchor="left",
1214
- font=dict(size=11, color=MUTED_COLOR),
1215
- )
1216
- fig.update_xaxes(title="Additive Poisson count-noise level λ", tickvals=[0, 0.2, 0.4, 0.6, 0.8])
1217
  fig.update_yaxes(title=metric)
1218
  return figure_layout(fig, height=540, legend_below=len(frame) <= 12)
1219
 
@@ -1261,12 +1087,12 @@ def compute_figures(
1261
  )
1262
  )
1263
  runtime.update_layout(
1264
- title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} training and inference time",
1265
  barmode="group",
1266
  )
1267
  runtime.update_xaxes(title="Elapsed time (seconds, log scale)", type="log")
1268
  runtime.update_yaxes(title="", showgrid=False)
1269
- figure_layout(runtime, height=max(470, 27 * len(frame) + 155))
1270
 
1271
  memory = go.Figure()
1272
  memory.add_trace(
@@ -1290,12 +1116,12 @@ def compute_figures(
1290
  )
1291
  )
1292
  memory.update_layout(
1293
- title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} peak RAM and GPU memory",
1294
  barmode="group",
1295
  )
1296
  memory.update_xaxes(title="Memory (GB)")
1297
  memory.update_yaxes(title="", showgrid=False)
1298
- figure_layout(memory, height=max(470, 27 * len(frame) + 155))
1299
 
1300
  table = frame[
1301
  [
@@ -1350,22 +1176,11 @@ def feature_frame(dataset: str, models: Sequence[str] | None) -> pd.DataFrame:
1350
 
1351
  def feature_figures(
1352
  dataset: str, models: Sequence[str] | None
1353
- ) -> tuple[go.Figure, go.Figure, pd.DataFrame]:
1354
  frame = feature_frame(dataset, models)
1355
  if frame.empty:
1356
- columns = [
1357
- "method",
1358
- "validation_target",
1359
- "validation_metric",
1360
- "validation_score",
1361
- "shap_mean_value",
1362
- "shap_median_value",
1363
- "shap_min_value",
1364
- "shap_max_value",
1365
- "shap_fraction_positive",
1366
- "shap_fraction_negative",
1367
- ]
1368
- return empty_figure("No feature-attribution validation result is available."), empty_figure("No signed Kernel SHAP summary is available."), pd.DataFrame(columns=columns)
1369
  score_column, target, metric, reference = feature_spec(dataset)
1370
  validation = frame.dropna(subset=["validation_score"]).sort_values(
1371
  ["validation_score", "model_order"], ascending=[True, False]
@@ -1388,57 +1203,16 @@ def feature_figures(
1388
  annotation_position="top",
1389
  )
1390
  validation_fig.update_layout(
1391
- title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} feature-attribution validation"
1392
  )
1393
  validation_fig.update_xaxes(title=metric)
1394
  validation_fig.update_yaxes(title="", showgrid=False)
1395
  figure_layout(validation_fig, height=max(430, 25 * len(validation) + 145))
1396
 
1397
- signed = frame.dropna(subset=["shap_mean_value"]).sort_values(
1398
- ["shap_mean_value", "model_order"], ascending=[True, False]
1399
- )
1400
- signed_colors = [FEATURE_COLOR if value >= 0 else NEGATIVE_COLOR for value in signed["shap_mean_value"]]
1401
- signed_fig = go.Figure(
1402
- go.Bar(
1403
- x=signed["shap_mean_value"],
1404
- y=signed["method"],
1405
- orientation="h",
1406
- marker=dict(color=signed_colors),
1407
- customdata=np.stack(
1408
- [
1409
- signed["shap_median_value"],
1410
- signed["shap_min_value"],
1411
- signed["shap_max_value"],
1412
- signed["shap_fraction_positive"],
1413
- ],
1414
- axis=-1,
1415
- ),
1416
- hovertemplate=(
1417
- "Method=%{y}<br>Mean signed value=%{x:.5g}<br>"
1418
- "Median=%{customdata[0]:.5g}<br>Range=[%{customdata[1]:.5g}, %{customdata[2]:.5g}]<br>"
1419
- "Fraction positive=%{customdata[3]:.3f}<extra></extra>"
1420
- ),
1421
- )
1422
  )
1423
- signed_fig.add_vline(x=0, line_color="#6F7882", line_width=1)
1424
- signed_fig.update_layout(title="Signed global Kernel SHAP summary")
1425
- signed_fig.update_xaxes(title="Mean signed contribution to task score")
1426
- signed_fig.update_yaxes(title="", showgrid=False)
1427
- figure_layout(signed_fig, height=max(430, 25 * len(signed) + 145))
1428
-
1429
- columns = [
1430
- "method",
1431
- "validation_target",
1432
- "validation_metric",
1433
- "validation_score",
1434
- "shap_mean_value",
1435
- "shap_median_value",
1436
- "shap_min_value",
1437
- "shap_max_value",
1438
- "shap_fraction_positive",
1439
- "shap_fraction_negative",
1440
- ]
1441
- return validation_fig, signed_fig, round_numeric(frame[columns].sort_values("validation_score", ascending=False))
1442
 
1443
 
1444
  def feature_heatmap(models: Sequence[str] | None) -> go.Figure:
@@ -1460,7 +1234,7 @@ def feature_heatmap(models: Sequence[str] | None) -> go.Figure:
1460
  return percentile_heatmap(
1461
  values,
1462
  models,
1463
- title="Feature-attribution validation across tasks (within-dataset percentiles)",
1464
  colorscale=FEATURE_SCALE,
1465
  raw_column="validation_score",
1466
  metric_column="metric_label",
@@ -1478,9 +1252,6 @@ def trial_frame(dataset: str, models: Sequence[str] | None) -> pd.DataFrame:
1478
  frame = add_method_columns(frame)
1479
  frame["method"] = frame["model"].map(lambda model: dataset_model_label(model, dataset))
1480
  frame = frame.rename(columns={"perturbation_auc": "corrupted_trial_auc"})
1481
- frame["converged"] = frame["converged"].fillna(False).astype(str).str.lower().map(
1482
- {"true": "Yes", "1": "Yes", "false": "No", "0": "No"}
1483
- ).fillna("No")
1484
  return round_numeric(frame)
1485
 
1486
 
@@ -1491,29 +1262,13 @@ def trial_detection_figure(dataset: str, models: Sequence[str] | None) -> go.Fig
1491
  frame = frame.dropna(subset=["corrupted_trial_auc"]).sort_values(
1492
  ["corrupted_trial_auc", "model_order"], ascending=[True, False]
1493
  )
1494
- patterns = ["" if value == "Yes" else "/" for value in frame["converged"]]
1495
  fig = go.Figure(
1496
  go.Bar(
1497
  x=frame["corrupted_trial_auc"],
1498
  y=frame["method"],
1499
  orientation="h",
1500
- marker=dict(color=TRIAL_COLOR, pattern=dict(shape=patterns)),
1501
- customdata=np.stack(
1502
- [
1503
- frame["converged"],
1504
- frame["iterations"],
1505
- frame["final_error"],
1506
- frame["shapley_mean_value"],
1507
- frame["shapley_fraction_positive"],
1508
- ],
1509
- axis=-1,
1510
- ),
1511
- hovertemplate=(
1512
- "Method=%{y}<br>Corrupted-trial ROC-AUC=%{x:.4f}<br>"
1513
- "Converged=%{customdata[0]}<br>Permutations=%{customdata[1]:.0f}<br>"
1514
- "Final error=%{customdata[2]:.4f}<br>Mean signed trial value=%{customdata[3]:.5g}<br>"
1515
- "Fraction positive=%{customdata[4]:.3f}<extra></extra>"
1516
- ),
1517
  )
1518
  )
1519
  fig.add_vline(
@@ -1524,7 +1279,7 @@ def trial_detection_figure(dataset: str, models: Sequence[str] | None) -> go.Fig
1524
  annotation_position="top",
1525
  )
1526
  fig.update_layout(
1527
- title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} corrupted-trial detection"
1528
  )
1529
  fig.update_xaxes(title="ROC-AUC from negative trial value")
1530
  fig.update_yaxes(title="", showgrid=False)
@@ -1537,7 +1292,7 @@ def trial_heatmap(models: Sequence[str] | None) -> go.Figure:
1537
  return percentile_heatmap(
1538
  values,
1539
  models,
1540
- title="Corrupted-trial detection across tasks (within-dataset percentiles)",
1541
  colorscale=TRIAL_SCALE,
1542
  raw_column="perturbation_auc",
1543
  metric_column="metric_label",
@@ -1579,10 +1334,8 @@ def equality_bounds(*series: pd.Series) -> tuple[float, float]:
1579
  return float(values.min() - padding), float(values.max() + padding)
1580
 
1581
 
1582
- def trial_retrain_figures(
1583
- models: Sequence[str] | None,
1584
- ) -> tuple[go.Figure, go.Figure, go.Figure, pd.DataFrame]:
1585
- within, historical = retrain_frames(models)
1586
  if within.empty:
1587
  removal = empty_figure("No macaque within-session removal summary is available.")
1588
  relation = empty_figure("No detection-versus-recovery summary is available.")
@@ -1600,11 +1353,11 @@ def trial_retrain_figures(
1600
  size=10,
1601
  line=dict(color="#FFFFFF", width=1),
1602
  ),
1603
- customdata=np.stack([within["method"], within["oracle"], within["recovery"]], axis=-1),
1604
  hovertemplate=(
1605
  "Method=%{customdata[0]}<br>Mixed trials R²=%{x:.4f}<br>"
1606
- "After trial-value removal R²=%{y:.4f}<br>Oracle removal R²=%{customdata[1]:.4f}<br>"
1607
- "Recovery ΔR²=%{customdata[2]:+.4f}<extra></extra>"
1608
  ),
1609
  )
1610
  )
@@ -1626,9 +1379,9 @@ def trial_retrain_figures(
1626
  bgcolor="rgba(255,255,255,0.88)",
1627
  font=dict(size=12, color="#2E7D32" if removal_change >= 0 else NEGATIVE_COLOR),
1628
  )
1629
- removal.update_layout(title="Macaque trial-value-guided removal")
1630
- removal.update_xaxes(title="Before filtering: mixed-trial test R²", range=[lower, upper])
1631
- removal.update_yaxes(title="After negative-value removal: test R²", range=[lower, upper])
1632
  figure_layout(removal, height=480)
1633
 
1634
  detection = active_rows(trial_shapley)
@@ -1683,7 +1436,7 @@ def trial_retrain_figures(
1683
  if pd.notna(rho)
1684
  else f"n = {len(relation_frame)}; select at least two methods for correlation"
1685
  )
1686
- if not models and pd.notna(rho):
1687
  relation_text += "<br>one-sided permutation P = 0.035"
1688
  relation.add_annotation(
1689
  text=relation_text,
@@ -1697,9 +1450,9 @@ def trial_retrain_figures(
1697
  bgcolor="rgba(255,255,255,0.85)",
1698
  font=dict(size=12),
1699
  )
1700
- relation.update_layout(title="Detection signal versus recovery after removal", showlegend=False)
1701
- relation.update_xaxes(title="Corrupted-trial detection ROC-AUC")
1702
- relation.update_yaxes(title="Recovery after removal (ΔR²)")
1703
  figure_layout(relation, height=480)
1704
 
1705
  if historical.empty:
@@ -1742,17 +1495,17 @@ def trial_retrain_figures(
1742
  bgcolor="rgba(255,255,255,0.88)",
1743
  font=dict(size=12, color="#2E7D32" if historical_change >= 0 else NEGATIVE_COLOR),
1744
  )
1745
- historical_fig.update_layout(title="Same-subject historical-trial selection")
1746
- historical_fig.update_xaxes(title="Current-session trials only: test R²", range=[lower, upper])
1747
  historical_fig.update_yaxes(
1748
- title="Add nonnegative-valued historical trials: test R²",
1749
  range=[lower, upper],
1750
  )
1751
  figure_layout(historical_fig, height=480)
1752
 
1753
  table = within[
1754
- [column for column in ["model", "method", "mixed_full", "data_shapley", "oracle", "recovery"] if column in within]
1755
- ].copy() if not within.empty else pd.DataFrame(columns=["model", "method", "mixed_full", "data_shapley", "oracle", "recovery"])
1756
  historical_columns = ["model", "target_only", "all_sessions", "historical_selected"]
1757
  if not historical.empty:
1758
  table = table.merge(historical[historical_columns], on="model", how="outer")
@@ -1761,14 +1514,7 @@ def trial_retrain_figures(
1761
  return removal, relation, historical_fig, round_numeric(table)
1762
 
1763
 
1764
- def historical_trajectory_figure(models: Sequence[str] | None) -> go.Figure:
1765
- if models and "rnn" not in set(models):
1766
- return empty_figure(
1767
- "The Figure 5e held-out trajectory example uses RNN. "
1768
- "Include RNN in Compare methods or clear the method filter.",
1769
- height=430,
1770
- )
1771
-
1772
  frame = trial_historical_trajectories.copy()
1773
  current_r2 = float(frame["current_only_r2"].iloc[0])
1774
  historical_r2 = float(frame["historical_selected_r2"].iloc[0])
@@ -1784,7 +1530,7 @@ def historical_trajectory_figure(models: Sequence[str] | None) -> go.Figure:
1784
  (
1785
  "historical_selected_x",
1786
  "historical_selected_y",
1787
- f"Current + nonnegative-valued historical trials<br>R² = {historical_r2:.2f}",
1788
  0.64,
1789
  2.5,
1790
  ),
@@ -1901,7 +1647,7 @@ def historical_trajectory_figure(models: Sequence[str] | None) -> go.Figure:
1901
 
1902
  figure_layout(fig, height=510, legend_below=True)
1903
  fig.update_layout(
1904
- title="Held-out target-session trajectories (RNN)",
1905
  margin=dict(l=28, r=28, t=76, b=118),
1906
  legend=dict(
1907
  orientation="h",
@@ -1988,6 +1734,11 @@ def add_latent_color_columns(df: pd.DataFrame, dataset: str, color_mode: str) ->
1988
 
1989
 
1990
  def latent_space_figure(dataset: str, model: str | None, color_mode: str) -> go.Figure:
 
 
 
 
 
1991
  if not model:
1992
  return empty_figure("Select an available method to view aligned coordinates.", height=500)
1993
  samples = latent_samples[
@@ -1995,7 +1746,7 @@ def latent_space_figure(dataset: str, model: str | None, color_mode: str) -> go.
1995
  & (latent_samples["model"].astype(str) == str(model))
1996
  ].copy()
1997
  if samples.empty:
1998
- return empty_figure("Figure 3-aligned display coordinates are unavailable for this entry.", height=500)
1999
  for column in ["x", "y", "z"]:
2000
  samples[column] = pd.to_numeric(samples[column], errors="coerce")
2001
  samples = add_latent_color_columns(samples, dataset, color_mode).dropna(subset=["x", "y", "z"])
@@ -2140,9 +1891,9 @@ def latent_space_figure(dataset: str, model: str | None, color_mode: str) -> go.
2140
  fig.update_layout(
2141
  **{
2142
  scene_id: dict(
2143
- xaxis=dict(title="Aligned dim. 1", range=[-extent, extent], showgrid=False, zeroline=False, showticklabels=False),
2144
- yaxis=dict(title="Aligned dim. 2", range=[-extent, extent], showgrid=False, zeroline=False, showticklabels=False),
2145
- zaxis=dict(title="Aligned dim. 3", range=[-extent, extent], showgrid=False, zeroline=False, showticklabels=False),
2146
  aspectmode="cube",
2147
  bgcolor="#FFFFFF",
2148
  camera=dict(eye=dict(x=1.5, y=1.4, z=1.0)),
@@ -2155,15 +1906,25 @@ def latent_space_figure(dataset: str, model: str | None, color_mode: str) -> go.
2155
  & (score_rows["model"].astype(str) == str(model))
2156
  ]
2157
  score = pd.to_numeric(score_rows.get("mean_r2"), errors="coerce").dropna()
2158
- suffix = "" if score.empty else f" · latent-consistency R² = {float(score.iloc[0]):.3f}"
 
 
2159
  fig.update_layout(
2160
- title=f"{model_label(model)} Figure 3-aligned representations{suffix}",
2161
- height=760 if rows > 1 else 540,
2162
  paper_bgcolor="#FFFFFF",
2163
  plot_bgcolor="#FFFFFF",
2164
- margin=dict(l=8, r=8, t=82, b=105),
2165
  font=dict(family="Arial, Helvetica, sans-serif", size=12, color=TEXT_COLOR),
2166
- legend=dict(orientation="h", yanchor="top", y=-0.07, xanchor="left", x=0, title=condition_name),
 
 
 
 
 
 
 
 
2167
  )
2168
  fig.for_each_annotation(lambda annotation: annotation.update(font=dict(size=12, color="#526171")))
2169
  return fig
@@ -2191,7 +1952,7 @@ def consistency_figures(
2191
  frame = consistency_frame(dataset, models)
2192
  if frame.empty:
2193
  message = (
2194
- "Cross-recording latent consistency is not defined for MC PacMan."
2195
  if dataset == "mc_pacman"
2196
  else "No latent-consistency result is available for this selection."
2197
  )
@@ -2212,8 +1973,8 @@ def consistency_figures(
2212
  ),
2213
  )
2214
  )
2215
- bar_fig.update_layout(title=f"{DATASET_SHORT_LABELS.get(dataset, dataset)} latent consistency")
2216
- bar_fig.update_xaxes(title="Symmetric linear-alignment R²", range=[0, 1.02])
2217
  bar_fig.update_yaxes(title="", showgrid=False)
2218
  figure_layout(bar_fig, height=max(400, 27 * len(bar) + 145))
2219
  columns = ["method", "latent_consistency_r2", "n_recordings", "latent_dim", "n_pairwise"]
@@ -2235,11 +1996,11 @@ def consistency_heatmap(models: Sequence[str] | None) -> go.Figure:
2235
  for row in range(pivot.shape[0]):
2236
  for column in range(pivot.shape[1]):
2237
  value = pivot.iloc[row, column]
2238
- text[row, column] = "×" if pd.isna(value) else f"{value:.2f}"
2239
  fig = go.Figure(
2240
  go.Heatmap(
2241
  z=pivot.to_numpy(dtype=float),
2242
- x=[DATASET_SHORT_LABELS[dataset] for dataset in pivot.columns],
2243
  y=[model_label(model) for model in pivot.index],
2244
  text=text,
2245
  texttemplate="%{text}",
@@ -2255,149 +2016,20 @@ def consistency_heatmap(models: Sequence[str] | None) -> go.Figure:
2255
  if len(missing_rows):
2256
  fig.add_trace(
2257
  go.Scatter(
2258
- x=[DATASET_SHORT_LABELS[pivot.columns[index]] for index in missing_columns],
2259
  y=[model_label(pivot.index[index]) for index in missing_rows],
2260
  mode="markers",
2261
- marker=dict(symbol="x", size=9, color="#7A858E", line=dict(width=1)),
2262
  showlegend=False,
2263
- hovertemplate="Method=%{y}<br>Dataset=%{x}<br>Status=Unavailable<extra></extra>",
2264
  )
2265
  )
2266
- fig.update_layout(title="Cross-recording latent consistency across tasks")
2267
  fig.update_xaxes(title="", side="top", showgrid=False)
2268
  fig.update_yaxes(title="", showgrid=False)
2269
  return heatmap_layout(fig, height=max(470, 27 * len(pivot) + 180))
2270
 
2271
 
2272
- def analysis_status(model: str, dataset: str, analysis: str) -> tuple[str, str]:
2273
- if analysis == "prediction":
2274
- available = row_exists(present_rows(prediction), model, dataset)
2275
- return ("Available", "") if available else ("Unavailable", missing_reason(analysis, model, dataset))
2276
- if analysis == "robustness":
2277
- available = row_exists(present_rows(robustness), model, dataset)
2278
- return ("Available", "") if available else ("Unavailable", missing_reason("prediction", model, dataset))
2279
- if analysis == "compute":
2280
- available = row_exists(present_rows(scalability), model, dataset)
2281
- return ("Available", "") if available else ("Unavailable", missing_reason("prediction", model, dataset))
2282
- if analysis == "consistency":
2283
- if dataset == "mc_pacman":
2284
- return "Not defined", "MC PacMan has no cross-recording consistency cohort."
2285
- if model not in CONSISTENCY_ELIGIBLE:
2286
- return "Not supported", "No predefined common three-dimensional representation for this analysis."
2287
- available = row_exists(active_rows(consistency), model, dataset)
2288
- return ("Available", "") if available else ("Unavailable", missing_reason(analysis, model, dataset))
2289
- if analysis == "feature":
2290
- if model == "marble":
2291
- if dataset == "allen_neuropixels":
2292
- return (
2293
- "Unavailable",
2294
- "MARBLE exceeded the host-memory allocation before feature attribution.",
2295
- )
2296
- return (
2297
- "Not supported",
2298
- "Masked-input evaluation changes MARBLE's transductive graph construction.",
2299
- )
2300
- if model not in FEATURE_ELIGIBLE:
2301
- return "Not supported", "No predefined feature-attribution interface."
2302
- available = row_exists(active_rows(neuron_shap), model, dataset)
2303
- return ("Available", "") if available else ("Unavailable", missing_reason(analysis, model, dataset))
2304
- if analysis == "trial":
2305
- if model not in TRIAL_ELIGIBLE:
2306
- return "Not supported", "The fixed-output/lightweight-rescoring Data Shapley rule is unavailable for this method."
2307
- available = row_exists(active_rows(trial_shapley), model, dataset)
2308
- return ("Available", "") if available else ("Unavailable", missing_reason(analysis, model, dataset))
2309
- raise ValueError(f"Unknown analysis: {analysis}")
2310
-
2311
-
2312
- def methods_frame(dataset: str, models: Sequence[str] | None) -> pd.DataFrame:
2313
- rows = []
2314
- for model in selected_models(models):
2315
- pred_rows = prediction[
2316
- (prediction["model"].astype(str) == model)
2317
- & (prediction["dataset"].astype(str) == str(dataset))
2318
- ]
2319
- pred = pred_rows.iloc[0] if not pred_rows.empty else pd.Series(dtype=object)
2320
- statuses = {}
2321
- notes = []
2322
- for analysis in ["prediction", "robustness", "compute", "consistency", "feature", "trial"]:
2323
- status, note = analysis_status(model, dataset, analysis)
2324
- statuses[analysis] = status
2325
- if note and status != "Available":
2326
- notes.append(f"{TABLE_LABELS.get(analysis + '_status', analysis.title())}: {note}")
2327
- rows.append(
2328
- {
2329
- "method": model_label(model),
2330
- "workflow": prediction_workflow(model, pred.get("decoder"), pred.get("status")),
2331
- "readout": readout_label(pred.get("decoder")),
2332
- "hardware": "CPU" if model in CPU_ONLY_MODELS else "GPU",
2333
- "prediction_status": statuses["prediction"],
2334
- "robustness_status": statuses["robustness"],
2335
- "compute_status": statuses["compute"],
2336
- "consistency_status": statuses["consistency"],
2337
- "feature_status": statuses["feature"],
2338
- "trial_status": statuses["trial"],
2339
- "coverage_notes": " ".join(notes) if notes else "Complete for configured analyses.",
2340
- "model_order": MODEL_INDEX[model],
2341
- }
2342
- )
2343
- return pd.DataFrame(rows).sort_values("model_order").drop(columns="model_order")
2344
-
2345
-
2346
- def coverage_cards(dataset: str, models: Sequence[str] | None) -> list[html.Div]:
2347
- chosen = selected_models(models)
2348
- cards = []
2349
- specs = [
2350
- ("Prediction", "prediction", set(MODELS), "prediction"),
2351
- ("Robustness", "robustness", set(MODELS), "robustness"),
2352
- ("Computational cost", "compute", set(MODELS), "compute"),
2353
- ("Latent consistency", "consistency", CONSISTENCY_ELIGIBLE, "consistency"),
2354
- ("Feature attribution", "feature", FEATURE_ELIGIBLE, "feature"),
2355
- ("Trial valuation", "trial", TRIAL_ELIGIBLE, "trial"),
2356
- ]
2357
- accents = {
2358
- "prediction": "prediction",
2359
- "robustness": "robustness",
2360
- "compute": "compute",
2361
- "consistency": "consistency",
2362
- "feature": "feature",
2363
- "trial": "trial",
2364
- }
2365
- for label, analysis, eligible, accent in specs:
2366
- if analysis == "consistency" and dataset == "mc_pacman":
2367
- cards.append(metric_card(label, "Not defined", "No cross-recording cohort.", accents[accent]))
2368
- continue
2369
- configured = [model for model in chosen if model in eligible]
2370
- available = sum(analysis_status(model, dataset, analysis)[0] == "Available" for model in configured)
2371
- unsupported = len(chosen) - len(configured)
2372
- detail = f"{len(configured)} configured"
2373
- if unsupported:
2374
- detail += f"; {unsupported} outside analysis"
2375
- cards.append(metric_card(label, f"{available}/{len(configured)}", detail, accents[accent]))
2376
- return cards
2377
-
2378
-
2379
- def release_detail() -> str:
2380
- stable_parts = []
2381
- manuscript_version = release_manifest.get("manuscript_working_version")
2382
- figure_set = release_manifest.get("figure_set")
2383
- if manuscript_version:
2384
- version_label = str(manuscript_version).replace("_", " ")
2385
- stable_parts.append(
2386
- version_label
2387
- if version_label.lower().startswith("manuscript ")
2388
- else f"manuscript {version_label}"
2389
- )
2390
- if figure_set:
2391
- stable_parts.append(str(figure_set))
2392
- if stable_parts:
2393
- return " Release alignment: " + "; ".join(stable_parts) + "."
2394
- for key in ["generated_at", "release_date", "created_at", "timestamp"]:
2395
- value = release_manifest.get(key)
2396
- if value:
2397
- return f" Release manifest: {value}."
2398
- return ""
2399
-
2400
-
2401
  app = Dash(__name__, title="BEND-BCI Interactive Benchmark")
2402
  server = app.server
2403
 
@@ -2415,39 +2047,41 @@ app.layout = html.Div(
2415
  [
2416
  html.Div(
2417
  [
2418
- html.Div("Tang Lab · Interactive companion to Figures 2–5", className="eyebrow"),
2419
- html.H1("BEND-BCI Interactive Benchmark"),
2420
- html.P(
2421
- "Compare held-out task prediction, robustness to noisy neural inputs, computational cost and cross-recording latent consistency across 23 neural decoding methods. Feature attribution and trial valuation are reported as separate validation-checked diagnostics.",
2422
- className="lede",
2423
- ),
2424
  html.Nav(
2425
  [
2426
  html.A(
2427
- "Benchmark repository",
2428
  href="https://github.com/TangLab-UBC/behavior_benchmarking",
2429
  target="_blank",
2430
  rel="noopener noreferrer",
2431
  ),
2432
- html.A(
2433
- "Hugging Face Space",
2434
- href="https://huggingface.co/spaces/Tang-Lab/benchdash",
2435
- target="_blank",
2436
- rel="noopener noreferrer",
 
 
 
 
2437
  ),
2438
  ],
2439
  className="hero-links",
2440
- **{"aria-label": "Project links"},
2441
  ),
2442
  ],
2443
- className="hero-copy",
2444
  ),
2445
  html.Div(
2446
  [
2447
- html.Span("Six reported views", className="hero-stat-value"),
2448
- html.Span("Four selection measurements + two diagnostics", className="hero-stat-label"),
 
 
 
2449
  ],
2450
- className="hero-stat",
2451
  ),
2452
  ],
2453
  className="hero",
@@ -2469,26 +2103,26 @@ app.layout = html.Div(
2469
  ),
2470
  html.Div(
2471
  [
2472
- html.Label("Compare methods", htmlFor="method-filter"),
2473
  dcc.Dropdown(
2474
  id="method-filter",
2475
  options=[{"label": model_label(model), "value": model} for model in FIGURE_MODEL_ORDER],
2476
  value=[],
2477
  multi=True,
2478
- placeholder="All 23 methods",
2479
  ),
2480
- html.Div("Leave empty to show all methods.", className="control-help"),
2481
  ],
2482
  className="control method-control",
2483
  ),
2484
  ],
2485
  className="toolbar",
2486
  ),
2487
- dcc.Tabs(
2488
- id="tabs",
2489
- value="overview",
2490
- className="tabs",
2491
- children=[
 
2492
  dcc.Tab(
2493
  label="Overview",
2494
  value="overview",
@@ -2496,35 +2130,34 @@ app.layout = html.Div(
2496
  selected_className="tab tab-selected",
2497
  children=[
2498
  panel(
2499
- "Held-out task prediction",
2500
  html.Div(id="overview-cards", className="metric-strip"),
2501
- html.Div(id="overview-coverage", className="coverage-note"),
2502
  html.Div(
2503
  [
2504
  graph_box("prediction-ranking", "Raw held-out prediction scores for the selected dataset."),
2505
- graph_box("prediction-heatmap", "Within-dataset prediction percentiles across five tasks."),
 
 
 
 
2506
  ],
2507
  className="chart-grid two",
2508
  ),
2509
- details_table("View and filter prediction rows", dataframe_table("overview-table", page_size=23)),
2510
  source_link("clean_prediction_summary.csv"),
2511
- eyebrow="Figure 2 · Prediction",
2512
- subtitle="Raw accuracy and R² remain visible within each task. The cross-task matrix uses within-dataset percentiles, with the raw task-specific value and metric on hover.",
2513
  class_name="axis-prediction",
2514
  ),
2515
  panel(
2516
- "Robustness to noisy neural inputs",
2517
- html.Div(id="robustness-coverage", className="coverage-note"),
2518
- graph_box("robustness-curve", "Task score as additive Poisson count-noise level increases."),
2519
- details_table("View and filter robustness rows", dataframe_table("robustness-table", page_size=23)),
2520
  source_link("robustness_summary.csv"),
2521
- eyebrow="Figure 2 · Robustness",
2522
- subtitle="The trained model and targets are held fixed while additive Poisson count noise is applied to test inputs at λ = 0, 0.2, 0.4, 0.6 and 0.8. The reported summary is the raw task-score-versus-noise area; it is unbounded, may be negative and is not normalized to λ = 0.",
2523
  class_name="axis-robustness",
2524
  ),
2525
  panel(
2526
  "Computational cost",
2527
- html.Div(id="compute-coverage", className="coverage-note"),
2528
  html.Div(
2529
  [
2530
  graph_box("runtime-bars", "Training and complete-held-out-split inference times."),
@@ -2532,10 +2165,9 @@ app.layout = html.Div(
2532
  ],
2533
  className="chart-grid two",
2534
  ),
2535
- details_table("View and filter computational-cost rows", dataframe_table("compute-table", page_size=23)),
2536
  source_link("scalability_summary.csv"),
2537
- eyebrow="Figure 2 · Supplementary Figure 11",
2538
- subtitle="Figure 2 reports the macaque-reaching resource block; the other selected datasets are per-dataset extensions reported in Supplementary Figure 11. Training time includes required training-side representation extraction and readout fitting. Inference time is one complete pass over the held-out split, including readout prediction and excluding metric calculation. CPU-only methods have no GPU-memory value.",
2539
  class_name="axis-compute",
2540
  ),
2541
  ],
@@ -2547,13 +2179,12 @@ app.layout = html.Div(
2547
  selected_className="tab tab-selected",
2548
  children=[
2549
  panel(
2550
- "Cross-recording latent consistency",
2551
- html.Div(id="consistency-coverage", className="coverage-note"),
2552
  html.Div(
2553
  [
2554
  html.Div(
2555
  [
2556
- html.Label("Representation", htmlFor="consistency-method"),
2557
  dcc.Dropdown(id="consistency-method", clearable=False),
2558
  ],
2559
  className="control",
@@ -2569,25 +2200,21 @@ app.layout = html.Div(
2569
  ],
2570
  className="inline-controls",
2571
  ),
2572
- html.Div(
2573
- [
2574
- html.Strong("Display-coordinate note: "),
2575
- "Representations are centered and whitened within recording and transformed into the Figure 3 display frame. The displayed coordinates illustrate matched structure; the reported symmetric R² is computed from bidirectional intercept-free linear alignment of task landmarks.",
2576
- ],
2577
- className="method-note",
2578
- ),
2579
- graph_box("latent-space", "Figure 3-aligned latent representations for each recording.", class_name="latent-graph"),
2580
  html.Div(
2581
  [
2582
  graph_box("consistency-bars", "Latent-consistency R-squared for the selected dataset."),
2583
- graph_box("consistency-heatmap", "Latent-consistency R-squared across four tasks."),
 
 
 
 
2584
  ],
2585
  className="chart-grid two",
2586
  ),
2587
- details_table("View and filter latent-consistency rows", dataframe_table("consistency-table", page_size=12)),
2588
  source_link("consistency_summary.csv"),
2589
- eyebrow="Figure 3",
2590
- subtitle="Latent consistency asks whether matched task-defined landmarks are linearly alignable across sessions, participants or independent simulations. It does not measure decoder transfer or identify a unique latent coordinate system.",
2591
  class_name="axis-consistency",
2592
  )
2593
  ],
@@ -2600,27 +2227,24 @@ app.layout = html.Div(
2600
  children=[
2601
  panel(
2602
  "Feature-attribution validation",
2603
- html.Div(id="feature-coverage", className="coverage-note"),
2604
  html.Div(
2605
  [
2606
  graph_box("feature-validation-bars", "Feature-attribution validation metric for the selected dataset."),
2607
- graph_box("feature-signed-bars", "Mean signed global Kernel SHAP values for the selected dataset."),
 
 
 
 
2608
  ],
2609
  className="chart-grid two",
2610
  ),
2611
- graph_box("feature-heatmap", "Within-dataset feature-attribution validation percentiles across tasks."),
2612
  html.Div(
2613
- [
2614
- html.Strong("Metric interpretation: "),
2615
- html.Span(id="feature-definition"),
2616
- " Validation scores and signed contribution summaries are distinct quantities. Signed values are preserved throughout; no absolute-value ranking is applied.",
2617
- ],
2618
  className="method-note",
2619
  ),
2620
- details_table("View validation and signed Kernel SHAP summaries", dataframe_table("feature-table", page_size=23)),
2621
  source_link("neuron_shap_summary.csv"),
2622
- eyebrow="Figure 4",
2623
- subtitle="Global Kernel SHAP estimates each neural feature’s signed contribution to the task score. Dataset-specific validation tests agreement with a constructed control, known simulated feature class or independently measured biological tuning proxy.",
2624
  class_name="axis-feature",
2625
  )
2626
  ],
@@ -2632,32 +2256,31 @@ app.layout = html.Div(
2632
  selected_className="tab tab-selected",
2633
  children=[
2634
  panel(
2635
- "Controlled corrupted-trial detection",
2636
- html.Div(id="trial-coverage", className="coverage-note"),
2637
- html.Div(id="trial-convergence", className="convergence-note"),
2638
  html.Div(
2639
  [
2640
  graph_box("trial-detection-bars", "Corrupted-trial detection ROC-AUC for the selected dataset."),
2641
- graph_box("trial-heatmap", "Within-dataset corrupted-trial detection percentiles across tasks."),
 
 
 
 
2642
  ],
2643
  className="chart-grid two",
2644
  ),
2645
  html.Div(
2646
  [
2647
- html.Strong("Controlled assay: "),
2648
- "Approximately one third of training trials were rotated by 75° in the full population-activity space while targets were unchanged. ROC-AUC uses negative trial value as the corruption-detection score. Hatched bars did not meet the TMC convergence threshold; they remain visible and are not counted as missing.",
2649
  ],
2650
  className="method-note",
2651
  ),
2652
- details_table("View detection, convergence and signed Data Shapley summaries", dataframe_table("trial-table", page_size=23)),
2653
  source_link("trial_shapley_summary.csv"),
2654
- eyebrow="Figure 5a",
2655
- subtitle="Data Shapley assigns each candidate training trial a signed marginal contribution to a specified held-out decoding utility. Values are model-, readout-, split- and metric-specific.",
2656
  class_name="axis-trial",
2657
  ),
2658
  panel(
2659
- "Macaque intervention case studies",
2660
- html.Div(intervention_coverage_note(), className="method-note caveat-note"),
2661
  html.Div(
2662
  [
2663
  graph_box("trial-removal", "Held-out R-squared before and after trial-value-guided corrupted-trial removal."),
@@ -2671,69 +2294,32 @@ app.layout = html.Div(
2671
  "Held-out RNN target-session trajectories for ground truth, current-session training, and nonnegative-valued historical-trial selection.",
2672
  class_name="historical-trajectory-graph",
2673
  ),
2674
- details_table("View within-session removal and historical-selection rows", dataframe_table("trial-retrain-table", page_size=17)),
2675
  html.Div(
2676
  [
2677
- source_link("trial_shapley_retrain_summary.csv", "Intervention summary CSV"),
2678
- source_link("trial_historical_trajectories.csv", "Figure 5e trajectory CSV"),
2679
  ],
2680
  className="download-grid panel-downloads",
2681
  ),
2682
- eyebrow="Figure 5b–e",
2683
- subtitle="The within-session case study removes negative-valued corrupted trials and retrains each decoder. The historical-selection case study retains current-session trials and adds nonnegative-valued trials from earlier sessions in a shared M1 feature space.",
2684
  class_name="axis-trial",
2685
  ),
2686
  ],
2687
  ),
2688
- dcc.Tab(
2689
- label="Methods & coverage",
2690
- value="methods",
2691
- className="tab",
2692
- selected_className="tab tab-selected",
2693
- children=[
2694
- panel(
2695
- "Prediction workflows and analysis coverage",
2696
- html.Div(id="coverage-cards", className="metric-strip coverage-cards"),
2697
- html.Div(
2698
- [
2699
- html.Strong("Prediction workflows: "),
2700
- "Each method–dataset cell follows one of three paper-defined paths: predictions generated directly by the trained model, a method-specific mapping from learned outputs to benchmark targets, or a shared ridge/logistic readout on fixed model outputs. The exact path and implementation can vary by dataset.",
2701
- ],
2702
- className="method-note",
2703
- ),
2704
- dataframe_table("methods-table", page_size=23, max_height="760px"),
2705
- html.Div(
2706
- [
2707
- html.Strong("Coverage terminology: "),
2708
- "Available means a completed summary is bundled. Unavailable means a configured analysis did not produce a completed result. Not supported means the method was outside a predefined analysis interface. Not defined means the dataset has no corresponding analysis cohort.",
2709
- ],
2710
- className="method-note",
2711
- ),
2712
- html.Div(
2713
- [
2714
- source_link("clean_prediction_summary.csv", "Prediction CSV"),
2715
- source_link("robustness_summary.csv", "Robustness CSV"),
2716
- source_link("consistency_summary.csv", "Consistency CSV"),
2717
- source_link("scalability_summary.csv", "Cost CSV"),
2718
- source_link("neuron_shap_summary.csv", "Feature-attribution CSV"),
2719
- source_link("trial_shapley_summary.csv", "Trial-valuation CSV"),
2720
- source_link("trial_shapley_retrain_summary.csv", "Intervention CSV"),
2721
- ],
2722
- className="download-grid",
2723
- ),
2724
- eyebrow="Methods and provenance",
2725
- subtitle="Coverage is shown per method and selected dataset. Unsupported and failed entries remain explicit so chart denominators can be interpreted.",
2726
- class_name="axis-methods",
2727
- )
2728
- ],
2729
- ),
2730
- ],
2731
  ),
2732
  html.Footer(
2733
  [
2734
- html.Strong("Data provenance. "),
2735
- "Dashboard tables are synchronized from the paper/results exports used to generate the current Figures 2–5 and Supplementary Figures. Latent coordinates are Figure-3-aligned display coordinates exported from the same analysis artifacts; display transforms do not change the reported consistency scores.",
2736
- release_detail(),
 
 
 
 
2737
  ],
2738
  className="provenance-footer",
2739
  ),
@@ -2744,16 +2330,13 @@ app.layout = html.Div(
2744
 
2745
  @app.callback(
2746
  Output("overview-cards", "children"),
2747
- Output("overview-coverage", "children"),
2748
  Output("overview-table", "columns"),
2749
  Output("overview-table", "data"),
2750
  Output("prediction-ranking", "figure"),
2751
  Output("prediction-heatmap", "figure"),
2752
- Output("robustness-coverage", "children"),
2753
  Output("robustness-curve", "figure"),
2754
  Output("robustness-table", "columns"),
2755
  Output("robustness-table", "data"),
2756
- Output("compute-coverage", "children"),
2757
  Output("runtime-bars", "figure"),
2758
  Output("memory-bars", "figure"),
2759
  Output("compute-table", "columns"),
@@ -2774,9 +2357,8 @@ def update_overview(dataset: str, models: list[str] | None):
2774
  "inference_time_sec",
2775
  "peak_ram_gb",
2776
  "peak_vram_gb",
2777
- "prediction_status",
2778
  ]
2779
- overview_table = frame.sort_values(
2780
  ["task_score", "model_order"],
2781
  ascending=[False, True],
2782
  na_position="last",
@@ -2793,22 +2375,15 @@ def update_overview(dataset: str, models: list[str] | None):
2793
  "robustness_auc", ascending=False
2794
  )
2795
  runtime, memory, compute_table = compute_figures(dataset, models)
2796
- chosen = selected_models(models)
2797
- prediction_available = int(frame["task_score"].notna().sum())
2798
- robustness_available = len(robustness_table)
2799
- compute_available = len(compute_table)
2800
  return (
2801
  overview_cards(dataset, models),
2802
- availability_note(prediction_available, len(chosen)),
2803
  column_defs(overview_columns),
2804
  records(round_numeric(overview_table)),
2805
  prediction_ranking_figure(dataset, models),
2806
  prediction_heatmap(models),
2807
- availability_note(robustness_available, len(chosen)),
2808
  robustness_figure(dataset, models),
2809
  column_defs(robustness_columns),
2810
  records(round_numeric(robustness_table)),
2811
- availability_note(compute_available, len(chosen)),
2812
  runtime,
2813
  memory,
2814
  column_defs(compute_table.columns),
@@ -2840,17 +2415,14 @@ def update_latent_color_control(dataset: str):
2840
  Output("consistency-method", "options"),
2841
  Output("consistency-method", "value"),
2842
  Output("consistency-method", "disabled"),
2843
- Output("consistency-coverage", "children"),
2844
  Input("dataset-filter", "value"),
2845
  Input("method-filter", "value"),
2846
  State("consistency-method", "value"),
2847
  )
2848
  def update_consistency_selector(dataset: str, models: list[str] | None, current: str | None):
2849
  dataset = dataset or DATASETS[0]
2850
- chosen = selected_models(models)
2851
- eligible = [model for model in chosen if model in CONSISTENCY_ELIGIBLE]
2852
  if dataset == "mc_pacman":
2853
- return [], None, True, "Not defined for MC PacMan: no cross-recording consistency cohort is configured."
2854
  frame = consistency_frame(dataset, models)
2855
  latent_pairs = set(zip(latent_samples["model"].astype(str), latent_samples["dataset"].astype(str)))
2856
  available = []
@@ -2862,8 +2434,7 @@ def update_consistency_selector(dataset: str, models: list[str] | None, current:
2862
  ]
2863
  options = [{"label": model_label(model), "value": model} for model in available]
2864
  value = current if current in available else (available[0] if available else None)
2865
- note = availability_note(len(available), len(eligible), unsupported=len(chosen) - len(eligible))
2866
- return options, value, not bool(options), note
2867
 
2868
 
2869
  @app.callback(
@@ -2895,10 +2466,8 @@ def update_consistency(
2895
 
2896
 
2897
  @app.callback(
2898
- Output("feature-coverage", "children"),
2899
  Output("feature-definition", "children"),
2900
  Output("feature-validation-bars", "figure"),
2901
- Output("feature-signed-bars", "figure"),
2902
  Output("feature-heatmap", "figure"),
2903
  Output("feature-table", "columns"),
2904
  Output("feature-table", "data"),
@@ -2907,21 +2476,26 @@ def update_consistency(
2907
  )
2908
  def update_feature(dataset: str, models: list[str] | None):
2909
  dataset = dataset or DATASETS[0]
2910
- chosen = selected_models(models)
2911
- eligible = [model for model in chosen if model in FEATURE_ELIGIBLE]
2912
- frame = feature_frame(dataset, models)
2913
- validation_fig, signed_fig, table = feature_figures(dataset, models)
2914
- _column, target, metric, reference = feature_spec(dataset)
2915
- definition = f"{metric} quantifies agreement with {target.lower()}."
2916
- if reference == 0.5:
2917
- definition += " Chance ROC-AUC is 0.5."
 
 
 
 
2918
  else:
2919
- definition += " Larger positive values indicate closer agreement; this proxy is not causal ground truth."
 
 
 
2920
  return (
2921
- feature_coverage_note(dataset, models),
2922
  definition,
2923
  validation_fig,
2924
- signed_fig,
2925
  feature_heatmap(models),
2926
  column_defs(table.columns),
2927
  records(table),
@@ -2929,8 +2503,6 @@ def update_feature(dataset: str, models: list[str] | None):
2929
 
2930
 
2931
  @app.callback(
2932
- Output("trial-coverage", "children"),
2933
- Output("trial-convergence", "children"),
2934
  Output("trial-detection-bars", "figure"),
2935
  Output("trial-heatmap", "figure"),
2936
  Output("trial-table", "columns"),
@@ -2946,43 +2518,13 @@ def update_feature(dataset: str, models: list[str] | None):
2946
  )
2947
  def update_trial(dataset: str, models: list[str] | None):
2948
  dataset = dataset or DATASETS[0]
2949
- chosen = selected_models(models)
2950
- eligible = [model for model in chosen if model in TRIAL_ELIGIBLE]
2951
  frame = trial_frame(dataset, models)
2952
- table_columns = [
2953
- "method",
2954
- "corrupted_trial_auc",
2955
- "converged",
2956
- "iterations",
2957
- "final_error",
2958
- "perturbation_fraction",
2959
- "rotation_angle_deg",
2960
- "rotation_subspace_dim_spec",
2961
- "shapley_mean_value",
2962
- "shapley_median_value",
2963
- "shapley_min_value",
2964
- "shapley_max_value",
2965
- "shapley_fraction_positive",
2966
- "shapley_fraction_negative",
2967
- ]
2968
  table = frame[[column for column in table_columns if column in frame.columns]].sort_values(
2969
  "corrupted_trial_auc", ascending=False
2970
  ) if not frame.empty else pd.DataFrame(columns=table_columns)
2971
- all_trial = active_rows(trial_shapley)
2972
- converged_mask = all_trial["converged"].fillna(False).astype(str).str.lower().isin({"true", "1", "yes"})
2973
- converged = int(converged_mask.sum())
2974
- nonconverged = int(len(all_trial) - converged)
2975
- selected_converged = int((frame.get("converged", pd.Series(dtype=str)) == "Yes").sum())
2976
- selected_nonconverged = int(len(frame) - selected_converged)
2977
- removal, relation, historical, retrain_table = trial_retrain_figures(models)
2978
  return (
2979
- availability_note(len(frame), len(eligible), unsupported=len(chosen) - len(eligible)),
2980
- (
2981
- f"Selected view: {selected_converged}/{len(frame)} summaries converged; "
2982
- f"{selected_nonconverged}/{len(frame)} did not. Bundled release overall: "
2983
- f"{converged}/{len(all_trial)} converged and {nonconverged}/{len(all_trial)} did not. "
2984
- "A non-converged summary remains visible when selected and is not treated as missing."
2985
- ),
2986
  trial_detection_figure(dataset, models),
2987
  trial_heatmap(models),
2988
  column_defs(table.columns),
@@ -2990,24 +2532,10 @@ def update_trial(dataset: str, models: list[str] | None):
2990
  removal,
2991
  relation,
2992
  historical,
2993
- historical_trajectory_figure(models),
2994
  column_defs(retrain_table.columns),
2995
  records(retrain_table),
2996
  )
2997
 
2998
-
2999
- @app.callback(
3000
- Output("coverage-cards", "children"),
3001
- Output("methods-table", "columns"),
3002
- Output("methods-table", "data"),
3003
- Input("dataset-filter", "value"),
3004
- Input("method-filter", "value"),
3005
- )
3006
- def update_methods(dataset: str, models: list[str] | None):
3007
- dataset = dataset or DATASETS[0]
3008
- table = methods_frame(dataset, models)
3009
- return coverage_cards(dataset, models), column_defs(table.columns), records(table)
3010
-
3011
-
3012
  if __name__ == "__main__":
3013
  app.run(host="0.0.0.0", port=7860, debug=False)
 
1
  from __future__ import annotations
2
 
 
3
  import re
4
  from pathlib import Path
5
  from typing import Iterable, Sequence
 
69
 
70
  DATASET_LABELS = {
71
  "monkey": "Macaque center-out reaching",
72
+ "allen_neuropixels": "Allen visual coding",
73
  "speech": "Attempted speech",
74
  "mc_pacman": "MC PacMan force decoding",
75
  "ratinabox": "RatInABox navigation",
76
  }
77
 
78
+ DATASET_TICK_LABELS = {
79
+ "monkey": "Macaque<br>center-out reaching",
80
+ "allen_neuropixels": "Allen<br>visual coding",
81
+ "speech": "Attempted<br>speech",
82
+ "mc_pacman": "MC PacMan<br>force decoding",
83
+ "ratinabox": "RatInABox<br>navigation",
84
  }
85
 
86
  DATASET_DESCRIPTIONS = {
 
225
  TABLE_LABELS = {
226
  "method": "Method",
227
  "workflow": "Prediction workflow",
 
228
  "hardware": "Primary hardware",
 
 
 
 
 
 
 
229
  "task_score": "Held-out task score",
230
  "prediction_percentile": "Within-dataset percentile",
231
+ "robustness_auc": "Area under task-score-versus-noise curve",
232
  "training_time_sec": "Training time (s)",
233
  "inference_time_sec": "Inference time (s)",
234
  "peak_ram_gb": "Peak RAM (GB)",
 
243
  "validation_target": "Validation target",
244
  "validation_metric": "Validation metric",
245
  "validation_score": "Validation score",
 
 
 
 
 
 
246
  "corrupted_trial_auc": "Corrupted-trial ROC-AUC",
 
 
 
 
 
 
 
 
 
 
 
 
247
  "mixed_full": "Mixed trials",
248
  "data_shapley": "After trial-value removal",
 
249
  "recovery": "Recovery ΔR²",
250
  "target_only": "Current session only",
251
  "all_sessions": "All-session pooling",
 
391
  return frame.sort_values(["direction_index", "trial_index", "time_index"]).reset_index(drop=True)
392
 
393
 
 
 
 
 
 
 
 
 
 
 
 
394
  prediction = load_csv("clean_prediction_summary.csv")
395
  robustness = load_csv("robustness_summary.csv")
396
  consistency = load_csv("consistency_summary.csv")
 
401
  trial_historical_trajectories = load_historical_trajectories()
402
  latent_samples = load_csv("latent_samples.csv")
403
  latent_trajectories = load_csv("latent_trajectories.csv")
 
404
 
405
 
406
  def present_rows(df: pd.DataFrame) -> pd.DataFrame:
 
602
  return html.Div(children, className=classes)
603
 
604
 
605
+ def source_link(filename: str, label: str = "Download CSV") -> html.A:
606
  return html.A(
607
  label,
608
  href=f"/download/{filename}",
 
673
  def heatmap_layout(fig: go.Figure, *, height: int) -> go.Figure:
674
  figure_layout(fig, height=height)
675
  fig.update_layout(
676
+ margin=dict(l=54, r=28, t=116, b=50),
677
  title=dict(y=0.985, yanchor="top", pad=dict(b=12)),
678
  )
679
+ fig.update_xaxes(tickangle=0, tickfont=dict(size=10), automargin=True)
680
  return fig
681
 
682
 
 
697
  return figure_layout(fig, height=height)
698
 
699
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
700
  def prediction_workflow(model: str, decoder: object, status: object) -> str:
701
  if str(status).lower() != "present":
702
  return "Unavailable"
703
  decoder_name = "" if decoder is None or pd.isna(decoder) else str(decoder)
704
+ native_prediction_decoders = {
705
  "native",
706
  "dnn",
707
  "gru",
 
714
  "xgboost_classification",
715
  "xgboost_regression",
716
  }
717
+ if decoder_name in native_prediction_decoders:
718
+ return "Native prediction output"
719
  # Manuscript v7 deliberately distinguishes LDNS task families: continuous
720
  # prediction uses the method recipe's ridge mapping on reconstructed rates
721
  # (alpha = 1e-6), while classification uses the standard logistic readout.
722
  if decoder_name in {"ridge", "logistic", "ldns_rate_sklearn_logistic"}:
723
+ return "Shared linear readout"
724
  if decoder_name in {"knn", "ole", "ldns_rate_sklearn_ridge"}:
725
+ return "Author-style task readout"
726
  raise ValueError(f"Unrecognized prediction decoder for {model}: {decoder_name!r}")
727
 
728
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
729
  def prediction_percentiles() -> pd.DataFrame:
730
  frame = present_rows(prediction)[["model", "dataset", "score"]].copy()
731
  frame["score"] = pd.to_numeric(frame["score"], errors="coerce")
 
779
  lambda row: prediction_workflow(row["model"], row.get("decoder"), row.get("prediction_status_raw")),
780
  axis=1,
781
  )
 
782
  frame.loc[frame["model"].isin(CPU_ONLY_MODELS), "peak_vram_gb"] = np.nan
783
  return round_numeric(frame)
784
 
 
794
  cards = [
795
  metric_card(
796
  "Dataset",
797
+ DATASET_LABELS.get(dataset, dataset),
798
  DATASET_DESCRIPTIONS.get(dataset),
799
  "prediction",
800
  ),
 
817
  "prediction",
818
  )
819
  )
 
 
 
 
 
 
 
 
820
  return cards
821
 
822
 
 
840
  ),
841
  )
842
  )
843
+ fig.update_layout(title="Held-out prediction")
844
  fig.update_xaxes(title=metric)
845
  fig.update_yaxes(title="", showgrid=False)
846
  return figure_layout(fig, height=max(440, 25 * len(frame) + 145))
 
884
  if eligible_models is not None:
885
  chosen = [model for model in chosen if model in eligible_models]
886
  if not chosen:
887
+ return empty_figure("No results for this selection.")
888
  frame = values.copy()
889
  if frame.empty:
890
  return empty_figure(empty_message)
 
912
  index=ordered_models, columns=DATASETS
913
  )
914
  display_text = np.empty(percentile_matrix.shape, dtype=object)
915
+ customdata = np.empty((*percentile_matrix.shape, 2), dtype=object)
916
  for row_index, model in enumerate(percentile_matrix.index):
917
  for column_index, dataset in enumerate(percentile_matrix.columns):
918
  percentile = percentile_matrix.iloc[row_index, column_index]
919
  raw_value = raw_matrix.iloc[row_index, column_index]
920
  metric = metric_matrix.iloc[row_index, column_index]
921
  available = pd.notna(raw_value)
922
+ display_text[row_index, column_index] = "" if not available else f"{percentile:.0f}"
923
  customdata[row_index, column_index, 0] = (
924
+ "" if not available else f"{float(raw_value):.4f}"
925
  )
926
+ customdata[row_index, column_index, 1] = "" if pd.isna(metric) else str(metric)
 
927
  fig = go.Figure(
928
  go.Heatmap(
929
  z=percentile_matrix.to_numpy(dtype=float),
930
+ x=[DATASET_TICK_LABELS[dataset] for dataset in percentile_matrix.columns],
931
  y=[model_label(model) for model in percentile_matrix.index],
932
  text=display_text,
933
  texttemplate="%{text}",
 
940
  hovertemplate=(
941
  "Method=%{y}<br>Dataset=%{x}<br>"
942
  "Within-dataset percentile=%{z:.1f}<br>"
943
+ "Raw value=%{customdata[0]}<br>Metric=%{customdata[1]}<extra></extra>"
 
944
  ),
945
  hoverongaps=False,
946
  )
 
949
  if len(missing_rows):
950
  fig.add_trace(
951
  go.Scatter(
952
+ x=[DATASET_TICK_LABELS[percentile_matrix.columns[index]] for index in missing_columns],
953
  y=[model_label(percentile_matrix.index[index]) for index in missing_rows],
954
  mode="markers",
955
+ marker=dict(symbol="x", size=8, color="#8A949C", line=dict(width=1)),
956
  showlegend=False,
957
+ hoverinfo="skip",
958
  )
959
  )
960
  fig.update_layout(title=title)
 
969
  return percentile_heatmap(
970
  values,
971
  models,
972
+ title="Prediction across tasks",
973
  colorscale=PREDICTION_SCALE,
974
  raw_column="score",
975
  metric_column="metric_label",
 
1028
  marker=dict(size=6, symbol=index % 8),
1029
  customdata=np.repeat(row.robustness_auc, len(levels)),
1030
  hovertemplate=(
1031
+ f"Method={row.method}<br>Input-noise level λ=%{{x:.1f}}<br>"
1032
+ "Task score=%{y:.4f}<br>Area under task-score-versus-noise curve=%{customdata:.4f}<extra></extra>"
1033
  ),
1034
  )
1035
  )
1036
  metric = metric_name(frame["metric"].dropna().iloc[0])
1037
  fig.update_layout(
1038
+ title="Robustness",
1039
  hovermode="closest",
1040
  showlegend=len(frame) <= 12,
1041
  )
1042
+ fig.update_xaxes(title="Input-noise level λ", tickvals=[0, 0.2, 0.4, 0.6, 0.8])
 
 
 
 
 
 
 
 
 
 
 
1043
  fig.update_yaxes(title=metric)
1044
  return figure_layout(fig, height=540, legend_below=len(frame) <= 12)
1045
 
 
1087
  )
1088
  )
1089
  runtime.update_layout(
1090
+ title="Training and inference time",
1091
  barmode="group",
1092
  )
1093
  runtime.update_xaxes(title="Elapsed time (seconds, log scale)", type="log")
1094
  runtime.update_yaxes(title="", showgrid=False)
1095
+ figure_layout(runtime, height=max(470, 27 * len(frame) + 155), legend_below=True)
1096
 
1097
  memory = go.Figure()
1098
  memory.add_trace(
 
1116
  )
1117
  )
1118
  memory.update_layout(
1119
+ title="Peak memory",
1120
  barmode="group",
1121
  )
1122
  memory.update_xaxes(title="Memory (GB)")
1123
  memory.update_yaxes(title="", showgrid=False)
1124
+ figure_layout(memory, height=max(470, 27 * len(frame) + 155), legend_below=True)
1125
 
1126
  table = frame[
1127
  [
 
1176
 
1177
  def feature_figures(
1178
  dataset: str, models: Sequence[str] | None
1179
+ ) -> tuple[go.Figure, pd.DataFrame]:
1180
  frame = feature_frame(dataset, models)
1181
  if frame.empty:
1182
+ columns = ["method", "validation_target", "validation_metric", "validation_score"]
1183
+ return empty_figure("No results for this selection."), pd.DataFrame(columns=columns)
 
 
 
 
 
 
 
 
 
 
 
1184
  score_column, target, metric, reference = feature_spec(dataset)
1185
  validation = frame.dropna(subset=["validation_score"]).sort_values(
1186
  ["validation_score", "model_order"], ascending=[True, False]
 
1203
  annotation_position="top",
1204
  )
1205
  validation_fig.update_layout(
1206
+ title="Feature-attribution validation"
1207
  )
1208
  validation_fig.update_xaxes(title=metric)
1209
  validation_fig.update_yaxes(title="", showgrid=False)
1210
  figure_layout(validation_fig, height=max(430, 25 * len(validation) + 145))
1211
 
1212
+ columns = ["method", "validation_target", "validation_metric", "validation_score"]
1213
+ return validation_fig, round_numeric(
1214
+ frame[columns].sort_values("validation_score", ascending=False)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1215
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1216
 
1217
 
1218
  def feature_heatmap(models: Sequence[str] | None) -> go.Figure:
 
1234
  return percentile_heatmap(
1235
  values,
1236
  models,
1237
+ title="Feature validation across tasks",
1238
  colorscale=FEATURE_SCALE,
1239
  raw_column="validation_score",
1240
  metric_column="metric_label",
 
1252
  frame = add_method_columns(frame)
1253
  frame["method"] = frame["model"].map(lambda model: dataset_model_label(model, dataset))
1254
  frame = frame.rename(columns={"perturbation_auc": "corrupted_trial_auc"})
 
 
 
1255
  return round_numeric(frame)
1256
 
1257
 
 
1262
  frame = frame.dropna(subset=["corrupted_trial_auc"]).sort_values(
1263
  ["corrupted_trial_auc", "model_order"], ascending=[True, False]
1264
  )
 
1265
  fig = go.Figure(
1266
  go.Bar(
1267
  x=frame["corrupted_trial_auc"],
1268
  y=frame["method"],
1269
  orientation="h",
1270
+ marker=dict(color=TRIAL_COLOR),
1271
+ hovertemplate="Method=%{y}<br>Corrupted-trial ROC-AUC=%{x:.4f}<extra></extra>",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1272
  )
1273
  )
1274
  fig.add_vline(
 
1279
  annotation_position="top",
1280
  )
1281
  fig.update_layout(
1282
+ title="Corrupted-trial detection"
1283
  )
1284
  fig.update_xaxes(title="ROC-AUC from negative trial value")
1285
  fig.update_yaxes(title="", showgrid=False)
 
1292
  return percentile_heatmap(
1293
  values,
1294
  models,
1295
+ title="Trial detection across tasks",
1296
  colorscale=TRIAL_SCALE,
1297
  raw_column="perturbation_auc",
1298
  metric_column="metric_label",
 
1334
  return float(values.min() - padding), float(values.max() + padding)
1335
 
1336
 
1337
+ def trial_retrain_figures() -> tuple[go.Figure, go.Figure, go.Figure, pd.DataFrame]:
1338
+ within, historical = retrain_frames(None)
 
 
1339
  if within.empty:
1340
  removal = empty_figure("No macaque within-session removal summary is available.")
1341
  relation = empty_figure("No detection-versus-recovery summary is available.")
 
1353
  size=10,
1354
  line=dict(color="#FFFFFF", width=1),
1355
  ),
1356
+ customdata=np.stack([within["method"], within["recovery"]], axis=-1),
1357
  hovertemplate=(
1358
  "Method=%{customdata[0]}<br>Mixed trials R²=%{x:.4f}<br>"
1359
+ "After trial-value removal R²=%{y:.4f}<br>"
1360
+ "Recovery ΔR²=%{customdata[1]:+.4f}<extra></extra>"
1361
  ),
1362
  )
1363
  )
 
1379
  bgcolor="rgba(255,255,255,0.88)",
1380
  font=dict(size=12, color="#2E7D32" if removal_change >= 0 else NEGATIVE_COLOR),
1381
  )
1382
+ removal.update_layout(title="Trial-value-guided removal")
1383
+ removal.update_xaxes(title="Before removal: test R²", range=[lower, upper])
1384
+ removal.update_yaxes(title="After removal: test R²", range=[lower, upper])
1385
  figure_layout(removal, height=480)
1386
 
1387
  detection = active_rows(trial_shapley)
 
1436
  if pd.notna(rho)
1437
  else f"n = {len(relation_frame)}; select at least two methods for correlation"
1438
  )
1439
+ if pd.notna(rho):
1440
  relation_text += "<br>one-sided permutation P = 0.035"
1441
  relation.add_annotation(
1442
  text=relation_text,
 
1450
  bgcolor="rgba(255,255,255,0.85)",
1451
  font=dict(size=12),
1452
  )
1453
+ relation.update_layout(title="Detection and recovery", showlegend=False)
1454
+ relation.update_xaxes(title="Detection ROC-AUC")
1455
+ relation.update_yaxes(title="Recovery (ΔR²)")
1456
  figure_layout(relation, height=480)
1457
 
1458
  if historical.empty:
 
1495
  bgcolor="rgba(255,255,255,0.88)",
1496
  font=dict(size=12, color="#2E7D32" if historical_change >= 0 else NEGATIVE_COLOR),
1497
  )
1498
+ historical_fig.update_layout(title="Historical-trial selection")
1499
+ historical_fig.update_xaxes(title="Current-session test R²", range=[lower, upper])
1500
  historical_fig.update_yaxes(
1501
+ title="Selected historical trials: test R²",
1502
  range=[lower, upper],
1503
  )
1504
  figure_layout(historical_fig, height=480)
1505
 
1506
  table = within[
1507
+ [column for column in ["model", "method", "mixed_full", "data_shapley", "recovery"] if column in within]
1508
+ ].copy() if not within.empty else pd.DataFrame(columns=["model", "method", "mixed_full", "data_shapley", "recovery"])
1509
  historical_columns = ["model", "target_only", "all_sessions", "historical_selected"]
1510
  if not historical.empty:
1511
  table = table.merge(historical[historical_columns], on="model", how="outer")
 
1514
  return removal, relation, historical_fig, round_numeric(table)
1515
 
1516
 
1517
+ def historical_trajectory_figure() -> go.Figure:
 
 
 
 
 
 
 
1518
  frame = trial_historical_trajectories.copy()
1519
  current_r2 = float(frame["current_only_r2"].iloc[0])
1520
  historical_r2 = float(frame["historical_selected_r2"].iloc[0])
 
1530
  (
1531
  "historical_selected_x",
1532
  "historical_selected_y",
1533
+ f"Selected historical trials<br>R² = {historical_r2:.2f}",
1534
  0.64,
1535
  2.5,
1536
  ),
 
1647
 
1648
  figure_layout(fig, height=510, legend_below=True)
1649
  fig.update_layout(
1650
+ title="Held-out trajectories · RNN",
1651
  margin=dict(l=28, r=28, t=76, b=118),
1652
  legend=dict(
1653
  orientation="h",
 
1734
 
1735
 
1736
  def latent_space_figure(dataset: str, model: str | None, color_mode: str) -> go.Figure:
1737
+ if dataset == "mc_pacman":
1738
+ return empty_figure(
1739
+ "Cross-recording consistency is not available for this dataset.",
1740
+ height=500,
1741
+ )
1742
  if not model:
1743
  return empty_figure("Select an available method to view aligned coordinates.", height=500)
1744
  samples = latent_samples[
 
1746
  & (latent_samples["model"].astype(str) == str(model))
1747
  ].copy()
1748
  if samples.empty:
1749
+ return empty_figure("No results for this selection.", height=500)
1750
  for column in ["x", "y", "z"]:
1751
  samples[column] = pd.to_numeric(samples[column], errors="coerce")
1752
  samples = add_latent_color_columns(samples, dataset, color_mode).dropna(subset=["x", "y", "z"])
 
1891
  fig.update_layout(
1892
  **{
1893
  scene_id: dict(
1894
+ xaxis=dict(range=[-extent, extent], visible=False),
1895
+ yaxis=dict(range=[-extent, extent], visible=False),
1896
+ zaxis=dict(range=[-extent, extent], visible=False),
1897
  aspectmode="cube",
1898
  bgcolor="#FFFFFF",
1899
  camera=dict(eye=dict(x=1.5, y=1.4, z=1.0)),
 
1906
  & (score_rows["model"].astype(str) == str(model))
1907
  ]
1908
  score = pd.to_numeric(score_rows.get("mean_r2"), errors="coerce").dropna()
1909
+ title = model_label(model)
1910
+ if not score.empty:
1911
+ title += f" · R² = {float(score.iloc[0]):.3f}"
1912
  fig.update_layout(
1913
+ title=title,
1914
+ height=700 if rows > 1 else 500,
1915
  paper_bgcolor="#FFFFFF",
1916
  plot_bgcolor="#FFFFFF",
1917
+ margin=dict(l=8, r=8, t=72, b=88),
1918
  font=dict(family="Arial, Helvetica, sans-serif", size=12, color=TEXT_COLOR),
1919
+ legend=dict(
1920
+ orientation="h",
1921
+ yanchor="top",
1922
+ y=-0.04,
1923
+ xanchor="center",
1924
+ x=0.5,
1925
+ entrywidth=46,
1926
+ entrywidthmode="pixels",
1927
+ ),
1928
  )
1929
  fig.for_each_annotation(lambda annotation: annotation.update(font=dict(size=12, color="#526171")))
1930
  return fig
 
1952
  frame = consistency_frame(dataset, models)
1953
  if frame.empty:
1954
  message = (
1955
+ "Cross-recording consistency is not available for this dataset."
1956
  if dataset == "mc_pacman"
1957
  else "No latent-consistency result is available for this selection."
1958
  )
 
1973
  ),
1974
  )
1975
  )
1976
+ bar_fig.update_layout(title="Latent consistency")
1977
+ bar_fig.update_xaxes(title="Latent-consistency R²", range=[0, 1.02])
1978
  bar_fig.update_yaxes(title="", showgrid=False)
1979
  figure_layout(bar_fig, height=max(400, 27 * len(bar) + 145))
1980
  columns = ["method", "latent_consistency_r2", "n_recordings", "latent_dim", "n_pairwise"]
 
1996
  for row in range(pivot.shape[0]):
1997
  for column in range(pivot.shape[1]):
1998
  value = pivot.iloc[row, column]
1999
+ text[row, column] = "" if pd.isna(value) else f"{value:.2f}"
2000
  fig = go.Figure(
2001
  go.Heatmap(
2002
  z=pivot.to_numpy(dtype=float),
2003
+ x=[DATASET_TICK_LABELS[dataset] for dataset in pivot.columns],
2004
  y=[model_label(model) for model in pivot.index],
2005
  text=text,
2006
  texttemplate="%{text}",
 
2016
  if len(missing_rows):
2017
  fig.add_trace(
2018
  go.Scatter(
2019
+ x=[DATASET_TICK_LABELS[pivot.columns[index]] for index in missing_columns],
2020
  y=[model_label(pivot.index[index]) for index in missing_rows],
2021
  mode="markers",
2022
+ marker=dict(symbol="x", size=8, color="#8A949C", line=dict(width=1)),
2023
  showlegend=False,
2024
+ hoverinfo="skip",
2025
  )
2026
  )
2027
+ fig.update_layout(title="Latent consistency across tasks")
2028
  fig.update_xaxes(title="", side="top", showgrid=False)
2029
  fig.update_yaxes(title="", showgrid=False)
2030
  return heatmap_layout(fig, height=max(470, 27 * len(pivot) + 180))
2031
 
2032
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2033
  app = Dash(__name__, title="BEND-BCI Interactive Benchmark")
2034
  server = app.server
2035
 
 
2047
  [
2048
  html.Div(
2049
  [
2050
+ html.A("BEND-BCI", href="#", className="site-brand"),
 
 
 
 
 
2051
  html.Nav(
2052
  [
2053
  html.A(
2054
+ "Code & data",
2055
  href="https://github.com/TangLab-UBC/behavior_benchmarking",
2056
  target="_blank",
2057
  rel="noopener noreferrer",
2058
  ),
2059
+ html.Span(
2060
+ ["Paper", html.Small("coming soon")],
2061
+ className="nav-placeholder",
2062
+ title="Manuscript link will be added on release.",
2063
+ ),
2064
+ html.Span(
2065
+ ["Submit a model", html.Small("planned")],
2066
+ className="nav-placeholder",
2067
+ title="A model-submission workflow is planned.",
2068
  ),
2069
  ],
2070
  className="hero-links",
2071
+ **{"aria-label": "Resources"},
2072
  ),
2073
  ],
2074
+ className="site-nav",
2075
  ),
2076
  html.Div(
2077
  [
2078
+ html.H1("Neural decoder selection beyond held-out performance"),
2079
+ html.P(
2080
+ "Interactive results for 23 methods across motor, visual, speech and spatial decoding tasks.",
2081
+ className="lede",
2082
+ ),
2083
  ],
2084
+ className="hero-copy",
2085
  ),
2086
  ],
2087
  className="hero",
 
2103
  ),
2104
  html.Div(
2105
  [
2106
+ html.Label("Methods", htmlFor="method-filter"),
2107
  dcc.Dropdown(
2108
  id="method-filter",
2109
  options=[{"label": model_label(model), "value": model} for model in FIGURE_MODEL_ORDER],
2110
  value=[],
2111
  multi=True,
2112
+ placeholder="All methods",
2113
  ),
 
2114
  ],
2115
  className="control method-control",
2116
  ),
2117
  ],
2118
  className="toolbar",
2119
  ),
2120
+ html.Main(
2121
+ dcc.Tabs(
2122
+ id="tabs",
2123
+ value="overview",
2124
+ className="tabs",
2125
+ children=[
2126
  dcc.Tab(
2127
  label="Overview",
2128
  value="overview",
 
2130
  selected_className="tab tab-selected",
2131
  children=[
2132
  panel(
2133
+ "Predictive performance",
2134
  html.Div(id="overview-cards", className="metric-strip"),
 
2135
  html.Div(
2136
  [
2137
  graph_box("prediction-ranking", "Raw held-out prediction scores for the selected dataset."),
2138
+ graph_box(
2139
+ "prediction-heatmap",
2140
+ "Within-dataset prediction percentiles across five tasks.",
2141
+ class_name="heatmap-graph",
2142
+ ),
2143
  ],
2144
  className="chart-grid two",
2145
  ),
2146
+ details_table("View data", dataframe_table("overview-table", page_size=23)),
2147
  source_link("clean_prediction_summary.csv"),
2148
+ subtitle="Raw task scores are shown by dataset; the cross-dataset view uses within-dataset percentiles.",
 
2149
  class_name="axis-prediction",
2150
  ),
2151
  panel(
2152
+ "Robustness to noisy inputs",
2153
+ graph_box("robustness-curve", "Task score as input-noise level increases."),
2154
+ details_table("View data", dataframe_table("robustness-table", page_size=23)),
 
2155
  source_link("robustness_summary.csv"),
2156
+ subtitle="Performance as controlled noise is added to held-out neural inputs.",
 
2157
  class_name="axis-robustness",
2158
  ),
2159
  panel(
2160
  "Computational cost",
 
2161
  html.Div(
2162
  [
2163
  graph_box("runtime-bars", "Training and complete-held-out-split inference times."),
 
2165
  ],
2166
  className="chart-grid two",
2167
  ),
2168
+ details_table("View data", dataframe_table("compute-table", page_size=23)),
2169
  source_link("scalability_summary.csv"),
2170
+ subtitle="Training time, inference time, RAM and GPU memory under the benchmark protocol.",
 
2171
  class_name="axis-compute",
2172
  ),
2173
  ],
 
2179
  selected_className="tab tab-selected",
2180
  children=[
2181
  panel(
2182
+ "Representation consistency across recordings",
 
2183
  html.Div(
2184
  [
2185
  html.Div(
2186
  [
2187
+ html.Label("Method", htmlFor="consistency-method"),
2188
  dcc.Dropdown(id="consistency-method", clearable=False),
2189
  ],
2190
  className="control",
 
2200
  ],
2201
  className="inline-controls",
2202
  ),
2203
+ graph_box("latent-space", "Aligned latent representations for each recording.", class_name="latent-graph"),
 
 
 
 
 
 
 
2204
  html.Div(
2205
  [
2206
  graph_box("consistency-bars", "Latent-consistency R-squared for the selected dataset."),
2207
+ graph_box(
2208
+ "consistency-heatmap",
2209
+ "Latent-consistency R-squared across four tasks.",
2210
+ class_name="heatmap-graph",
2211
+ ),
2212
  ],
2213
  className="chart-grid two",
2214
  ),
2215
+ details_table("View data", dataframe_table("consistency-table", page_size=12)),
2216
  source_link("consistency_summary.csv"),
2217
+ subtitle="Plots show whitened latent spaces aligned to a common display frame. Consistency measures linear alignment of matched task landmarks across recordings, participants or simulations.",
 
2218
  class_name="axis-consistency",
2219
  )
2220
  ],
 
2227
  children=[
2228
  panel(
2229
  "Feature-attribution validation",
 
2230
  html.Div(
2231
  [
2232
  graph_box("feature-validation-bars", "Feature-attribution validation metric for the selected dataset."),
2233
+ graph_box(
2234
+ "feature-heatmap",
2235
+ "Within-dataset feature-attribution validation percentiles across tasks.",
2236
+ class_name="heatmap-graph",
2237
+ ),
2238
  ],
2239
  className="chart-grid two",
2240
  ),
 
2241
  html.Div(
2242
+ html.Span(id="feature-definition"),
 
 
 
 
2243
  className="method-note",
2244
  ),
2245
+ details_table("View data", dataframe_table("feature-table", page_size=23)),
2246
  source_link("neuron_shap_summary.csv"),
2247
+ subtitle="Agreement with predefined, dataset-specific validation targets.",
 
2248
  class_name="axis-feature",
2249
  )
2250
  ],
 
2256
  selected_className="tab tab-selected",
2257
  children=[
2258
  panel(
2259
+ "Corrupted-trial detection",
 
 
2260
  html.Div(
2261
  [
2262
  graph_box("trial-detection-bars", "Corrupted-trial detection ROC-AUC for the selected dataset."),
2263
+ graph_box(
2264
+ "trial-heatmap",
2265
+ "Within-dataset corrupted-trial detection percentiles across tasks.",
2266
+ class_name="heatmap-graph",
2267
+ ),
2268
  ],
2269
  className="chart-grid two",
2270
  ),
2271
  html.Div(
2272
  [
2273
+ "One third of training trials were rotated 75° in population-activity space while targets were unchanged. ROC-AUC uses negative trial value as the detection score.",
 
2274
  ],
2275
  className="method-note",
2276
  ),
2277
+ details_table("View data", dataframe_table("trial-table", page_size=23)),
2278
  source_link("trial_shapley_summary.csv"),
2279
+ subtitle="ROC-AUC measures whether lower trial values identify training trials with rotated neural activity.",
 
2280
  class_name="axis-trial",
2281
  ),
2282
  panel(
2283
+ "Macaque center-out reaching training-data interventions",
 
2284
  html.Div(
2285
  [
2286
  graph_box("trial-removal", "Held-out R-squared before and after trial-value-guided corrupted-trial removal."),
 
2294
  "Held-out RNN target-session trajectories for ground truth, current-session training, and nonnegative-valued historical-trial selection.",
2295
  class_name="historical-trajectory-graph",
2296
  ),
2297
+ details_table("View data", dataframe_table("trial-retrain-table", page_size=17)),
2298
  html.Div(
2299
  [
2300
+ source_link("trial_shapley_retrain_summary.csv", "Summary CSV"),
2301
+ source_link("trial_historical_trajectories.csv", "RNN trajectory CSV"),
2302
  ],
2303
  className="download-grid panel-downloads",
2304
  ),
2305
+ subtitle="Removing negative-valued trials improved 13 of 17 methods (mean R² 0.775→0.797). Selecting nonnegative-valued historical trials raised mean held-out R² to 0.617, versus 0.532 for current-only training and 0.546 for all-session pooling.",
 
2306
  class_name="axis-trial",
2307
  ),
2308
  ],
2309
  ),
2310
+ ],
2311
+ ),
2312
+ className="main-content",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2313
  ),
2314
  html.Footer(
2315
  [
2316
+ html.Span("BEND-BCI · Tang Lab"),
2317
+ html.A(
2318
+ "Code & data",
2319
+ href="https://github.com/TangLab-UBC/behavior_benchmarking",
2320
+ target="_blank",
2321
+ rel="noopener noreferrer",
2322
+ ),
2323
  ],
2324
  className="provenance-footer",
2325
  ),
 
2330
 
2331
  @app.callback(
2332
  Output("overview-cards", "children"),
 
2333
  Output("overview-table", "columns"),
2334
  Output("overview-table", "data"),
2335
  Output("prediction-ranking", "figure"),
2336
  Output("prediction-heatmap", "figure"),
 
2337
  Output("robustness-curve", "figure"),
2338
  Output("robustness-table", "columns"),
2339
  Output("robustness-table", "data"),
 
2340
  Output("runtime-bars", "figure"),
2341
  Output("memory-bars", "figure"),
2342
  Output("compute-table", "columns"),
 
2357
  "inference_time_sec",
2358
  "peak_ram_gb",
2359
  "peak_vram_gb",
 
2360
  ]
2361
+ overview_table = frame.dropna(subset=["task_score"]).sort_values(
2362
  ["task_score", "model_order"],
2363
  ascending=[False, True],
2364
  na_position="last",
 
2375
  "robustness_auc", ascending=False
2376
  )
2377
  runtime, memory, compute_table = compute_figures(dataset, models)
 
 
 
 
2378
  return (
2379
  overview_cards(dataset, models),
 
2380
  column_defs(overview_columns),
2381
  records(round_numeric(overview_table)),
2382
  prediction_ranking_figure(dataset, models),
2383
  prediction_heatmap(models),
 
2384
  robustness_figure(dataset, models),
2385
  column_defs(robustness_columns),
2386
  records(round_numeric(robustness_table)),
 
2387
  runtime,
2388
  memory,
2389
  column_defs(compute_table.columns),
 
2415
  Output("consistency-method", "options"),
2416
  Output("consistency-method", "value"),
2417
  Output("consistency-method", "disabled"),
 
2418
  Input("dataset-filter", "value"),
2419
  Input("method-filter", "value"),
2420
  State("consistency-method", "value"),
2421
  )
2422
  def update_consistency_selector(dataset: str, models: list[str] | None, current: str | None):
2423
  dataset = dataset or DATASETS[0]
 
 
2424
  if dataset == "mc_pacman":
2425
+ return [], None, True
2426
  frame = consistency_frame(dataset, models)
2427
  latent_pairs = set(zip(latent_samples["model"].astype(str), latent_samples["dataset"].astype(str)))
2428
  available = []
 
2434
  ]
2435
  options = [{"label": model_label(model), "value": model} for model in available]
2436
  value = current if current in available else (available[0] if available else None)
2437
+ return options, value, not bool(options)
 
2438
 
2439
 
2440
  @app.callback(
 
2466
 
2467
 
2468
  @app.callback(
 
2469
  Output("feature-definition", "children"),
2470
  Output("feature-validation-bars", "figure"),
 
2471
  Output("feature-heatmap", "figure"),
2472
  Output("feature-table", "columns"),
2473
  Output("feature-table", "data"),
 
2476
  )
2477
  def update_feature(dataset: str, models: list[str] | None):
2478
  dataset = dataset or DATASETS[0]
2479
+ validation_fig, table = feature_figures(dataset, models)
2480
+ _column, _target, _metric, _reference = feature_spec(dataset)
2481
+ if dataset == "allen_neuropixels":
2482
+ definition = (
2483
+ "Spearman’s ρ measures association with drifting-gratings orientation "
2484
+ "selectivity, a biological proxy."
2485
+ )
2486
+ elif dataset == "ratinabox":
2487
+ definition = (
2488
+ "ROC-AUC measures whether place cells rank above head-direction and speed "
2489
+ "cells. Chance ROC-AUC is 0.5."
2490
+ )
2491
  else:
2492
+ definition = (
2493
+ "ROC-AUC measures whether recorded neural features rank above appended "
2494
+ "synthetic controls. Chance ROC-AUC is 0.5."
2495
+ )
2496
  return (
 
2497
  definition,
2498
  validation_fig,
 
2499
  feature_heatmap(models),
2500
  column_defs(table.columns),
2501
  records(table),
 
2503
 
2504
 
2505
  @app.callback(
 
 
2506
  Output("trial-detection-bars", "figure"),
2507
  Output("trial-heatmap", "figure"),
2508
  Output("trial-table", "columns"),
 
2518
  )
2519
  def update_trial(dataset: str, models: list[str] | None):
2520
  dataset = dataset or DATASETS[0]
 
 
2521
  frame = trial_frame(dataset, models)
2522
+ table_columns = ["method", "corrupted_trial_auc"]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2523
  table = frame[[column for column in table_columns if column in frame.columns]].sort_values(
2524
  "corrupted_trial_auc", ascending=False
2525
  ) if not frame.empty else pd.DataFrame(columns=table_columns)
2526
+ removal, relation, historical, retrain_table = trial_retrain_figures()
 
 
 
 
 
 
2527
  return (
 
 
 
 
 
 
 
2528
  trial_detection_figure(dataset, models),
2529
  trial_heatmap(models),
2530
  column_defs(table.columns),
 
2532
  removal,
2533
  relation,
2534
  historical,
2535
+ historical_trajectory_figure(),
2536
  column_defs(retrain_table.columns),
2537
  records(retrain_table),
2538
  )
2539
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2540
  if __name__ == "__main__":
2541
  app.run(host="0.0.0.0", port=7860, debug=False)
assets/styles.css CHANGED
@@ -1,18 +1,18 @@
1
  :root {
2
- --ink: #17202a;
3
- --muted: #607080;
4
- --line: #d8e1e7;
5
- --soft-line: #e9eef2;
6
  --paper: #ffffff;
7
- --canvas: #f4f6f8;
8
  --prediction: #1565c0;
9
  --robustness: #2e7d32;
10
  --compute: #e65100;
11
  --consistency: #007c91;
12
  --feature: #6a51a3;
13
  --trial: #6a1b9a;
14
- --shadow: 0 12px 34px rgba(24, 43, 59, 0.075);
15
- --radius: 12px;
16
  }
17
 
18
  * {
@@ -27,7 +27,7 @@ body {
27
  margin: 0;
28
  background: var(--canvas);
29
  color: var(--ink);
30
- font-family: Arial, Helvetica, sans-serif;
31
  font-size: 15px;
32
  line-height: 1.5;
33
  text-rendering: optimizeLegibility;
@@ -57,50 +57,47 @@ p {
57
 
58
  .app-shell {
59
  min-height: 100vh;
60
- padding: 20px 24px 34px;
61
  }
62
 
63
  .hero,
64
  .toolbar,
65
  .tabs,
66
  .provenance-footer {
67
- width: min(100%, 1540px);
68
  margin-right: auto;
69
  margin-left: auto;
70
  }
71
 
72
  .hero {
73
- position: relative;
74
- display: grid;
75
- grid-template-columns: minmax(0, 1fr) minmax(210px, 280px);
76
- gap: 36px;
77
- align-items: end;
78
- overflow: hidden;
79
- padding: 36px 40px 34px;
80
- border: 1px solid #173f59;
81
- border-radius: 16px;
82
- background:
83
- radial-gradient(circle at 92% 12%, rgba(86, 180, 233, 0.18), transparent 34%),
84
- linear-gradient(130deg, #10283a 0%, #12364b 63%, #154960 100%);
85
- box-shadow: 0 18px 44px rgba(15, 39, 56, 0.16);
86
- color: #ffffff;
87
  }
88
 
89
- .hero::after {
90
- position: absolute;
91
- right: -68px;
92
- bottom: -104px;
93
- width: 310px;
94
- height: 310px;
95
- border: 1px solid rgba(255, 255, 255, 0.09);
96
- border-radius: 50%;
97
- content: "";
98
  }
99
 
100
- .hero-copy,
101
- .hero-stat {
102
- position: relative;
103
- z-index: 1;
 
 
 
 
 
 
 
 
 
 
 
104
  }
105
 
106
  .eyebrow,
@@ -117,33 +114,34 @@ p {
117
  }
118
 
119
  h1 {
120
- max-width: 900px;
121
- margin-bottom: 12px;
122
- font-size: clamp(34px, 4vw, 51px);
123
- font-weight: 750;
124
- line-height: 1.04;
125
- letter-spacing: -0.035em;
126
  }
127
 
128
  .lede {
129
- max-width: 960px;
130
- margin-bottom: 18px;
131
- color: #d6e4ec;
132
- font-size: 16px;
133
- line-height: 1.55;
134
  }
135
 
136
  .hero-links {
137
  display: flex;
138
- gap: 18px;
 
139
  flex-wrap: wrap;
140
  }
141
 
142
  .hero-links a {
143
- color: #ffffff;
144
  font-size: 13px;
145
- font-weight: 700;
146
- text-decoration-color: rgba(255, 255, 255, 0.5);
147
  }
148
 
149
  .hero-links a::after {
@@ -152,42 +150,42 @@ h1 {
152
  font-size: 11px;
153
  }
154
 
155
- .hero-stat {
156
  display: flex;
157
- flex-direction: column;
158
- gap: 4px;
159
- padding: 18px 20px;
160
- border: 1px solid rgba(255, 255, 255, 0.18);
161
- border-radius: 12px;
162
- background: rgba(255, 255, 255, 0.075);
163
- backdrop-filter: blur(8px);
164
- }
165
-
166
- .hero-stat-value {
167
- font-size: 18px;
168
- font-weight: 800;
169
  }
170
 
171
- .hero-stat-label {
172
- color: #c9dce7;
173
- font-size: 12px;
 
 
 
 
 
 
174
  line-height: 1.4;
 
175
  }
176
 
177
  .toolbar {
178
  position: sticky;
179
- top: 0;
180
  z-index: 20;
181
  display: grid;
182
  grid-template-columns: minmax(300px, 0.72fr) minmax(420px, 1.28fr);
183
  gap: 18px;
184
- margin-top: 16px;
185
  margin-bottom: 14px;
186
  padding: 13px 16px 14px;
187
  border: 1px solid var(--line);
188
  border-radius: var(--radius);
189
  background: rgba(255, 255, 255, 0.96);
190
- box-shadow: 0 10px 28px rgba(18, 38, 53, 0.09);
191
  backdrop-filter: blur(10px);
192
  }
193
 
@@ -205,46 +203,31 @@ h1 {
205
  text-transform: uppercase;
206
  }
207
 
208
- .control-help {
209
- margin: 5px 1px 0;
210
- color: #70808e;
211
- font-size: 11px;
212
- line-height: 1.25;
213
- }
214
-
215
- .Select-control,
216
- .select__control {
217
  min-height: 39px;
218
- border-color: #cbd5dd !important;
219
- border-radius: 7px !important;
220
- box-shadow: none !important;
 
221
  }
222
 
223
- .is-focused:not(.is-open) > .Select-control {
224
- border-color: var(--prediction) !important;
225
- box-shadow: 0 0 0 3px rgba(21, 101, 192, 0.16) !important;
226
  }
227
 
228
- .Select-placeholder,
229
- .Select-value-label,
230
- .Select-input,
231
- .Select-option {
232
- font-size: 13px !important;
233
  }
234
 
235
- .Select--multi .Select-value {
236
- margin-top: 4px;
237
  border-color: #c9d9e7;
238
  border-radius: 5px;
239
- background: #edf5fb;
240
  color: #154f7b;
241
  }
242
 
243
- .Select-multi-value-wrapper {
244
- max-height: 82px;
245
- overflow-y: auto;
246
- }
247
-
248
  .tabs {
249
  display: flex;
250
  flex-direction: row !important;
@@ -295,9 +278,9 @@ h1 {
295
  position: relative;
296
  margin-top: 16px;
297
  overflow: hidden;
298
- padding: 24px 26px 26px;
299
  border: 1px solid var(--line);
300
- border-top: 4px solid #71808d;
301
  border-radius: var(--radius);
302
  background: var(--paper);
303
  box-shadow: var(--shadow);
@@ -333,7 +316,7 @@ h1 {
333
 
334
  .panel-heading {
335
  max-width: 1120px;
336
- margin-bottom: 18px;
337
  }
338
 
339
  .section-eyebrow {
@@ -367,7 +350,7 @@ h1 {
367
 
368
  h2 {
369
  margin-bottom: 7px;
370
- font-size: 23px;
371
  font-weight: 750;
372
  line-height: 1.2;
373
  letter-spacing: -0.018em;
@@ -383,7 +366,7 @@ h2 {
383
 
384
  .metric-strip {
385
  display: grid;
386
- grid-template-columns: repeat(4, minmax(180px, 1fr));
387
  gap: 11px;
388
  margin-bottom: 14px;
389
  }
@@ -456,44 +439,12 @@ h2 {
456
  line-height: 1.4;
457
  }
458
 
459
- .coverage-note,
460
- .convergence-note,
461
  .method-note {
462
  border-radius: 8px;
463
  font-size: 12px;
464
  line-height: 1.5;
465
  }
466
 
467
- .coverage-note {
468
- display: inline-flex;
469
- align-items: center;
470
- min-height: 32px;
471
- margin-bottom: 10px;
472
- padding: 6px 11px;
473
- border: 1px solid #d5e0e7;
474
- background: #f7fafb;
475
- color: #3f5260;
476
- font-weight: 700;
477
- }
478
-
479
- .coverage-note::before {
480
- width: 7px;
481
- height: 7px;
482
- margin-right: 7px;
483
- border-radius: 50%;
484
- background: #5f7585;
485
- content: "";
486
- }
487
-
488
- .convergence-note {
489
- margin: 0 0 12px;
490
- padding: 9px 12px;
491
- border: 1px solid #e4d5eb;
492
- background: #faf7fc;
493
- color: #5b4169;
494
- font-weight: 700;
495
- }
496
-
497
  .method-note {
498
  margin: 12px 0;
499
  padding: 11px 13px;
@@ -709,31 +660,29 @@ h2 {
709
  }
710
 
711
  .provenance-footer {
 
 
 
 
712
  margin-top: 18px;
713
- padding: 17px 20px;
714
- border: 1px solid #d3dde4;
715
- border-radius: 10px;
716
- background: #edf2f5;
717
- color: #4e606d;
718
- font-size: 11px;
719
  line-height: 1.55;
720
  }
721
 
 
 
 
 
 
722
  [data-dash-is-loading="true"] {
723
  opacity: 0.56;
724
  transition: opacity 140ms ease;
725
  }
726
 
727
  @media (max-width: 1180px) {
728
- .hero {
729
- grid-template-columns: 1fr;
730
- gap: 22px;
731
- }
732
-
733
- .hero-stat {
734
- max-width: 420px;
735
- }
736
-
737
  .metric-strip {
738
  grid-template-columns: repeat(2, minmax(180px, 1fr));
739
  }
@@ -745,12 +694,27 @@ h2 {
745
 
746
  @media (max-width: 820px) {
747
  .app-shell {
748
- padding: 12px 12px 24px;
749
  }
750
 
751
  .hero {
752
- padding: 28px 24px;
753
- border-radius: 13px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
754
  }
755
 
756
  .toolbar {
@@ -776,21 +740,47 @@ h2 {
776
  .inline-controls {
777
  grid-template-columns: 1fr;
778
  }
 
 
 
 
 
 
 
 
 
 
779
  }
780
 
781
  @media (max-width: 560px) {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
782
  h1 {
783
- font-size: 34px;
784
  }
785
 
786
  .lede {
787
  font-size: 14px;
788
  }
789
 
790
- .hero-stat {
791
- padding: 14px 15px;
792
- }
793
-
794
  .metric-strip {
795
  grid-template-columns: 1fr;
796
  }
@@ -847,16 +837,13 @@ h2 {
847
 
848
  .toolbar,
849
  .tabs,
850
- .source-link,
851
- .hero-stat {
852
  display: none !important;
853
  }
854
 
855
  .hero {
856
  padding: 18px 0;
857
- border: 0;
858
  background: #ffffff;
859
- box-shadow: none;
860
  color: #000000;
861
  }
862
 
 
1
  :root {
2
+ --ink: #142535;
3
+ --muted: #60717f;
4
+ --line: #dce3e8;
5
+ --soft-line: #edf1f4;
6
  --paper: #ffffff;
7
+ --canvas: #f7f8f9;
8
  --prediction: #1565c0;
9
  --robustness: #2e7d32;
10
  --compute: #e65100;
11
  --consistency: #007c91;
12
  --feature: #6a51a3;
13
  --trial: #6a1b9a;
14
+ --shadow: 0 5px 18px rgba(24, 43, 59, 0.055);
15
+ --radius: 10px;
16
  }
17
 
18
  * {
 
27
  margin: 0;
28
  background: var(--canvas);
29
  color: var(--ink);
30
+ font-family: Inter, ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
31
  font-size: 15px;
32
  line-height: 1.5;
33
  text-rendering: optimizeLegibility;
 
57
 
58
  .app-shell {
59
  min-height: 100vh;
60
+ padding: 0 28px 34px;
61
  }
62
 
63
  .hero,
64
  .toolbar,
65
  .tabs,
66
  .provenance-footer {
67
+ width: min(100%, 1380px);
68
  margin-right: auto;
69
  margin-left: auto;
70
  }
71
 
72
  .hero {
73
+ padding-bottom: 32px;
74
+ border-bottom: 1px solid var(--line);
 
 
 
 
 
 
 
 
 
 
 
 
75
  }
76
 
77
+ .site-nav {
78
+ display: flex;
79
+ min-height: 68px;
80
+ align-items: center;
81
+ justify-content: space-between;
82
+ gap: 24px;
83
+ border-bottom: 1px solid var(--soft-line);
 
 
84
  }
85
 
86
+ .site-brand {
87
+ color: #12334a;
88
+ font-size: 18px;
89
+ font-weight: 800;
90
+ letter-spacing: -0.015em;
91
+ text-decoration: none;
92
+ }
93
+
94
+ .site-brand:hover {
95
+ color: #12334a;
96
+ }
97
+
98
+ .hero-copy {
99
+ max-width: 980px;
100
+ padding-top: 46px;
101
  }
102
 
103
  .eyebrow,
 
114
  }
115
 
116
  h1 {
117
+ max-width: 960px;
118
+ margin-bottom: 14px;
119
+ font-size: clamp(34px, 4.1vw, 50px);
120
+ font-weight: 760;
121
+ line-height: 1.08;
122
+ letter-spacing: -0.032em;
123
  }
124
 
125
  .lede {
126
+ max-width: 820px;
127
+ margin-bottom: 0;
128
+ color: #526777;
129
+ font-size: 17px;
130
+ line-height: 1.6;
131
  }
132
 
133
  .hero-links {
134
  display: flex;
135
+ gap: 22px;
136
+ align-items: center;
137
  flex-wrap: wrap;
138
  }
139
 
140
  .hero-links a {
141
+ color: #244d69;
142
  font-size: 13px;
143
+ font-weight: 680;
144
+ text-decoration: none;
145
  }
146
 
147
  .hero-links a::after {
 
150
  font-size: 11px;
151
  }
152
 
153
+ .nav-placeholder {
154
  display: flex;
155
+ gap: 7px;
156
+ align-items: center;
157
+ color: #586b79;
158
+ font-size: 13px;
159
+ font-weight: 650;
 
 
 
 
 
 
 
160
  }
161
 
162
+ .nav-placeholder small {
163
+ padding: 2px 6px;
164
+ border: 1px solid #d5dde3;
165
+ border-radius: 999px;
166
+ background: #ffffff;
167
+ color: #71808b;
168
+ font-size: 9px;
169
+ font-weight: 750;
170
+ letter-spacing: 0.035em;
171
  line-height: 1.4;
172
+ text-transform: uppercase;
173
  }
174
 
175
  .toolbar {
176
  position: sticky;
177
+ top: 10px;
178
  z-index: 20;
179
  display: grid;
180
  grid-template-columns: minmax(300px, 0.72fr) minmax(420px, 1.28fr);
181
  gap: 18px;
182
+ margin-top: 18px;
183
  margin-bottom: 14px;
184
  padding: 13px 16px 14px;
185
  border: 1px solid var(--line);
186
  border-radius: var(--radius);
187
  background: rgba(255, 255, 255, 0.96);
188
+ box-shadow: 0 5px 18px rgba(18, 38, 53, 0.07);
189
  backdrop-filter: blur(10px);
190
  }
191
 
 
203
  text-transform: uppercase;
204
  }
205
 
206
+ .dash-dropdown-grid-container {
 
 
 
 
 
 
 
 
207
  min-height: 39px;
208
+ border: 1px solid #cbd5dd;
209
+ border-radius: 7px;
210
+ background: #ffffff;
211
+ box-shadow: none;
212
  }
213
 
214
+ .dash-dropdown:focus-within .dash-dropdown-grid-container {
215
+ border-color: var(--prediction);
216
+ box-shadow: 0 0 0 3px rgba(21, 101, 192, 0.14);
217
  }
218
 
219
+ .dash-dropdown-value,
220
+ .dash-dropdown-value-item,
221
+ .dash-dropdown-trigger {
222
+ font-size: 13px;
 
223
  }
224
 
225
+ .dash-dropdown-value-item {
 
226
  border-color: #c9d9e7;
227
  border-radius: 5px;
 
228
  color: #154f7b;
229
  }
230
 
 
 
 
 
 
231
  .tabs {
232
  display: flex;
233
  flex-direction: row !important;
 
278
  position: relative;
279
  margin-top: 16px;
280
  overflow: hidden;
281
+ padding: 25px 27px 27px;
282
  border: 1px solid var(--line);
283
+ border-top: 3px solid #71808d;
284
  border-radius: var(--radius);
285
  background: var(--paper);
286
  box-shadow: var(--shadow);
 
316
 
317
  .panel-heading {
318
  max-width: 1120px;
319
+ margin-bottom: 20px;
320
  }
321
 
322
  .section-eyebrow {
 
350
 
351
  h2 {
352
  margin-bottom: 7px;
353
+ font-size: 22px;
354
  font-weight: 750;
355
  line-height: 1.2;
356
  letter-spacing: -0.018em;
 
366
 
367
  .metric-strip {
368
  display: grid;
369
+ grid-template-columns: repeat(3, minmax(180px, 1fr));
370
  gap: 11px;
371
  margin-bottom: 14px;
372
  }
 
439
  line-height: 1.4;
440
  }
441
 
 
 
442
  .method-note {
443
  border-radius: 8px;
444
  font-size: 12px;
445
  line-height: 1.5;
446
  }
447
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
448
  .method-note {
449
  margin: 12px 0;
450
  padding: 11px 13px;
 
660
  }
661
 
662
  .provenance-footer {
663
+ display: flex;
664
+ align-items: center;
665
+ justify-content: space-between;
666
+ gap: 18px;
667
  margin-top: 18px;
668
+ padding: 18px 2px 4px;
669
+ border-top: 1px solid var(--line);
670
+ color: #62727e;
671
+ font-size: 12px;
 
 
672
  line-height: 1.55;
673
  }
674
 
675
+ .provenance-footer a {
676
+ font-weight: 700;
677
+ text-decoration: none;
678
+ }
679
+
680
  [data-dash-is-loading="true"] {
681
  opacity: 0.56;
682
  transition: opacity 140ms ease;
683
  }
684
 
685
  @media (max-width: 1180px) {
 
 
 
 
 
 
 
 
 
686
  .metric-strip {
687
  grid-template-columns: repeat(2, minmax(180px, 1fr));
688
  }
 
694
 
695
  @media (max-width: 820px) {
696
  .app-shell {
697
+ padding: 0 14px 24px;
698
  }
699
 
700
  .hero {
701
+ padding-bottom: 25px;
702
+ }
703
+
704
+ .site-nav {
705
+ min-height: 62px;
706
+ }
707
+
708
+ .hero-copy {
709
+ padding-top: 34px;
710
+ }
711
+
712
+ .hero-links {
713
+ gap: 14px;
714
+ }
715
+
716
+ .nav-placeholder small {
717
+ display: none;
718
  }
719
 
720
  .toolbar {
 
740
  .inline-controls {
741
  grid-template-columns: 1fr;
742
  }
743
+
744
+ .heatmap-graph,
745
+ .latent-graph {
746
+ overflow-x: auto;
747
+ }
748
+
749
+ .heatmap-graph > div,
750
+ .latent-graph > div {
751
+ min-width: 660px;
752
+ }
753
  }
754
 
755
  @media (max-width: 560px) {
756
+ .site-nav {
757
+ align-items: flex-start;
758
+ padding: 15px 0;
759
+ }
760
+
761
+ .site-brand {
762
+ padding-top: 1px;
763
+ }
764
+
765
+ .hero-links {
766
+ flex-direction: column;
767
+ gap: 4px;
768
+ align-items: flex-end;
769
+ }
770
+
771
+ .hero-links a,
772
+ .nav-placeholder {
773
+ font-size: 11px;
774
+ }
775
+
776
  h1 {
777
+ font-size: 32px;
778
  }
779
 
780
  .lede {
781
  font-size: 14px;
782
  }
783
 
 
 
 
 
784
  .metric-strip {
785
  grid-template-columns: 1fr;
786
  }
 
837
 
838
  .toolbar,
839
  .tabs,
840
+ .source-link {
 
841
  display: none !important;
842
  }
843
 
844
  .hero {
845
  padding: 18px 0;
 
846
  background: #ffffff;
 
847
  color: #000000;
848
  }
849