josephsoo commited on
Commit
77d097c
·
1 Parent(s): 5acd535

Align Space with manuscript figures

Browse files
Dockerfile CHANGED
@@ -4,6 +4,8 @@ COPY --from=ghcr.io/astral-sh/uv:0.4.20 /uv /bin/uv
4
  RUN useradd -m -u 1000 user
5
  ENV PATH="/home/user/.local/bin:$PATH"
6
  ENV UV_SYSTEM_PYTHON=1
 
 
7
 
8
  WORKDIR /app
9
 
@@ -13,4 +15,4 @@ RUN uv pip install -r requirements.txt
13
  COPY --chown=user . /app
14
  USER user
15
 
16
- CMD ["gunicorn", "app:server", "--workers", "4", "--bind", "0.0.0.0:7860"]
 
4
  RUN useradd -m -u 1000 user
5
  ENV PATH="/home/user/.local/bin:$PATH"
6
  ENV UV_SYSTEM_PYTHON=1
7
+ ENV PYTHONDONTWRITEBYTECODE=1
8
+ ENV PYTHONUNBUFFERED=1
9
 
10
  WORKDIR /app
11
 
 
15
  COPY --chown=user . /app
16
  USER user
17
 
18
+ CMD ["gunicorn", "app:server", "--workers", "2", "--threads", "2", "--timeout", "120", "--bind", "0.0.0.0:7860"]
README.md CHANGED
@@ -1,17 +1,69 @@
1
  ---
2
- title: Neural Model Benchmark
3
  sdk: docker
4
  app_port: 7860
 
5
  ---
6
 
7
- # Neural Model Benchmark
8
 
9
- Interactive dashboard for the Tang Lab neural model benchmark.
 
 
10
 
11
- The dashboard compares 23 methods across five datasets, with sortable decoding
12
- performance, robustness, cross-session alignment, neuron and trial influence,
13
- compute cost, and 3D latent-space views.
14
 
15
- The bundled files in `data/` are lightweight summary tables for interactive
16
- inspection. The full benchmark artifacts and reproduction workflow live in the
17
- main benchmark repository.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: BEND-BCI Interactive Benchmark
3
  sdk: docker
4
  app_port: 7860
5
+ short_description: Interactive BEND-BCI neural decoder benchmark
6
  ---
7
 
8
+ # BEND-BCI Interactive Benchmark
9
 
10
+ This Space is the interactive companion to BEND-BCI (Benchmarking Neural
11
+ Decoders for Brain-Computer Interfaces). It presents 23 evaluated methods on
12
+ five primary tasks.
13
 
14
+ The application keeps the benchmark's scientific organization explicit:
 
 
15
 
16
+ - decoder-selection measurements: held-out task prediction, robustness to
17
+ added count noise, computational cost, and cross-recording latent
18
+ consistency;
19
+ - diagnostic assays: feature attribution and trial-level data valuation.
20
+
21
+ Interactive matrices use within-dataset percentile ranks, with raw values and
22
+ task-specific metrics retained on hover. Unavailable analyses remain visible
23
+ as coverage limits and are not imputed.
24
+
25
+ ## Data and figure provenance
26
+
27
+ The bundled tables in `data/` are synchronized from the canonical
28
+ `paper/results/*.csv` exports in the
29
+ [main BEND-BCI repository](https://github.com/TangLab-UBC/behavior_benchmarking).
30
+ Each source table retains artifact paths for provenance.
31
+
32
+ The interactive sections follow the current generated manuscript figures:
33
+
34
+ - Figure 2: prediction, robustness, and computational cost;
35
+ - Figure 3: cross-recording latent consistency;
36
+ - Figure 4: feature-attribution validation;
37
+ - Figure 5: Data Shapley trial valuation, retraining case studies, and the
38
+ held-out RNN historical-selection trajectory example.
39
+
40
+ Latent display coordinates are exported after the same per-session whitening
41
+ and matched-landmark display alignment used for Figure 3. The reported
42
+ consistency value is symmetric alignment R²; it does not measure decoder
43
+ transfer or identify a unique latent coordinate system.
44
+
45
+ Feature and trial values remain signed. Feature-attribution validation is
46
+ dataset-specific: Allen Neuropixels uses Spearman correlation with measured
47
+ orientation selectivity, while the other primary tasks use their predefined
48
+ ROC-AUC validation assays. Trial valuation reports corrupted-trial detection
49
+ ROC-AUC using negative signed trial value as the detection score.
50
+
51
+ ## Local validation and launch
52
+
53
+ ```bash
54
+ python -m venv .venv
55
+ .venv/bin/pip install -r requirements.txt
56
+ .venv/bin/python validate_data.py
57
+ .venv/bin/python app.py
58
+ ```
59
+
60
+ From a checkout nested inside the main benchmark repository, exact equality
61
+ with the manuscript-facing tables can also be checked with:
62
+
63
+ ```bash
64
+ .venv/bin/python validate_data.py --canonical-root ..
65
+ ```
66
+
67
+ The complete artifacts, dataset preparation, benchmark execution, figure
68
+ builders, and result-export workflow live in the main repository. The Space
69
+ contains lightweight summaries intended for interactive inspection.
app.py CHANGED
The diff for this file is too large to render. See raw diff
 
assets/styles.css CHANGED
@@ -1,280 +1,640 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  * {
2
  box-sizing: border-box;
3
  }
4
 
 
 
 
 
5
  body {
6
  margin: 0;
7
- background: #f4f6f7;
8
- color: #17202a;
9
- font-family: Inter, Arial, sans-serif;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  }
11
 
12
  .app-shell {
13
  min-height: 100vh;
14
- padding: 18px 22px 28px;
15
  }
16
 
17
  .hero,
18
  .toolbar,
19
- .tabs {
20
- max-width: 1560px;
21
- margin-left: auto;
22
  margin-right: auto;
 
23
  }
24
 
25
  .hero {
26
- padding: 8px 2px 14px;
27
- border-bottom: 1px solid #d8e1e7;
28
- }
29
-
30
- .hero-copy {
31
- min-width: 0;
32
- }
33
-
34
- .eyebrow {
35
- margin-bottom: 7px;
36
- color: #526171;
37
- font-size: 12px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  font-weight: 800;
39
- letter-spacing: 0;
40
  text-transform: uppercase;
41
  }
42
 
43
- h1,
44
- h2,
45
- h3,
46
- p {
47
- margin-top: 0;
48
  }
49
 
50
  h1 {
51
- margin-bottom: 8px;
52
- font-size: 40px;
53
- line-height: 1.05;
54
- letter-spacing: 0;
55
- }
56
-
57
- h2 {
58
- margin-bottom: 5px;
59
- font-size: 20px;
60
- letter-spacing: 0;
61
  }
62
 
63
  .lede {
64
- max-width: 840px;
65
- margin-bottom: 0;
66
- color: #526171;
67
- font-size: 15px;
68
- line-height: 1.45;
69
  }
70
 
71
- .top-strip {
72
  display: flex;
73
- gap: 10px;
74
- align-items: stretch;
75
  flex-wrap: wrap;
76
  }
77
 
78
- .metric-card {
79
- min-width: 145px;
80
- padding: 12px 14px;
81
- border: 1px solid #d8e1e7;
82
- border-radius: 8px;
83
- background: #ffffff;
84
  }
85
 
86
- .metric-label {
87
- color: #607080;
 
88
  font-size: 11px;
89
- font-weight: 800;
90
- letter-spacing: 0;
91
- text-transform: uppercase;
92
  }
93
 
94
- .metric-value {
95
- margin-top: 7px;
96
- color: #101820;
97
- font-size: 22px;
 
 
 
 
 
 
 
 
 
98
  font-weight: 800;
99
- line-height: 1.1;
100
  }
101
 
102
- .metric-detail {
103
- margin-top: 6px;
104
- color: #647383;
105
  font-size: 12px;
106
- line-height: 1.35;
107
  }
108
 
109
  .toolbar {
110
  position: sticky;
111
  top: 0;
112
- z-index: 10;
113
  display: grid;
114
- grid-template-columns: minmax(230px, 310px);
115
- gap: 16px;
116
- margin-top: 14px;
117
  margin-bottom: 14px;
118
- padding: 12px;
119
- border: 1px solid #d8e1e7;
120
- border-radius: 8px;
121
  background: rgba(255, 255, 255, 0.96);
122
- box-shadow: 0 8px 24px rgba(18, 28, 38, 0.06);
123
- backdrop-filter: blur(6px);
 
 
 
 
124
  }
125
 
126
  .control label {
127
  display: block;
128
  margin-bottom: 6px;
129
  color: #344452;
130
- font-size: 12px;
131
  font-weight: 800;
132
- letter-spacing: 0;
133
  text-transform: uppercase;
134
  }
135
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
  .Select--multi .Select-value {
137
- margin-top: 3px;
 
 
 
 
138
  }
139
 
140
  .Select-multi-value-wrapper {
141
- max-height: 86px;
142
  overflow-y: auto;
143
  }
144
 
145
  .tabs {
146
- border: 0;
 
 
 
 
 
 
 
 
 
147
  }
148
 
149
  .tab {
150
  display: flex !important;
 
 
151
  align-items: center !important;
152
  justify-content: center !important;
153
- min-height: 44px;
154
- padding: 11px 18px !important;
155
  border: 0 !important;
156
- border-bottom: 2px solid transparent !important;
157
- background: transparent !important;
158
- color: #637381 !important;
159
- font-size: 14px;
 
160
  font-weight: 700;
 
 
 
 
 
 
 
 
 
 
161
  }
162
 
163
  .tab-selected {
164
- color: #12302f !important;
165
- border-bottom-color: #1565C0 !important;
166
- background: #ffffff !important;
167
  }
168
 
169
  .panel {
170
- margin-top: 14px;
171
- padding: 18px;
172
- border: 1px solid #d8e1e7;
173
- border-radius: 8px;
174
- background: #ffffff;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
175
  }
176
 
177
  .panel-heading {
178
- margin-bottom: 14px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
179
  }
180
 
181
  .panel-subtitle {
182
- max-width: 980px;
183
  margin-bottom: 0;
184
- color: #526171;
185
  font-size: 13px;
186
- line-height: 1.45;
187
  }
188
 
189
- .leaderboard-panel {
190
- padding-top: 16px;
 
 
 
191
  }
192
 
193
- .top-strip {
194
- margin-bottom: 14px;
 
 
 
 
 
 
195
  }
196
 
197
- .leaderboard-grid {
198
- display: grid;
199
- grid-template-columns: minmax(560px, 1.08fr) minmax(360px, 0.92fr);
200
- gap: 16px;
201
- align-items: start;
 
 
 
202
  }
203
 
204
- .leaderboard-table-wrap,
205
- .consistency-table-wrap {
206
- min-width: 0;
 
 
 
 
 
 
 
 
 
 
 
207
  }
208
 
209
- .sort-state {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
210
  display: inline-flex;
211
  align-items: center;
212
- min-height: 30px;
213
- margin: 0 0 8px;
214
- padding: 6px 10px;
215
- border: 1px solid #cfd8df;
216
- border-radius: 6px;
217
  background: #f7fafb;
218
- color: #344452;
219
- font-size: 12px;
220
- font-weight: 800;
221
- line-height: 1.2;
222
  }
223
 
224
- .latent-grid {
225
- display: grid;
226
- grid-template-columns: minmax(360px, 0.42fr) minmax(620px, 1fr);
227
- gap: 16px;
228
- align-items: start;
 
 
229
  }
230
 
231
- .latent-panel {
232
- min-width: 0;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
233
  }
234
 
235
- .latent-color-control {
236
- width: min(260px, 100%);
237
- margin-bottom: 8px;
238
  }
239
 
240
  .chart-grid {
241
  display: grid;
242
- gap: 16px;
243
  align-items: start;
244
  }
245
 
246
  .chart-grid.two {
247
- grid-template-columns: repeat(2, minmax(320px, 1fr));
248
  }
249
 
250
- .table-hint {
251
- margin: 10px 2px 0;
252
- color: #647383;
253
- font-size: 12px;
 
 
254
  }
255
 
256
- .details-table {
 
 
 
 
 
 
257
  margin-top: 12px;
258
- border-top: 1px solid #e5ebef;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
259
  }
260
 
261
  .details-table summary {
262
  cursor: pointer;
263
- padding: 12px 0 8px;
264
- color: #344452;
265
- font-size: 13px;
266
  font-weight: 800;
 
 
 
 
 
 
 
 
 
267
  }
268
 
269
  .details-body {
270
- padding-top: 4px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
271
  }
272
 
273
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th {
274
- background: #f3f6f8 !important;
275
  padding: 0 !important;
276
  border-right: 1px solid #d8e1e7 !important;
277
- overflow: visible !important;
278
  }
279
 
280
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th:last-child {
@@ -283,84 +643,231 @@ h2 {
283
 
284
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th.dash-header > div {
285
  display: flex !important;
 
286
  align-items: center !important;
287
  justify-content: flex-start !important;
288
- gap: 8px !important;
289
- min-height: 43px;
290
- padding: 10px 12px !important;
291
  }
292
 
293
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-header-name {
294
  flex: 1 1 auto !important;
295
  min-width: 0;
296
- color: #26323f;
297
- line-height: 1.15;
298
  white-space: normal;
299
  }
300
 
301
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-actions {
302
  order: -1;
303
- flex: 0 0 auto !important;
304
  display: inline-flex !important;
 
305
  align-items: center;
306
- justify-content: center;
307
  }
308
 
309
- .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-header--sort {
 
310
  display: inline-flex !important;
311
  align-items: center;
312
  justify-content: center;
313
- width: 22px;
314
- height: 22px;
315
  border: 1px solid #c7d3dc;
316
- border-radius: 5px;
317
  background: #ffffff;
318
  color: #526171 !important;
319
- font-size: 13px;
320
- font-weight: 900;
321
- line-height: 1;
322
  opacity: 1 !important;
323
  }
324
 
325
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th:hover .column-header--sort {
326
- border-color: #1565C0;
327
  background: #edf6ff;
328
- color: #1565C0 !important;
 
 
 
 
 
 
 
 
 
 
329
  }
330
 
331
  .dash-table-container .previous-next-container {
332
  margin-top: 8px;
333
  color: #526171;
334
- font-size: 12px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
335
  }
336
 
337
  @media (max-width: 1180px) {
338
- .hero,
339
- .toolbar,
340
- .leaderboard-grid,
341
- .latent-grid,
342
- .chart-grid.two {
343
  grid-template-columns: 1fr;
 
 
 
 
 
 
 
 
 
344
  }
345
 
 
 
 
346
  }
347
 
348
- @media (max-width: 680px) {
349
  .app-shell {
 
 
 
 
 
 
 
 
 
 
 
 
350
  padding: 12px;
351
  }
352
 
353
- h1 {
354
- font-size: 32px;
 
 
 
 
 
355
  }
356
 
357
  .panel {
358
- padding: 14px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
359
  }
360
 
361
  .tab {
362
- padding-left: 10px !important;
363
- padding-right: 10px !important;
364
- font-size: 13px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
365
  }
366
  }
 
1
+ :root {
2
+ --ink: #17202a;
3
+ --muted: #607080;
4
+ --line: #d8e1e7;
5
+ --soft-line: #e9eef2;
6
+ --paper: #ffffff;
7
+ --canvas: #f4f6f8;
8
+ --prediction: #1565c0;
9
+ --robustness: #2e7d32;
10
+ --compute: #e65100;
11
+ --consistency: #007c91;
12
+ --feature: #6a51a3;
13
+ --trial: #6a1b9a;
14
+ --shadow: 0 12px 34px rgba(24, 43, 59, 0.075);
15
+ --radius: 12px;
16
+ }
17
+
18
  * {
19
  box-sizing: border-box;
20
  }
21
 
22
+ html {
23
+ scroll-behavior: smooth;
24
+ }
25
+
26
  body {
27
  margin: 0;
28
+ background: var(--canvas);
29
+ color: var(--ink);
30
+ font-family: Arial, Helvetica, sans-serif;
31
+ font-size: 15px;
32
+ line-height: 1.5;
33
+ text-rendering: optimizeLegibility;
34
+ -webkit-font-smoothing: antialiased;
35
+ }
36
+
37
+ a {
38
+ color: #0b5da8;
39
+ text-underline-offset: 3px;
40
+ }
41
+
42
+ a:hover {
43
+ color: #083e70;
44
+ }
45
+
46
+ :focus-visible {
47
+ outline: 3px solid rgba(21, 101, 192, 0.34) !important;
48
+ outline-offset: 2px !important;
49
+ }
50
+
51
+ h1,
52
+ h2,
53
+ h3,
54
+ p {
55
+ margin-top: 0;
56
  }
57
 
58
  .app-shell {
59
  min-height: 100vh;
60
+ padding: 20px 24px 34px;
61
  }
62
 
63
  .hero,
64
  .toolbar,
65
+ .tabs,
66
+ .provenance-footer {
67
+ width: min(100%, 1540px);
68
  margin-right: auto;
69
+ margin-left: auto;
70
  }
71
 
72
  .hero {
73
+ position: relative;
74
+ display: grid;
75
+ grid-template-columns: minmax(0, 1fr) minmax(210px, 280px);
76
+ gap: 36px;
77
+ align-items: end;
78
+ overflow: hidden;
79
+ padding: 36px 40px 34px;
80
+ border: 1px solid #173f59;
81
+ border-radius: 16px;
82
+ background:
83
+ radial-gradient(circle at 92% 12%, rgba(86, 180, 233, 0.18), transparent 34%),
84
+ linear-gradient(130deg, #10283a 0%, #12364b 63%, #154960 100%);
85
+ box-shadow: 0 18px 44px rgba(15, 39, 56, 0.16);
86
+ color: #ffffff;
87
+ }
88
+
89
+ .hero::after {
90
+ position: absolute;
91
+ right: -68px;
92
+ bottom: -104px;
93
+ width: 310px;
94
+ height: 310px;
95
+ border: 1px solid rgba(255, 255, 255, 0.09);
96
+ border-radius: 50%;
97
+ content: "";
98
+ }
99
+
100
+ .hero-copy,
101
+ .hero-stat {
102
+ position: relative;
103
+ z-index: 1;
104
+ }
105
+
106
+ .eyebrow,
107
+ .section-eyebrow {
108
+ font-size: 11px;
109
  font-weight: 800;
110
+ letter-spacing: 0.105em;
111
  text-transform: uppercase;
112
  }
113
 
114
+ .eyebrow {
115
+ margin-bottom: 10px;
116
+ color: #b9d7e7;
 
 
117
  }
118
 
119
  h1 {
120
+ max-width: 900px;
121
+ margin-bottom: 12px;
122
+ font-size: clamp(34px, 4vw, 51px);
123
+ font-weight: 750;
124
+ line-height: 1.04;
125
+ letter-spacing: -0.035em;
 
 
 
 
126
  }
127
 
128
  .lede {
129
+ max-width: 960px;
130
+ margin-bottom: 18px;
131
+ color: #d6e4ec;
132
+ font-size: 16px;
133
+ line-height: 1.55;
134
  }
135
 
136
+ .hero-links {
137
  display: flex;
138
+ gap: 18px;
 
139
  flex-wrap: wrap;
140
  }
141
 
142
+ .hero-links a {
143
+ color: #ffffff;
144
+ font-size: 13px;
145
+ font-weight: 700;
146
+ text-decoration-color: rgba(255, 255, 255, 0.5);
 
147
  }
148
 
149
+ .hero-links a::after {
150
+ margin-left: 5px;
151
+ content: "↗";
152
  font-size: 11px;
 
 
 
153
  }
154
 
155
+ .hero-stat {
156
+ display: flex;
157
+ flex-direction: column;
158
+ gap: 4px;
159
+ padding: 18px 20px;
160
+ border: 1px solid rgba(255, 255, 255, 0.18);
161
+ border-radius: 12px;
162
+ background: rgba(255, 255, 255, 0.075);
163
+ backdrop-filter: blur(8px);
164
+ }
165
+
166
+ .hero-stat-value {
167
+ font-size: 18px;
168
  font-weight: 800;
 
169
  }
170
 
171
+ .hero-stat-label {
172
+ color: #c9dce7;
 
173
  font-size: 12px;
174
+ line-height: 1.4;
175
  }
176
 
177
  .toolbar {
178
  position: sticky;
179
  top: 0;
180
+ z-index: 20;
181
  display: grid;
182
+ grid-template-columns: minmax(300px, 0.72fr) minmax(420px, 1.28fr);
183
+ gap: 18px;
184
+ margin-top: 16px;
185
  margin-bottom: 14px;
186
+ padding: 13px 16px 14px;
187
+ border: 1px solid var(--line);
188
+ border-radius: var(--radius);
189
  background: rgba(255, 255, 255, 0.96);
190
+ box-shadow: 0 10px 28px rgba(18, 38, 53, 0.09);
191
+ backdrop-filter: blur(10px);
192
+ }
193
+
194
+ .control {
195
+ min-width: 0;
196
  }
197
 
198
  .control label {
199
  display: block;
200
  margin-bottom: 6px;
201
  color: #344452;
202
+ font-size: 11px;
203
  font-weight: 800;
204
+ letter-spacing: 0.075em;
205
  text-transform: uppercase;
206
  }
207
 
208
+ .control-help {
209
+ margin: 5px 1px 0;
210
+ color: #70808e;
211
+ font-size: 11px;
212
+ line-height: 1.25;
213
+ }
214
+
215
+ .Select-control,
216
+ .select__control {
217
+ min-height: 39px;
218
+ border-color: #cbd5dd !important;
219
+ border-radius: 7px !important;
220
+ box-shadow: none !important;
221
+ }
222
+
223
+ .is-focused:not(.is-open) > .Select-control {
224
+ border-color: var(--prediction) !important;
225
+ box-shadow: 0 0 0 3px rgba(21, 101, 192, 0.16) !important;
226
+ }
227
+
228
+ .Select-placeholder,
229
+ .Select-value-label,
230
+ .Select-input,
231
+ .Select-option {
232
+ font-size: 13px !important;
233
+ }
234
+
235
  .Select--multi .Select-value {
236
+ margin-top: 4px;
237
+ border-color: #c9d9e7;
238
+ border-radius: 5px;
239
+ background: #edf5fb;
240
+ color: #154f7b;
241
  }
242
 
243
  .Select-multi-value-wrapper {
244
+ max-height: 82px;
245
  overflow-y: auto;
246
  }
247
 
248
  .tabs {
249
+ display: flex;
250
+ flex-direction: row !important;
251
+ flex-wrap: nowrap;
252
+ overflow-x: auto;
253
+ border: 1px solid var(--line);
254
+ border-radius: 11px;
255
+ background: #ffffff;
256
+ box-shadow: 0 5px 16px rgba(24, 43, 59, 0.045);
257
+ scrollbar-width: thin;
258
+ -webkit-overflow-scrolling: touch;
259
  }
260
 
261
  .tab {
262
  display: flex !important;
263
+ flex: 1 0 auto !important;
264
+ width: auto !important;
265
  align-items: center !important;
266
  justify-content: center !important;
267
+ min-height: 50px;
268
+ padding: 12px 20px !important;
269
  border: 0 !important;
270
+ border-right: 1px solid #edf1f4 !important;
271
+ border-bottom: 3px solid transparent !important;
272
+ background: #ffffff !important;
273
+ color: #61717f !important;
274
+ font-size: 13px;
275
  font-weight: 700;
276
+ white-space: nowrap;
277
+ }
278
+
279
+ .tab:last-child {
280
+ border-right: 0 !important;
281
+ }
282
+
283
+ .tab:hover {
284
+ background: #f8fafb !important;
285
+ color: #213442 !important;
286
  }
287
 
288
  .tab-selected {
289
+ border-bottom-color: var(--prediction) !important;
290
+ background: #f7fafc !important;
291
+ color: #102d41 !important;
292
  }
293
 
294
  .panel {
295
+ position: relative;
296
+ margin-top: 16px;
297
+ overflow: hidden;
298
+ padding: 24px 26px 26px;
299
+ border: 1px solid var(--line);
300
+ border-top: 4px solid #71808d;
301
+ border-radius: var(--radius);
302
+ background: var(--paper);
303
+ box-shadow: var(--shadow);
304
+ }
305
+
306
+ .panel.axis-prediction {
307
+ border-top-color: var(--prediction);
308
+ }
309
+
310
+ .panel.axis-robustness {
311
+ border-top-color: var(--robustness);
312
+ }
313
+
314
+ .panel.axis-compute {
315
+ border-top-color: var(--compute);
316
+ }
317
+
318
+ .panel.axis-consistency {
319
+ border-top-color: var(--consistency);
320
+ }
321
+
322
+ .panel.axis-feature {
323
+ border-top-color: var(--feature);
324
+ }
325
+
326
+ .panel.axis-trial {
327
+ border-top-color: var(--trial);
328
+ }
329
+
330
+ .panel.axis-methods {
331
+ border-top-color: #455a64;
332
  }
333
 
334
  .panel-heading {
335
+ max-width: 1120px;
336
+ margin-bottom: 18px;
337
+ }
338
+
339
+ .section-eyebrow {
340
+ margin-bottom: 5px;
341
+ color: var(--muted);
342
+ }
343
+
344
+ .axis-prediction .section-eyebrow {
345
+ color: var(--prediction);
346
+ }
347
+
348
+ .axis-robustness .section-eyebrow {
349
+ color: var(--robustness);
350
+ }
351
+
352
+ .axis-compute .section-eyebrow {
353
+ color: var(--compute);
354
+ }
355
+
356
+ .axis-consistency .section-eyebrow {
357
+ color: var(--consistency);
358
+ }
359
+
360
+ .axis-feature .section-eyebrow {
361
+ color: var(--feature);
362
+ }
363
+
364
+ .axis-trial .section-eyebrow {
365
+ color: var(--trial);
366
+ }
367
+
368
+ h2 {
369
+ margin-bottom: 7px;
370
+ font-size: 23px;
371
+ font-weight: 750;
372
+ line-height: 1.2;
373
+ letter-spacing: -0.018em;
374
  }
375
 
376
  .panel-subtitle {
377
+ max-width: 1080px;
378
  margin-bottom: 0;
379
+ color: #526473;
380
  font-size: 13px;
381
+ line-height: 1.55;
382
  }
383
 
384
+ .metric-strip {
385
+ display: grid;
386
+ grid-template-columns: repeat(4, minmax(180px, 1fr));
387
+ gap: 11px;
388
+ margin-bottom: 14px;
389
  }
390
 
391
+ .metric-card {
392
+ position: relative;
393
+ min-width: 0;
394
+ overflow: hidden;
395
+ padding: 15px 16px 14px;
396
+ border: 1px solid #dce4e9;
397
+ border-radius: 9px;
398
+ background: linear-gradient(180deg, #ffffff 0%, #fbfcfd 100%);
399
  }
400
 
401
+ .metric-card::before {
402
+ position: absolute;
403
+ top: 0;
404
+ right: 0;
405
+ left: 0;
406
+ height: 3px;
407
+ background: #7b8995;
408
+ content: "";
409
  }
410
 
411
+ .metric-card-prediction::before {
412
+ background: var(--prediction);
413
+ }
414
+
415
+ .metric-card-robustness::before {
416
+ background: var(--robustness);
417
+ }
418
+
419
+ .metric-card-compute::before {
420
+ background: var(--compute);
421
+ }
422
+
423
+ .metric-card-consistency::before {
424
+ background: var(--consistency);
425
  }
426
 
427
+ .metric-card-feature::before {
428
+ background: var(--feature);
429
+ }
430
+
431
+ .metric-card-trial::before {
432
+ background: var(--trial);
433
+ }
434
+
435
+ .metric-label {
436
+ color: #657582;
437
+ font-size: 10px;
438
+ font-weight: 800;
439
+ letter-spacing: 0.075em;
440
+ text-transform: uppercase;
441
+ }
442
+
443
+ .metric-value {
444
+ margin-top: 7px;
445
+ overflow-wrap: anywhere;
446
+ color: #101820;
447
+ font-size: 20px;
448
+ font-weight: 800;
449
+ line-height: 1.15;
450
+ }
451
+
452
+ .metric-detail {
453
+ margin-top: 6px;
454
+ color: #647583;
455
+ font-size: 11px;
456
+ line-height: 1.4;
457
+ }
458
+
459
+ .coverage-note,
460
+ .convergence-note,
461
+ .method-note {
462
+ border-radius: 8px;
463
+ font-size: 12px;
464
+ line-height: 1.5;
465
+ }
466
+
467
+ .coverage-note {
468
  display: inline-flex;
469
  align-items: center;
470
+ min-height: 32px;
471
+ margin-bottom: 10px;
472
+ padding: 6px 11px;
473
+ border: 1px solid #d5e0e7;
 
474
  background: #f7fafb;
475
+ color: #3f5260;
476
+ font-weight: 700;
 
 
477
  }
478
 
479
+ .coverage-note::before {
480
+ width: 7px;
481
+ height: 7px;
482
+ margin-right: 7px;
483
+ border-radius: 50%;
484
+ background: #5f7585;
485
+ content: "";
486
  }
487
 
488
+ .convergence-note {
489
+ margin: 0 0 12px;
490
+ padding: 9px 12px;
491
+ border: 1px solid #e4d5eb;
492
+ background: #faf7fc;
493
+ color: #5b4169;
494
+ font-weight: 700;
495
+ }
496
+
497
+ .method-note {
498
+ margin: 12px 0;
499
+ padding: 11px 13px;
500
+ border-left: 3px solid #6f8190;
501
+ background: #f6f8fa;
502
+ color: #445664;
503
+ }
504
+
505
+ .axis-consistency .method-note {
506
+ border-left-color: var(--consistency);
507
+ }
508
+
509
+ .axis-feature .method-note {
510
+ border-left-color: var(--feature);
511
  }
512
 
513
+ .axis-trial .method-note {
514
+ border-left-color: var(--trial);
 
515
  }
516
 
517
  .chart-grid {
518
  display: grid;
519
+ gap: 18px;
520
  align-items: start;
521
  }
522
 
523
  .chart-grid.two {
524
+ grid-template-columns: repeat(2, minmax(0, 1fr));
525
  }
526
 
527
+ .graph-box {
528
+ min-width: 0;
529
+ overflow: hidden;
530
+ border: 1px solid #e3e9ed;
531
+ border-radius: 9px;
532
+ background: #ffffff;
533
  }
534
 
535
+ .graph-box + .graph-box,
536
+ .chart-grid + .graph-box,
537
+ .graph-box + .chart-grid {
538
+ margin-top: 0;
539
+ }
540
+
541
+ .panel > .graph-box {
542
  margin-top: 12px;
543
+ }
544
+
545
+ .chart-grid > .graph-box {
546
+ margin-top: 0;
547
+ }
548
+
549
+ .latent-graph {
550
+ margin-top: 14px !important;
551
+ }
552
+
553
+ .inline-controls {
554
+ display: grid;
555
+ grid-template-columns: minmax(240px, 420px) minmax(200px, 300px);
556
+ gap: 14px;
557
+ align-items: end;
558
+ margin-bottom: 12px;
559
+ padding: 13px 14px;
560
+ border: 1px solid #e0e6ea;
561
+ border-radius: 9px;
562
+ background: #fafbfc;
563
+ }
564
+
565
+ .details-table {
566
+ margin-top: 15px;
567
+ border: 1px solid #e1e7eb;
568
+ border-radius: 8px;
569
+ background: #ffffff;
570
  }
571
 
572
  .details-table summary {
573
  cursor: pointer;
574
+ padding: 11px 14px;
575
+ color: #344a59;
576
+ font-size: 12px;
577
  font-weight: 800;
578
+ user-select: none;
579
+ }
580
+
581
+ .details-table summary:hover {
582
+ background: #f7f9fa;
583
+ }
584
+
585
+ .details-table[open] summary {
586
+ border-bottom: 1px solid #e5eaee;
587
  }
588
 
589
  .details-body {
590
+ padding: 9px;
591
+ }
592
+
593
+ .source-link {
594
+ display: inline-flex;
595
+ align-items: center;
596
+ gap: 5px;
597
+ margin-top: 12px;
598
+ padding: 6px 10px;
599
+ border: 1px solid #cdd9e1;
600
+ border-radius: 6px;
601
+ background: #ffffff;
602
+ color: #355b77;
603
+ font-size: 11px;
604
+ font-weight: 750;
605
+ text-decoration: none;
606
+ }
607
+
608
+ .source-link::before {
609
+ content: "↓";
610
+ font-size: 13px;
611
+ }
612
+
613
+ .source-link:hover {
614
+ border-color: #93adbf;
615
+ background: #f4f8fa;
616
+ }
617
+
618
+ .download-grid {
619
+ display: flex;
620
+ gap: 8px;
621
+ flex-wrap: wrap;
622
+ margin-top: 4px;
623
+ }
624
+
625
+ .download-grid .source-link {
626
+ margin-top: 0;
627
+ }
628
+
629
+ .dash-table-container {
630
+ color: var(--ink);
631
  }
632
 
633
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th {
634
+ overflow: visible !important;
635
  padding: 0 !important;
636
  border-right: 1px solid #d8e1e7 !important;
637
+ background: #f3f6f8 !important;
638
  }
639
 
640
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th:last-child {
 
643
 
644
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th.dash-header > div {
645
  display: flex !important;
646
+ gap: 7px !important;
647
  align-items: center !important;
648
  justify-content: flex-start !important;
649
+ min-height: 42px;
650
+ padding: 9px 10px !important;
 
651
  }
652
 
653
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-header-name {
654
  flex: 1 1 auto !important;
655
  min-width: 0;
656
+ color: #263844;
657
+ line-height: 1.2;
658
  white-space: normal;
659
  }
660
 
661
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-actions {
662
  order: -1;
 
663
  display: inline-flex !important;
664
+ flex: 0 0 auto !important;
665
  align-items: center;
 
666
  }
667
 
668
+ .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-header--sort,
669
+ .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner .column-header--hide {
670
  display: inline-flex !important;
671
  align-items: center;
672
  justify-content: center;
673
+ width: 21px;
674
+ height: 21px;
675
  border: 1px solid #c7d3dc;
676
+ border-radius: 4px;
677
  background: #ffffff;
678
  color: #526171 !important;
679
+ font-size: 12px;
 
 
680
  opacity: 1 !important;
681
  }
682
 
683
  .dash-table-container .dash-spreadsheet-container .dash-spreadsheet-inner th:hover .column-header--sort {
684
+ border-color: var(--prediction);
685
  background: #edf6ff;
686
+ color: var(--prediction) !important;
687
+ }
688
+
689
+ .dash-table-container .dash-filter input {
690
+ min-height: 29px;
691
+ padding: 5px 7px !important;
692
+ border: 1px solid #d5dee4 !important;
693
+ border-radius: 4px;
694
+ background: #ffffff !important;
695
+ font-family: Arial, Helvetica, sans-serif !important;
696
+ font-size: 11px !important;
697
  }
698
 
699
  .dash-table-container .previous-next-container {
700
  margin-top: 8px;
701
  color: #526171;
702
+ font-size: 11px;
703
+ }
704
+
705
+ .dash-table-container .previous-next-container button {
706
+ border: 1px solid #d4dde3;
707
+ border-radius: 5px;
708
+ background: #ffffff;
709
+ }
710
+
711
+ .provenance-footer {
712
+ margin-top: 18px;
713
+ padding: 17px 20px;
714
+ border: 1px solid #d3dde4;
715
+ border-radius: 10px;
716
+ background: #edf2f5;
717
+ color: #4e606d;
718
+ font-size: 11px;
719
+ line-height: 1.55;
720
+ }
721
+
722
+ [data-dash-is-loading="true"] {
723
+ opacity: 0.56;
724
+ transition: opacity 140ms ease;
725
  }
726
 
727
  @media (max-width: 1180px) {
728
+ .hero {
 
 
 
 
729
  grid-template-columns: 1fr;
730
+ gap: 22px;
731
+ }
732
+
733
+ .hero-stat {
734
+ max-width: 420px;
735
+ }
736
+
737
+ .metric-strip {
738
+ grid-template-columns: repeat(2, minmax(180px, 1fr));
739
  }
740
 
741
+ .chart-grid.two {
742
+ grid-template-columns: 1fr;
743
+ }
744
  }
745
 
746
+ @media (max-width: 820px) {
747
  .app-shell {
748
+ padding: 12px 12px 24px;
749
+ }
750
+
751
+ .hero {
752
+ padding: 28px 24px;
753
+ border-radius: 13px;
754
+ }
755
+
756
+ .toolbar {
757
+ position: relative;
758
+ grid-template-columns: 1fr;
759
+ gap: 12px;
760
  padding: 12px;
761
  }
762
 
763
+ .tabs {
764
+ justify-content: flex-start;
765
+ }
766
+
767
+ .tab {
768
+ flex: 0 0 auto !important;
769
+ min-width: 148px;
770
  }
771
 
772
  .panel {
773
+ padding: 20px 17px;
774
+ }
775
+
776
+ .inline-controls {
777
+ grid-template-columns: 1fr;
778
+ }
779
+ }
780
+
781
+ @media (max-width: 560px) {
782
+ h1 {
783
+ font-size: 34px;
784
+ }
785
+
786
+ .lede {
787
+ font-size: 14px;
788
+ }
789
+
790
+ .hero-stat {
791
+ padding: 14px 15px;
792
+ }
793
+
794
+ .metric-strip {
795
+ grid-template-columns: 1fr;
796
+ }
797
+
798
+ .metric-card {
799
+ padding: 13px 14px;
800
+ }
801
+
802
+ h2 {
803
+ font-size: 20px;
804
+ }
805
+
806
+ .panel-subtitle,
807
+ .method-note {
808
+ font-size: 12px;
809
  }
810
 
811
  .tab {
812
+ min-width: 134px;
813
+ padding-right: 12px !important;
814
+ padding-left: 12px !important;
815
+ font-size: 12px;
816
+ }
817
+
818
+ .details-body {
819
+ padding: 4px;
820
+ }
821
+
822
+ .source-link {
823
+ width: 100%;
824
+ justify-content: center;
825
+ }
826
+ }
827
+
828
+ @media (prefers-reduced-motion: reduce) {
829
+ html {
830
+ scroll-behavior: auto;
831
+ }
832
+
833
+ *,
834
+ *::before,
835
+ *::after {
836
+ transition-duration: 0.01ms !important;
837
+ animation-duration: 0.01ms !important;
838
+ animation-iteration-count: 1 !important;
839
+ }
840
+ }
841
+
842
+ @media print {
843
+ body,
844
+ .app-shell {
845
+ background: #ffffff;
846
+ }
847
+
848
+ .toolbar,
849
+ .tabs,
850
+ .source-link,
851
+ .hero-stat {
852
+ display: none !important;
853
+ }
854
+
855
+ .hero {
856
+ padding: 18px 0;
857
+ border: 0;
858
+ background: #ffffff;
859
+ box-shadow: none;
860
+ color: #000000;
861
+ }
862
+
863
+ .eyebrow,
864
+ .lede,
865
+ .hero-links a {
866
+ color: #333333;
867
+ }
868
+
869
+ .panel {
870
+ break-inside: avoid;
871
+ box-shadow: none;
872
  }
873
  }
data/consistency_summary.csv CHANGED
@@ -35,7 +35,7 @@ marble,monkey,monkey,True,4,sub-C_ses-CO-20151104_behavior+ecephys;sub-J_ses-CO-
35
  marble,ratinabox,ratinabox,True,4,ratinabox_nav;ratinabox_nav_s123;ratinabox_nav_s456;ratinabox_nav_s789,3,0.5023447353328528,0.15119647919292642,12,direct_labels,per_session_whitening,100.0,100.0,benchmark_results/marble_cross_ratinabox_consistency.json
36
  marble,speech,speech_threshold_crossings,True,4,t12;t15;t16;t17,3,0.5995980059621582,0.4348672589667975,12,centroid,per_session_whitening,8.0,8.0,benchmark_results/marble_cross_speech_threshold_crossings_consistency.json
37
  neuro_behavior_conditioning,allen_neuropixels,allen_neuropixels,True,3,721123822;715093703;732592105,3,0.5341513273436873,0.3258140816630879,6,centroid,per_session_whitening,8.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_allen_neuropixels_consistency.json
38
- neuro_behavior_conditioning,monkey,monkey,True,4,sub-C_ses-CO-20151104_behavior+ecephys;sub-J_ses-CO-20160405_behavior+ecephys;sub-M_ses-CO-20140203_behavior+ecephys;sub-T_ses-CO-20130819_behavior+ecephys,3,0.187690323745175,-0.22244557265455436,12,position_binned,per_session_whitening,80.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_monkey_consistency.json
39
  neuro_behavior_conditioning,ratinabox,ratinabox,True,4,ratinabox_nav;ratinabox_nav_s123;ratinabox_nav_s456;ratinabox_nav_s789,3,0.12144475733839015,-0.38850488980683845,12,direct_labels,per_session_whitening,100.0,100.0,benchmark_results/neuro_behavior_conditioning_cross_ratinabox_consistency.json
40
  neuro_behavior_conditioning,speech,speech_threshold_crossings,True,4,t12;t15;t16;t17,3,0.4050875872751732,-30679.955330283454,12,centroid,per_session_whitening,8.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_speech_threshold_crossings_consistency.json
41
  pca,allen_neuropixels,allen_neuropixels,True,3,721123822;715093703;732592105,3,0.5668164187589818,0.3842947844321292,6,centroid,per_session_whitening,8.0,8.0,benchmark_results/pca_cross_allen_neuropixels_consistency.json
 
35
  marble,ratinabox,ratinabox,True,4,ratinabox_nav;ratinabox_nav_s123;ratinabox_nav_s456;ratinabox_nav_s789,3,0.5023447353328528,0.15119647919292642,12,direct_labels,per_session_whitening,100.0,100.0,benchmark_results/marble_cross_ratinabox_consistency.json
36
  marble,speech,speech_threshold_crossings,True,4,t12;t15;t16;t17,3,0.5995980059621582,0.4348672589667975,12,centroid,per_session_whitening,8.0,8.0,benchmark_results/marble_cross_speech_threshold_crossings_consistency.json
37
  neuro_behavior_conditioning,allen_neuropixels,allen_neuropixels,True,3,721123822;715093703;732592105,3,0.5341513273436873,0.3258140816630879,6,centroid,per_session_whitening,8.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_allen_neuropixels_consistency.json
38
+ neuro_behavior_conditioning,monkey,monkey,True,4,sub-C_ses-CO-20151104_behavior+ecephys;sub-J_ses-CO-20160405_behavior+ecephys;sub-M_ses-CO-20140203_behavior+ecephys;sub-T_ses-CO-20130819_behavior+ecephys,3,0.18909089587238706,-0.22025913182256643,12,position_binned,per_session_whitening,80.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_monkey_consistency.json
39
  neuro_behavior_conditioning,ratinabox,ratinabox,True,4,ratinabox_nav;ratinabox_nav_s123;ratinabox_nav_s456;ratinabox_nav_s789,3,0.12144475733839015,-0.38850488980683845,12,direct_labels,per_session_whitening,100.0,100.0,benchmark_results/neuro_behavior_conditioning_cross_ratinabox_consistency.json
40
  neuro_behavior_conditioning,speech,speech_threshold_crossings,True,4,t12;t15;t16;t17,3,0.4050875872751732,-30679.955330283454,12,centroid,per_session_whitening,8.0,8.0,benchmark_results/neuro_behavior_conditioning_cross_speech_threshold_crossings_consistency.json
41
  pca,allen_neuropixels,allen_neuropixels,True,3,721123822;715093703;732592105,3,0.5668164187589818,0.3842947844321292,6,centroid,per_session_whitening,8.0,8.0,benchmark_results/pca_cross_allen_neuropixels_consistency.json
data/latent_samples.csv CHANGED
The diff for this file is too large to render. See raw diff
 
data/latent_trajectories.csv CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:cf78532b6d37c6a05f3f93baa348f45908d1eb6d6072aec26657423a69f499b5
3
- size 13882186
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d15a19b50d7dbc517b4bf5de60a550fda6870327e5e59fb4a5ec37f01921abf5
3
+ size 16913534
data/release_manifest.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "figure_set": "Figures 2-5 and Supplementary Figures 1-27",
3
+ "latent_coordinate_spaces": [
4
+ "per_session_whitened_reference_aligned_3d"
5
+ ],
6
+ "manuscript_working_version": "manuscript_v7",
7
+ "schema_version": 1,
8
+ "source": "paper/results, active consistency embedding artifacts, and Figure 5 prediction sidecars",
9
+ "source_git_revision": "80def770609a52f45a835c6b62500b04cdc2b548",
10
+ "source_repository": "https://github.com/TangLab-UBC/behavior_benchmarking",
11
+ "source_worktree_dirty": true,
12
+ "tables": {
13
+ "clean_prediction_summary.csv": {
14
+ "rows": 115,
15
+ "sha256": "1be57a892d7033367897bb065ec75f7d6942d5b1a1970136f958df08cacb43df"
16
+ },
17
+ "consistency_summary.csv": {
18
+ "rows": 54,
19
+ "sha256": "7039c0d38c288576e39ea6a1147a41eb2d395559b1d9b1de1b14eac3b3801b7f"
20
+ },
21
+ "latent_samples.csv": {
22
+ "rows": 31140,
23
+ "sha256": "bb484c47b78bd7a7d0a2bfea94296e726aa80034f87368b32302aaba0c5aa622"
24
+ },
25
+ "latent_trajectories.csv": {
26
+ "rows": 75040,
27
+ "sha256": "d15a19b50d7dbc517b4bf5de60a550fda6870327e5e59fb4a5ec37f01921abf5"
28
+ },
29
+ "neuron_shap_summary.csv": {
30
+ "rows": 105,
31
+ "sha256": "a36430bdd193040c48e2ab9fd48570d9c37b9ad58025d8a02d2c84293c768389"
32
+ },
33
+ "robustness_summary.csv": {
34
+ "rows": 115,
35
+ "sha256": "6e67a53dd3f3629c1280609aaad3f3203a0c7ab4938bdedb05d0c207cfd6e98f"
36
+ },
37
+ "scalability_summary.csv": {
38
+ "rows": 115,
39
+ "sha256": "04838e17f653394e8d861b8698c66d907864bce44c3561343a8a6489021dbd23"
40
+ },
41
+ "trial_historical_trajectories.csv": {
42
+ "rows": 4725,
43
+ "sha256": "605373599d4bd46c1379ab99b791bd82060544843be8eb880a9f9bcb523d1cda"
44
+ },
45
+ "trial_shapley_retrain_summary.csv": {
46
+ "rows": 99,
47
+ "sha256": "069fcb01b5ebd3dd290553d9373a7c1a9841a390d08a30eb9cc024ed8eab7eb0"
48
+ },
49
+ "trial_shapley_summary.csv": {
50
+ "rows": 81,
51
+ "sha256": "bba3435afa6a0e2755d30199e995f53bfa33df8b7200b066ea9b575d6125133b"
52
+ }
53
+ }
54
+ }
data/trial_historical_trajectories.csv ADDED
The diff for this file is too large to render. See raw diff
 
requirements.txt CHANGED
@@ -1,5 +1,5 @@
1
- dash>=2.17,<5
2
- gunicorn>=22,<24
3
- numpy>=1.26,<3
4
- pandas>=2.1,<3
5
- plotly>=5.22,<7
 
1
+ dash==4.4.0
2
+ gunicorn==23.0.0
3
+ numpy==2.4.2
4
+ pandas==2.3.3
5
+ plotly==6.9.0
validate_data.py ADDED
@@ -0,0 +1,592 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Validate the bundled BEND-BCI Space tables and their paper provenance.
3
+
4
+ The local checks run in the standalone Hugging Face Space. Passing
5
+ ``--canonical-root`` additionally compares every manuscript-facing summary
6
+ against ``paper/results`` in the main benchmark repository.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ import argparse
12
+ import hashlib
13
+ import json
14
+ from pathlib import Path
15
+ from typing import Iterable
16
+
17
+ import numpy as np
18
+ import pandas as pd
19
+
20
+
21
+ DATASETS = {
22
+ "monkey",
23
+ "allen_neuropixels",
24
+ "speech",
25
+ "mc_pacman",
26
+ "ratinabox",
27
+ }
28
+ CONSISTENCY_DATASETS = DATASETS - {"mc_pacman"}
29
+ LATENT_COORDINATE_SPACE = "per_session_whitened_reference_aligned_3d"
30
+ FIGURE5_TARGET_SESSION = "sub-C_ses-CO-20150716_behavior+ecephys"
31
+
32
+ # Coverage is defined in manuscript v7 Supplementary Table 2.
33
+ EXPECTED_COVERAGE = {
34
+ "clean_prediction_summary.csv": (115, 112),
35
+ "robustness_summary.csv": (115, 112),
36
+ "scalability_summary.csv": (115, 112),
37
+ "consistency_summary.csv": (54, 46),
38
+ "neuron_shap_summary.csv": (105, 105),
39
+ "trial_shapley_summary.csv": (81, 81),
40
+ "trial_shapley_retrain_summary.csv": (99, 99),
41
+ }
42
+
43
+ REQUIRED_COLUMNS = {
44
+ "clean_prediction_summary.csv": {
45
+ "model", "dataset", "status", "metric", "score", "decoder",
46
+ },
47
+ "robustness_summary.csv": {
48
+ "model", "dataset", "status", "metric", "noise_levels", "scores",
49
+ "raw_auc",
50
+ },
51
+ "scalability_summary.csv": {
52
+ "model", "dataset", "status", "training_time_sec",
53
+ "inference_time_sec", "peak_ram_gb", "peak_vram_gb",
54
+ },
55
+ "consistency_summary.csv": {
56
+ "model", "dataset", "is_active_model", "mean_r2", "n_sessions",
57
+ "sessions", "latent_dim", "scoring_modes", "normalizations",
58
+ },
59
+ "neuron_shap_summary.csv": {
60
+ "model", "dataset", "is_active_model", "auc", "spearman_corr",
61
+ "shap_mean_value", "shap_min_value", "shap_max_value",
62
+ "shap_fraction_positive", "shap_fraction_negative",
63
+ },
64
+ "trial_shapley_summary.csv": {
65
+ "model", "dataset", "is_active_model", "analysis", "perturbation_auc",
66
+ "rotation_angle_deg", "rotation_subspace_dim_spec",
67
+ "trial_selection_mode", "converged", "shapley_mean_value",
68
+ "shapley_min_value", "shapley_max_value",
69
+ "shapley_fraction_positive", "shapley_fraction_negative",
70
+ },
71
+ "trial_shapley_retrain_summary.csv": {
72
+ "analysis", "model", "is_active_model", "condition", "metric", "score",
73
+ },
74
+ "trial_historical_trajectories.csv": {
75
+ "model", "target_session", "trial_index", "trial_id",
76
+ "direction_index", "direction_label", "time_index", "target_x",
77
+ "target_y", "current_only_x", "current_only_y",
78
+ "historical_selected_x", "historical_selected_y", "current_only_r2",
79
+ "historical_selected_r2",
80
+ },
81
+ "latent_samples.csv": {
82
+ "model", "dataset", "session", "session_label", "x", "y", "z",
83
+ "condition", "trial_index", "time_index", "eval_time_index",
84
+ "coordinate_space", "reference_session", "alignment", "landmark_type",
85
+ "n_alignment_landmarks", "is_reference", "session_order",
86
+ },
87
+ "latent_trajectories.csv": {
88
+ "model", "dataset", "session", "session_label", "x", "y", "z",
89
+ "condition", "time_index", "eval_time_index", "n_points",
90
+ "coordinate_space", "reference_session", "alignment", "landmark_type",
91
+ "n_alignment_landmarks", "is_reference", "session_order",
92
+ },
93
+ }
94
+
95
+ UNIQUE_KEYS = {
96
+ "clean_prediction_summary.csv": ["model", "dataset"],
97
+ "robustness_summary.csv": ["model", "dataset"],
98
+ "scalability_summary.csv": ["model", "dataset"],
99
+ "consistency_summary.csv": ["model", "dataset"],
100
+ "neuron_shap_summary.csv": ["model", "dataset"],
101
+ "trial_shapley_summary.csv": ["model", "dataset"],
102
+ "trial_shapley_retrain_summary.csv": ["analysis", "model", "condition"],
103
+ }
104
+
105
+ CANONICAL_NAMES = {
106
+ "clean_prediction_summary.csv": "metrics_summary.csv",
107
+ "robustness_summary.csv": "robustness_summary.csv",
108
+ "scalability_summary.csv": "scalability_summary.csv",
109
+ "consistency_summary.csv": "consistency_summary.csv",
110
+ "neuron_shap_summary.csv": "neuron_shap_summary.csv",
111
+ "trial_shapley_summary.csv": "trial_shapley_summary.csv",
112
+ "trial_shapley_retrain_summary.csv": "trial_shapley_retrain_summary.csv",
113
+ }
114
+
115
+
116
+ class ValidationError(RuntimeError):
117
+ """Raised when Space data violates its manuscript-facing contract."""
118
+
119
+
120
+ def _sha256(path: Path) -> str:
121
+ digest = hashlib.sha256()
122
+ with path.open("rb") as handle:
123
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
124
+ digest.update(chunk)
125
+ return digest.hexdigest()
126
+
127
+
128
+ def _active_mask(frame: pd.DataFrame) -> pd.Series:
129
+ if "status" in frame.columns:
130
+ return frame["status"].fillna("").eq("present")
131
+ if "is_active_model" in frame.columns:
132
+ return frame["is_active_model"].astype(str).str.lower().eq("true")
133
+ return pd.Series(True, index=frame.index)
134
+
135
+
136
+ def _require(condition: bool, message: str, errors: list[str]) -> None:
137
+ if not condition:
138
+ errors.append(message)
139
+
140
+
141
+ def _same_values(left: pd.DataFrame, right: pd.DataFrame) -> None:
142
+ pd.testing.assert_frame_equal(
143
+ left.reset_index(drop=True),
144
+ right.reset_index(drop=True),
145
+ check_dtype=False,
146
+ check_exact=True,
147
+ check_categorical=False,
148
+ )
149
+
150
+
151
+ def _validate_latent_cell(
152
+ frame: pd.DataFrame,
153
+ *,
154
+ table_label: str,
155
+ model: str,
156
+ dataset: str,
157
+ sessions: list[str],
158
+ landmark_type: str,
159
+ errors: list[str],
160
+ ) -> None:
161
+ """Validate reference/alignment metadata for one method-dataset cell."""
162
+ cell = frame[
163
+ frame["model"].astype(str).eq(model)
164
+ & frame["dataset"].astype(str).eq(dataset)
165
+ ]
166
+ for session_order, session in enumerate(sessions):
167
+ session_rows = cell[cell["session"].astype(str).eq(session)]
168
+ if session_rows.empty:
169
+ errors.append(f"{table_label}: missing {model}/{dataset}/{session}")
170
+ continue
171
+
172
+ expected_reference = "true" if session_order == 0 else "false"
173
+ observed_reference = set(
174
+ session_rows["is_reference"].dropna().astype(str).str.lower()
175
+ )
176
+ _require(
177
+ observed_reference == {expected_reference},
178
+ f"{table_label}: {model}/{dataset}/{session} is_reference "
179
+ f"values {sorted(observed_reference)}",
180
+ errors,
181
+ )
182
+ expected_alignment = "identity" if session_order == 0 else "proper_similarity_procrustes"
183
+ observed_alignment = set(session_rows["alignment"].dropna().astype(str))
184
+ _require(
185
+ observed_alignment == {expected_alignment},
186
+ f"{table_label}: {model}/{dataset}/{session} alignment "
187
+ f"values {sorted(observed_alignment)}",
188
+ errors,
189
+ )
190
+ observed_reference_sessions = set(
191
+ session_rows["reference_session"].dropna().astype(str)
192
+ )
193
+ _require(
194
+ observed_reference_sessions == {sessions[0]},
195
+ f"{table_label}: {model}/{dataset}/{session} reference metadata "
196
+ f"{sorted(observed_reference_sessions)}",
197
+ errors,
198
+ )
199
+ observed_landmarks = set(session_rows["landmark_type"].dropna().astype(str))
200
+ _require(
201
+ observed_landmarks == {landmark_type},
202
+ f"{table_label}: {model}/{dataset}/{session} landmarks "
203
+ f"{sorted(observed_landmarks)}",
204
+ errors,
205
+ )
206
+ orders = pd.to_numeric(session_rows["session_order"], errors="coerce")
207
+ _require(
208
+ orders.notna().all()
209
+ and np.isfinite(orders.to_numpy()).all()
210
+ and orders.eq(session_order).all(),
211
+ f"{table_label}: {model}/{dataset}/{session} has invalid session_order",
212
+ errors,
213
+ )
214
+ landmark_counts = pd.to_numeric(
215
+ session_rows["n_alignment_landmarks"], errors="coerce"
216
+ )
217
+ _require(
218
+ landmark_counts.notna().all()
219
+ and np.isfinite(landmark_counts.to_numpy()).all()
220
+ and landmark_counts.ge(3).all(),
221
+ f"{table_label}: {model}/{dataset}/{session} has invalid landmark count",
222
+ errors,
223
+ )
224
+
225
+
226
+ def validate_local(data_dir: Path) -> dict[str, pd.DataFrame]:
227
+ """Validate schemas, coverage, uniqueness, and fixed analysis conventions."""
228
+
229
+ frames: dict[str, pd.DataFrame] = {}
230
+ errors: list[str] = []
231
+
232
+ for name, columns in REQUIRED_COLUMNS.items():
233
+ path = data_dir / name
234
+ if not path.exists():
235
+ errors.append(f"missing required table: {path}")
236
+ continue
237
+ frame = pd.read_csv(path)
238
+ missing = sorted(columns - set(frame.columns))
239
+ _require(not missing, f"{name}: missing columns {missing}", errors)
240
+ if missing:
241
+ continue
242
+ frames[name] = frame
243
+
244
+ manifest_path = data_dir / "release_manifest.json"
245
+ if not manifest_path.exists():
246
+ errors.append(f"missing release manifest: {manifest_path}")
247
+ else:
248
+ try:
249
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
250
+ except (json.JSONDecodeError, OSError) as exc:
251
+ errors.append(f"invalid release manifest: {exc}")
252
+ manifest = {}
253
+ if not isinstance(manifest, dict):
254
+ errors.append("invalid release manifest: top-level value must be an object")
255
+ manifest = {}
256
+ raw_manifest_spaces = manifest.get("latent_coordinate_spaces", [])
257
+ if not isinstance(raw_manifest_spaces, list):
258
+ errors.append("release manifest: latent_coordinate_spaces must be a list")
259
+ raw_manifest_spaces = []
260
+ elif not all(isinstance(value, str) for value in raw_manifest_spaces):
261
+ errors.append("release manifest: coordinate-space values must be strings")
262
+ raw_manifest_spaces = []
263
+ manifest_spaces = set(raw_manifest_spaces)
264
+ _require(
265
+ manifest_spaces == {LATENT_COORDINATE_SPACE},
266
+ f"release manifest: coordinate spaces {sorted(manifest_spaces)}",
267
+ errors,
268
+ )
269
+ manifest_tables = manifest.get("tables", {})
270
+ if not isinstance(manifest_tables, dict):
271
+ errors.append("release manifest: tables must be an object")
272
+ manifest_tables = {}
273
+ for name in REQUIRED_COLUMNS:
274
+ entry = manifest_tables.get(name)
275
+ if not isinstance(entry, dict):
276
+ errors.append(f"release manifest: missing table entry {name}")
277
+ continue
278
+ path = data_dir / name
279
+ if not path.exists():
280
+ continue
281
+ expected_hash = entry.get("sha256")
282
+ actual_hash = _sha256(path)
283
+ _require(
284
+ expected_hash == actual_hash,
285
+ f"release manifest: SHA-256 mismatch for {name}",
286
+ errors,
287
+ )
288
+ if name in frames:
289
+ _require(
290
+ entry.get("rows") == len(frames[name]),
291
+ f"release manifest: row-count mismatch for {name}",
292
+ errors,
293
+ )
294
+
295
+ for name, (n_rows, n_active) in EXPECTED_COVERAGE.items():
296
+ if name not in frames:
297
+ continue
298
+ frame = frames[name]
299
+ _require(len(frame) == n_rows, f"{name}: expected {n_rows} rows, found {len(frame)}", errors)
300
+ active = int(_active_mask(frame).sum())
301
+ _require(active == n_active, f"{name}: expected {n_active} available rows, found {active}", errors)
302
+
303
+ for name, keys in UNIQUE_KEYS.items():
304
+ if name not in frames or not set(keys).issubset(frames[name].columns):
305
+ continue
306
+ duplicates = frames[name].duplicated(keys, keep=False)
307
+ _require(not duplicates.any(), f"{name}: duplicate keys for {keys}", errors)
308
+
309
+ for name in ("clean_prediction_summary.csv", "robustness_summary.csv", "scalability_summary.csv"):
310
+ if name in frames:
311
+ observed = set(frames[name]["dataset"].dropna().astype(str))
312
+ _require(observed == DATASETS, f"{name}: dataset set is {sorted(observed)}", errors)
313
+
314
+ if "clean_prediction_summary.csv" in frames:
315
+ prediction = frames["clean_prediction_summary.csv"]
316
+ _require(prediction["model"].nunique() == 23, "prediction: expected 23 methods", errors)
317
+ metrics = set(prediction.loc[_active_mask(prediction), "metric"].dropna())
318
+ _require(metrics == {"accuracy", "r2"}, f"prediction: unexpected metrics {sorted(metrics)}", errors)
319
+
320
+ if "consistency_summary.csv" in frames:
321
+ consistency = frames["consistency_summary.csv"]
322
+ active = consistency.loc[_active_mask(consistency)]
323
+ observed = set(active["dataset"].dropna().astype(str))
324
+ _require(observed == CONSISTENCY_DATASETS, f"consistency: dataset set is {sorted(observed)}", errors)
325
+ norms = set(active["normalizations"].dropna().astype(str))
326
+ _require(norms == {"per_session_whitening"}, f"consistency: unexpected normalization {sorted(norms)}", errors)
327
+
328
+ if "neuron_shap_summary.csv" in frames:
329
+ feature = frames["neuron_shap_summary.csv"]
330
+ allen = feature[feature["dataset"].eq("allen_neuropixels")]
331
+ _require(not allen.empty and allen["spearman_corr"].notna().all(), "feature attribution: Allen Spearman values missing", errors)
332
+ other = feature[~feature["dataset"].eq("allen_neuropixels")]
333
+ _require(other["auc"].notna().all(), "feature attribution: ROC-AUC values missing", errors)
334
+ _require(feature["shap_min_value"].lt(0).any(), "feature attribution: signed negative values absent", errors)
335
+
336
+ if "trial_shapley_summary.csv" in frames:
337
+ trial = frames["trial_shapley_summary.csv"]
338
+ _require(set(trial["analysis"].dropna()) == {"subspace_rotation"}, "trial valuation: noncanonical analysis", errors)
339
+ angles = set(pd.to_numeric(trial["rotation_angle_deg"], errors="coerce").dropna())
340
+ _require(angles == {75.0}, f"trial valuation: rotation angles {sorted(angles)}", errors)
341
+ dims = set(trial["rotation_subspace_dim_spec"].dropna().astype(str))
342
+ _require(dims == {"full"}, f"trial valuation: subspace specs {sorted(dims)}", errors)
343
+ modes = set(trial["trial_selection_mode"].dropna().astype(str))
344
+ _require(modes == {"random"}, f"trial valuation: selection modes {sorted(modes)}", errors)
345
+ _require(trial["perturbation_auc"].notna().all(), "trial valuation: detection AUC missing", errors)
346
+ _require(trial["shapley_min_value"].lt(0).any(), "trial valuation: signed negative values absent", errors)
347
+
348
+ if "trial_shapley_retrain_summary.csv" in frames:
349
+ retrain = frames["trial_shapley_retrain_summary.csv"]
350
+ expected = {
351
+ "within_session_cleaning": {"mixed_full", "data_shapley", "oracle"},
352
+ "cross_session_old_trial_selection": {
353
+ "target_only", "all_sessions", "oldonly_dshap_negative_removal",
354
+ },
355
+ }
356
+ observed = {
357
+ analysis: set(group["condition"].dropna().astype(str))
358
+ for analysis, group in retrain.groupby("analysis")
359
+ }
360
+ _require(observed == expected, f"trial retraining: conditions {observed}", errors)
361
+
362
+ if "trial_historical_trajectories.csv" in frames:
363
+ historical = frames["trial_historical_trajectories.csv"]
364
+ _require(
365
+ len(historical) == 135 * 35,
366
+ "historical trajectories: expected 135 trials × 35 time bins",
367
+ errors,
368
+ )
369
+ keys = ["trial_index", "time_index"]
370
+ _require(
371
+ not historical.duplicated(keys, keep=False).any(),
372
+ f"historical trajectories: duplicate keys for {keys}",
373
+ errors,
374
+ )
375
+ _require(
376
+ set(historical["model"].dropna().astype(str)) == {"rnn"},
377
+ "historical trajectories: expected the Figure 5e RNN example",
378
+ errors,
379
+ )
380
+ _require(
381
+ set(historical["target_session"].dropna().astype(str))
382
+ == {FIGURE5_TARGET_SESSION},
383
+ "historical trajectories: unexpected target session",
384
+ errors,
385
+ )
386
+ _require(
387
+ historical["trial_index"].nunique() == 135
388
+ and historical["trial_id"].nunique() == 135,
389
+ "historical trajectories: expected 135 held-out trials and trial IDs",
390
+ errors,
391
+ )
392
+ time_counts = historical.groupby("trial_index")["time_index"].nunique()
393
+ _require(
394
+ len(time_counts) == 135 and time_counts.eq(35).all(),
395
+ "historical trajectories: every trial must contain 35 time bins",
396
+ errors,
397
+ )
398
+ direction_indices = pd.to_numeric(
399
+ historical["direction_index"], errors="coerce"
400
+ )
401
+ _require(
402
+ direction_indices.notna().all()
403
+ and set(direction_indices.astype(int)) == set(range(8)),
404
+ "historical trajectories: expected all eight reach directions",
405
+ errors,
406
+ )
407
+ for column in (
408
+ "target_x",
409
+ "target_y",
410
+ "current_only_x",
411
+ "current_only_y",
412
+ "historical_selected_x",
413
+ "historical_selected_y",
414
+ "current_only_r2",
415
+ "historical_selected_r2",
416
+ ):
417
+ values = pd.to_numeric(historical[column], errors="coerce")
418
+ _require(
419
+ values.notna().all() and np.isfinite(values.to_numpy()).all(),
420
+ f"historical trajectories: non-finite {column} values",
421
+ errors,
422
+ )
423
+ _require(
424
+ historical["current_only_r2"].nunique() == 1
425
+ and historical["historical_selected_r2"].nunique() == 1,
426
+ "historical trajectories: expected one R² value per condition",
427
+ errors,
428
+ )
429
+
430
+ for name in ("latent_samples.csv", "latent_trajectories.csv"):
431
+ if name not in frames:
432
+ continue
433
+ latent = frames[name]
434
+ observed = set(latent["dataset"].dropna().astype(str))
435
+ _require(observed.issubset(CONSISTENCY_DATASETS), f"{name}: unsupported datasets {sorted(observed - CONSISTENCY_DATASETS)}", errors)
436
+ for column in ("x", "y", "z"):
437
+ values = pd.to_numeric(latent[column], errors="coerce")
438
+ _require(
439
+ values.notna().all() and np.isfinite(values.to_numpy()).all(),
440
+ f"{name}: non-finite {column} values",
441
+ errors,
442
+ )
443
+ for column in (
444
+ "coordinate_space",
445
+ "reference_session",
446
+ "alignment",
447
+ "landmark_type",
448
+ "n_alignment_landmarks",
449
+ "is_reference",
450
+ "session_order",
451
+ ):
452
+ _require(
453
+ latent[column].notna().all(),
454
+ f"{name}: missing {column} values",
455
+ errors,
456
+ )
457
+ spaces = set(latent["coordinate_space"].dropna().astype(str))
458
+ _require(
459
+ spaces == {LATENT_COORDINATE_SPACE},
460
+ f"{name}: coordinate spaces {sorted(spaces)}",
461
+ errors,
462
+ )
463
+
464
+ if {
465
+ "consistency_summary.csv", "latent_samples.csv", "latent_trajectories.csv"
466
+ }.issubset(frames):
467
+ consistency = frames["consistency_summary.csv"]
468
+ consistency = consistency.loc[_active_mask(consistency)].copy()
469
+ samples = frames["latent_samples.csv"]
470
+ trajectories = frames["latent_trajectories.csv"]
471
+ expected_sample_sessions: set[tuple[str, str, str]] = set()
472
+ expected_trajectory_sessions: set[tuple[str, str, str]] = set()
473
+ for row in consistency.itertuples(index=False):
474
+ sessions = (
475
+ []
476
+ if pd.isna(row.sessions)
477
+ else [item.strip() for item in str(row.sessions).split(";") if item.strip()]
478
+ )
479
+ if not sessions:
480
+ errors.append(f"consistency: {row.model}/{row.dataset} has no sessions")
481
+ continue
482
+ try:
483
+ expected_n_sessions = int(row.n_sessions)
484
+ except (TypeError, ValueError):
485
+ expected_n_sessions = -1
486
+ _require(
487
+ len(sessions) == expected_n_sessions,
488
+ f"consistency: {row.model}/{row.dataset} lists {len(sessions)} "
489
+ f"sessions but n_sessions={row.n_sessions}",
490
+ errors,
491
+ )
492
+ expected_sample_sessions.update(
493
+ (str(row.model), str(row.dataset), session) for session in sessions
494
+ )
495
+ if str(row.dataset) != "ratinabox":
496
+ expected_trajectory_sessions.update(
497
+ (str(row.model), str(row.dataset), session) for session in sessions
498
+ )
499
+ _validate_latent_cell(
500
+ samples,
501
+ table_label="latent samples",
502
+ model=str(row.model),
503
+ dataset=str(row.dataset),
504
+ sessions=sessions,
505
+ landmark_type=str(row.scoring_modes),
506
+ errors=errors,
507
+ )
508
+ if str(row.dataset) != "ratinabox":
509
+ _validate_latent_cell(
510
+ trajectories,
511
+ table_label="latent trajectories",
512
+ model=str(row.model),
513
+ dataset=str(row.dataset),
514
+ sessions=sessions,
515
+ landmark_type=str(row.scoring_modes),
516
+ errors=errors,
517
+ )
518
+
519
+ observed_sample_sessions = set(
520
+ samples[["model", "dataset", "session"]].astype(str).itertuples(index=False, name=None)
521
+ )
522
+ observed_trajectory_sessions = set(
523
+ trajectories[["model", "dataset", "session"]].astype(str).itertuples(index=False, name=None)
524
+ )
525
+ _require(
526
+ observed_sample_sessions == expected_sample_sessions,
527
+ "latent samples: active consistency session coverage differs",
528
+ errors,
529
+ )
530
+ _require(
531
+ observed_trajectory_sessions == expected_trajectory_sessions,
532
+ "latent trajectories: applicable consistency session coverage differs",
533
+ errors,
534
+ )
535
+ _require(
536
+ len(observed_sample_sessions) == 173,
537
+ f"latent samples: expected 173 sessions, found {len(observed_sample_sessions)}",
538
+ errors,
539
+ )
540
+
541
+ if errors:
542
+ raise ValidationError("\n".join(f"- {item}" for item in errors))
543
+ return frames
544
+
545
+
546
+ def validate_canonical(data_dir: Path, canonical_root: Path) -> None:
547
+ """Require Space summaries to equal the current paper result tables."""
548
+
549
+ results_dir = canonical_root / "paper" / "results"
550
+ errors: list[str] = []
551
+ for dashboard_name, paper_name in CANONICAL_NAMES.items():
552
+ dashboard_path = data_dir / dashboard_name
553
+ paper_path = results_dir / paper_name
554
+ if not dashboard_path.exists():
555
+ errors.append(f"missing dashboard table: {dashboard_path}")
556
+ continue
557
+ if not paper_path.exists():
558
+ errors.append(f"missing canonical table: {paper_path}")
559
+ continue
560
+ try:
561
+ _same_values(pd.read_csv(dashboard_path), pd.read_csv(paper_path))
562
+ except AssertionError as exc:
563
+ first_line = str(exc).splitlines()[0] if str(exc) else "values differ"
564
+ errors.append(f"{dashboard_name} != {paper_name}: {first_line}")
565
+ if errors:
566
+ raise ValidationError("\n".join(f"- {item}" for item in errors))
567
+
568
+
569
+ def build_parser() -> argparse.ArgumentParser:
570
+ parser = argparse.ArgumentParser(description=__doc__)
571
+ parser.add_argument("--data-dir", type=Path, default=Path(__file__).resolve().parent / "data")
572
+ parser.add_argument(
573
+ "--canonical-root",
574
+ type=Path,
575
+ help="Main benchmark repository root; enables exact paper/results comparisons.",
576
+ )
577
+ return parser
578
+
579
+
580
+ def main(argv: Iterable[str] | None = None) -> int:
581
+ args = build_parser().parse_args(argv)
582
+ frames = validate_local(args.data_dir)
583
+ if args.canonical_root is not None:
584
+ validate_canonical(args.data_dir, args.canonical_root.resolve())
585
+ print(f"Validated {len(frames)} BEND-BCI Space tables in {args.data_dir}")
586
+ if args.canonical_root is not None:
587
+ print("Canonical paper/results comparison passed")
588
+ return 0
589
+
590
+
591
+ if __name__ == "__main__":
592
+ raise SystemExit(main())