eugenehp commited on
Commit
6d33f2e
·
verified ·
1 Parent(s): 5e969dd

Add model card, safetensors weights, config, build_args and fsaverage5 mesh

Browse files
.DS_Store ADDED
Binary file (8.2 kB). View file
 
.gitattributes CHANGED
@@ -33,3 +33,18 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ e2e_output/brain_coolwarm.png filter=lfs diff=lfs merge=lfs -text
37
+ e2e_output/brain_dorsal.png filter=lfs diff=lfs merge=lfs -text
38
+ e2e_output/brain_left.png filter=lfs diff=lfs merge=lfs -text
39
+ e2e_output/brain_mosaic.png filter=lfs diff=lfs merge=lfs -text
40
+ e2e_output/brain_rgb_overlay.png filter=lfs diff=lfs merge=lfs -text
41
+ e2e_output/brain_right.png filter=lfs diff=lfs merge=lfs -text
42
+ e2e_output/timesteps/frame_0001.png filter=lfs diff=lfs merge=lfs -text
43
+ e2e_output/timesteps/frame_0002.png filter=lfs diff=lfs merge=lfs -text
44
+ e2e_output/timesteps/frame_0004.png filter=lfs diff=lfs merge=lfs -text
45
+ fsaverage5/surf/lh.infl filter=lfs diff=lfs merge=lfs -text
46
+ fsaverage5/surf/lh.pial filter=lfs diff=lfs merge=lfs -text
47
+ fsaverage5/surf/lh.white filter=lfs diff=lfs merge=lfs -text
48
+ fsaverage5/surf/rh.infl filter=lfs diff=lfs merge=lfs -text
49
+ fsaverage5/surf/rh.pial filter=lfs diff=lfs merge=lfs -text
50
+ fsaverage5/surf/rh.white filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -1,3 +1,140 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ language:
4
+ - en
5
+ tags:
6
+ - neuroscience
7
+ - fmri
8
+ - brain-encoding
9
+ - multimodal
10
+ - rust
11
+ - safetensors
12
+ base_model: facebook/tribev2
13
+ ---
14
+
15
+ <div align="center">
16
+
17
+ # TRIBE v2 — Rust Edition
18
+
19
+ **A Foundation Model of Vision, Audition, and Language for In-Silico Neuroscience**
20
+
21
+ [![License: CC BY-NC 4.0](https://img.shields.io/badge/License-CC%20BY--NC%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc/4.0/)
22
+ [![Rust](https://img.shields.io/badge/inference-Rust-orange.svg)](https://www.rust-lang.org/)
23
+ [![Base model](https://img.shields.io/badge/base%20model-facebook%2Ftribev2-blue.svg)](https://huggingface.co/facebook/tribev2)
24
+
25
+ 📄 [Paper](https://ai.meta.com/research/publications/a-foundation-model-of-vision-audition-and-language-for-in-silico-neuroscience/) ·
26
+ 🤗 [Original weights](https://huggingface.co/facebook/tribev2) ·
27
+ 🦀 [Rust implementation](https://github.com/eugenehp/tribev2-rs)
28
+
29
+ </div>
30
+
31
+ ## Overview
32
+
33
+ This directory contains the **same pretrained weights** as [`facebook/tribev2`](https://huggingface.co/facebook/tribev2), converted to the [safetensors](https://github.com/huggingface/safetensors) format for use with the pure-Rust inference engine **tribev2-rs**.
34
+
35
+ No fine-tuning, quantisation, or architectural changes have been made.
36
+ The model is **bit-for-bit equivalent** to the original Python checkpoint — every layer has been independently verified for numerical parity.
37
+
38
+ ## Model description
39
+
40
+ TRIBE v2 is a deep multimodal brain encoding model that predicts fMRI responses to naturalistic stimuli (video, audio, text).
41
+ It combines three state-of-the-art feature extractors:
42
+
43
+ | Modality | Extractor | Dim |
44
+ |----------|-----------|----:|
45
+ | Text | LLaMA 3.2-3B | 3 072 |
46
+ | Audio | Wav2Vec-BERT 2.0 | 1 024 |
47
+ | Video | V-JEPA2 ViT-G | 1 408 |
48
+
49
+ These multimodal representations are projected and fused by a **Transformer encoder** (8 layers, 1 152-d, ScaleNorm, Rotary PE) that outputs predicted BOLD responses on the **fsaverage5** cortical mesh (~20 484 vertices).
50
+
51
+ Full architectural details are in the [paper](https://ai.meta.com/research/publications/a-foundation-model-of-vision-audition-and-language-for-in-silico-neuroscience/) and in the [`facebook/tribev2`](https://huggingface.co/facebook/tribev2) model card.
52
+
53
+ ## Files
54
+
55
+ | File | Description |
56
+ |------|-------------|
57
+ | `model.safetensors` | Pretrained weights (safetensors, converted from the original PyTorch Lightning checkpoint) |
58
+ | `config.yaml` | Model hyper-parameters (hidden dim, depth, heads, modalities, …) |
59
+ | `build_args.json` | Feature-extractor build arguments used at training time |
60
+ | `fsaverage5/` | FreeSurfer fsaverage5 cortical mesh files (`.pial`, `.inflated`, `.sulc`, `.curv`) for brain visualisation |
61
+
62
+ ## Rust usage
63
+
64
+ ```rust
65
+ use std::collections::BTreeMap;
66
+ use tribev2_rs::model::tribe::TribeV2;
67
+ use tribev2_rs::tensor::Tensor;
68
+
69
+ // Load model from this data directory
70
+ let model = TribeV2::from_pretrained(
71
+ "data/config.yaml",
72
+ "data/model.safetensors",
73
+ Some("data/build_args.json"),
74
+ ).unwrap();
75
+
76
+ // Build multi-modal feature tensors [1, dim, T]
77
+ let mut features = BTreeMap::new();
78
+ features.insert("text".to_string(), Tensor::zeros(&[1, 6144, 100]));
79
+ features.insert("audio".to_string(), Tensor::zeros(&[1, 2048, 100]));
80
+ features.insert("video".to_string(), Tensor::zeros(&[1, 2816, 100]));
81
+
82
+ // Forward pass → [1, 20484, 100]
83
+ let output = model.forward(&features, None, true);
84
+ println!("{:?}", output.shape()); // [1, 20484, 100]
85
+ ```
86
+
87
+ See the [tribev2-rs README](../README.md) for the full CLI, feature flags, benchmarks, and brain-visualisation API.
88
+
89
+ ## Converting weights from the original checkpoint
90
+
91
+ ```bash
92
+ # 1. Download the original checkpoint from HuggingFace
93
+ cargo run --bin tribev2-download --features hf-download -- --repo facebook/tribev2
94
+
95
+ # 2. Convert to safetensors (requires Python ≥ 3.9, torch, safetensors)
96
+ python3 scripts/convert_checkpoint.py weights/best.ckpt data/model.safetensors
97
+ # → data/model.safetensors + data/build_args.json
98
+ ```
99
+
100
+ ## Pretrained model parameters
101
+
102
+ | Parameter | Value |
103
+ |-----------|-------|
104
+ | Hidden dim | 1 152 |
105
+ | Encoder depth | 8 |
106
+ | Attention heads | 8 |
107
+ | FF multiplier | 4× |
108
+ | Norm | ScaleNorm |
109
+ | Position encoding | Rotary (dim = 72) |
110
+ | Low-rank head | 2 048 |
111
+ | Subjects (released) | 1 (average subject) |
112
+ | Output surface | fsaverage5 (20 484 vertices) |
113
+ | Output timesteps | 100 TRs |
114
+
115
+ ## Citation
116
+
117
+ If you use these weights or the Rust inference engine, please cite the original paper:
118
+
119
+ ```bibtex
120
+ @article{dAscoli2026TribeV2,
121
+ title={A foundation model of vision, audition, and language for in-silico neuroscience},
122
+ author={d'Ascoli, St{\'e}phane and Rapin, J{\'e}r{\'e}my and Benchetrit, Yohann and
123
+ Brookes, Teon and Begany, Katelyn and Raugel, Jos{\'e}phine and
124
+ Banville, Hubert and King, Jean-R{\'e}mi},
125
+ year={2026}
126
+ }
127
+ ```
128
+
129
+ ## License
130
+
131
+ The **model weights** (all files in this directory) are released under the
132
+ [Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license,
133
+ identical to the original [`facebook/tribev2`](https://huggingface.co/facebook/tribev2) release.
134
+
135
+ > You are free to share and adapt the weights for **non-commercial** purposes,
136
+ > provided you give appropriate credit and indicate if changes were made.
137
+ > **Commercial use is not permitted.**
138
+
139
+ The Rust source code of **tribev2-rs** is separately licensed under Apache-2.0.
140
+ See [LICENSE](../LICENSE) in the repository root.
build_args.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "feature_dims": {
3
+ "text": [
4
+ 2,
5
+ 3072
6
+ ],
7
+ "audio": [
8
+ 2,
9
+ 1024
10
+ ],
11
+ "video": [
12
+ 2,
13
+ 1408
14
+ ]
15
+ },
16
+ "n_outputs": 20484,
17
+ "n_output_timesteps": 100
18
+ }
config.yaml ADDED
@@ -0,0 +1,619 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ infra:
2
+ folder: /checkpoint/sdascoli/results/tribe_release/half_depth/brain_model_config.subject_layers.subject_dropout=0.10,data.study.names=(Algonauts2025Bold,Lahner2024Bold,Lebel2023Bold,Wen2017)-ecdd89b2
3
+ cluster: auto
4
+ logs: '{folder}/logs/{user}/%j'
5
+ job_name: brain_model_config.subject_layers.subject_dropout=0.10,data.study.names=(Algonauts2025Bold,Lahner2024Bold,Lebel2023Bold,Wen2017)-ecdd89b2
6
+ timeout_min: 4320
7
+ nodes: 1
8
+ tasks_per_node: 1
9
+ cpus_per_task: 20
10
+ gpus_per_node: 1
11
+ mem_gb: 128.0
12
+ max_pickle_size_gb: null
13
+ slurm_constraint: volta32gb
14
+ slurm_partition: learnfair
15
+ slurm_account: null
16
+ slurm_qos: null
17
+ slurm_use_srun: false
18
+ slurm_additional_parameters: null
19
+ conda_env: null
20
+ workdir:
21
+ copied:
22
+ - neuralset
23
+ - neuraltrain
24
+ - neuralhub
25
+ - tribe
26
+ folder: null
27
+ log_commit: false
28
+ includes:
29
+ - '*.py'
30
+ - '*.txt'
31
+ excludes: !!python/tuple
32
+ - __pycache__
33
+ - .git
34
+ permissions: 511
35
+ version: '1'
36
+ mode: retry
37
+ keep_in_ram: false
38
+ data:
39
+ study:
40
+ infra: null
41
+ names:
42
+ - Algonauts2025Bold
43
+ - Lahner2024Bold
44
+ - Lebel2023Bold
45
+ - Wen2017
46
+ path: /large_experiments/brainai/shared/studies
47
+ transforms:
48
+ extractaudio:
49
+ infra: null
50
+ overwrite: false
51
+ name: ExtractAudioFromVideo
52
+ extractwords:
53
+ infra: null
54
+ language: english
55
+ overwrite: false
56
+ name: ExtractWordsFromAudio
57
+ addtext:
58
+ infra: null
59
+ name: AddText
60
+ addsentence:
61
+ infra: null
62
+ max_unmatched_ratio: 0.05
63
+ override_sentences: false
64
+ name: AddSentenceToWords
65
+ addcontext:
66
+ infra: null
67
+ sentence_only: false
68
+ max_context_len: 1024
69
+ split_field: ''
70
+ name: AddContextToWords
71
+ removemissing:
72
+ infra: null
73
+ event_types: Word
74
+ field: context
75
+ name: RemoveMissing
76
+ chunksounds:
77
+ infra: null
78
+ event_type_to_chunk: Audio
79
+ event_type_to_use: null
80
+ min_duration: 30.0
81
+ max_duration: 60.0
82
+ name: ChunkEvents
83
+ chunkvideos:
84
+ infra:
85
+ folder: !!python/object/apply:pathlib.PosixPath
86
+ - /
87
+ - checkpoint
88
+ - sdascoli
89
+ - cache
90
+ - tribe_release
91
+ cache_type: ValidatedParquet
92
+ mode: cached
93
+ keep_in_ram: false
94
+ backend: Cached
95
+ event_type_to_chunk: Video
96
+ event_type_to_use: null
97
+ min_duration: 30.0
98
+ max_duration: 60.0
99
+ name: ChunkEvents
100
+ query:
101
+ infra: null
102
+ query: null
103
+ name: QueryEvents
104
+ split:
105
+ infra: null
106
+ val_ratio: 0.1
107
+ name: SplitEvents
108
+ query: null
109
+ studies_to_include: null
110
+ infra_timelines:
111
+ folder: /checkpoint/sdascoli/cache/tribe_release
112
+ cluster: processpool
113
+ logs: '{folder}/logs/{user}/%j'
114
+ job_name: null
115
+ timeout_min: 720
116
+ nodes: 1
117
+ tasks_per_node: 1
118
+ cpus_per_task: null
119
+ gpus_per_node: null
120
+ mem_gb: null
121
+ max_pickle_size_gb: null
122
+ slurm_constraint: null
123
+ slurm_partition: null
124
+ slurm_account: null
125
+ slurm_qos: null
126
+ slurm_use_srun: false
127
+ slurm_additional_parameters: null
128
+ conda_env: null
129
+ workdir: null
130
+ permissions: 511
131
+ version: final
132
+ keep_in_ram: true
133
+ max_jobs: 1024
134
+ min_samples_per_job: 4
135
+ forbid_single_item_computation: false
136
+ mode: cached
137
+ name: MultiStudyLoader
138
+ neuro:
139
+ event_types: Fmri
140
+ aggregation: single
141
+ allow_missing: true
142
+ frequency: 1.0
143
+ offset: 5.0
144
+ projection:
145
+ mesh: fsaverage5
146
+ radius: 3.0
147
+ interpolation: linear
148
+ kind: ball
149
+ n_samples: null
150
+ mask_img: null
151
+ depth: null
152
+ center_depth: 0.5
153
+ extract_fsaverage_from_mni: true
154
+ name: TribeSurfaceProjector
155
+ cleaning:
156
+ standardize: zscore_sample
157
+ detrend: true
158
+ high_pass: null
159
+ low_pass: null
160
+ filter: null
161
+ ensure_finite: true
162
+ padding: null
163
+ input_space: null
164
+ fwhm: null
165
+ infra:
166
+ folder: /checkpoint/sdascoli/cache/tribe_release
167
+ cluster: slurm
168
+ logs: '{folder}/logs/{user}/%j'
169
+ job_name: null
170
+ timeout_min: 720
171
+ nodes: 1
172
+ tasks_per_node: 1
173
+ cpus_per_task: 10
174
+ gpus_per_node: 1
175
+ mem_gb: null
176
+ max_pickle_size_gb: null
177
+ slurm_constraint: volta32gb
178
+ slurm_partition: scavenge
179
+ slurm_account: null
180
+ slurm_qos: null
181
+ slurm_use_srun: false
182
+ slurm_additional_parameters: null
183
+ conda_env: null
184
+ workdir: null
185
+ permissions: 511
186
+ version: release
187
+ keep_in_ram: true
188
+ max_jobs: 256
189
+ min_samples_per_job: 1
190
+ forbid_single_item_computation: false
191
+ mode: cached
192
+ name: FmriExtractor
193
+ text_feature:
194
+ model_name: meta-llama/Llama-3.2-3B
195
+ device: cuda
196
+ layers:
197
+ - 0.5
198
+ - 0.75
199
+ - 1.0
200
+ cache_all_layers: false
201
+ cache_n_layers: 20
202
+ layer_aggregation: group_mean
203
+ token_aggregation: mean
204
+ event_types: Word
205
+ aggregation: sum
206
+ allow_missing: true
207
+ frequency: 2.0
208
+ infra:
209
+ folder: /checkpoint/sdascoli/cache/tribe_release
210
+ cluster: slurm
211
+ logs: '{folder}/logs/{user}/%j'
212
+ job_name: null
213
+ timeout_min: 720
214
+ nodes: 1
215
+ tasks_per_node: 1
216
+ cpus_per_task: 10
217
+ gpus_per_node: 1
218
+ mem_gb: null
219
+ max_pickle_size_gb: null
220
+ slurm_constraint: volta32gb
221
+ slurm_partition: scavenge
222
+ slurm_account: null
223
+ slurm_qos: null
224
+ slurm_use_srun: false
225
+ slurm_additional_parameters: null
226
+ conda_env: null
227
+ workdir: null
228
+ permissions: 511
229
+ version: release
230
+ keep_in_ram: true
231
+ max_jobs: 256
232
+ min_samples_per_job: 32
233
+ forbid_single_item_computation: false
234
+ mode: cached
235
+ batch_size: 4
236
+ contextualized: true
237
+ pretrained: true
238
+ name: HuggingFaceText
239
+ image_feature:
240
+ event_types: Video
241
+ aggregation: sum
242
+ allow_missing: true
243
+ frequency: 2.0
244
+ image:
245
+ model_name: facebook/dinov2-large
246
+ device: cuda
247
+ layers: 0.6666666666666666
248
+ cache_all_layers: false
249
+ cache_n_layers: 20
250
+ layer_aggregation: mean
251
+ token_aggregation: mean
252
+ event_types: Image
253
+ aggregation: single
254
+ allow_missing: false
255
+ frequency: 0.0
256
+ batch_size: 2
257
+ imsize: null
258
+ pretrained: true
259
+ infra:
260
+ folder: null
261
+ cluster: null
262
+ logs: '{folder}/logs/{user}/%j'
263
+ job_name: null
264
+ timeout_min: 25
265
+ nodes: 1
266
+ tasks_per_node: 1
267
+ cpus_per_task: 8
268
+ gpus_per_node: 1
269
+ mem_gb: null
270
+ max_pickle_size_gb: null
271
+ slurm_constraint: null
272
+ slurm_partition: null
273
+ slurm_account: null
274
+ slurm_qos: null
275
+ slurm_use_srun: false
276
+ slurm_additional_parameters: null
277
+ conda_env: null
278
+ workdir: null
279
+ permissions: 511
280
+ version: v5
281
+ keep_in_ram: false
282
+ max_jobs: 128
283
+ min_samples_per_job: 4096
284
+ forbid_single_item_computation: false
285
+ mode: cached
286
+ name: HuggingFaceImage
287
+ use_audio: true
288
+ clip_duration: null
289
+ max_imsize: null
290
+ layer_type: ''
291
+ num_frames: null
292
+ infra:
293
+ folder: /checkpoint/sdascoli/cache/tribe_release
294
+ cluster: slurm
295
+ logs: '{folder}/logs/{user}/%j'
296
+ job_name: null
297
+ timeout_min: 1440
298
+ nodes: 1
299
+ tasks_per_node: 1
300
+ cpus_per_task: 10
301
+ gpus_per_node: 1
302
+ mem_gb: null
303
+ max_pickle_size_gb: null
304
+ slurm_constraint: volta32gb
305
+ slurm_partition: scavenge
306
+ slurm_account: null
307
+ slurm_qos: null
308
+ slurm_use_srun: false
309
+ slurm_additional_parameters: null
310
+ conda_env: null
311
+ workdir: null
312
+ permissions: 511
313
+ version: release
314
+ keep_in_ram: true
315
+ max_jobs: 1024
316
+ min_samples_per_job: 1
317
+ forbid_single_item_computation: false
318
+ mode: cached
319
+ name: HuggingFaceVideo
320
+ audio_feature:
321
+ model_name: facebook/w2v-bert-2.0
322
+ device: cuda
323
+ layers:
324
+ - 0.5
325
+ - 0.75
326
+ - 1.0
327
+ cache_all_layers: false
328
+ cache_n_layers: 20
329
+ layer_aggregation: group_mean
330
+ token_aggregation: mean
331
+ event_types: Audio
332
+ aggregation: sum
333
+ allow_missing: true
334
+ frequency: 2.0
335
+ norm_audio: true
336
+ infra:
337
+ folder: /checkpoint/sdascoli/cache/tribe_release
338
+ cluster: slurm
339
+ logs: '{folder}/logs/{user}/%j'
340
+ job_name: null
341
+ timeout_min: 720
342
+ nodes: 1
343
+ tasks_per_node: 1
344
+ cpus_per_task: 10
345
+ gpus_per_node: 1
346
+ mem_gb: null
347
+ max_pickle_size_gb: null
348
+ slurm_constraint: volta32gb
349
+ slurm_partition: scavenge
350
+ slurm_account: null
351
+ slurm_qos: null
352
+ slurm_use_srun: false
353
+ slurm_additional_parameters: null
354
+ conda_env: null
355
+ workdir: null
356
+ permissions: 511
357
+ version: release
358
+ keep_in_ram: true
359
+ max_jobs: 256
360
+ min_samples_per_job: 1
361
+ forbid_single_item_computation: false
362
+ mode: cached
363
+ normalized: true
364
+ layer_type: transformer
365
+ name: Wav2VecBert
366
+ video_feature:
367
+ event_types: Video
368
+ aggregation: sum
369
+ allow_missing: true
370
+ frequency: 2.0
371
+ image:
372
+ model_name: facebook/vjepa2-vitg-fpc64-256
373
+ device: cuda
374
+ layers:
375
+ - 0.5
376
+ - 0.75
377
+ - 1.0
378
+ cache_all_layers: false
379
+ cache_n_layers: 20
380
+ layer_aggregation: group_mean
381
+ token_aggregation: mean
382
+ event_types: Image
383
+ aggregation: single
384
+ allow_missing: false
385
+ frequency: 0.0
386
+ batch_size: 8
387
+ imsize: null
388
+ pretrained: true
389
+ infra:
390
+ folder: null
391
+ cluster: null
392
+ logs: '{folder}/logs/{user}/%j'
393
+ job_name: null
394
+ timeout_min: 25
395
+ nodes: 1
396
+ tasks_per_node: 1
397
+ cpus_per_task: 8
398
+ gpus_per_node: 1
399
+ mem_gb: null
400
+ max_pickle_size_gb: null
401
+ slurm_constraint: null
402
+ slurm_partition: null
403
+ slurm_account: null
404
+ slurm_qos: null
405
+ slurm_use_srun: false
406
+ slurm_additional_parameters: null
407
+ conda_env: null
408
+ workdir: null
409
+ permissions: 511
410
+ version: v5
411
+ keep_in_ram: false
412
+ max_jobs: 128
413
+ min_samples_per_job: 4096
414
+ forbid_single_item_computation: false
415
+ mode: cached
416
+ name: HuggingFaceImage
417
+ use_audio: true
418
+ clip_duration: 4.0
419
+ max_imsize: null
420
+ layer_type: ''
421
+ num_frames: null
422
+ infra:
423
+ folder: /checkpoint/sdascoli/cache/tribe_release
424
+ cluster: slurm
425
+ logs: '{folder}/logs/{user}/%j'
426
+ job_name: null
427
+ timeout_min: 1440
428
+ nodes: 1
429
+ tasks_per_node: 1
430
+ cpus_per_task: 10
431
+ gpus_per_node: 1
432
+ mem_gb: null
433
+ max_pickle_size_gb: null
434
+ slurm_constraint: volta32gb
435
+ slurm_partition: scavenge
436
+ slurm_account: null
437
+ slurm_qos: null
438
+ slurm_use_srun: false
439
+ slurm_additional_parameters: null
440
+ conda_env: null
441
+ workdir: null
442
+ permissions: 511
443
+ version: release
444
+ keep_in_ram: true
445
+ max_jobs: 1024
446
+ min_samples_per_job: 1
447
+ forbid_single_item_computation: false
448
+ mode: cached
449
+ name: HuggingFaceVideo
450
+ subject_id:
451
+ event_types: Event
452
+ aggregation: first
453
+ allow_missing: true
454
+ frequency: 0.0
455
+ event_field: subject
456
+ treat_missing_as_separate_class: false
457
+ return_one_hot: false
458
+ predefined_mapping:
459
+ Algonauts2025Bold/sub-01: 0
460
+ Algonauts2025Bold/sub-02: 1
461
+ Algonauts2025Bold/sub-03: 2
462
+ Algonauts2025Bold/sub-05: 3
463
+ Lahner2024Bold/1: 4
464
+ Lahner2024Bold/10: 5
465
+ Lahner2024Bold/2: 6
466
+ Lahner2024Bold/3: 7
467
+ Lahner2024Bold/4: 8
468
+ Lahner2024Bold/5: 9
469
+ Lahner2024Bold/6: 10
470
+ Lahner2024Bold/7: 11
471
+ Lahner2024Bold/8: 12
472
+ Lahner2024Bold/9: 13
473
+ Lebel2023Bold/UTS01: 14
474
+ Lebel2023Bold/UTS02: 15
475
+ Lebel2023Bold/UTS03: 16
476
+ Lebel2023Bold/UTS04: 17
477
+ Lebel2023Bold/UTS05: 18
478
+ Lebel2023Bold/UTS06: 19
479
+ Lebel2023Bold/UTS07: 20
480
+ Lebel2023Bold/UTS08: 21
481
+ Wen2017/subject1: 22
482
+ Wen2017/subject2: 23
483
+ Wen2017/subject3: 24
484
+ name: LabelEncoder
485
+ frequency: 2.0
486
+ features_to_use:
487
+ - text
488
+ - audio
489
+ - video
490
+ features_to_mask: []
491
+ n_layers_to_use: null
492
+ layers_to_use:
493
+ - 0.5
494
+ - 0.75
495
+ - 1.0
496
+ layer_aggregation: group_mean
497
+ duration_trs: 100
498
+ overlap_trs_train: 0
499
+ overlap_trs_val: 0
500
+ batch_size: 8
501
+ num_workers: 20
502
+ shuffle_train: true
503
+ shuffle_val: true
504
+ stride_drop_incomplete: false
505
+ split_segments_by_time: false
506
+ seed: 33
507
+ brain_model_config:
508
+ projector:
509
+ input_size: null
510
+ hidden_sizes: null
511
+ norm_layer: layer
512
+ activation_layer: gelu
513
+ bias: true
514
+ dropout: 0.0
515
+ name: Mlp
516
+ combiner: null
517
+ encoder:
518
+ heads: 8
519
+ depth: 8
520
+ cross_attend: false
521
+ causal: false
522
+ attn_flash: false
523
+ attn_dropout: 0.0
524
+ ff_mult: 4
525
+ ff_dropout: 0.0
526
+ use_scalenorm: true
527
+ use_rmsnorm: false
528
+ rel_pos_bias: false
529
+ alibi_pos_bias: false
530
+ rotary_pos_emb: true
531
+ rotary_xpos: false
532
+ residual_attn: false
533
+ scale_residual: true
534
+ layer_dropout: 0.0
535
+ name: TransformerEncoder
536
+ time_pos_embedding: true
537
+ subject_embedding: false
538
+ subject_layers:
539
+ n_subjects: 25
540
+ bias: true
541
+ init_id: false
542
+ average_subjects: false
543
+ mode: gather
544
+ subject_dropout: 0.1
545
+ name: SubjectLayers
546
+ hidden: 1152
547
+ max_seq_len: 1024
548
+ dropout: 0.0
549
+ extractor_aggregation: cat
550
+ layer_aggregation: cat
551
+ linear_baseline: false
552
+ modality_dropout: 0.3
553
+ temporal_dropout: 0.0
554
+ low_rank_head: 2048
555
+ temporal_smoothing: null
556
+ name: FmriEncoder
557
+ loss:
558
+ kwargs:
559
+ reduction: none
560
+ name: MSELoss
561
+ optim:
562
+ optimizer:
563
+ lr: 0.0001
564
+ kwargs:
565
+ weight_decay: 0.0
566
+ name: Adam
567
+ scheduler:
568
+ kwargs:
569
+ max_lr: 0.0001
570
+ pct_start: 0.1
571
+ name: OneCycleLR
572
+ interval: step
573
+ name: LightningOptimizer
574
+ metrics:
575
+ - log_name: pearson
576
+ dim: 0
577
+ reduction: mean
578
+ torchmetrics_kwargs: null
579
+ name: OnlinePearsonCorr
580
+ - log_name: subj_pearson
581
+ metric_name: OnlinePearsonCorr
582
+ kwargs:
583
+ dim: 0
584
+ name: GroupedMetric
585
+ - log_name: retrieval_top1
586
+ topk: 1
587
+ name: TopkAcc
588
+ monitor: val/pearson
589
+ wandb_config:
590
+ name: brain_model_config.subject_layers.subject_dropout=0.10,data.study.names=(Algonauts2025Bold,Lahner2024Bold,Lebel2023Bold,Wen2017)-ecdd89b2
591
+ group: half_depth
592
+ entity: sdascoli
593
+ project: tribe_release
594
+ offline: false
595
+ host: https://fairwandb.org
596
+ id: null
597
+ dir: null
598
+ anonymous: null
599
+ log_model: false
600
+ experiment: null
601
+ prefix: ''
602
+ resume: allow
603
+ accelerator: gpu
604
+ n_epochs: 15
605
+ max_steps: -1
606
+ patience: null
607
+ limit_train_batches: null
608
+ accumulate_grad_batches: 1
609
+ enable_progress_bar: true
610
+ log_every_n_steps: 5
611
+ fast_dev_run: false
612
+ save_checkpoints: true
613
+ checkpoint_filename: best
614
+ resize_subject_layer: false
615
+ freeze_backbone: false
616
+ average_subjects: false
617
+ checkpoint_path: null
618
+ load_checkpoint: true
619
+ test_only: false
e2e_output/.DS_Store ADDED
Binary file (6.15 kB). View file
 
e2e_output/brain_coolwarm.png ADDED

Git LFS Details

  • SHA256: c369a50c8bb303cc119d133a35806289e18fe8e0347199ef4d5e8df7d861c56d
  • Pointer size: 131 Bytes
  • Size of remote file: 266 kB
e2e_output/brain_coolwarm.svg ADDED
e2e_output/brain_dorsal.png ADDED

Git LFS Details

  • SHA256: f71d4bf7cbd524a050559f10f0c66a5603f0684540c961679d06253a6d6bc5b0
  • Pointer size: 131 Bytes
  • Size of remote file: 221 kB
e2e_output/brain_dorsal.svg ADDED
e2e_output/brain_left.png ADDED

Git LFS Details

  • SHA256: a6f079103ba47d08d2b468f92ae58de20a68c31f8cd1768ea487c69e031fe617
  • Pointer size: 131 Bytes
  • Size of remote file: 278 kB
e2e_output/brain_left.svg ADDED
e2e_output/brain_mosaic.png ADDED

Git LFS Details

  • SHA256: c4251d81d8841a4c45c7184b008df333a5798151dd0a044ba11b4a4afbcac4fd
  • Pointer size: 131 Bytes
  • Size of remote file: 242 kB
e2e_output/brain_mosaic.svg ADDED
e2e_output/brain_rgb_overlay.png ADDED

Git LFS Details

  • SHA256: 5d92a65121d8cfaca6580030ec886d55c6abc65f98d6d83ca4cdc597a78c2eef
  • Pointer size: 131 Bytes
  • Size of remote file: 419 kB
e2e_output/brain_rgb_overlay.svg ADDED
e2e_output/brain_right.png ADDED

Git LFS Details

  • SHA256: 13879499edbca06cde2e48b2fb41a5aaeef44f9679b1880e755a1946b6840fb6
  • Pointer size: 131 Bytes
  • Size of remote file: 277 kB
e2e_output/brain_right.svg ADDED
e2e_output/predictions.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:822ef4161933ddf101970c19ebfabcb55b42393cbeb82847060baab38da95125
3
+ size 8193600
e2e_output/predictions_segmented.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7aaebd4979b25ea967599e6f303991c6754aeb85c447942534289cc7d8e457f
3
+ size 16387200
e2e_output/timesteps/frame_0000.png ADDED
e2e_output/timesteps/frame_0000.svg ADDED
e2e_output/timesteps/frame_0001.png ADDED

Git LFS Details

  • SHA256: fd28436dd9707ac1b288b6c750d3fff75b8ec26c8892c526e2ae91d227867c6d
  • Pointer size: 131 Bytes
  • Size of remote file: 102 kB
e2e_output/timesteps/frame_0001.svg ADDED
e2e_output/timesteps/frame_0002.png ADDED

Git LFS Details

  • SHA256: 9f49f87ccd31f31ebdab7d129d8e13fba8dbed2a34c58162e06615e4aa080563
  • Pointer size: 131 Bytes
  • Size of remote file: 103 kB
e2e_output/timesteps/frame_0002.svg ADDED
e2e_output/timesteps/frame_0003.png ADDED
e2e_output/timesteps/frame_0003.svg ADDED
e2e_output/timesteps/frame_0004.png ADDED

Git LFS Details

  • SHA256: 1a67b1563a83edb388a507bd3cf5d54ee66167dec41c8aa9e16a2918e0059bdf
  • Pointer size: 131 Bytes
  • Size of remote file: 102 kB
e2e_output/timesteps/frame_0004.svg ADDED
fsaverage5/surf/lh.curv ADDED
Binary file (41 kB). View file
 
fsaverage5/surf/lh.infl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c521035f63cff7eb9b2be1778ec7db5accfa7247f4e0626ecf06f147eedf1192
3
+ size 368720
fsaverage5/surf/lh.pial ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34cccc5b7f64aaef670beb90ae3423aa258c2d6d81275fc8372d9a34ed97fb83
3
+ size 368720
fsaverage5/surf/lh.sulc ADDED
Binary file (41 kB). View file
 
fsaverage5/surf/lh.white ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46756fc30763b48ca69fc4b2bd0a2c7fe640a5f32f2d8920f63bd42c59f773f0
3
+ size 368720
fsaverage5/surf/rh.curv ADDED
Binary file (41 kB). View file
 
fsaverage5/surf/rh.infl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd829b1b09a1324a71b9a5a7ba4a9dfb2c67ca25c906e5453cdac7e1cd29bc79
3
+ size 368720
fsaverage5/surf/rh.pial ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:174fc904af97a4a38619355e3211427a57ae336922e5b941a15f2a0d7daabf50
3
+ size 368720
fsaverage5/surf/rh.sulc ADDED
Binary file (41 kB). View file
 
fsaverage5/surf/rh.white ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fafaff5f1cadc4342ac6b9c7b3cfc9884d208f57bab8815edf0c6097eab303e5
3
+ size 368720
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c48e25476616b955eb4ba40ad65200e9688b29ec050f72bdd9492471bbf946a
3
+ size 708832652