DM-Diaz commited on
Commit
381844a
Β·
verified Β·
1 Parent(s): f499163

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +72 -27
README.md CHANGED
@@ -26,18 +26,18 @@ This repository contains **subject-specific voxelwise encoding-model fits** from
26
  **arXiv:** `2607.19316`<br>
27
  **Contributed Talk:** [CCN 2026 presentation on YouTube](https://www.youtube.com/watch?v=Lb4S3FWqd2M&t=2545s)
28
 
29
- The encoding models were fit to fMRI responses from the **Natural Scenes Dataset (NSD)** using representations extracted from four [VEDB-pretrained SimCLR ResNet-18 models](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb):
30
 
31
- * **Baseline**
32
- * **Fovea-Gaze**
33
- * **Periph**
34
- * **Periph-NF**
35
 
36
- These four models used a **ResNet-18 visual encoder pretrained with SimCLR self-supervised contrastive learning** on naturalistic egocentric imagery with synchronized human gaze data from the [**Visual Experience Dataset (VEDB)**](https://jov.arvojournals.org/article.aspx?articleid=2802101).
37
 
38
- The associated study investigated whether constraining visual experience to different portions of the visual field produces systematic differences in learned representations, downstream task performance, and alignment with human visual cortex.
39
 
40
- ### Conditions
41
 
42
  <p align="center">
43
  <img
@@ -47,7 +47,8 @@ The associated study investigated whether constraining visual experience to diff
47
  >
48
  </p>
49
 
50
- The associated model variants manipulate the same source frames in complementary ways:
 
51
  - **Baseline:** full-field input without an eccentricity-specific restriction
52
  - **Fovea-Gaze:** gaze-centered central-only input
53
  - **Periph:** peripheral-only input produced by masking the gaze-centered central region
@@ -61,9 +62,9 @@ The associated model variants manipulate the same source frames in complementary
61
  | Model card | Available |
62
  | Feature-extraction / model-fitting code | Forthcoming |
63
  | Analysis code | Forthcoming |
64
- | VEDB imagery | Not redistributed; available via [Databrary](https://www.databrary.org/volume/1612) |
65
 
66
- The complete feature-extraction, model-fitting, and evaluation codebase is currently being consolidated and documented and will be linked here upon public release.
67
 
68
  ## Repository Structure
69
 
@@ -96,9 +97,20 @@ fovea-gaze/
96
  <em>Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.</em>
97
  </p>
98
 
99
- ## Encoding Models
 
 
 
 
100
 
101
- For each subject and visual-field condition, features were extracted from:
 
 
 
 
 
 
 
102
 
103
  ```text
104
  conv1
@@ -109,18 +121,40 @@ layer4.1
109
  avgpool
110
  ```
111
 
112
- Layer features were dimensionally reduced with PCA, concatenated, and used to fit **voxelwise L2-regularized linear regression (ridge) encoding models**.
 
 
 
 
 
 
 
 
113
 
114
- Each `.npy` file contains a saved Python dictionary including:
115
 
116
- * fitted voxelwise `weights`
117
- * held-out `r2`
118
- * held-out `corr`
119
- * candidate `lambdas`
120
- * `best_lambda_inds`
121
- * voxel mask and index information
122
- * voxel noise ceilings
123
- * subject and model metadata
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
  ## Loading a Fit
126
 
@@ -135,9 +169,20 @@ fit = np.load(
135
  weights = fit["weights"]
136
  r2 = fit["r2"]
137
  corr = fit["corr"]
 
 
138
  best_lambda_inds = fit["best_lambda_inds"]
 
 
 
139
  ```
140
 
 
 
 
 
 
 
141
  ## Related Models
142
 
143
  This repository is associated with the **[Eccentricity-Constrained SimCLR Models (VEDB)](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb)** collection.
@@ -146,11 +191,11 @@ This repository is associated with the **[Eccentricity-Constrained SimCLR Models
146
  - [VEDB SimCLR ResNet-18 β€” Fovea-Gaze](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Fovea-Gaze)
147
  - [VEDB SimCLR ResNet-18 β€” Periph](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph)
148
  - [VEDB SimCLR ResNet-18 β€” Periph-NF](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph-NF)
149
- - [VEDB NSD ResNet-18 β€” Encoding-Models](https://huggingface.co/DM-Diaz/VEDB-NSD-ResNet18-Encoding-Models)
150
 
151
  ## Citation
152
 
153
- If you use these encoding model fits in academic work, please cite the associated study:
154
 
155
  ```bibtex
156
  @inproceedings{diaz2026eccentricity,
@@ -176,6 +221,6 @@ See the [VEDB dataset paper](https://jov.arvojournals.org/article.aspx?articleid
176
 
177
  ## License
178
 
179
- The encoding model fits in this repository are released under the **Apache License 2.0**.
180
 
181
- The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included for illustration of the published visual-field manipulations.
 
26
  **arXiv:** `2607.19316`<br>
27
  **Contributed Talk:** [CCN 2026 presentation on YouTube](https://www.youtube.com/watch?v=Lb4S3FWqd2M&t=2545s)
28
 
29
+ The encoding models were fit to fMRI responses from the **Natural Scenes Dataset (NSD)** using features extracted from four [VEDB-pretrained SimCLR ResNet-18 models](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb):
30
 
31
+ - **Baseline**
32
+ - **Fovea-Gaze**
33
+ - **Periph**
34
+ - **Periph-NF**
35
 
36
+ The four visual models use a ResNet-18 encoder pretrained with SimCLR self-supervised contrastive learning on naturalistic egocentric imagery from the [**Visual Experience Dataset (VEDB)**](https://jov.arvojournals.org/article.aspx?articleid=2802101). VEDB includes synchronized gaze measurements, allowing the training images to be modified according to gaze-centered visual-field conditions.
37
 
38
+ The associated study tested whether these differences in visual experience produced systematic changes in learned representations, downstream classification performance, and correspondence with human visual cortex.
39
 
40
+ ### Visual-Field Conditions
41
 
42
  <p align="center">
43
  <img
 
47
  >
48
  </p>
49
 
50
+ The four pretrained models were trained on complementary versions of the same source imagery:
51
+
52
  - **Baseline:** full-field input without an eccentricity-specific restriction
53
  - **Fovea-Gaze:** gaze-centered central-only input
54
  - **Periph:** peripheral-only input produced by masking the gaze-centered central region
 
62
  | Model card | Available |
63
  | Feature-extraction / model-fitting code | Forthcoming |
64
  | Analysis code | Forthcoming |
65
+ | VEDB imagery | Full dataset not redistributed; example frames included for illustration |
66
 
67
+ The feature-extraction, model-fitting, and analysis code is currently being consolidated and documented and will be linked here when released.
68
 
69
  ## Repository Structure
70
 
 
97
  <em>Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.</em>
98
  </p>
99
 
100
+ ## Encoding-Model Procedure
101
+
102
+ The encoding models were fit separately for each **NSD subject (S1–S8)** and each VEDB-pretrained model condition.
103
+
104
+ NSD stimuli were presented to the pretrained ResNet-18 models as intact `224 Γ— 224` images. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were used during SimCLR pretraining and were **not reapplied to NSD images** during encoding-model evaluation.
105
 
106
+ Before feature extraction, NSD images were rescaled to `[0, 1]` and normalized using ImageNet channel statistics:
107
+
108
+ ```text
109
+ mean = [0.485, 0.456, 0.406]
110
+ std = [0.229, 0.224, 0.225]
111
+ ```
112
+
113
+ Features were extracted from six points in the ResNet-18 encoder:
114
 
115
  ```text
116
  conv1
 
121
  avgpool
122
  ```
123
 
124
+ Convolutional feature maps were reduced with adaptive average pooling and then flattened. PCA was applied separately to each layer, retaining the top **200 principal components**. The six 200-component representations were then concatenated before voxelwise model fitting.
125
+
126
+ PCA was fit separately for each subject, model condition, and feature layer. In the analysis used for the study, PCA was applied to the full subject-specific feature matrix before the encoding-model train and holdout partitions were applied.
127
+
128
+ ### Feature Normalization and Ridge Fitting
129
+
130
+ For each subject, the feature matrix was divided into a training partition, a nested holdout partition for ridge-penalty selection, and a final held-out evaluation partition containing the **1,000 NSD images shared across participants**.
131
+
132
+ Feature-wise means and standard deviations were estimated from the combined training and nested-holdout partitions. The same parameters were then used to z-score all three partitions. The final held-out evaluation set was excluded when estimating these normalization parameters.
133
 
134
+ A column of ones was appended to the feature matrix to provide an intercept term.
135
 
136
+ Voxel responses were modeled with **L2-regularized linear regression (ridge regression)**. Twenty candidate ridge penalties were evaluated. The penalty that minimized prediction error on the nested holdout partition was selected independently for each voxel.
137
+
138
+ Performance on the final held-out set was quantified with voxelwise coefficient of determination (`RΒ²`) and Pearson correlation (`corr`).
139
+
140
+ ## Saved Fit Contents
141
+
142
+ Each `.npy` file contains a Python dictionary with the fitted encoding model and associated metadata:
143
+
144
+ - `subject` β€” NSD subject number
145
+ - `model` β€” visual model condition
146
+ - `features_file_list` β€” feature files used for model fitting
147
+ - `lambdas` β€” candidate ridge penalties
148
+ - `weights` β€” fitted voxelwise ridge-regression weights
149
+ - `best_lambda_inds` β€” selected ridge-penalty index for each voxel
150
+ - `r2` β€” held-out voxelwise coefficient of determination
151
+ - `corr` β€” held-out voxelwise correlation
152
+ - `voxel_mask` β€” subject voxel mask
153
+ - `voxel_index` β€” voxel indices
154
+ - `voxel_nc` β€” voxel noise-ceiling estimates
155
+ - `brain_nii_shape` β€” original volumetric brain-image shape
156
+
157
+ Because the intercept was added as a final column in the feature matrix, the **final row of `weights` contains the fitted intercept term**.
158
 
159
  ## Loading a Fit
160
 
 
169
  weights = fit["weights"]
170
  r2 = fit["r2"]
171
  corr = fit["corr"]
172
+
173
+ lambdas = fit["lambdas"]
174
  best_lambda_inds = fit["best_lambda_inds"]
175
+
176
+ voxel_index = fit["voxel_index"]
177
+ voxel_nc = fit["voxel_nc"]
178
  ```
179
 
180
+ ## Scope of the Released Artifacts
181
+
182
+ The `.npy` files contain the fitted voxelwise encoding models, prediction metrics, ridge-penalty information, and voxel metadata used in the study.
183
+
184
+ Reproducing voxel predictions for new images also requires the corresponding pretrained ResNet-18 checkpoint and the feature-extraction, spatial-pooling, PCA, concatenation, and normalization procedures used during fitting. These steps will be documented in the accompanying public code release.
185
+
186
  ## Related Models
187
 
188
  This repository is associated with the **[Eccentricity-Constrained SimCLR Models (VEDB)](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb)** collection.
 
191
  - [VEDB SimCLR ResNet-18 β€” Fovea-Gaze](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Fovea-Gaze)
192
  - [VEDB SimCLR ResNet-18 β€” Periph](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph)
193
  - [VEDB SimCLR ResNet-18 β€” Periph-NF](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph-NF)
194
+ - [VEDB NSD ResNet-18 β€” Encoding Models](https://huggingface.co/DM-Diaz/VEDB-NSD-ResNet18-Encoding-Models)
195
 
196
  ## Citation
197
 
198
+ If you use these encoding-model fits in academic work, please cite the associated study:
199
 
200
  ```bibtex
201
  @inproceedings{diaz2026eccentricity,
 
221
 
222
  ## License
223
 
224
+ The encoding-model fits in this repository are released under the **Apache License 2.0**.
225
 
226
+ The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included to illustrate the published visual-field manipulations.