Update README.md
Browse files
README.md
CHANGED
|
@@ -26,18 +26,18 @@ This repository contains **subject-specific voxelwise encoding-model fits** from
|
|
| 26 |
**arXiv:** `2607.19316`<br>
|
| 27 |
**Contributed Talk:** [CCN 2026 presentation on YouTube](https://www.youtube.com/watch?v=Lb4S3FWqd2M&t=2545s)
|
| 28 |
|
| 29 |
-
The encoding models were fit to fMRI responses from the **Natural Scenes Dataset (NSD)** using
|
| 30 |
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
-
The associated study
|
| 39 |
|
| 40 |
-
### Conditions
|
| 41 |
|
| 42 |
<p align="center">
|
| 43 |
<img
|
|
@@ -47,7 +47,8 @@ The associated study investigated whether constraining visual experience to diff
|
|
| 47 |
>
|
| 48 |
</p>
|
| 49 |
|
| 50 |
-
The
|
|
|
|
| 51 |
- **Baseline:** full-field input without an eccentricity-specific restriction
|
| 52 |
- **Fovea-Gaze:** gaze-centered central-only input
|
| 53 |
- **Periph:** peripheral-only input produced by masking the gaze-centered central region
|
|
@@ -61,9 +62,9 @@ The associated model variants manipulate the same source frames in complementary
|
|
| 61 |
| Model card | Available |
|
| 62 |
| Feature-extraction / model-fitting code | Forthcoming |
|
| 63 |
| Analysis code | Forthcoming |
|
| 64 |
-
| VEDB imagery |
|
| 65 |
|
| 66 |
-
The
|
| 67 |
|
| 68 |
## Repository Structure
|
| 69 |
|
|
@@ -96,9 +97,20 @@ fovea-gaze/
|
|
| 96 |
<em>Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.</em>
|
| 97 |
</p>
|
| 98 |
|
| 99 |
-
## Encoding
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
|
| 101 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
|
| 103 |
```text
|
| 104 |
conv1
|
|
@@ -109,18 +121,40 @@ layer4.1
|
|
| 109 |
avgpool
|
| 110 |
```
|
| 111 |
|
| 112 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
|
| 114 |
-
|
| 115 |
|
| 116 |
-
*
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
## Loading a Fit
|
| 126 |
|
|
@@ -135,9 +169,20 @@ fit = np.load(
|
|
| 135 |
weights = fit["weights"]
|
| 136 |
r2 = fit["r2"]
|
| 137 |
corr = fit["corr"]
|
|
|
|
|
|
|
| 138 |
best_lambda_inds = fit["best_lambda_inds"]
|
|
|
|
|
|
|
|
|
|
| 139 |
```
|
| 140 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
## Related Models
|
| 142 |
|
| 143 |
This repository is associated with the **[Eccentricity-Constrained SimCLR Models (VEDB)](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb)** collection.
|
|
@@ -146,11 +191,11 @@ This repository is associated with the **[Eccentricity-Constrained SimCLR Models
|
|
| 146 |
- [VEDB SimCLR ResNet-18 β Fovea-Gaze](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Fovea-Gaze)
|
| 147 |
- [VEDB SimCLR ResNet-18 β Periph](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph)
|
| 148 |
- [VEDB SimCLR ResNet-18 β Periph-NF](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph-NF)
|
| 149 |
-
- [VEDB NSD ResNet-18 β Encoding
|
| 150 |
|
| 151 |
## Citation
|
| 152 |
|
| 153 |
-
If you use these encoding
|
| 154 |
|
| 155 |
```bibtex
|
| 156 |
@inproceedings{diaz2026eccentricity,
|
|
@@ -176,6 +221,6 @@ See the [VEDB dataset paper](https://jov.arvojournals.org/article.aspx?articleid
|
|
| 176 |
|
| 177 |
## License
|
| 178 |
|
| 179 |
-
The encoding
|
| 180 |
|
| 181 |
-
The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included
|
|
|
|
| 26 |
**arXiv:** `2607.19316`<br>
|
| 27 |
**Contributed Talk:** [CCN 2026 presentation on YouTube](https://www.youtube.com/watch?v=Lb4S3FWqd2M&t=2545s)
|
| 28 |
|
| 29 |
+
The encoding models were fit to fMRI responses from the **Natural Scenes Dataset (NSD)** using features extracted from four [VEDB-pretrained SimCLR ResNet-18 models](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb):
|
| 30 |
|
| 31 |
+
- **Baseline**
|
| 32 |
+
- **Fovea-Gaze**
|
| 33 |
+
- **Periph**
|
| 34 |
+
- **Periph-NF**
|
| 35 |
|
| 36 |
+
The four visual models use a ResNet-18 encoder pretrained with SimCLR self-supervised contrastive learning on naturalistic egocentric imagery from the [**Visual Experience Dataset (VEDB)**](https://jov.arvojournals.org/article.aspx?articleid=2802101). VEDB includes synchronized gaze measurements, allowing the training images to be modified according to gaze-centered visual-field conditions.
|
| 37 |
|
| 38 |
+
The associated study tested whether these differences in visual experience produced systematic changes in learned representations, downstream classification performance, and correspondence with human visual cortex.
|
| 39 |
|
| 40 |
+
### Visual-Field Conditions
|
| 41 |
|
| 42 |
<p align="center">
|
| 43 |
<img
|
|
|
|
| 47 |
>
|
| 48 |
</p>
|
| 49 |
|
| 50 |
+
The four pretrained models were trained on complementary versions of the same source imagery:
|
| 51 |
+
|
| 52 |
- **Baseline:** full-field input without an eccentricity-specific restriction
|
| 53 |
- **Fovea-Gaze:** gaze-centered central-only input
|
| 54 |
- **Periph:** peripheral-only input produced by masking the gaze-centered central region
|
|
|
|
| 62 |
| Model card | Available |
|
| 63 |
| Feature-extraction / model-fitting code | Forthcoming |
|
| 64 |
| Analysis code | Forthcoming |
|
| 65 |
+
| VEDB imagery | Full dataset not redistributed; example frames included for illustration |
|
| 66 |
|
| 67 |
+
The feature-extraction, model-fitting, and analysis code is currently being consolidated and documented and will be linked here when released.
|
| 68 |
|
| 69 |
## Repository Structure
|
| 70 |
|
|
|
|
| 97 |
<em>Overview of the VEDB preprocessing, SimCLR pretraining, downstream linear probes, and voxelwise encoding workflow.</em>
|
| 98 |
</p>
|
| 99 |
|
| 100 |
+
## Encoding-Model Procedure
|
| 101 |
+
|
| 102 |
+
The encoding models were fit separately for each **NSD subject (S1βS8)** and each VEDB-pretrained model condition.
|
| 103 |
+
|
| 104 |
+
NSD stimuli were presented to the pretrained ResNet-18 models as intact `224 Γ 224` images. The Baseline, Fovea-Gaze, Periph, and Periph-NF transformations were used during SimCLR pretraining and were **not reapplied to NSD images** during encoding-model evaluation.
|
| 105 |
|
| 106 |
+
Before feature extraction, NSD images were rescaled to `[0, 1]` and normalized using ImageNet channel statistics:
|
| 107 |
+
|
| 108 |
+
```text
|
| 109 |
+
mean = [0.485, 0.456, 0.406]
|
| 110 |
+
std = [0.229, 0.224, 0.225]
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
Features were extracted from six points in the ResNet-18 encoder:
|
| 114 |
|
| 115 |
```text
|
| 116 |
conv1
|
|
|
|
| 121 |
avgpool
|
| 122 |
```
|
| 123 |
|
| 124 |
+
Convolutional feature maps were reduced with adaptive average pooling and then flattened. PCA was applied separately to each layer, retaining the top **200 principal components**. The six 200-component representations were then concatenated before voxelwise model fitting.
|
| 125 |
+
|
| 126 |
+
PCA was fit separately for each subject, model condition, and feature layer. In the analysis used for the study, PCA was applied to the full subject-specific feature matrix before the encoding-model train and holdout partitions were applied.
|
| 127 |
+
|
| 128 |
+
### Feature Normalization and Ridge Fitting
|
| 129 |
+
|
| 130 |
+
For each subject, the feature matrix was divided into a training partition, a nested holdout partition for ridge-penalty selection, and a final held-out evaluation partition containing the **1,000 NSD images shared across participants**.
|
| 131 |
+
|
| 132 |
+
Feature-wise means and standard deviations were estimated from the combined training and nested-holdout partitions. The same parameters were then used to z-score all three partitions. The final held-out evaluation set was excluded when estimating these normalization parameters.
|
| 133 |
|
| 134 |
+
A column of ones was appended to the feature matrix to provide an intercept term.
|
| 135 |
|
| 136 |
+
Voxel responses were modeled with **L2-regularized linear regression (ridge regression)**. Twenty candidate ridge penalties were evaluated. The penalty that minimized prediction error on the nested holdout partition was selected independently for each voxel.
|
| 137 |
+
|
| 138 |
+
Performance on the final held-out set was quantified with voxelwise coefficient of determination (`RΒ²`) and Pearson correlation (`corr`).
|
| 139 |
+
|
| 140 |
+
## Saved Fit Contents
|
| 141 |
+
|
| 142 |
+
Each `.npy` file contains a Python dictionary with the fitted encoding model and associated metadata:
|
| 143 |
+
|
| 144 |
+
- `subject` β NSD subject number
|
| 145 |
+
- `model` β visual model condition
|
| 146 |
+
- `features_file_list` β feature files used for model fitting
|
| 147 |
+
- `lambdas` β candidate ridge penalties
|
| 148 |
+
- `weights` β fitted voxelwise ridge-regression weights
|
| 149 |
+
- `best_lambda_inds` β selected ridge-penalty index for each voxel
|
| 150 |
+
- `r2` β held-out voxelwise coefficient of determination
|
| 151 |
+
- `corr` β held-out voxelwise correlation
|
| 152 |
+
- `voxel_mask` β subject voxel mask
|
| 153 |
+
- `voxel_index` β voxel indices
|
| 154 |
+
- `voxel_nc` β voxel noise-ceiling estimates
|
| 155 |
+
- `brain_nii_shape` β original volumetric brain-image shape
|
| 156 |
+
|
| 157 |
+
Because the intercept was added as a final column in the feature matrix, the **final row of `weights` contains the fitted intercept term**.
|
| 158 |
|
| 159 |
## Loading a Fit
|
| 160 |
|
|
|
|
| 169 |
weights = fit["weights"]
|
| 170 |
r2 = fit["r2"]
|
| 171 |
corr = fit["corr"]
|
| 172 |
+
|
| 173 |
+
lambdas = fit["lambdas"]
|
| 174 |
best_lambda_inds = fit["best_lambda_inds"]
|
| 175 |
+
|
| 176 |
+
voxel_index = fit["voxel_index"]
|
| 177 |
+
voxel_nc = fit["voxel_nc"]
|
| 178 |
```
|
| 179 |
|
| 180 |
+
## Scope of the Released Artifacts
|
| 181 |
+
|
| 182 |
+
The `.npy` files contain the fitted voxelwise encoding models, prediction metrics, ridge-penalty information, and voxel metadata used in the study.
|
| 183 |
+
|
| 184 |
+
Reproducing voxel predictions for new images also requires the corresponding pretrained ResNet-18 checkpoint and the feature-extraction, spatial-pooling, PCA, concatenation, and normalization procedures used during fitting. These steps will be documented in the accompanying public code release.
|
| 185 |
+
|
| 186 |
## Related Models
|
| 187 |
|
| 188 |
This repository is associated with the **[Eccentricity-Constrained SimCLR Models (VEDB)](https://hf.co/collections/DM-Diaz/eccentricity-constrained-simclr-models-vedb)** collection.
|
|
|
|
| 191 |
- [VEDB SimCLR ResNet-18 β Fovea-Gaze](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Fovea-Gaze)
|
| 192 |
- [VEDB SimCLR ResNet-18 β Periph](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph)
|
| 193 |
- [VEDB SimCLR ResNet-18 β Periph-NF](https://huggingface.co/DM-Diaz/VEDB-SimCLR-ResNet18-Periph-NF)
|
| 194 |
+
- [VEDB NSD ResNet-18 β Encoding Models](https://huggingface.co/DM-Diaz/VEDB-NSD-ResNet18-Encoding-Models)
|
| 195 |
|
| 196 |
## Citation
|
| 197 |
|
| 198 |
+
If you use these encoding-model fits in academic work, please cite the associated study:
|
| 199 |
|
| 200 |
```bibtex
|
| 201 |
@inproceedings{diaz2026eccentricity,
|
|
|
|
| 221 |
|
| 222 |
## License
|
| 223 |
|
| 224 |
+
The encoding-model fits in this repository are released under the **Apache License 2.0**.
|
| 225 |
|
| 226 |
+
The VEDB dataset and other third-party resources used in the associated study remain subject to their respective licenses, access requirements, and terms of use. This repository does not redistribute the full VEDB dataset; a small number of example frames are included to illustrate the published visual-field manipulations.
|