encoder-tvsd

These models predict how the macaque ventral visual stream responds to an image, as recorded in the THINGS Ventral Stream Spiking Dataset (TVSD): the multi-unit activity in V1, V4 and IT of two monkeys, one number per image for each of 1,704 reliable electrodes. Four variants trade accuracy for speed. All were fitted on the 22,248 TVSD training images and scored on its 100 test images.

Quick start

pip install torch transformers pillow

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "epfl-neuroai/encoder-tvsd", revision="v1", trust_remote_code=True
).to("cuda")

images = [Image.open("dog.jpg"), Image.open("teapot.jpg")]
responses = model(images)  # {"monkeyN": {"V1": (2, 437), "V4": (2, 247), "IT": (2, 178)}, "monkeyF": {...}}
it = model(images, areas="IT")["monkeyF"]["IT"]  # (2, 241)

model.monkeys  # "monkeyN", "monkeyF"
model.config.electrode_index["monkeyN"]["V4"]  # each output's 0-based channel in THINGS_normMUA.mat

This loads the default variant, efficient. Add variant="tiny", "cnn" or "accurate" for another; only the selected variant's weights are downloaded (for accurate, the Qwen weights follow on the first call). The model takes a PIL image or a list of them and returns float32 tensors; batch_size (default 32) sets how many images go through the GPU at once.

trust_remote_code=True runs this repository's Python files on your machine: modeling.py, the model in plain PyTorch, and two short transformers wrappers. Review them and pin revision. Without transformers (pip install torch pillow safetensors huggingface-hub):

import importlib.util
from huggingface_hub import snapshot_download

code = snapshot_download("epfl-neuroai/encoder-tvsd", revision="v1", allow_patterns=["*.py"])
spec = importlib.util.spec_from_file_location("encoder_tvsd", f"{code}/modeling.py")
modeling = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modeling)
model = modeling.load("epfl-neuroai/encoder-tvsd", revision="v1", device="cuda")

The electrode indices are then in the repository's config.json, under electrode_index.

Variants

variant	image networks	test r / ceiling: V1	V4	IT	download	images/s	GPU memory
efficient (default)	ResNet-50 + ViT-S	0.923	0.877	0.868	6.7 GB	53	8.4 GiB
tiny	ResNet-18 + ViT-S	0.924	0.876	0.865	5.1 GB	63	6.7 GiB
cnn	ResNet-152	0.913	0.844	0.862	9.3 GB	45	10.8 GiB
accurate	ResNet-152 + DINOv2-B + Qwen3-VL-2B vision	0.935	0.898	0.888	12.6 GB + 4.3 GB from Qwen	6.3	16.5 GiB

efficient and tiny score almost the same; tiny is smaller and a little faster. cnn uses a single network and is weakest in V4. accurate is best in every area, 0.01 to 0.02 above efficient, at about a tenth of the speed, and downloads the Qwen3-VL-2B weights from Qwen's repository at a pinned revision. Speed is end to end from PIL images on one NVIDIA A100 (batch 32). Most of each download is the V1 and V4 readout weights, whose input features are large.

Scores. For each electrode, the Pearson correlation between predicted and recorded responses across the 100 test images, divided by the electrode's noise ceiling, then averaged over an area's electrodes and over the two monkeys. The best single model on the TVSD benchmark, an adversarially trained ResNet-152 without fine-tuning, reaches 0.842, 0.781 and 0.789 in V1, V4 and IT.

What the outputs are

TVSD releases, for every electrode, the multi-unit activity normalized and averaged in a time window after image onset, one value per image (training images averaged over their presentations, test images over their repetitions). The models predict these values for the electrodes whose released reliability is at least 0.1: 437, 247 and 178 in monkey N's V1, V4 and IT, and 462, 139 and 241 in monkey F's. model.config.electrode_index gives each output's 0-based index into the 1,024 channels of that monkey's THINGS_normMUA.mat, so the predictions line up with the dataset.

Using the features with other data

The readouts are specific to the recorded electrodes of these two monkeys. For other animals, recording sites or datasets, fit your own linear readout on the features the readouts use:

features = model.features(images, "IT")  # (images, dims): the input of the IT readout

The V1 and V4 features have many dimensions, so use a regularized regression such as ridge. We did this for other macaque recordings, using the V4 and IT features for V4 and IT targets and the V1 features for V1 and V2. Scores are r / ceiling on each benchmark's test images (for MacaqueITBench, r / √reliability), compared with the best single layer of these networks before fine-tuning. MacaqueITBench has no area labels; its row covers the 200 of its 222 recording sessions whose responses are best predicted by late layers.

dataset	accurate	cnn	efficient	tiny	best layer, not fine-tuned
Majaj-Hong V4 (Majaj et al., 2015)	0.794	0.788	0.777	0.779	0.733
Majaj-Hong IT	0.757	0.751	0.753	0.743	0.731
MacaqueITBench IT (Madan et al., 2024)	0.479	0.474	0.474	0.474	0.461
Freeman-Ziemba V1 (Freeman et al., 2013)	0.312	0.397	0.306	0.310	0.400
Freeman-Ziemba V2	0.413	0.432	0.401	0.401	0.468

The fine-tuning carries over to other recordings of V4 and IT with object images, rendered (Majaj-Hong) or photographed (MacaqueITBench). It does not carry over to the texture images of Freeman-Ziemba: there, the best layer without fine-tuning scores higher than the released features, and only cnn comes close in V1. For stimuli far from THINGS photographs, an off-the-shelf network may be the better basis.

Why DINOv2 and not DINOv3

We evaluated DINOv3 (ViT-B/16 and ViT-L/16) in the same parts of the models, with the same layer search and fine-tuning. It did not help: in our development ensembles, DINOv3-B matched DINOv2-B on held-out training images, and on the TVSD test images DINOv3-B and -L reached 0.907 and 0.908 against 0.909 with DINOv2-B (means over the three areas). DINOv2 is released under Apache 2.0 and can be downloaded freely, while DINOv3's weights are distributed under the DINOv3 License, with gated access on Meta's repositories and conditions that would carry over to these models.

How the encoders were built

• Data: TVSD (two macaques; 22,248 THINGS training images, 100 test images), the released normalized multi-unit activity.
• Image networks: adversarially trained ResNets from Gokce & Schrimpf (2025), ViT-S/16 and DINOv2-B from timm, and the Qwen3-VL-2B vision encoder, each cut at the layer that predicts an area best, so each area has its own layers (listed per variant in config.json). The blocks leading to that layer were fine-tuned to the training images' responses with low-rank updates (LoRA); some members also learn a small attention pooling layer over that feature map, trained together with the low-rank updates.
• Readout: each member's features (the layer's map, average-pooled at several grid sizes, or the attention pooling output) are mapped to an area's electrodes by a ridge regression per monkey, with a penalty per electrode. Each area combines its members with fixed weights chosen on held-out training images, and the final models were refitted on all training images.
• Preprocessing: ResNets and ViTs see a 224 x 224 center crop after resizing the shorter side to 256 pixels; Qwen sees a square center crop resized to 500 pixels, then to its 512 x 512 input.
• Export check: in float32 with TF32 off, the released files reproduce the original test predictions to within about 1e-4 of their standard deviation (5.3e-3 for accurate, whose Qwen blocks run in bfloat16), and the test scores to within 1e-5.

Limitations

• Fitted on two monkeys and one presentation paradigm. The outputs are specific to their electrodes; for other animals, fit a readout on model.features.
• Fitted on THINGS object photographs. Other image types (textures, scenes, faces in close-up) are out of the training distribution; see the Freeman-Ziemba results above.
• The ResNets and ViT-S were trained on ImageNet, licensed for non-commercial research; check this for commercial use.

License

Apache 2.0 for the weights and code in this repository. TVSD is CC-BY 4.0; the THINGS images are not redistributed. ViT-S and DINOv2 (timm) and Qwen3-VL-2B are Apache 2.0; the Qwen weights are downloaded from Qwen's repository, not redistributed here.

Citation

Please cite the benchmark and the ResNet checkpoints:

@inproceedings{gokce2026multimodal_brain_scaling,
  title     = {Multimodal Scaling Laws for Task \& Data-Optimized Models of Visual Cortex},
  author    = {Abdulkadir Gokce and Yingtian Tang and Martin Schrimpf},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=OQ6jQHJPTT}
}

@inproceedings{gokce2025scalingprimatevvs,
  title     = {Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream},
  author    = {Abdulkadir Gokce and Martin Schrimpf},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=WxY61MmHYo}
}

Also cite TVSD (Papale et al., 2025, Neuron 113, 539-553), THINGS (Hebart et al., 2019, PLOS ONE 14, e0223792), DINOv2 (Oquab et al., 2024, TMLR), ViT-S (Steiner et al., 2022, TMLR) and Qwen3-VL for accurate (Qwen Team, 2025, arXiv:2511.21631). Transfer datasets: Majaj-Hong (Majaj et al., 2015, J. Neurosci. 35, 13402-13418), Freeman-Ziemba (Freeman et al., 2013, Nat. Neurosci. 16, 974-981) and MacaqueITBench (Madan et al., 2024, arXiv:2406.16935).