encoder-things-eeg2

These models predict the EEG response to an image as recorded in THINGS-EEG2: 63 channels from 0 to 0.79 s after image onset, sampled at 100 Hz, in microvolts. By default the prediction is the average over the dataset's ten participants, the best guess for someone who was not in the dataset. Each participant's own prediction is available too.

The repository holds four variants that trade accuracy for speed. All were fitted on the 16,540 THINGS-EEG2 training images and scored on its 200 test images, whose 200 object concepts never appear in training.

Quick start

pip install torch transformers pillow

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "epfl-neuroai/encoder-things-eeg2", subfolder="efficient", revision="v1", trust_remote_code=True
).to("cuda")

images = [Image.open("dog.jpg"), Image.open("teapot.jpg")]
eeg = model(images)  # (2, 63, 80): average over participants
eeg_all = model(images, subjects="all")  # (2, 10, 63, 80)
eeg_one = model(images, subjects=["sub-03"])  # (2, 1, 63, 80)

model.subjects  # "sub-01" ... "sub-10"
model.channels  # channel names, in output order
model.times  # 0.0, 0.01, ..., 0.79 seconds after image onset

The model takes PIL images (one or a list; any mode, converted to RGB) and returns a float32 torch.Tensor on its device; use .cpu().numpy() for NumPy. batch_size (default 64) sets how many images go through the GPU at once; lower it on smaller GPUs. For efficient, tiny and cnn, wrapping the call in torch.autocast("cuda", torch.float16) speeds up the GPU part 1.5 to 4 times and changes outputs by at most 0.3% of their standard deviation. accurate runs part of its network in bfloat16, so its outputs change slightly (up to 0.3% of their standard deviation) with batch_size.

trust_remote_code=True runs the Python files of this repository on your machine: modeling.py, the model in plain PyTorch, and two short transformers wrappers. Review them, and pin revision so the reviewed code is the code that runs.

Without transformers (pip install torch pillow safetensors huggingface-hub numpy), load the same files with the loader in modeling.py:

import importlib.util
from huggingface_hub import snapshot_download
from PIL import Image

repo = snapshot_download(
    "epfl-neuroai/encoder-things-eeg2", revision="v1", allow_patterns=["*.py", "efficient/*"]
)
spec = importlib.util.spec_from_file_location("encoder_things_eeg2", f"{repo}/modeling.py")
modeling = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modeling)

model = modeling.load(repo, variant="efficient", device="cuda")
eeg = model([Image.open("dog.jpg")])  # (1, 63, 80), same as above

Variants

variant	image networks	test r, all channels	test r, occipital and parietal	test r, new participant	download
efficient (default)	ResNet-50 + ViT-S	0.241	0.345	0.168	1.3 GB
tiny	ResNet-18 + ViT-S	0.235	0.335	0.164	0.25 GB
cnn	ResNet-152	0.245	0.350	0.171	1.9 GB
accurate	ResNet-152 + Qwen3-Omni vision	0.261	0.373	0.184	7.5 GB + 4.0 GB from Qwen

variant	images/s, end to end	images/s, GPU only	peak GPU memory
efficient	68	3,700	2.0 GiB
tiny	73	6,700	0.7 GiB
cnn	63	490	2.7 GiB
accurate	15	45*	9.0 GiB

efficient is a good default: close to the best scores, and fast. tiny is a fifth of the download and a third of the memory, and its networks run almost twice as fast, but end to end it is only slightly faster, because resizing the images on the CPU sets the pace for both. cnn scores about the same as efficient with a single convolutional network, at an eighth of its GPU speed. accurate scores best and runs at about a fifth of efficient's speed; the first call downloads the Qwen3-Omni vision weights (one 4.0 GB file) from Qwen's repository at a pinned revision.

Scores. Each participant has their own readout, a linear map from image features to their EEG. Test r is the Pearson correlation between predicted and recorded responses across the 200 test images, computed for every channel and time point, averaged over the channel-time points whose noise ceiling exceeds 0.1, then over the ten participants. The mean noise ceiling of those points is 0.47 over all channels and 0.59 over occipital and parietal channels (O*, PO*, P*); the scores are raw r, not divided by the ceiling. The first two score columns use each participant's own readout. "New participant" scores each participant with the average of the other nine readouts, which is what the default output (the average of all ten) does for someone outside the dataset.

A bootstrap over participants and test images gives 95% intervals of about ±0.03 for each score (efficient: 0.241 [0.213, 0.265]). Differences between variants, measured on the same images, are more precise: accurate exceeds efficient by 0.020 [0.012, 0.028] and efficient exceeds tiny by 0.006 [0.002, 0.011], while cnn and efficient do not differ (0.004 [-0.004, 0.012]). The THINGS-EEG2 benchmark of Gokce et al. (2026), where frozen networks are read out with one ridge penalty per region, has a leaderboard best of 0.220 (all channels) and 0.324 (occipital and parietal).

Speed. Measured on one NVIDIA A100 80 GB (AMD EPYC 7543 CPU) with PyTorch defaults and batches of 64 images (16 for accurate); medians over 9 runs. "End to end" starts from decoded PIL images and includes resizing them on the CPU, so it depends on the CPU. "GPU only" starts from resized images and uses float16 autocast; *accurate runs its Qwen part in bfloat16 and the rest in float32, without autocast. Peak GPU memory is for an end-to-end call at that batch size, weights included.

Why the score is a correlation across images

Most of the signal in both the recorded and the predicted EEG is the same for every image. Any image evokes roughly the same event-related potential (ERP). Images followed each other every 200 ms, so each 0.8 s epoch also holds the responses to the images before and after it, which add a strong 5 Hz rhythm. On top of that, the signal drifts slowly up or down within a recording run. The 0.1 Hz high-pass applied during recording removes the slowest drift and baseline correction removes each epoch's offset, but a slope remains within each epoch. None of this depends on which image was shown, although the slope and the neighbouring images change from trial to trial and so add noise across images.

The score asks a narrower question: at each channel and time point, do the predicted responses go up and down across images with the recorded ones? Pearson r across images measures this linear agreement. It ignores the mean over images (the shared waveform) and the scale.

R² (one minus the mean squared error divided by the variance of the recordings across images) asks more: the predicted values must also have the right offset and scale. Even a prediction with the best possible offset and scale and r = 0.24 would explain only about 6% of the variance across images. And because the image-independent part of the signal is many times larger than the image-dependent part, a small mismatch in it, such as a different ERP amplitude or latency, can be as large as the whole spread across images. So a positive r with a negative R² does not mean the model is broken. The table under "Comparing amplitudes" below shows this for efficient.

Using the predictions with your own data

Matching your recordings

The output follows the THINGS-EEG2 benchmark of Gokce et al. (2026): the trial-averaged response to an image shown for 100 ms in a rapid sequence of five images per second, on the recording reference (Fz, which is not among the 63 outputs), in microvolts. To compare it with your recordings:

1. Cut epochs from -0.2 to 0.8 s around image onset, subtract the mean of the 200 ms before onset, resample to 100 Hz, keep 0 to 0.79 s (80 samples) and average each image's repetitions. If your amplifier records from DC, high-pass the continuous recording at 0.1 Hz first, as THINGS-EEG2 did; on the datasets below this changed scores by less than 0.003.
2. Keep the channels you share with model.channels, in the same order, and re-reference both the predictions and your data to their average. To include Fz, append a channel of zeros to the predictions first.
3. Check for a constant delay. Consumer headsets and other presentation setups often add tens of milliseconds between the trigger and the image.

import numpy as np

predicted = model(images).cpu().numpy()  # (images, 63, 80)
# recorded: your (images, channels, 80) trial averages; names: their channel names
shared = [name for name in model.channels if name in names]
p = predicted[:, [model.channels.index(name) for name in shared]]
q = recorded[:, [names.index(name) for name in shared]]
p, q = p - p.mean(1, keepdims=True), q - q.mean(1, keepdims=True)  # average reference
p, q = p - p.mean(0), q - q.mean(0)
r = (p * q).sum(0) / np.sqrt((p**2).sum(0) * (q**2).sum(0))  # (channels, 80), r across images

Comparing amplitudes

For R², mean squared error or overlaid waveforms, remove the leftover slope from the predictions and from your trial averages in the same way: detrend each 0 to 0.79 s epoch (pip install scipy).

from scipy.signal import detrend

predicted = detrend(model(images).cpu().numpy(), axis=-1)
recorded = detrend(recorded, axis=-1)  # your (images, 63, 80) trial averages

For efficient on the test images, with the same processing applied to both sides:

processing	own readout: r	own readout: R²	new participant: r	new participant: R²
none	0.241	-0.005	0.168	-1.27
mean over the test images subtracted	0.241	0.085	0.168	0.031
linear detrend of each epoch	0.293	0.052	0.195	-0.78
1 Hz high-pass of each epoch	0.272	0.034	0.178	-0.91
1 Hz high-pass, then detrend	0.281	0.033	0.182	-0.97

Subtracting each side's mean over the test images gives the best possible offset, but it uses the images being scored. Detrending needs no such information and makes R² positive for 9 of the 10 participants' own readouts. It does not do so for a new participant, whose ERP amplitude and timing differ from the average. The step has to be the same on both sides. If your recordings were high-passed at 1 Hz or detrended before epoching, processing the 0.8 s predictions cannot reproduce that. When we filtered THINGS-EEG2 this way, r was about 0.25, but R² stayed between -0.13 and -0.06 with processed predictions (own readout) and fell to about -4 with unprocessed ones. For calibrated amplitudes, fit an offset and gain per channel and time point on images you do not evaluate on (with few images, one gain per channel).

Other datasets

We applied efficient without refitting to three other image-EEG datasets, following the steps above: shared channels re-referenced to their average, and one constant delay, chosen on half of the images, with scores from the other half (and then the halves swapped). Scores are Pearson r across images for the default output, against each participant's trial averages and against their mean over participants, over channel-time points whose noise ceiling in that dataset exceeds 0.1.

dataset	participants	shared channels	images	repetitions	presentation
Alljoined-1.6M (Xu et al., 2025)	20	32 (consumer headset)	the 200 THINGS-EEG2 test images	about 80	100 ms on, 100 ms off
THINGS-EEG1 (Grootswagers et al., 2022)	48	61	200 THINGS images not in THINGS-EEG2	12	50 ms on, 50 ms off
Alljoined1 (Xu et al., 2024)	8	59	960 natural scenes from NSD	7 or 8	300 ms on, 300-350 ms off

dataset	recordings lag the markers by	r, each participant	r, mean over participants
Alljoined-1.6M	40 ms	0.098	0.272
THINGS-EEG1	0-10 ms	0.037	0.047
Alljoined1	60-70 ms	0.040	0.034

Where the two halves chose different delays, both are given. On Alljoined-1.6M, which showed the same images, the THINGS-EEG2 recordings themselves (averaged over its ten participants) reach r = 0.298 against the Alljoined-1.6M mean, so the model gets about 90% of the way to that cross-dataset ceiling. THINGS-EEG1 presents images twice as fast and Alljoined1 shows scenes instead of single objects, with slower presentation and a different system; both keep much less of the image-specific signal.

How the encoders were built

EEG data. THINGS-EEG2 (Gifford et al., 2022) recorded 10 participants viewing 16,540 training images (1,654 concepts), each shown 4 times, and 200 test images (200 other concepts), each shown 80 times. The recording software band-passed the EEG at 0.1-100 Hz. The benchmark then cut epochs from -0.2 to 0.8 s around image onset, subtracted the mean of the 200 ms before onset, rejected no trials, resampled to 100 Hz (with anti-aliasing) and kept 0-0.79 s; there is no other filtering. Each image's repetitions were averaged.

Image features.

• Adversarially trained ResNets from Gokce & Schrimpf (2025), trained on ImageNet: ResNet-50 at the end of its third stage (block 5), ResNet-152 at block 33 of the 36 in its third stage (counting from 0), ResNet-18 after the first block of its fourth stage. ResNet-50 and ResNet-18 use the feature map both in full and average-pooled to 1 x 1, 2 x 2 and 4 x 4 grids; ResNet-152 uses the full map only.
• ViT-S/16 from timm (ImageNet-21k, fine-tuned on ImageNet-1k): the mean over patch tokens after block 7 (the 8th of 12), before the final norm.
• The Qwen3-Omni vision encoder: the mean over patch tokens of each of blocks 11 to 18.

Fine-tuning. The last blocks of each network were adapted to the EEG of the THINGS-EEG2 training images with low-rank updates (LoRA, rank 4). cnn and accurate combine several fine-tuned copies of the same network, whose features are concatenated.

Readout. Each network's features are centered and scaled on the training images, and the ResNet features are compressed to their top principal components (1,024 for ResNet-50, 256 otherwise). The networks are combined with fixed weights (ResNet / transformer: 0.7 / 0.3 for efficient, 0.6 / 0.4 for tiny, 0.4 / 0.6 for accurate), chosen on held-out training concepts. A ridge regression per participant maps the result to the 5,040 channel-time values, with a separate penalty for each participant, channel and time point, chosen by leave-one-concept-out cross-validation. The default output averages the ten readouts.

Image preprocessing. ResNets and ViT-S see the image resized so its shorter side is 256 pixels (Lanczos), center-cropped to 224 x 224 and normalized. The Qwen encoder sees a square center crop resized to 500 x 500 (the size of the THINGS-EEG2 image files) and then Qwen's own processing at 512 x 512. Images far from square lose their edges.

Export check. In float32 with TF32 off, the released files reproduce the original pipeline's test predictions to within 1e-5 of their standard deviation (about 3e-5 for accurate at batch size 16). PyTorch uses TF32 for float32 convolutions by default on NVIDIA Ampere and newer GPUs (the speeds above were measured that way); with that setting, outputs differ from the TF32-off results by up to 0.9% of their standard deviation.

Limitations

• The encoders learned from ten adults, one lab and one presentation paradigm. New participants, devices and paradigms lower accuracy (see "Other datasets" above); the average readout is the best starting point for them.
• The predicted waveform includes the average overlap from neighbouring images in a 5 Hz sequence. For slower or single-image presentation, the waveform after about 200 ms will not match, which hurts R² more than r.
• THINGS images show single objects in natural contexts. Predictions for scenes, text, close-up faces or other image types have not been validated.
• The ResNets and ViT-S were trained on ImageNet, whose images are licensed for non-commercial research; whether that restricts models trained on them is unsettled. Check this for commercial use.

License

Apache 2.0 for the weights and code in this repository. The EEG data the readouts were fitted on, THINGS-EEG2, is CC-BY 4.0; the THINGS images are not redistributed. The ViT-S weights (timm, vit_small_patch16_224.augreg_in21k_ft_in1k) are Apache 2.0. The Qwen3-Omni weights used by accurate are Apache 2.0 and are downloaded from Qwen's repository, not redistributed here.

Citation

If you use these encoders, please cite the benchmark (OpenReview) and the ResNet checkpoints (arXiv:2411.05712):

@inproceedings{gokce2026multimodal_brain_scaling,
  title     = {Multimodal Scaling Laws for Task \& Data-Optimized Models of Visual Cortex},
  author    = {Abdulkadir Gokce and Yingtian Tang and Martin Schrimpf},
  booktitle = {Forty-third International Conference on Machine Learning},
  year      = {2026},
  url       = {https://openreview.net/forum?id=OQ6jQHJPTT}
}

@inproceedings{gokce2025scalingprimatevvs,
  title     = {Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream},
  author    = {Abdulkadir Gokce and Martin Schrimpf},
  booktitle = {Forty-second International Conference on Machine Learning},
  year      = {2025},
  url       = {https://openreview.net/forum?id=WxY61MmHYo}
}

and the data, images and networks:

• Gifford, A. T., Dwivedi, K., Roig, G., & Cichy, R. M. (2022). A large and rich EEG dataset for modeling human visual object recognition. NeuroImage, 264, 119754.
• Hebart, M. N., Dickter, A. H., Kidder, A., Kwok, W. Y., Corriveau, A., Van Wicklin, C., & Baker, C. I. (2019). THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLOS ONE, 14(10), e0223792.
• For efficient and tiny: Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2022). How to train your ViT? Data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research.
• For accurate: Qwen Team (2025). Qwen3-Omni Technical Report. arXiv:2509.17765.

The datasets used only for the transfer results above:

• Grootswagers, T., Zhou, I., Robinson, A. K., Hebart, M. N., & Carlson, T. A. (2022). Human EEG recordings for 1,854 concepts presented in rapid serial visual presentation streams. Scientific Data, 9, 3.
• Xu, J., Aristimunha, B., Feucht, M. E., Qian, E., Liu, C., Shahjahan, T., et al. (2024). Alljoined: A dataset for EEG-to-image decoding. CVPR 2024 Workshop on Data Curation and Augmentation in Medical Imaging. arXiv:2404.05553.
• Xu, J., Bruzadin Nunes, U., Jiang, W., Ryther, S., Pringle, J., Scotti, P. S., Delorme, A., & Kneeland, R. (2025). Alljoined-1.6M: A million-trial EEG-image dataset for evaluating affordable brain-computer interfaces. arXiv:2508.18571.