DeepMReye 2.0
Decode eye gaze from fMRI without an eye tracker. The model reads gaze from the eyeballs in a BOLD run: a frozen linear basis fitted on 2,060 unlabelled participants from 652 OpenNeuro datasets by canonical correlation between the left and right orbit, and a ridge readout fitted on 337 gaze-labelled participants from 9 datasets.
pip install git+https://github.com/DeepMReye/DeepMReye.git@deepmreye-unsupervised
deepmreye decode sub-01_task-movie_bold.nii.gz -o gaze/
import deepmreye
result = deepmreye.decode("sub-01_task-movie_bold.nii.gz")
result["gaze"] # [T, 10, 2]: ten gaze samples per volume, x then y
result["confidence"] # agreement of the two eyes' decodings
Files
| file | what |
|---|---|
decoder.npz |
the decoder (basis at 32 pairs, +-1 TR of context, ridge readout); Decoder.pretrained() |
basis.npz |
the basis at 256 pairs, for training your own readout and for evaluation |
evaluation.json |
every number of the evaluation below |
Output
Ten gaze samples per volume (sample j covers [(t + j/10) TR, (t + (j+1)/10) TR)),
x then y, x to the right and y downwards (screen coordinates). Units are the
training datasets' standard deviations: gaze was z-scored per dataset before the
readout was fitted, because screen size and viewing distance differ between
studies. For degrees, calibrate on a few participants with an eye tracker
(deepmreye calibrate), which fits one gain and offset per axis.
Evaluation
Leave-one-dataset-out: the readout is refitted on eight labelled datasets and scored on every participant of the ninth. Pearson r between decoded and measured gaze per participant (mean of x and y), median over a dataset's participants, then over datasets.
| median over datasets | mean over datasets | |
|---|---|---|
| sub-TR (10 samples per volume) | 0.771 | 0.721 |
| per TR | 0.835 | 0.777 |
Per dataset (sub-TR / per TR):
| dataset | n | sub-TR | per TR |
|---|---|---|---|
dsL01_guided_fixations |
170 | 0.771 | 0.787 |
dsL02_pursuit |
9 | 0.920 | 0.930 |
dsL03_pursuit |
24 | 0.776 | 0.810 |
dsL04_pursuit |
34 | 0.851 | 0.863 |
dsL05_free_viewing |
27 | 0.803 | 0.839 |
dsL06_sequences |
6 | 0.667 | 0.731 |
dsL07_deepmreye_calib |
15 | 0.743 | 0.835 |
dsL08_studyforrest_movie |
14 | 0.285 | 0.351 |
dsL11_backtothefuture |
37 | 0.672 | 0.845 |
The labelled datasets are the dsL* folders of
DeepMReye/eyeballs. No
participant of them, nor any run of the OpenNeuro accessions three of them were
taken from, is in the basis's corpus.
Confidence
The readout is linear, so applying it to each orbit alone gives two decodings
whose average is the real one. Their agreement ranks a dataset's participants by
how well they decode, without labels (Decoder.confidence). It does not compare
datasets, and does not flag single volumes.
Limits
- The eyes must be inside the field of view and the registration must hold:
check the QA image (
deepmreye decode --qa). - Gaze is decoded relative to each participant's own gaze distribution; absolute position needs calibration.
- Built 2026-10-14 from deepmreye 2.0.0.