| --- |
| license: lgpl-3.0 |
| library_name: deepmreye |
| tags: |
| - fmri |
| - neuroimaging |
| - eye-tracking |
| - gaze |
| datasets: |
| - DeepMReye/eyeballs |
| --- |
| |
| # DeepMReye 2.0 |
|
|
| Decode eye gaze from fMRI without an eye tracker. The model reads gaze from |
| the eyeballs in a BOLD run: a frozen linear basis fitted on **2,060 |
| unlabelled participants from 652 OpenNeuro datasets** by canonical |
| correlation between the left and right orbit, and a ridge readout fitted on |
| **337 gaze-labelled participants from 9 datasets**. |
|
|
| ```bash |
| pip install git+https://github.com/DeepMReye/DeepMReye.git@deepmreye-unsupervised |
| deepmreye decode sub-01_task-movie_bold.nii.gz -o gaze/ |
| ``` |
|
|
| ```python |
| import deepmreye |
| result = deepmreye.decode("sub-01_task-movie_bold.nii.gz") |
| result["gaze"] # [T, 10, 2]: ten gaze samples per volume, x then y |
| result["confidence"] # agreement of the two eyes' decodings |
| ``` |
|
|
| ## Files |
|
|
| | file | what | |
| |---|---| |
| | `decoder.npz` | the decoder (basis at 32 pairs, +-1 TR of context, ridge readout); `Decoder.pretrained()` | |
| | `basis.npz` | the basis at 256 pairs, for training your own readout and for evaluation | |
| | `evaluation.json` | every number of the evaluation below | |
|
|
| ## Output |
|
|
| Ten gaze samples per volume (sample j covers [(t + j/10) TR, (t + (j+1)/10) TR)), |
| x then y, x to the right and y **downwards** (screen coordinates). Units are the |
| training datasets' standard deviations: gaze was z-scored per dataset before the |
| readout was fitted, because screen size and viewing distance differ between |
| studies. For degrees, calibrate on a few participants with an eye tracker |
| (`deepmreye calibrate`), which fits one gain and offset per axis. |
|
|
| ## Evaluation |
|
|
| Leave-one-dataset-out: the readout is refitted on eight labelled datasets and |
| scored on every participant of the ninth. Pearson r between decoded and measured |
| gaze per participant (mean of x and y), median over a dataset's participants, |
| then over datasets. |
|
|
| | | median over datasets | mean over datasets | |
| |---|---|---| |
| | sub-TR (10 samples per volume) | 0.771 | 0.721 | |
| | per TR | 0.835 | 0.777 | |
|
|
| Per dataset (sub-TR / per TR): |
|
|
| | dataset | n | sub-TR | per TR | |
| |---|---|---|---| |
| | `dsL01_guided_fixations` | 170 | 0.771 | 0.787 | |
| | `dsL02_pursuit` | 9 | 0.920 | 0.930 | |
| | `dsL03_pursuit` | 24 | 0.776 | 0.810 | |
| | `dsL04_pursuit` | 34 | 0.851 | 0.863 | |
| | `dsL05_free_viewing` | 27 | 0.803 | 0.839 | |
| | `dsL06_sequences` | 6 | 0.667 | 0.731 | |
| | `dsL07_deepmreye_calib` | 15 | 0.743 | 0.835 | |
| | `dsL08_studyforrest_movie` | 14 | 0.285 | 0.351 | |
| | `dsL11_backtothefuture` | 37 | 0.672 | 0.845 | |
|
|
| The labelled datasets are the `dsL*` folders of |
| [DeepMReye/eyeballs](https://huggingface.co/datasets/DeepMReye/eyeballs). No |
| participant of them, nor any run of the OpenNeuro accessions three of them were |
| taken from, is in the basis's corpus. |
|
|
| ## Confidence |
|
|
| The readout is linear, so applying it to each orbit alone gives two decodings |
| whose average is the real one. Their agreement ranks a dataset's participants by |
| how well they decode, without labels (`Decoder.confidence`). It does not compare |
| datasets, and does not flag single volumes. |
|
|
| ## Limits |
|
|
| - The eyes must be inside the field of view and the registration must hold: |
| check the QA image (`deepmreye decode --qa`). |
| - Gaze is decoded relative to each participant's own gaze distribution; absolute |
| position needs calibration. |
| - Built 2026-10-14 from deepmreye 2.0.0. |
|
|
