File size: 3,378 Bytes
54f4032 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
license: lgpl-3.0
library_name: deepmreye
tags:
- fmri
- neuroimaging
- eye-tracking
- gaze
datasets:
- DeepMReye/eyeballs
---
# DeepMReye 2.0
Decode eye gaze from fMRI without an eye tracker. The model reads gaze from
the eyeballs in a BOLD run: a frozen linear basis fitted on **2,060
unlabelled participants from 652 OpenNeuro datasets** by canonical
correlation between the left and right orbit, and a ridge readout fitted on
**337 gaze-labelled participants from 9 datasets**.
```bash
pip install git+https://github.com/DeepMReye/DeepMReye.git@deepmreye-unsupervised
deepmreye decode sub-01_task-movie_bold.nii.gz -o gaze/
```
```python
import deepmreye
result = deepmreye.decode("sub-01_task-movie_bold.nii.gz")
result["gaze"] # [T, 10, 2]: ten gaze samples per volume, x then y
result["confidence"] # agreement of the two eyes' decodings
```
## Files
| file | what |
|---|---|
| `decoder.npz` | the decoder (basis at 32 pairs, +-1 TR of context, ridge readout); `Decoder.pretrained()` |
| `basis.npz` | the basis at 256 pairs, for training your own readout and for evaluation |
| `evaluation.json` | every number of the evaluation below |
## Output
Ten gaze samples per volume (sample j covers [(t + j/10) TR, (t + (j+1)/10) TR)),
x then y, x to the right and y **downwards** (screen coordinates). Units are the
training datasets' standard deviations: gaze was z-scored per dataset before the
readout was fitted, because screen size and viewing distance differ between
studies. For degrees, calibrate on a few participants with an eye tracker
(`deepmreye calibrate`), which fits one gain and offset per axis.
## Evaluation
Leave-one-dataset-out: the readout is refitted on eight labelled datasets and
scored on every participant of the ninth. Pearson r between decoded and measured
gaze per participant (mean of x and y), median over a dataset's participants,
then over datasets.
| | median over datasets | mean over datasets |
|---|---|---|
| sub-TR (10 samples per volume) | 0.771 | 0.721 |
| per TR | 0.835 | 0.777 |
Per dataset (sub-TR / per TR):
| dataset | n | sub-TR | per TR |
|---|---|---|---|
| `dsL01_guided_fixations` | 170 | 0.771 | 0.787 |
| `dsL02_pursuit` | 9 | 0.920 | 0.930 |
| `dsL03_pursuit` | 24 | 0.776 | 0.810 |
| `dsL04_pursuit` | 34 | 0.851 | 0.863 |
| `dsL05_free_viewing` | 27 | 0.803 | 0.839 |
| `dsL06_sequences` | 6 | 0.667 | 0.731 |
| `dsL07_deepmreye_calib` | 15 | 0.743 | 0.835 |
| `dsL08_studyforrest_movie` | 14 | 0.285 | 0.351 |
| `dsL11_backtothefuture` | 37 | 0.672 | 0.845 |
The labelled datasets are the `dsL*` folders of
[DeepMReye/eyeballs](https://huggingface.co/datasets/DeepMReye/eyeballs). No
participant of them, nor any run of the OpenNeuro accessions three of them were
taken from, is in the basis's corpus.
## Confidence
The readout is linear, so applying it to each orbit alone gives two decodings
whose average is the real one. Their agreement ranks a dataset's participants by
how well they decode, without labels (`Decoder.confidence`). It does not compare
datasets, and does not flag single volumes.
## Limits
- The eyes must be inside the field of view and the registration must hold:
check the QA image (`deepmreye decode --qa`).
- Gaze is decoded relative to each participant's own gaze distribution; absolute
position needs calibration.
- Built 2026-10-14 from deepmreye 2.0.0.
|