---
license: lgpl-3.0
library_name: deepmreye
tags:
- fmri
- neuroimaging
- eye-tracking
- gaze
datasets:
- DeepMReye/eyeballs
---

# DeepMReye 2.0

Decode eye gaze from fMRI without an eye tracker. The model reads gaze from
the eyeballs in a BOLD run: a frozen linear basis fitted on **2,060
unlabelled participants from 652 OpenNeuro datasets** by canonical
correlation between the left and right orbit, and a ridge readout fitted on
**337 gaze-labelled participants from 9 datasets**.

```bash
pip install git+https://github.com/DeepMReye/DeepMReye.git@deepmreye-unsupervised
deepmreye decode sub-01_task-movie_bold.nii.gz -o gaze/
```

```python
import deepmreye
result = deepmreye.decode("sub-01_task-movie_bold.nii.gz")
result["gaze"]        # [T, 10, 2]: ten gaze samples per volume, x then y
result["confidence"]  # agreement of the two eyes' decodings
```

## Files

| file | what |
|---|---|
| `decoder.npz` | the decoder (basis at 32 pairs, +-1 TR of context, ridge readout); `Decoder.pretrained()` |
| `basis.npz` | the basis at 256 pairs, for training your own readout and for evaluation |
| `evaluation.json` | every number of the evaluation below |

## Output

Ten gaze samples per volume (sample j covers [(t + j/10) TR, (t + (j+1)/10) TR)),
x then y, x to the right and y **downwards** (screen coordinates). Units are the
training datasets' standard deviations: gaze was z-scored per dataset before the
readout was fitted, because screen size and viewing distance differ between
studies. For degrees, calibrate on a few participants with an eye tracker
(`deepmreye calibrate`), which fits one gain and offset per axis.

## Evaluation

Leave-one-dataset-out: the readout is refitted on eight labelled datasets and
scored on every participant of the ninth. Pearson r between decoded and measured
gaze per participant (mean of x and y), median over a dataset's participants,
then over datasets.

| | median over datasets | mean over datasets |
|---|---|---|
| sub-TR (10 samples per volume) | 0.771 | 0.721 |
| per TR | 0.835 | 0.777 |

Per dataset (sub-TR / per TR):

| dataset | n | sub-TR | per TR |
|---|---|---|---|
| `dsL01_guided_fixations` | 170 | 0.771 | 0.787 |
| `dsL02_pursuit` | 9 | 0.920 | 0.930 |
| `dsL03_pursuit` | 24 | 0.776 | 0.810 |
| `dsL04_pursuit` | 34 | 0.851 | 0.863 |
| `dsL05_free_viewing` | 27 | 0.803 | 0.839 |
| `dsL06_sequences` | 6 | 0.667 | 0.731 |
| `dsL07_deepmreye_calib` | 15 | 0.743 | 0.835 |
| `dsL08_studyforrest_movie` | 14 | 0.285 | 0.351 |
| `dsL11_backtothefuture` | 37 | 0.672 | 0.845 |

The labelled datasets are the `dsL*` folders of
[DeepMReye/eyeballs](https://huggingface.co/datasets/DeepMReye/eyeballs). No
participant of them, nor any run of the OpenNeuro accessions three of them were
taken from, is in the basis's corpus.

## Confidence

The readout is linear, so applying it to each orbit alone gives two decodings
whose average is the real one. Their agreement ranks a dataset's participants by
how well they decode, without labels (`Decoder.confidence`). It does not compare
datasets, and does not flag single volumes.

## Limits

- The eyes must be inside the field of view and the registration must hold:
  check the QA image (`deepmreye decode --qa`).
- Gaze is decoded relative to each participant's own gaze distribution; absolute
  position needs calibration.
- Built 2026-10-14 from deepmreye 2.0.0.