---
tags:
- neuroscience
- fmri
- brain-decoding
- stimulus-synthesis
- v-jepa
- diffusers
library_name: diffusers
pipeline_tag: text-to-video
---

> 🚧 **Work in progress** — this model is still being transferred from its main development repository, so the model card and API are subject to change.

# NEvo — Neural-Guided Evolutionary Video Synthesis

**🌐 Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/)  ·  📄 Paper: [arXiv:2607.02317](https://arxiv.org/abs/2607.02317)**

NEvo is a self-contained Hugging Face custom [Diffusers](https://github.com/huggingface/diffusers) pipeline for **neural-response-guided visual stimulus synthesis**. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

It orchestrates three frozen models. The models below are **placeholders / defaults** and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

| Role | Default model |
|------|---------------|
| Encoder (image/video → fMRI) | [`epfl-neuroai/vjepa2-encoder-basic`](https://huggingface.co/epfl-neuroai/vjepa2-encoder-basic) (`predict_fmri`) |
| Text → image | [`stabilityai/sdxl-turbo`](https://huggingface.co/stabilityai/sdxl-turbo) |
| Image → video | [`Lightricks/LTX-Video-0.9.8-13B-distilled`](https://huggingface.co/Lightricks/LTX-Video-0.9.8-13B-distilled) |

## Gallery

Each clip is from the **top results of a NEvo search targeting one visual region** — the model discovers, from scratch, stimuli that drive that region's known selectivity.

| Region | Stimulus | Region | Stimulus |
|:------:|:--------:|:------:|:--------:|
| **FFA** · faces | ![FFA](assets/gallery/FFA.gif) | **PPA** · places | ![PPA](assets/gallery/PPA.gif) |
| **MT** · motion | ![MT](assets/gallery/MT.gif) | **EBA** · bodies | ![EBA](assets/gallery/EBA.gif) |
| **pSTS** · social motion | ![pSTS](assets/gallery/pSTS.gif) | **V1** · early visual | ![V1](assets/gallery/V1.gif) |

Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.

## Installation

**Off-the-shelf — no install.** Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

```python
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)
```

**Or install the package** (for cleaner `from stimulus_synthesis import ...` imports / development):

```bash
conda create -n nevo python=3.10 -y
conda activate nevo
pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
# or from a local clone:
#   git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
# then:  from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")
```

Runtime dependencies (either way): `torch`, `diffusers`, `transformers`, `huggingface_hub`, `numpy`, `pillow`, `av` (`pytest` for tests). No `nilearn` / atlas downloads — ROI masks are shipped as small precomputed data files.

## Quickstart

Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

```python
from diffusers import DiffusionPipeline

# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

out = pipe(roi="FFA", progress=True)                             # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
print(out.best_prompt, out.best_score)

from stimulus_synthesis.media import save_video, video_to_t_c_h_w   # importable once the pipeline has loaded
save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4")   # save the synthesized video
out.best.image.save("best_stimulus.png")                            # and the stage-1 best image (PIL)
```

This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.

### Faster run

For a quicker first result, shrink the search and the video:

```python
out = pipe(
    roi="FFA",
    progress=True,
    image_max_evals=80,       # stage-1 (image) evaluation budget (default: 400)
    video_max_evals=40,       # stage-2 (video) evaluation budget (default: 200)
    population_size=8,        # GA population per generation (default: 20)
    seed=0,                   # RNG seed, for reproducibility
    video_kwargs={            # merged over the fast defaults (8 steps / 25 frames / 512²); override any key
        "num_inference_steps": 8,   # denoising steps — the distilled LTX model needs only a few
        "num_frames": 25,           # clip length; LTX requires 8*k + 1 frames
        "height": 256, "width": 256,
    },
)
```

**Enhanced search space.** Selecting a region (`roi=...`) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass `enforce_general_search_space=True` to search the full general space instead.

Available ROI tokens (comma-separated tokens are unioned):

- **Named ROIs:** `FFA`, `PPA`, `MT`, `EBA`, `LOC`, `RSC`, `pSTS`, `aSTS`, `V1`, `V2`, `V3`, `V4` — optionally hemisphere-suffixed (`FFA_lh`, `MT_rh`).
- **Searchlight regions:** `SL-<n>` (both hemispheres), `SL-<n>_lh`, `SL-<n>_rh` (58 both / 28 lh / 30 rh).

```python
from stimulus_synthesis.neuro import available_rois, searchlight_counts
available_rois()        # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
searchlight_counts()    # {'both': 58, 'lh': 28, 'rh': 30}
```

> **fsaverage5 only.** The bundled ROI/searchlight masks are defined on the **fsaverage5** cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose `predict_fmri` output lives in that same space — the default `epfl-neuroai/vjepa2-encoder-basic`. A custom encoder with a different output space can still be driven with explicit `vector`/`indices` targets, but not with the named-ROI helper.

### Custom targets

Instead of a region name, pass raw voxel indices or a full target fMRI vector:

```python
import numpy as np
from stimulus_synthesis import resolve_driving_voxels

mask = resolve_driving_voxels("FFA")                 # boolean mask, length 20484
out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})
```

### Target types & objectives

| Target | Objective (default) | Meaning |
|--------|--------------------|---------|
| `{"type": "indices", "indices": [...]}` | `indices_mean` | mean predicted response over ROI voxels |
| `{"type": "vector", "vector": [...]}` (len 20484) | `target_vector_cosine` / `vector_dot` | match a full target fMRI vector |
| `{"type": "weights", "weights": [...]}` | `weighted_mean` | weighted voxel objective |

## Search parameters (defaults)

Set in `stimulus_synthesis_config.json`:

| Param | Default | Notes |
|-------|---------|-------|
| `default_image_max_evals` | 400 | stage-1 (image) evaluation budget (GA `max_evals`) |
| `default_video_max_evals` | 200 | stage-2 (video) evaluation budget |
| `default_population_size` | 20 | GA population per generation (= `n_init`) |
| `default_score_frames` | 24 | number of frames the encoder scores (a still image is replicated to this) |
| `default_score_size` | 224 | resolution the clip is resized to for the encoder (call-time: `score_size=`) |
| `default_mutation_rate` | 0.25 | |
| `default_elite_frac` | 0.35 | |
| `default_objective` | `indices_mean` | |
| `default_score_transform` | disabled | robust augmentation off by default (clean single pass) |
| `default_image_kwargs` | `{num_inference_steps: 1, guidance_scale: 0, height: 512, width: 512}` | fast SDXL-Turbo settings (merged under call-time `image_kwargs`) |
| `default_video_kwargs` | `{num_inference_steps: 8, num_frames: 25, height: 512, width: 512}` | fast LTX settings (merged under call-time `video_kwargs`) |

Each stage runs a genetic search with population `population_size` (default 20) until it hits its evaluation budget — `image_max_evals` (default 400) and `video_max_evals` (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (`default_image_kwargs` / `default_video_kwargs`); anything you pass as `image_kwargs` / `video_kwargs` is merged over them, so you only override the keys you care about.

### Robust scoring

By default each candidate is scored with a single clean encoder pass. An optional **robust mode** — the mean over 4 augmented draws (random crop `0.8`, Gaussian `σ=0.1`) via `RobustTransformScorer` — reduces sensitivity to encoder artifacts; turn it on by setting `"enabled": true` in `default_score_transform`.

## Cache configuration

Model weights and outputs cache location resolves in priority order:

1. `NEvo_CACHE_DIR` — set it in a repo-root `.env` file (see `.env.example`) or the environment.
2. Otherwise the **system/user-default HuggingFace cache** (`HF_HOME`, else `~/.cache/huggingface`) is used and left untouched.
3. Only if no default is resolvable, a repo-local `cache/` is used.

`cache/` and `.env` are git-ignored.

## Batch runners

Two ROI-driven, two-stage (image-search → video-search) runners are included:

- **`run_roi_samples.py`** — genetic search per ROI/seed, scoring in-memory tensors; writes `best_image.png` / `best_video.mp4` / scores.
- **`run_regional_asset_pilot.py`** — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the *decoded file* — producing provenance-tracked, reproducible published assets with manifests.

Both take `--rois`, `--seeds`, `--image-evals` / `--video-evals`, `--encoder-model`, `--out-dir`, etc., and default to the config's encoder and a cache-relative output directory.

## Reproducibility

The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a *target-matching* signal, not ground-truth reconstruction quality.

## Intended use & limitations

- **Research use** in visual neuroscience / brain-decoding. Outputs are *predicted* to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
- Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
- Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

## Citation

If you use NEvo, please cite:

```bibtex
@article{tang2026nevo,
  title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
  author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
  journal={arXiv preprint arXiv:2607.02317},
  year={2026}
}
```

Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/)

## Acknowledgements

Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).