---
license: cc-by-nc-4.0
tags:
- neuroscience
- fmri
- brain-decoding
- stimulus-synthesis
- v-jepa
- diffusers
library_name: diffusers
pipeline_tag: text-to-video
---

> **Community fork maintained by `mzx`.** The original NEvo project and paper are by EPFL NeuroAI. This fork is an independent adaptation and is not affiliated with or endorsed by EPFL or the original authors. See the [original model](https://huggingface.co/epfl-neuroai/NEvo), [project website](https://nevo-project.epfl.ch/), and [paper](https://arxiv.org/abs/2607.02317).

> 🚧 **Work in progress** — this model is still being transferred from its main development repository, so the model card and API are subject to change.

# NEvo — Neural-Guided Evolutionary Video Synthesis

**🌐 Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/)  ·  📄 Paper: [arXiv:2607.02317](https://arxiv.org/abs/2607.02317)**

NEvo is a self-contained Hugging Face custom [Diffusers](https://github.com/huggingface/diffusers) pipeline for **neural-response-guided visual stimulus synthesis**. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

It orchestrates three frozen models. The models below are **placeholders / defaults** and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

| Role | Default model |
|------|---------------|
| Encoder (image/video → fMRI) | [`epfl-neuroai/vjepa2-encoder-basic`](https://huggingface.co/epfl-neuroai/vjepa2-encoder-basic) (`predict_fmri`) |
| Text → image | [`stabilityai/sdxl-turbo`](https://huggingface.co/stabilityai/sdxl-turbo) |
| Image → video | [`Lightricks/LTX-Video-0.9.8-13B-distilled`](https://huggingface.co/Lightricks/LTX-Video-0.9.8-13B-distilled) |

## Gallery

Each clip is from the **top results of a NEvo search targeting one visual region** — the model discovers, from scratch, stimuli that drive that region's known selectivity.

| Region | Stimulus | Region | Stimulus |
|:------:|:--------:|:------:|:--------:|
| **FFA** · faces | ![FFA](assets/gallery/FFA.gif) | **PPA** · places | ![PPA](assets/gallery/PPA.gif) |
| **MT** · motion | ![MT](assets/gallery/MT.gif) | **EBA** · bodies | ![EBA](assets/gallery/EBA.gif) |
| **pSTS** · social motion | ![pSTS](assets/gallery/pSTS.gif) | **V1** · early visual | ![V1](assets/gallery/V1.gif) |

Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.

## Installation

**Off-the-shelf — no install.** Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

```python
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)
```

**Or install the package** (for cleaner `from stimulus_synthesis import ...` imports / development):

```bash
conda create -n nevo python=3.10 -y
conda activate nevo
pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
# or from a local clone:
#   git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
# then:  from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")
```

Runtime dependencies (either way): `torch`, `diffusers`, `transformers`, `huggingface_hub`, `numpy`, `pillow`, `av` (`pytest` for tests). No `nilearn` / atlas downloads — ROI masks are shipped as small precomputed data files.

## Quickstart

Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

```python
from diffusers import DiffusionPipeline

# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

out = pipe(roi="FFA", progress=True)                             # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
print(out.best_prompt, out.best_score)

from stimulus_synthesis.media import save_video, video_to_t_c_h_w   # importable once the pipeline has loaded
save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4", fps=12)   # save the synthesized video (mp4 playback fps; default 12)
out.best.image.save("best_stimulus.png")                            # and the stage-1 best image (PIL)
```

This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.

### Faster run

For a quicker first result, shrink the search and the video:

```python
out = pipe(
    roi="FFA",
    progress=True,
    image_max_evals=80,       # stage-1 (image) evaluation budget (default: 400)
    video_max_evals=40,       # stage-2 (video) evaluation budget (default: 200)
    population_size=8,        # GA population per generation (default: 20)
    image_batch_size=8,       # images generated per batch (default: 16)
    video_batch_size=4,       # videos generated per batch (default: 8)
    seed=0,                   # RNG seed, for reproducibility
    video_kwargs={            # merged over the fast defaults (8 steps / 25 frames / 512²); override any key
        "num_inference_steps": 8,   # denoising steps — the distilled LTX model needs only a few
        "num_frames": 25,           # clip length; LTX requires 8*k + 1 frames
        "height": 256, "width": 256,
    },
)
```

**Enhanced search space.** Selecting a region (`roi=...`) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass `enforce_general_search_space=True` to search the full general space instead.

Available ROI tokens (comma-separated tokens are unioned):

- **Named ROIs:** `FFA`, `PPA`, `MT`, `EBA`, `LOC`, `RSC`, `pSTS`, `aSTS`, `V1`, `V2`, `V3`, `V4` — optionally hemisphere-suffixed (`FFA_lh`, `MT_rh`).
- **Searchlight regions:** `SL-<n>` (both hemispheres), `SL-<n>_lh`, `SL-<n>_rh` (58 both / 28 lh / 30 rh).

```python
from stimulus_synthesis.neuro import available_rois, searchlight_counts
available_rois()        # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
searchlight_counts()    # {'both': 58, 'lh': 28, 'rh': 30}
```

> **fsaverage5 only.** The bundled ROI/searchlight masks are defined on the **fsaverage5** cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose `predict_fmri` output lives in that same space — the default `epfl-neuroai/vjepa2-encoder-basic`. A custom encoder with a different output space can still be driven with explicit `vector`/`indices` targets, but not with the named-ROI helper.

### Custom targets

Instead of a region name, pass raw voxel indices or a full target fMRI vector:

```python
import numpy as np
from stimulus_synthesis import resolve_driving_voxels

mask = resolve_driving_voxels("FFA")                 # boolean mask, length 20484
out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})
```

### Target types & objectives

| Target | Objective (default) | Meaning |
|--------|--------------------|---------|
| `{"type": "indices", "indices": [...]}` | `indices_mean` | mean predicted response over ROI voxels |
| `{"type": "vector", "vector": [...]}` (len 20484) | `target_vector_cosine` / `vector_dot` | match a full target fMRI vector |
| `{"type": "weights", "weights": [...]}` | `weighted_mean` | weighted voxel objective |

## Search parameters (defaults)

Set in `stimulus_synthesis_config.json`:

| Param | Default | Notes |
|-------|---------|-------|
| `default_image_max_evals` | 400 | stage-1 (image) evaluation budget (GA `max_evals`) |
| `default_video_max_evals` | 200 | stage-2 (video) evaluation budget |
| `default_population_size` | 20 | GA population per generation (= `n_init`) |
| `default_score_frames` | 24 | number of frames the encoder scores (a still image is replicated to this) |
| `default_score_size` | 224 | resolution the clip is resized to for the encoder (call-time: `score_size=`) |
| `default_mutation_rate` | 0.25 | |
| `default_elite_frac` | 0.35 | |
| `default_objective` | `indices_mean` | |
| `default_score_transform` | disabled | robust augmentation off by default (clean single pass) |
| `default_image_kwargs` | `{num_inference_steps: 1, guidance_scale: 0, height: 512, width: 512}` | fast SDXL-Turbo settings (merged under call-time `image_kwargs`) |
| `default_video_kwargs` | `{num_inference_steps: 8, num_frames: 25, height: 512, width: 512}` | fast LTX settings (merged under call-time `video_kwargs`) |

Each stage runs a genetic search with population `population_size` (default 20) until it hits its evaluation budget — `image_max_evals` (default 400) and `video_max_evals` (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (`default_image_kwargs` / `default_video_kwargs`); anything you pass as `image_kwargs` / `video_kwargs` is merged over them, so you only override the keys you care about.

### Robust scoring

By default each candidate is scored with a single clean encoder pass. An optional **robust mode** — the mean over 4 augmented draws (random crop `0.8`, Gaussian `σ=0.1`) via `RobustTransformScorer` — reduces sensitivity to encoder artifacts; turn it on by setting `"enabled": true` in `default_score_transform`.

## Cache configuration

Model weights and outputs cache location resolves in priority order:

1. `NEvo_CACHE_DIR` — set it in a repo-root `.env` file (see `.env.example`) or the environment.
2. Otherwise the **system/user-default HuggingFace cache** (`HF_HOME`, else `~/.cache/huggingface`) is used and left untouched.
3. Only if no default is resolvable, a repo-local `cache/` is used.

`cache/` and `.env` are git-ignored.

## Batch runners

Two ROI-driven, two-stage (image-search → video-search) runners are included:

- **`run_roi_samples.py`** — genetic search per ROI/seed, scoring in-memory tensors; writes `best_image.png` / `best_video.mp4` / scores.
- **`run_regional_asset_pilot.py`** — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the *decoded file* — producing provenance-tracked, reproducible published assets with manifests.

Both take `--rois`, `--seeds`, `--image-evals` / `--video-evals`, `--encoder-model`, `--out-dir`, etc., and default to the config's encoder and a cache-relative output directory.

## Reproducibility

The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a *target-matching* signal, not ground-truth reconstruction quality.

## Intended use & limitations

- **Research use** in visual neuroscience / brain-decoding. Outputs are *predicted* to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
- Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
- Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

## Citation

If you use NEvo, please cite:

```bibtex
@article{tang2026nevo,
  title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
  author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
  journal={arXiv preprint arXiv:2607.02317},
  year={2026}
}
```

Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/)

## Acknowledgements

Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).

## Intended Use, Out-of-Scope Use & Risks

NEvo is a **research-only** tool for computational visual neuroscience: it searches for stimuli predicted to drive a target brain region under a fixed image/video→fMRI encoder, and its outputs are **unvalidated hypotheses**, not scientific findings. It is **not a medical device** and must **not** be used for diagnosis, treatment, or to design, guide, or deliver any real brain stimulation (TMS, tDCS/tACS, DBS, intracranial, etc.) or in-vivo neuromodulation — it only optimizes a *predicted* fMRI response, not effects on a living brain, and any use involving human or animal subjects requires independent ethical review (e.g. IRB). Optimizing against a single encoder can exploit artifacts, and the generative models may produce biased or disturbing imagery; the model is provided **as is**, without warranty, for **non-commercial** use under CC BY-NC 4.0 (upstream models carry their own licenses).