NEvo / README.md

Duplicate from epfl-neuroai/NEvo

1e2bb2f about 19 hours ago

11.8 kB

	---
	tags:
	- neuroscience
	- fmri
	- brain-decoding
	- stimulus-synthesis
	- v-jepa
	- diffusers
	library_name: diffusers
	pipeline_tag: text-to-video
	---

	> 🚧 Work in progress — this model is still being transferred from its main development repository, so the model card and API are subject to change.

	# NEvo — Neural-Guided Evolutionary Video Synthesis

	🌐 Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/) · 📄 Paper: [arXiv:2607.02317](https://arxiv.org/abs/2607.02317)

	NEvo is a self-contained Hugging Face custom [Diffusers](https://github.com/huggingface/diffusers) pipeline for neural-response-guided visual stimulus synthesis. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

	It orchestrates three frozen models. The models below are placeholders / defaults and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

	\| Role \| Default model \|
	\|------\|---------------\|
	\| Encoder (image/video → fMRI) \| [`epfl-neuroai/vjepa2-encoder-basic`](https://huggingface.co/epfl-neuroai/vjepa2-encoder-basic) (`predict_fmri`) \|
	\| Text → image \| [`stabilityai/sdxl-turbo`](https://huggingface.co/stabilityai/sdxl-turbo) \|
	\| Image → video \| [`Lightricks/LTX-Video-0.9.8-13B-distilled`](https://huggingface.co/Lightricks/LTX-Video-0.9.8-13B-distilled) \|

	## Gallery

	Each clip is from the top results of a NEvo search targeting one visual region — the model discovers, from scratch, stimuli that drive that region's known selectivity.

	\| Region \| Stimulus \| Region \| Stimulus \|
	\|:------:\|:--------:\|:------:\|:--------:\|
	\| FFA · faces \| ![FFA](assets/gallery/FFA.gif) \| PPA · places \| ![PPA](assets/gallery/PPA.gif) \|
	\| MT · motion \| ![MT](assets/gallery/MT.gif) \| EBA · bodies \| ![EBA](assets/gallery/EBA.gif) \|
	\| pSTS · social motion \| ![pSTS](assets/gallery/pSTS.gif) \| V1 · early visual \| ![V1](assets/gallery/V1.gif) \|

	Explore the full interactive gallery and 3D brain maps at [nevo-project.epfl.ch](https://nevo-project.epfl.ch/).

	## Installation

	Off-the-shelf — no install. Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

	```python
	from diffusers import DiffusionPipeline

	pipe = DiffusionPipeline.from_pretrained(
	"epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
	)
	```

	Or install the package (for cleaner `from stimulus_synthesis import ...` imports / development):

	```bash
	conda create -n nevo python=3.10 -y
	conda activate nevo
	pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
	# or from a local clone:
	# git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
	# then: from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")
	```

	Runtime dependencies (either way): `torch`, `diffusers`, `transformers`, `huggingface_hub`, `numpy`, `pillow`, `av` (`pytest` for tests). No `nilearn` / atlas downloads — ROI masks are shipped as small precomputed data files.

	## Quickstart

	Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

	```python
	from diffusers import DiffusionPipeline

	# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
	pipe = DiffusionPipeline.from_pretrained(
	"epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
	)

	out = pipe(roi="FFA", progress=True) # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
	print(out.best_prompt, out.best_score)

	from stimulus_synthesis.media import save_video, video_to_t_c_h_w # importable once the pipeline has loaded
	save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4") # save the synthesized video
	out.best.image.save("best_stimulus.png") # and the stage-1 best image (PIL)
	```

	This runs the two-stage search with the defaults — up to 400 image evaluations then 200 video evaluations (population 20), using the fast distilled-model defaults (1-step 512×512 SDXL-Turbo, 8-step 512×512 LTX). A run takes a few minutes and a good amount of GPU memory.

	### Faster run

	For a quicker first result, shrink the search and the video:

	```python
	out = pipe(
	roi="FFA",
	progress=True,
	image_max_evals=80, # stage-1 (image) evaluation budget (default: 400)
	video_max_evals=40, # stage-2 (video) evaluation budget (default: 200)
	population_size=8, # GA population per generation (default: 20)
	seed=0, # RNG seed, for reproducibility
	video_kwargs={ # merged over the fast defaults (8 steps / 25 frames / 512²); override any key
	"num_inference_steps": 8, # denoising steps — the distilled LTX model needs only a few
	"num_frames": 25, # clip length; LTX requires 8*k + 1 frames
	"height": 256, "width": 256,
	},
	)
	```

	Enhanced search space. Selecting a region (`roi=...`) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Pass `enforce_general_search_space=True` to search the full general space instead.

	Available ROI tokens (comma-separated tokens are unioned):

	- Named ROIs: `FFA`, `PPA`, `MT`, `EBA`, `LOC`, `RSC`, `pSTS`, `aSTS`, `V1`, `V2`, `V3`, `V4` — optionally hemisphere-suffixed (`FFA_lh`, `MT_rh`).
	- Searchlight regions: `SL-<n>` (both hemispheres), `SL-<n>_lh`, `SL-<n>_rh` (58 both / 28 lh / 30 rh).

	```python
	from stimulus_synthesis.neuro import available_rois, searchlight_counts
	available_rois() # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
	searchlight_counts() # {'both': 58, 'lh': 28, 'rh': 30}
	```

	> fsaverage5 only. The bundled ROI/searchlight masks are defined on the fsaverage5 cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose `predict_fmri` output lives in that same space — the default `epfl-neuroai/vjepa2-encoder-basic`. A custom encoder with a different output space can still be driven with explicit `vector`/`indices` targets, but not with the named-ROI helper.

	### Custom targets

	Instead of a region name, pass raw voxel indices or a full target fMRI vector:

	```python
	import numpy as np
	from stimulus_synthesis import resolve_driving_voxels

	mask = resolve_driving_voxels("FFA") # boolean mask, length 20484
	out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})
	```

	### Target types & objectives

	\| Target \| Objective (default) \| Meaning \|
	\|--------\|--------------------\|---------\|
	\| `{"type": "indices", "indices": [...]}` \| `indices_mean` \| mean predicted response over ROI voxels \|
	\| `{"type": "vector", "vector": [...]}` (len 20484) \| `target_vector_cosine` / `vector_dot` \| match a full target fMRI vector \|
	\| `{"type": "weights", "weights": [...]}` \| `weighted_mean` \| weighted voxel objective \|

	## Search parameters (defaults)

	Set in `stimulus_synthesis_config.json`:

	\| Param \| Default \| Notes \|
	\|-------\|---------\|-------\|
	\| `default_image_max_evals` \| 400 \| stage-1 (image) evaluation budget (GA `max_evals`) \|
	\| `default_video_max_evals` \| 200 \| stage-2 (video) evaluation budget \|
	\| `default_population_size` \| 20 \| GA population per generation (= `n_init`) \|
	\| `default_score_frames` \| 24 \| number of frames the encoder scores (a still image is replicated to this) \|
	\| `default_score_size` \| 224 \| resolution the clip is resized to for the encoder (call-time: `score_size=`) \|
	\| `default_mutation_rate` \| 0.25 \| \|
	\| `default_elite_frac` \| 0.35 \| \|
	\| `default_objective` \| `indices_mean` \| \|
	\| `default_score_transform` \| disabled \| robust augmentation off by default (clean single pass) \|
	\| `default_image_kwargs` \| `{num_inference_steps: 1, guidance_scale: 0, height: 512, width: 512}` \| fast SDXL-Turbo settings (merged under call-time `image_kwargs`) \|
	\| `default_video_kwargs` \| `{num_inference_steps: 8, num_frames: 25, height: 512, width: 512}` \| fast LTX settings (merged under call-time `video_kwargs`) \|

	Each stage runs a genetic search with population `population_size` (default 20) until it hits its evaluation budget — `image_max_evals` (default 400) and `video_max_evals` (default 200) generate→score passes. Image and video generation use fast distilled defaults out of the box (`default_image_kwargs` / `default_video_kwargs`); anything you pass as `image_kwargs` / `video_kwargs` is merged over them, so you only override the keys you care about.

	### Robust scoring

	By default each candidate is scored with a single clean encoder pass. An optional robust mode — the mean over 4 augmented draws (random crop `0.8`, Gaussian `σ=0.1`) via `RobustTransformScorer` — reduces sensitivity to encoder artifacts; turn it on by setting `"enabled": true` in `default_score_transform`.

	## Cache configuration

	Model weights and outputs cache location resolves in priority order:

	1. `NEvo_CACHE_DIR` — set it in a repo-root `.env` file (see `.env.example`) or the environment.
	2. Otherwise the system/user-default HuggingFace cache (`HF_HOME`, else `~/.cache/huggingface`) is used and left untouched.
	3. Only if no default is resolvable, a repo-local `cache/` is used.

	`cache/` and `.env` are git-ignored.

	## Batch runners

	Two ROI-driven, two-stage (image-search → video-search) runners are included:

	- `run_roi_samples.py` — genetic search per ROI/seed, scoring in-memory tensors; writes `best_image.png` / `best_video.mp4` / scores.
	- `run_regional_asset_pilot.py` — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the decoded file — producing provenance-tracked, reproducible published assets with manifests.

	Both take `--rois`, `--seeds`, `--image-evals` / `--video-evals`, `--encoder-model`, `--out-dir`, etc., and default to the config's encoder and a cache-relative output directory.

	## Reproducibility

	The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a target-matching signal, not ground-truth reconstruction quality.

	## Intended use & limitations

	- Research use in visual neuroscience / brain-decoding. Outputs are predicted to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
	- Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
	- Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

	## Citation

	If you use NEvo, please cite:

	```bibtex
	@article{tang2026nevo,
	title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
	author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
	journal={arXiv preprint arXiv:2607.02317},
	year={2026}
	}
	```

	Project website: [nevo-project.epfl.ch](https://nevo-project.epfl.ch/)

	## Acknowledgements

	Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).