🚧 Work in progress — this model is still being transferred from its main development repository, so the model card and API are subject to change.

NEvo — Neural-Guided Evolutionary Video Synthesis

🌐 Project website: nevo-project.epfl.ch · 📄 Paper: arXiv:2607.02317

NEvo is a self-contained Hugging Face custom Diffusers pipeline for neural-response-guided visual stimulus synthesis. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.

It orchestrates three frozen models. The models below are placeholders / defaults and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):

Role	Default model
Encoder (image/video → fMRI)	epfl-neuroai/vjepa2-encoder-basic (predict_fmri)
Text → image	stabilityai/sdxl-turbo
Image → video	Lightricks/LTX-Video-0.9.8-13B-distilled

Gallery

Each clip is from the top results of a NEvo search targeting one visual region — the model discovers, from scratch, stimuli that drive that region's known selectivity.

Region	Stimulus	Region	Stimulus
FFA · faces		PPA · places
MT · motion		EBA · bodies
pSTS · social motion		V1 · early visual

Explore the full interactive gallery and 3D brain maps at nevo-project.epfl.ch.

Installation

Off-the-shelf — no install. Load NEvo as a custom Diffusers pipeline; the package and its bundled data are fetched from the Hub automatically (you only need the usual dependencies below):

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

Or install the package (for cleaner from stimulus_synthesis import ... imports / development):

conda create -n nevo python=3.10 -y
conda activate nevo
pip install "git+https://huggingface.co/epfl-neuroai/NEvo"
# or from a local clone:
#   git clone https://huggingface.co/epfl-neuroai/NEvo && pip install ./NEvo
# then:  from stimulus_synthesis import NevoPipeline; pipe = NevoPipeline.from_pretrained("epfl-neuroai/NEvo")

Install runtime dependencies with pip install -r requirements.txt when loading the custom pipeline without installing the package. They include the native LTX dependencies (accelerate, einops, imageio, opencv-python-headless, pyyaml, safetensors, sentencepiece, timm, torchvision) as well as torch, diffusers, transformers, huggingface_hub, numpy, pillow and av. No nilearn / atlas downloads — ROI masks are shipped as small precomputed data files.

The default LTX backend uses the native rendering recipe: native multi-scale rendering, seven first-pass timesteps, learned spatial upsampling, then three refinement timesteps, with CFG=1 and STG=0 in both passes. Its public native source is pinned to Lightricks/ltx-video-distilled Space revision 8a42e93469c66b62d794b83a4233f5fc8439e2b5; source and original checkpoints are downloaded on first use. nevo-study is not a dependency. The exact recipe is shipped in stimulus_synthesis/data/ltx_native_config.json.

For an existing native installation, optionally set NEVO_LTX_REPO_DIR to its source directory and NEVO_LTX_MODELS_DIR to the directory containing the original distilled and spatial-upscaler .safetensors files. These overrides reuse the supplied local files; omit them to use the pinned public source/checkpoints.

Quickstart

Target a brain region by name — NEvo resolves its voxels and searches for a video predicted to drive it:

from diffusers import DiffusionPipeline

# fetches the pipeline (and package) from the Hub; model weights are pulled on first use
pipe = DiffusionPipeline.from_pretrained(
    "epfl-neuroai/NEvo", custom_pipeline="epfl-neuroai/NEvo", trust_remote_code=True,
)

out = pipe(roi="FFA", progress=True)                             # omit seed (default) -> different result each run; pass seed=<int> to reproduce (the seed used is in out.metadata["seed"])
print(out.best_prompt, out.best_score)

from stimulus_synthesis.media import save_video, video_to_t_c_h_w   # importable once the pipeline has loaded
save_video(video_to_t_c_h_w(out.best.video), "best_stimulus.mp4", fps=out.metadata["video_fps"])
out.best.image.save("best_stimulus.png")                            # and the stage-1 best image (PIL)

This runs up to 400 image evaluations then 200 video evaluations (population 20). Images use one-step 512×512 SDXL-Turbo. Videos use the native LTX recipe above at 512×512: 49 frames generated, trimmed to 48 frames / 2 seconds at 24 fps, with decoder timestep 0.05, decoder noise 0.025 and image-conditioning noise 0.15. The stage-1 scene description is prepended to every motion prompt. out.best.metadata["effective_video_prompt"] records the text actually sent to LTX; out.metadata["video_generation_settings"] records the native recipe. This is more expensive than the old single-pass 25-frame default.

Faster run

For a quicker first result, shrink the search while preserving the same LTX quality settings:

out = pipe(
    roi="FFA",
    progress=True,
    image_max_evals=80,       # stage-1 (image) evaluation budget (default: 400)
    video_max_evals=40,       # stage-2 (video) evaluation budget (default: 200)
    population_size=8,        # GA population per generation (default: 20)
    image_batch_size=8,       # images generated per batch (default: 16)
    video_batch_size=1,       # native videos are generated sequentially
    seed=0,                   # RNG seed, for reproducibility
)

Enhanced search space. Selecting a region (roi=...) restricts the prompt search to the categories relevant to that region — a smaller space that converges faster. Case-insensitive names, hemisphere suffixes (for example FFA_lh), and comma-separated ROI unions use the corresponding category filters. Pass enforce_general_search_space=True to search the full general space instead.

Available ROI tokens (comma-separated tokens are unioned):

• Named ROIs: FFA, PPA, MT, EBA, LOC, RSC, pSTS, aSTS, V1, V2, V3, V4 — optionally hemisphere-suffixed (FFA_lh, MT_rh).
• Searchlight regions: SL-<n> (both hemispheres), SL-<n>_lh, SL-<n>_rh (58 both / 28 lh / 30 rh).

from stimulus_synthesis.neuro import available_rois, searchlight_counts
available_rois()        # ['EBA','FFA','LOC','MT','PPA','RSC','V1','V2','V3','V4','aSTS','pSTS']
searchlight_counts()    # {'both': 58, 'lh': 28, 'rh': 30}

fsaverage5 only. The bundled ROI/searchlight masks are defined on the fsaverage5 cortical surface (20 484 vertices). Targeting a region by name therefore requires an encoder whose predict_fmri output lives in that same space — the default epfl-neuroai/vjepa2-encoder-basic. A custom encoder with a different output space can still be driven with explicit vector/indices targets, but not with the named-ROI helper.

Custom targets

Instead of a region name, pass raw voxel indices or a full target fMRI vector:

import numpy as np
from stimulus_synthesis import resolve_driving_voxels

mask = resolve_driving_voxels("FFA")                 # boolean mask, length 20484
out = pipe(target={"type": "indices", "indices": np.flatnonzero(mask).tolist()})

Target types & objectives

Target	Objective (default)	Meaning
{"type": "indices", "indices": [...]}	indices_mean	mean predicted response over ROI voxels
{"type": "vector", "vector": [...]} (len 20484)	target_vector_cosine / vector_dot	match a full target fMRI vector
{"type": "weights", "weights": [...]}	weighted_mean	weighted voxel objective

Search parameters (defaults)

Set in stimulus_synthesis_config.json:

Param	Default	Notes
default_image_max_evals	400	stage-1 (image) evaluation budget (GA max_evals)
default_video_max_evals	200	stage-2 (video) evaluation budget
default_population_size	20	GA population per generation (= n_init)
default_score_frames	24	number of frames the encoder scores (a still image is replicated to this)
default_score_size	224	resolution the clip is resized to for the encoder (call-time: score_size=)
default_mutation_rate	0.25
default_elite_frac	0.35
default_objective	indices_mean
default_score_transform	disabled	robust augmentation off by default (clean single pass)
default_image_kwargs	{num_inference_steps: 1, guidance_scale: 0, height: 512, width: 512}	fast SDXL-Turbo settings (merged under call-time image_kwargs)
default_video_kwargs	{duration_s: 2.0, frame_rate: 24, height: 512, width: 512}	native multi-scale LTX; 49 generated / 48 returned frames

Each stage runs a genetic search with population population_size (default 20) until it hits its evaluation budget — image_max_evals (default 400) and video_max_evals (default 200) generate→score passes. image_kwargs / video_kwargs merge over the defaults. For native LTX, use duration_s and frame_rate to control clip length. An explicit num_frames is the generated frame count and must be 8*k+1 and cover the requested duration. Generic num_inference_steps / guidance_scale overrides are not native multi-scale options: the shipped profile sets both pass schedules and guidance explicitly. Other video models continue to use the Diffusers adapter. To deliberately use the legacy Diffusers path for LTX, set image_to_video_backend="diffusers" and replace default_video_kwargs through synthesis_config with the options supported by that pipeline.

Robust scoring

By default each candidate is scored with a single clean encoder pass. An optional robust mode — the mean over 4 augmented draws (random crop 0.8, Gaussian σ=0.1) via RobustTransformScorer — reduces sensitivity to encoder artifacts; turn it on by setting "enabled": true in default_score_transform.

Cache configuration

Model weights and outputs cache location resolves in priority order:

1. NEvo_CACHE_DIR — set it in a repo-root .env file (see .env.example) or the environment.
2. Otherwise the system/user-default HuggingFace cache (HF_HOME, else ~/.cache/huggingface) is used and left untouched.
3. Only if no default is resolvable, a repo-local cache/ is used.

cache/ and .env are git-ignored.

Batch runners

Two ROI-driven, two-stage (image-search → video-search) runners are included:

• run_roi_samples.py — genetic search per ROI/seed, scoring in-memory tensors; writes best_image.png / best_video.mp4 / scores.
• run_regional_asset_pilot.py — same search but exports every candidate to a deterministically-encoded file (PNG/MP4), hashes it (sha256), and scores the decoded file — producing provenance-tracked, reproducible published assets with manifests.

Both take --rois, --seeds, --image-evals / --video-evals, --encoder-model, --out-dir, etc., and default to the config's encoder and a cache-relative output directory.

Reproducibility

The pipeline is deterministic for a fixed seed/config: the shipped ROI masks reproduce the original atlas masks bit-for-bit, and a fixed-seed run reproduces prior scores exactly. Encoder scores are a target-matching signal, not ground-truth reconstruction quality.

The native LTX decoder also consumes PyTorch's global RNG, in addition to the supplied diffusion generator. NEvo seeds global RNG state at the start of each search stage. For standalone adapter comparisons, match both the generator seed and global CPU/CUDA RNG states; a matching generator seed alone is insufficient. The native profile replaces the old rendering recipe, so scores and videos from the old Diffusers default are not expected to replay under the new backend.

Intended use & limitations

• Research use in visual neuroscience / brain-decoding. Outputs are predicted to drive a target region under a specific encoder — they are hypotheses to validate, not ground truth.
• Optimizing hard against a single encoder can exploit encoder artifacts; inspect images and use held-out validation.
• Requires a CUDA GPU with enough memory for the 13B video model; you must accept the license/access terms of the referenced upstream models.

Citation

If you use NEvo, please cite:

@article{tang2026nevo,
  title={NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity},
  author={Tang, Yingtian and Salehi, Sogand and Zhou, Ming and Zamir, Amir and Isik, Leyla and Schrimpf, Martin},
  journal={arXiv preprint arXiv:2607.02317},
  year={2026}
}

Project website: nevo-project.epfl.ch

Acknowledgements

Builds on BrainDiVE-style encoder-guided synthesis, vJEPA-2, SDXL-Turbo, and LTX-Video. ROI/searchlight definitions derive from an fsaverage-space group atlas (precomputed and bundled).

Intended Use, Out-of-Scope Use & Risks

NEvo is a research-only tool for computational visual neuroscience: it searches for stimuli predicted to drive a target brain region under a fixed image/video→fMRI encoder, and its outputs are unvalidated hypotheses, not scientific findings. It is not a medical device and must not be used for diagnosis, treatment, or to design, guide, or deliver any real brain stimulation (TMS, tDCS/tACS, DBS, intracranial, etc.) or in-vivo neuromodulation — it only optimizes a predicted fMRI response, not effects on a living brain, and any use involving human or animal subjects requires independent ethical review (e.g. IRB). Optimizing against a single encoder can exploit artifacts, and the generative models may produce biased or disturbing imagery; the model is provided as is, without warranty, for non-commercial use under CC BY-NC 4.0 (upstream models carry their own licenses).