Instructions to use mzx/NEvo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use mzx/NEvo with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("mzx/NEvo", torch_dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Simplify How-it-works, drop gallery caption column, halve gallery videos
Browse files- README.md +12 -57
- assets/gallery/EBA.gif +2 -2
- assets/gallery/FFA.gif +2 -2
- assets/gallery/MT.gif +2 -2
- assets/gallery/PPA.gif +2 -2
- assets/gallery/V1.gif +2 -2
- assets/gallery/pSTS.gif +2 -2
README.md
CHANGED
|
@@ -28,69 +28,24 @@ It orchestrates three frozen models (weights are **not** bundled β they are pu
|
|
| 28 |
|
| 29 |
Each clip is the **top result of a NEvo search targeting one visual region** β the model discovers, from scratch, stimuli that drive that region's known selectivity.
|
| 30 |
|
| 31 |
-
| Region | Synthesized stimulus |
|
| 32 |
-
|--------|:--------------------:|
|
| 33 |
-
| **FFA** Β· faces |  |
|
| 34 |
-
| **PPA** Β· places |  |
|
| 35 |
-
| **MT** Β· motion |  |
|
| 36 |
-
| **EBA** Β· bodies |  |
|
| 37 |
-
| **pSTS** Β· social motion |  |
|
| 38 |
-
| **V1** Β· early visual |  |
|
| 39 |
|
| 40 |
Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
|
| 41 |
|
| 42 |
## How it works
|
| 43 |
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
target (voxels / fMRI vector)
|
| 49 |
-
β
|
| 50 |
-
βΌ
|
| 51 |
-
image prompt gene pool ββcrossoverβββΊ child genes ββmutationβββΊ conditioning
|
| 52 |
-
β² β
|
| 53 |
-
β βΌ
|
| 54 |
-
β SDXL-Turbo (textβimage)
|
| 55 |
-
β β
|
| 56 |
-
β βΌ
|
| 57 |
-
β image
|
| 58 |
-
β β
|
| 59 |
-
β βΌ
|
| 60 |
-
ββββββββββββββββββ activation reward βββ vJEPA-2 fMRI encoder
|
| 61 |
-
|
| 62 |
-
(repeat over generations)
|
| 63 |
-
β
|
| 64 |
-
βΌ
|
| 65 |
-
best image (highest predicted activation)
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
STAGE 2 β Genetic search over prompts (video, fixed image conditioning)
|
| 69 |
-
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 70 |
-
|
| 71 |
-
best image ββββββββββββββΊ fixed image conditioning (constant across search)
|
| 72 |
-
β
|
| 73 |
-
target (voxels / fMRI vector) β
|
| 74 |
-
β β
|
| 75 |
-
βΌ βΌ
|
| 76 |
-
video prompt gene pool ββcrossoverβββΊ child genes ββmutationβββΊ conditioning
|
| 77 |
-
β² β
|
| 78 |
-
β βΌ
|
| 79 |
-
β LTX-Video (imageβvideo, fixed image)
|
| 80 |
-
β β
|
| 81 |
-
β βΌ
|
| 82 |
-
β 2-second video
|
| 83 |
-
β β
|
| 84 |
-
β βΌ
|
| 85 |
-
ββββββββββββββββββ activation reward βββ vJEPA-2 fMRI encoder
|
| 86 |
-
|
| 87 |
-
(repeat over generations)
|
| 88 |
-
β
|
| 89 |
-
βΌ
|
| 90 |
-
ranked candidate videos (best_prompt, best_score, best video)
|
| 91 |
-
```
|
| 92 |
|
| 93 |
-
|
| 94 |
|
| 95 |
## Installation
|
| 96 |
|
|
|
|
| 28 |
|
| 29 |
Each clip is the **top result of a NEvo search targeting one visual region** β the model discovers, from scratch, stimuli that drive that region's known selectivity.
|
| 30 |
|
| 31 |
+
| Region | Synthesized stimulus |
|
| 32 |
+
|--------|:--------------------:|
|
| 33 |
+
| **FFA** Β· faces |  |
|
| 34 |
+
| **PPA** Β· places |  |
|
| 35 |
+
| **MT** Β· motion |  |
|
| 36 |
+
| **EBA** Β· bodies |  |
|
| 37 |
+
| **pSTS** Β· social motion |  |
|
| 38 |
+
| **V1** Β· early visual |  |
|
| 39 |
|
| 40 |
Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
|
| 41 |
|
| 42 |
## How it works
|
| 43 |
|
| 44 |
+
NEvo runs an **evolutionary (genetic) search** over prompts, guided by a brain encoder:
|
| 45 |
+
|
| 46 |
+
`prompt population β SDXL-Turbo (image) β LTX-Video (video) β vJEPA-2 fMRI encoder β activation score β keep best β recombine β repeat`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
+
Each generation turns candidate prompts into short videos and scores how strongly the **target region** is predicted to respond; the highest-scoring prompts survive and are mutated/recombined into the next generation. Over generations the search converges on stimuli that maximally drive the target voxels (default objective `indices_mean`: the mean predicted response across them). It runs in two stages β first evolving the image prompt, then the motion prompt on top of the best image.
|
| 49 |
|
| 50 |
## Installation
|
| 51 |
|
assets/gallery/EBA.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/gallery/FFA.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/gallery/MT.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/gallery/PPA.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/gallery/V1.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/gallery/pSTS.gif
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|