dawidtang commited on
Commit
36c2be2
Β·
verified Β·
1 Parent(s): 75c3413

Simplify How-it-works, drop gallery caption column, halve gallery videos

Browse files
README.md CHANGED
@@ -28,69 +28,24 @@ It orchestrates three frozen models (weights are **not** bundled β€” they are pu
28
 
29
  Each clip is the **top result of a NEvo search targeting one visual region** β€” the model discovers, from scratch, stimuli that drive that region's known selectivity.
30
 
31
- | Region | Synthesized stimulus | What the search converged on |
32
- |--------|:--------------------:|------------------------------|
33
- | **FFA** Β· faces | ![FFA](assets/gallery/FFA.gif) | *fashion editorial, multiple faces, crowd of people* |
34
- | **PPA** Β· places | ![PPA](assets/gallery/PPA.gif) | *unreal-engine render, subway station, looking-up view* |
35
- | **MT** Β· motion | ![MT](assets/gallery/MT.gif) | *sprinting athlete, body in action, time-lapse* |
36
- | **EBA** Β· bodies | ![EBA](assets/gallery/EBA.gif) | *two figures, legs and feet, three people* |
37
- | **pSTS** Β· social motion | ![pSTS](assets/gallery/pSTS.gif) | *hugging, greeting handshake, three people* |
38
- | **V1** Β· early visual | ![V1](assets/gallery/V1.gif) | *pop art, circuitry PCB traces, monochromatic* |
39
 
40
  Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
41
 
42
  ## How it works
43
 
44
- ```
45
- STAGE 1 β€” Genetic search over prompts (image)
46
- ════════════════════════════════════════════════════
47
-
48
- target (voxels / fMRI vector)
49
- β”‚
50
- β–Ό
51
- image prompt gene pool ──crossover──► child genes ──mutation──► conditioning
52
- β–² β”‚
53
- β”‚ β–Ό
54
- │ SDXL-Turbo (text→image)
55
- β”‚ β”‚
56
- β”‚ β–Ό
57
- β”‚ image
58
- β”‚ β”‚
59
- β”‚ β–Ό
60
- └───────────────── activation reward ◄── vJEPA-2 fMRI encoder
61
-
62
- (repeat over generations)
63
- β”‚
64
- β–Ό
65
- best image (highest predicted activation)
66
-
67
-
68
- STAGE 2 β€” Genetic search over prompts (video, fixed image conditioning)
69
- ════════════════════════════════════════════════════════════════════════
70
-
71
- best image ─────────────► fixed image conditioning (constant across search)
72
- β”‚
73
- target (voxels / fMRI vector) β”‚
74
- β”‚ β”‚
75
- β–Ό β–Ό
76
- video prompt gene pool ──crossover──► child genes ──mutation──► conditioning
77
- β–² β”‚
78
- β”‚ β–Ό
79
- │ LTX-Video (image→video, fixed image)
80
- β”‚ β”‚
81
- β”‚ β–Ό
82
- β”‚ 2-second video
83
- β”‚ β”‚
84
- β”‚ β–Ό
85
- └───────────────── activation reward ◄── vJEPA-2 fMRI encoder
86
-
87
- (repeat over generations)
88
- β”‚
89
- β–Ό
90
- ranked candidate videos (best_prompt, best_score, best video)
91
- ```
92
 
93
- The search maximizes an **objective over the encoder's predicted fMRI response** (default `indices_mean`: the mean predicted response across the target voxels).
94
 
95
  ## Installation
96
 
 
28
 
29
  Each clip is the **top result of a NEvo search targeting one visual region** β€” the model discovers, from scratch, stimuli that drive that region's known selectivity.
30
 
31
+ | Region | Synthesized stimulus |
32
+ |--------|:--------------------:|
33
+ | **FFA** Β· faces | ![FFA](assets/gallery/FFA.gif) |
34
+ | **PPA** Β· places | ![PPA](assets/gallery/PPA.gif) |
35
+ | **MT** Β· motion | ![MT](assets/gallery/MT.gif) |
36
+ | **EBA** Β· bodies | ![EBA](assets/gallery/EBA.gif) |
37
+ | **pSTS** Β· social motion | ![pSTS](assets/gallery/pSTS.gif) |
38
+ | **V1** Β· early visual | ![V1](assets/gallery/V1.gif) |
39
 
40
  Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
41
 
42
  ## How it works
43
 
44
+ NEvo runs an **evolutionary (genetic) search** over prompts, guided by a brain encoder:
45
+
46
+ `prompt population β†’ SDXL-Turbo (image) β†’ LTX-Video (video) β†’ vJEPA-2 fMRI encoder β†’ activation score β†’ keep best β†’ recombine β†’ repeat`
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
 
48
+ Each generation turns candidate prompts into short videos and scores how strongly the **target region** is predicted to respond; the highest-scoring prompts survive and are mutated/recombined into the next generation. Over generations the search converges on stimuli that maximally drive the target voxels (default objective `indices_mean`: the mean predicted response across them). It runs in two stages β€” first evolving the image prompt, then the motion prompt on top of the best image.
49
 
50
  ## Installation
51
 
assets/gallery/EBA.gif CHANGED

Git LFS Details

  • SHA256: f4fa8b6191f22fdfbe60fd3ebd7f318af350760fac9f59ba00af6531fd944cfb
  • Pointer size: 132 Bytes
  • Size of remote file: 3.03 MB

Git LFS Details

  • SHA256: 683bbc7d810684fbb404d40e33138b1bd006337604e4d3fe099ce2514fb1e13f
  • Pointer size: 132 Bytes
  • Size of remote file: 1.01 MB
assets/gallery/FFA.gif CHANGED

Git LFS Details

  • SHA256: 25c3731a895d9346b0538870218f027d4117039c7a7aaace20f14123b16b2787
  • Pointer size: 132 Bytes
  • Size of remote file: 2.87 MB

Git LFS Details

  • SHA256: 4fed64effefc8fc7f344f93e6b4d9f1e49a1cd7907d6844d9f4fff43eeacf4ab
  • Pointer size: 131 Bytes
  • Size of remote file: 845 kB
assets/gallery/MT.gif CHANGED

Git LFS Details

  • SHA256: e476431a15f55a34c7cdca246c3b3d619e748f4ff4f1b0d3b62aca248f797058
  • Pointer size: 132 Bytes
  • Size of remote file: 3.81 MB

Git LFS Details

  • SHA256: 5089ec8bb2513a5601896b585c9b10d96502b8ecb392fe4b9fd43889d37ecd0b
  • Pointer size: 132 Bytes
  • Size of remote file: 1.21 MB
assets/gallery/PPA.gif CHANGED

Git LFS Details

  • SHA256: c2db7610121120840966aec0adcb228da2e4733f1020f4625f9df63f73647917
  • Pointer size: 132 Bytes
  • Size of remote file: 3.56 MB

Git LFS Details

  • SHA256: 3e0be6ceec22838fd2ade0f0c505f2136e87b70c9a90235dcb0a43eb7c5b51a1
  • Pointer size: 132 Bytes
  • Size of remote file: 1.07 MB
assets/gallery/V1.gif CHANGED

Git LFS Details

  • SHA256: 87ce37b1a90329b520c3a3e03bae7ebbb51bc4cee31f76e475cc5c06d851c0f5
  • Pointer size: 132 Bytes
  • Size of remote file: 2.97 MB

Git LFS Details

  • SHA256: d4ceee8ad5903bb98b602a231f5bd2b8c5fc1bf6cbb36998a39a3ae2b5be2d13
  • Pointer size: 132 Bytes
  • Size of remote file: 1.08 MB
assets/gallery/pSTS.gif CHANGED

Git LFS Details

  • SHA256: 9ec04603024c6d8499e819d80bdcca102127954a81315734e7b72266d0249b91
  • Pointer size: 132 Bytes
  • Size of remote file: 2.52 MB

Git LFS Details

  • SHA256: 17f5f6992c66447468f896d6d5ba6f7334e67acbe8777163d27f53e9e44a4746
  • Pointer size: 131 Bytes
  • Size of remote file: 831 kB