ykt668 commited on
Commit
1b46308
·
verified ·
1 Parent(s): 5bb0042

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: feature-extraction
4
+ tags:
5
+ - fmri
6
+ - mindeye2
7
+ - brain-decoding
8
+ - multimodal
9
+ - text-alignment
10
+ ---
11
+
12
+ # TextAlign Model for MindEye2
13
+
14
+ This repository contains the pre-trained weights and derived features for **[TextAlign-mindeye2](https://github.com/YKT-668/TextAlign-mindeye2)**.
15
+
16
+ **GitHub Codebase:** [YKT-668/TextAlign-mindeye2](https://github.com/YKT-668/TextAlign-mindeye2)
17
+ **Aligned Commit:** \`579ab6e1cb31f5e9e539fdccfef4c29984f5e870\`
18
+
19
+ ## Model Description
20
+ TextAlign improves fMRI-to-image and fMRI-to-text retrieval by aligning brain representations with fine-grained text embeddings. It is built on top of MindEye2 (Scotti et al., 2024).
21
+
22
+ - **Input:** fMRI betas (flattened cortical surface vertices).
23
+ - **Output:** CLIP L/14 latent embeddings (Vision & Text aligned).
24
+
25
+ ## Directory Structure
26
+
27
+ ### `checkpoints/`
28
+ - **`s1_textalign_stage1_FINAL_BEST_32/last.pth`** (25GB)
29
+ - The final Stage 1 model.
30
+ - Trained with counterfactual hard negatives.
31
+ - **Use this for inference.**
32
+ - **`s1_textalign_stage0_repair_80G/last.pth`** (23GB)
33
+ - The intermediate Stage 0 model (pre-training).
34
+
35
+ ### `features/`
36
+ Contains pre-computed text features required to run training or evaluation without access to the full NSD captions (which are restricted).
37
+ - `train_coco_text_clip.pt`
38
+ - `train_coco_captions.json`
39
+
40
+ ## Usage (Inference)
41
+
42
+ Please refer to the [GitHub Repository](https://github.com/YKT-668/TextAlign-mindeye2) for installation.
43
+
44
+ ```bash
45
+ # Example: Reconstruction Inference
46
+ python src/recon_inference_run.py \
47
+ --subject 1 \
48
+ --ckpt_path checkpoints/s1_textalign_stage1_FINAL_BEST_32/last.pth \
49
+ --eval_only
50
+ ```
51
+
52
+ ## Licensing
53
+ - Weights are released under MIT License.
54
+ - Derived features (`features/`) respect the original NSD/COCO terms. Do not redistribute primitive data.
checkpoints/s1_textalign_stage1_FINAL_BEST_32/last.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03db5c9e20d845ff00ace9c4ac8398a1dfaeec1fdf9f3810ef923234731a0c38
3
+ size 8614838272