Eigenbrain: frozen brain-decoding inference bundle

This repository contains frozen inference assets for three independent brain-decoding demonstrations: EEG classification, neural language decoding, and NSD Subject-1 fMRI-to-image reconstruction. It is an inference/replay bundle; it does not train a model and does not fit preprocessing transforms.

Contents

Task	Input	Frozen route	Main output
EEG classification	SEED-V eigenmode EEG	temporal/mode-attention encoder + 5-class head	predicted class
Language decoding	Alice BCI observation and prior	PoE + subject layer + QFormer + BGE tokens + Phi-4 LoRA	decoded English text
Image reconstruction	NSD Subject-1 CIFTI-4096 and modes-2000	anchored PoE + MindEye2 BrainNetwork + diffusion prior + SDXL-unCLIP	reconstructed PNG

The language route uses the public base models microsoft/Phi-4-mini-instruct and BAAI/bge-m3; the files in this repository are the project-specific alignment, projector, and LoRA weights. The image route uses a large frozen unCLIP checkpoint and requires a CUDA GPU with substantial storage and memory.

Download

Download this repository together with the companion inference code. The project code expects the following local layout:

assets/
weights/
weights/alice_lora/
weights/nsd_sub01/
models/Phi-4-mini-instruct/
models/bge-m3/

From the companion GitHub repository, run cd demo first. The companion download_assets.py restores the local layout above and downloads the project files from this repository and the two public base models:

export EIGEN_HF_REPO=SSp1ash/Eigenbrain
python download_assets.py

For classification and language only, omit the large NSD image files with python download_assets.py --skip-nsd.

Run

python run_inference.py --task classification --device cuda

python run_inference.py --task language --device cuda \
  --phi-path models/Phi-4-mini-instruct \
  --bge-path models/bge-m3

python run_inference.py --task image --device cuda

The image command evaluates the default ten held-out NSD examples and writes the selected qualitative reconstructions and image-level metrics under outputs/nsd_sub01/.

Files

Demo data are stored under assets/; checkpoints are organized by task under demo/checkpoints/. The largest file is demo/checkpoints/visual_decoding/unclip.ckpt (about 18 GB). Public base models are intentionally not duplicated here.

License and limitations

The code and frozen assets are provided for research demonstration and reproducibility. Check the licenses and terms of the public Phi-4, BGE-M3, MindEye2/SDXL-unCLIP components and the NSD data before redistribution or commercial use. The NSD route is within-subject reconstruction for Subject 1, not a cross-subject general-purpose image decoder.

Checkpoint naming

demo/checkpoint_manifest.json maps the task-based Hub names to the local filenames expected by the inference code and records file checksums. Renaming does not modify checkpoint bytes, training history, or evaluation status. Standard PEFT adapter filenames are retained for library compatibility.