MODEL_CARD.md

---
license: cc-by-nc-4.0
library_name: cortexlab
tags:
  - neuroscience
  - fmri
  - brain-encoding
  - multimodal
  - tribe-v2
  - brain-alignment
  - cognitive-load
language:
  - en
pipeline_tag: other
---

# CortexLab

Enhanced multimodal fMRI brain encoding toolkit built on [Meta's TRIBE v2](https://github.com/facebookresearch/tribev2).

CortexLab extends TRIBE v2 with streaming inference, interpretability tools, cross-subject adaptation, brain-alignment benchmarking, and cognitive load scoring.

## What This Repo Contains

This is a **code-only** repository. It does not contain pretrained weights. The pretrained TRIBE v2 model is hosted by Meta at [`facebook/tribev2`](https://huggingface.co/facebook/tribev2).

## Features

| Feature | Description |
|---|---|
| **Streaming Inference** | Sliding-window real-time predictions from live feature streams |
| **ROI Attention Maps** | Visualize which brain regions attend to which temporal moments |
| **Modality Attribution** | Per-vertex importance scores for text, audio, and video |
| **Cross-Subject Adaptation** | Ridge regression or nearest-neighbour adaptation for new subjects |
| **Brain-Alignment Benchmark** | Score how "brain-like" any AI model's representations are (RSA, CKA, Procrustes) |
| **Cognitive Load Scorer** | Predict cognitive demand of media from predicted brain activation patterns |

## Prerequisites

The pretrained TRIBE v2 model uses **LLaMA 3.2-3B** as its text encoder. You must:

1. Accept Meta's LLaMA license at [llama.meta.com](https://llama.meta.com/)
2. Request access on [HuggingFace](https://huggingface.co/meta-llama/Llama-3.2-3B)
3. Authenticate: `huggingface-cli login`

## Installation

```bash
git clone https://github.com/siddhant-rajhans/cortexlab.git
cd cortexlab
pip install -e ".[analysis]"
```

## Quick Start

### Inference

```python
from cortexlab.inference.predictor import TribeModel

model = TribeModel.from_pretrained("facebook/tribev2", device="auto")
events = model.get_events_dataframe(video_path="clip.mp4")
preds, segments = model.predict(events)
```

### Brain-Alignment Benchmark

```python
from cortexlab.analysis import BrainAlignmentBenchmark

bench = BrainAlignmentBenchmark(brain_predictions, roi_indices=roi_indices)
result = bench.score_model(clip_features, method="rsa")
print(f"Alignment: {result.aggregate_score:.3f}")
```

### Cognitive Load Scoring

```python
from cortexlab.analysis import CognitiveLoadScorer

scorer = CognitiveLoadScorer(roi_indices)
result = scorer.score_predictions(predictions)
print(f"Overall load: {result.overall_load:.2f}")
```

## Compute Requirements

| Component | VRAM | Notes |
|---|---|---|
| TRIBE v2 encoder | ~1 GB | Small (1.15M params) |
| LLaMA 3.2-3B (text) | ~8 GB | Features cached after first run |
| V-JEPA2 (video) | ~6 GB | Features cached after first run |
| Wav2Vec-BERT (audio) | ~3 GB | Features cached after first run |

Minimum: 16 GB VRAM GPU for full inference. CPU works but is slow. Analysis tools (benchmark, cognitive load) work with zero GPU on precomputed predictions.

## Architecture

```
src/cortexlab/
  core/          Model architecture, attention extraction, subject adaptation
  data/          Dataset loading, transforms, HCP ROI utilities
  training/      PyTorch Lightning training pipeline
  inference/     Predictor, streaming, modality attribution
  analysis/      Brain-alignment benchmark, cognitive load scorer
  viz/           Brain surface visualization (nilearn, pyvista)
```

## License

CC BY-NC 4.0 (non-commercial use only), inherited from TRIBE v2.

This project does not redistribute pretrained weights. Users must download weights directly from [`facebook/tribev2`](https://huggingface.co/facebook/tribev2).

## Citation

If you use CortexLab in your research, please cite the original TRIBE v2 paper:

```bibtex
@article{dascoli2026tribe,
  title={A foundation model of vision, audition, and language for in-silico neuroscience},
  author={d'Ascoli, St{\'e}phane and others},
  year={2026}
}
```

## Links

- **GitHub**: [siddhant-rajhans/cortexlab](https://github.com/siddhant-rajhans/cortexlab)
- **TRIBE v2**: [facebookresearch/tribev2](https://github.com/facebookresearch/tribev2)
- **Pretrained weights**: [facebook/tribev2](https://huggingface.co/facebook/tribev2)