|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- Sleep-EDF |
|
|
- TUAB |
|
|
- MOABB |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- eeg |
|
|
- brain |
|
|
- timeseries |
|
|
- self-supervised |
|
|
- transformer |
|
|
- biomedical |
|
|
- neuroscience |
|
|
--- |
|
|
|
|
|
# BENDR: BErt-inspired Neural Data Representations |
|
|
|
|
|
Pretrained BENDR model for EEG classification tasks. This is the official Braindecode implementation |
|
|
of BENDR from Kostas et al. (2021). |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model Type**: Transformer-based EEG encoder |
|
|
- **Pretraining**: Self-supervised learning on masked sequence reconstruction |
|
|
- **Architecture**: |
|
|
- Convolutional Encoder: 6 blocks with 512 hidden units |
|
|
- Transformer Contextualizer: 8 layers, 8 attention heads |
|
|
- Total Parameters: ~157M |
|
|
- **Input**: Raw EEG signals (20 channels, variable length) |
|
|
- **Output**: Contextualized representations or class predictions |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from braindecode.models import BENDR |
|
|
import torch |
|
|
|
|
|
# Load pretrained model |
|
|
model = BENDR(n_chans=20, n_outputs=2) |
|
|
|
|
|
# Load pretrained weights from Hugging Face |
|
|
from huggingface_hub import hf_hub_download |
|
|
checkpoint_path = hf_hub_download(repo_id="braindecode/bendr-pretrained-v1", filename="pytorch_model.bin") |
|
|
checkpoint = torch.load(checkpoint_path) |
|
|
model.load_state_dict(checkpoint["model_state_dict"], strict=False) |
|
|
|
|
|
# Use for inference |
|
|
model.eval() |
|
|
with torch.no_grad(): |
|
|
eeg_data = torch.randn(1, 20, 600) # (batch, channels, time) |
|
|
predictions = model(eeg_data) |
|
|
``` |
|
|
|
|
|
## Fine-tuning |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from torch.optim import Adam |
|
|
|
|
|
# Freeze encoder for transfer learning |
|
|
for param in model.encoder.parameters(): |
|
|
param.requires_grad = False |
|
|
|
|
|
# Fine-tune on downstream task |
|
|
optimizer = Adam(model.parameters(), lr=0.0001) |
|
|
``` |
|
|
|
|
|
## Paper |
|
|
|
|
|
[BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://doi.org/10.3389/fnhum.2021.653659) |
|
|
|
|
|
Kostas, D., Aroca-Ouellette, S., & Rudzicz, F. (2021). |
|
|
Frontiers in Human Neuroscience, 15, 653659. |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@article{kostas2021bendr, |
|
|
title={BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data}, |
|
|
author={Kostas, Demetres and Aroca-Ouellette, St{\'e}phane and Rudzicz, Frank}, |
|
|
journal={Frontiers in Human Neuroscience}, |
|
|
volume={15}, |
|
|
pages={653659}, |
|
|
year={2021}, |
|
|
publisher={Frontiers} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Implementation Notes |
|
|
|
|
|
- Start token is correctly extracted at index 0 (BERT [CLS] convention) |
|
|
- Uses T-Fixup weight initialization for stability |
|
|
- Includes LayerDrop for regularization |
|
|
- All architectural improvements from original paper maintained |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
|
|
## Authors |
|
|
|
|
|
- Braindecode Team |
|
|
- Original paper: Kostas et al. (2021) |
|
|
|
