README.md

---
license: cc-by-nd-4.0
language:
- en
library_name: pytorch
tags:
- eeg 
- biosignal 
- foundation-model 
- self-supervised 
- masked-modeling 
- windowed-alternating-attention 
- neuroscience
datasets:
- TUEG 
- SEED 
- BOAS 
- SleepEDFx 
- BCI-NER 
- GWD 
- CHB-MIT
- Neonate
- PhysioNet-MI 
- SHU-MI 
- TUAB 
- STEW 
- SEED-VIG 
- ISRUC 
- Mumtaz 
- MentalArithmetic 
- SEED-V 
metrics:
- roc_auc 
- f1
- nrmse 
model-index:
  - name: S-CEReBrO
    results:
      - task:
          type: time-series-classification
          name: Seizure Detection
        dataset:
          type: CHB-MIT
          name: CHB-MIT
        metrics:
          - type: roc_auc
            value: 87.45
            name: AUROC
      - task:
          type: time-series-classification
          name: Seizure Detection
        dataset:
          type: Neonate
          name: Neonate
        metrics:
          - type: roc_auc
            value: 85.76
            name: AUROC
      - task:
          type: time-series-classification
          name: Motor Imagery Classification
        dataset:
          type: PhysioNet-MI
          name: PhysioNet-MI
        metrics:
          - type: f1
            value: 60.93
            name: Weighted F1
      - task:
          type: time-series-classification
          name: Motor Imagery Classification
        dataset:
          type: SHU-MI
          name: SHU-MI
        metrics:
          - type: roc_auc
            value: 66.48
            name: AUROC
      - task:
          type: time-series-classification
          name: Abnormal Classification
        dataset:
          type: TUAB
          name: TUAB
        metrics:
          - type: roc_auc
            value: 89.30
            name: AUROC
      - task:
          type: time-series-classification
          name: Mental Workload Classification
        dataset:
          type: STEW
          name: STEW
        metrics:
          - type: f1
            value: 52.90
            name: Weighted F1
      - task:
          type: time-series-regression
          name: Vigilance Estimation
        dataset:
          type: SEED-VIG
          name: SEED-VIG
        metrics:
          - type: nrmse
            value: 0.93
            name: NRMSE
      - task:
          type: time-series-classification
          name: Sleep Stage Classification
        dataset:
          type: ISRUC
          name: ISRUC
        metrics:
          - type: f1
            value: 78.23
            name: Weighted F1
      - task:
          type: time-series-classification
          name: Mental Disorder Diagnosis
        dataset:
          type: Mumtaz
          name: Mumtaz
        metrics:
          - type: roc_auc
            value: 98.23
            name: AUROC
      - task:
          type: time-series-classification
          name: Mental Stress Detection
        dataset:
          type: MentalArithmetic
          name: MentalArithmetic
        metrics:
          - type: roc_auc
            value: 68.97
            name: AUROC
      - task:
          type: time-series-classification
          name: Emotion Recognition
        dataset:
          type: SEED-V
          name: SEED-V
        metrics:
          - type: f1
            value: 28.16
            name: Weighted F1
---
<div align="center">
  <h1>S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring</h1>
</div>
<p align="center">
  <a href="https://github.com/pulp-bio/biofoundation">
    <img src ="https://img.shields.io/github/stars/pulp-bio/biofoundation?color=ccf" alt="Github">
  </a>
  <a href="https://creativecommons.org/licenses/by-nd/4.0/">
    <img src="https://img.shields.io/badge/License-CC_BY--ND_4.0-lightgrey.svg" alt="License">
  </a>
  <a href="https://arxiv.org/abs/2607.27913">
    <img src="https://img.shields.io/badge/arXiv-2607.27913-b31b1b.svg" alt="Paper">
  </a>
</p>

**S-CEReBrO** (Streaming CEReBrO) is an **EEG foundation model** designed for **continuous monitoring**. 
S-CEReBrO addresses the memory overflow bottleneck of continuous monitoring by utilizing a novel **Windowed Alternating Attention mechanism**. This factorizes attention computation into fixed-size spatiotemporal windows, guaranteeing a **constant KV cache memory** as only the active window requires resident attention maps.

---

## 🔒 License & Usage Policy (Weights)

**Weights license:** The released model weights are licensed under **Creative Commons Attribution–NoDerivatives 4.0 (CC BY-ND 4.0)**. This section summarizes the practical implications for users. *This is not legal advice; please read the full license text.*

### ✅ You may
- **Use** and **redistribute** the **unmodified** S-CEReBrO weights (including in commercial settings) **with proper attribution** to the S-CEReBrO authors.
- **Fine-tune / adapt** the weights **for your internal use** (research or production) **without redistributing** the modified weights.
- **Publish your code, configs, logs, and papers** describing experiments with S-CEReBrO (please cite the paper).

### 🚫 You may not
- **Share, host, or redistribute any modified weights** (including LoRA/adapter/delta checkpoints or pruned/quantized variants). Any parameter set that encodes an adaptation is considered a derivative and cannot be shared under CC BY-ND 4.0.
- **Imply endorsement** by the S-CEReBrO authors for any derivative or evaluation without our written permission.
- **Use the S-CEReBrO name** in a way that suggests your modified model is an official S-CEReBrO release.

### 🤝 How to contribute improvements (PR-gated releases)
We welcome community improvements via a **pull-request (PR)** workflow. If you believe your improvements should become an **official S-CEReBrO release**:
1. **Open a PR** in the [BioFoundation repository](https://github.com/pulp-bio/biofoundation) describing the change (architecture/head/training recipe, datasets, preprocessing, compute).
2. Include **reproducibility artifacts**: configs, seeds, scripts, environment details, training/validation logs, and the **evaluation protocol** with exact splits.
3. Provide **comprehensive results** (AUROC/AUPR/BA, FLOPs, memory) vs. the baselines reported in the S-CEReBrO paper.
4. After **maintainer review**, approved changes will be **retrained/validated** and, if accepted, **released by the maintainers** as a new **official S-CEReBrO** checkpoint under **CC BY-ND 4.0**.

> Rationale: CC BY-ND protects users from fragmented, lower-quality variants, while still enabling internal fine-tuning and a path for the community to upstream improvements through review.

---

## 🔎 Model Summary

- **Goal:** Realize efficient, generalizable, and continuous EEG monitoring while breaking the memory barrier of Transformer-based architectures.
- **Core idea:** **Windowed Alternating Attention** interleaves spatial and temporal attention blocks. This restricts interactions to local neighborhoods in time and space, allowing the model to process long signals while maintaining a constant resident memory. Complexity is $\mathcal{O}(CT)$.
- **Pre-training data:** **>25,000 hours** of EEG from **>12,000 subjects**. Includes TUEG, SEED series, BOAS, SleepEDFx, BCI-NER, and GWD. Downstream subjects (e.g. TUAB) are strictly excluded to prevent leakage.
- **Downstream tasks:** Seizure detection (CHB-MIT, Neonate), Motor Imagery (PhysioNet-MI, SHU-MI), Abnormal classification (TUAB), Mental Workload (STEW), Vigilance Estimation (SEED-VIG), Sleep Staging (ISRUC), Mental Disorder Diagnosis (Mumtaz), Mental Stress (Mental Arithmetic), and Emotion Recognition (SEED-V).

---

## 🚀 Model Variants

S-CEReBrO is highly compact, utilizing only 2.4M parameters. This design allows it to achieve state-of-the-art performance with up to 60% fewer parameters than existing foundation models.

---

## 📊 Results

S-CEReBrO achieves state-of-the-art performance on 7 of 11 downstream public benchmarks.

- **CHB-MIT (seizure detection):** 87.45 AUROC.
- **Neonate (seizure detection):** 85.76 AUROC.
- **PhysioNet-MI (motor imagery):** 60.93 Weighted F1.
- **SHU-MI (motor imagery):** 66.48 AUROC.
- **TUAB (abnormal classification):** 89.30 AUROC.
- **STEW (mental workload):** 52.90 Weighted F1.
- **SEED-VIG (vigilance estimation):** 0.93 NRMSE.
- **ISRUC (sleep staging):** 78.23 Weighted F1.
- **Mumtaz (mental disorder diagnosis):** 98.23 AUROC.
- **Mental Arithmetic (mental stress detection):** 68.97 AUROC.
- **SEED-V (emotion recognition):** 28.16 Weighted F1.

**Efficiency & Scaling:** Without low-level optimzations, windowed Alternating Attention can process signals 100x longer than full self-attention (up to 14 hours / 50,000 seconds). It also processes signals 3x longer than low-rank linear attention. Compared to low-rank linear attention on long contexts (15,000 seconds), it requires 55% of the memory while increasing inference throughput by 2.1x.

---

## 🧠 Intended Use & Limitations

**Intended use.** Research on continuous EEG representation learning and classification (clinical diagnostics, BCI, state monitoring) where long context and memory efficiency are necessary.

**Limitations.**
- **Not a medical device.** Do **not** use for clinical decisions without proper validation & regulatory clearance.  
- **Unseen topologies:** Zero-shot transfer to **very different/dense** layouts can underperform SOTA despite positive scaling; consider augmenting pre-training montage diversity and spatial encodings.
- **Distribution shifts:** Performance varies across cohorts, devices, and label protocols; validate locally and consider domain adaptation.

---

## 🏗️ Architecture & Training

**Spatiotemporal Tokenization.** The raw EEG recording is tokenized using a sliding window. A shared convolutional encoder projects each patch into a latent embedding. Spatiotemporal topology is preserved via an index-based matrix for temporal order and a continuous spatial embedding mapped via an MLP from 3D electrode coordinates.

**Windowed Temporal Attention (WTA).** WTA layers model temporal dependencies independently for each channel. Each query attends only to keys within a local neighborhood, avoiding memory accumulation.

**Windowed Spatial Attention (WSA).** WSA layers facilitate cross-channel information exchange at a fixed time index. Attention computation is restricted to a local neighborhood. Layer-specific dilation and shift factors stagger receptive fields to enable distant, physiologically related channels to interact.

**Pre-training objectives.** The model is pre-trained using Masked Autoencoding (MAE). The objective minimizes the weighted sum of reconstruction losses for masked and visible patch positions.

---

## 🔧 How to Use

S-CEReBrO is supported within the BioFoundation framework. Experiments and setups follow similar pipeline structures:

1. **Install & read data prep**: clone the [BioFoundation repo](https://github.com/pulp-bio/biofoundation). Set up the environment as described there, then open `make_datasets/README.md` for dataset-specific notes.
2. **Point to weights**: set `pretrained_safetensors_path` to the downloaded weights in your experiment YAML.
3. **Preprocess data**: acquire fine-tuning dataset and follow preprocessing protocol to generate `train/test/val.lmdb` files.
4. **Update data module config**
5. **Task settings**: Configure `classification_type`, `model.num_classes`, and task modes inside your YAML config based on downstream objectives.
6. **Launch fine-tuning (Hydra)**:

```bash
python -u run_train.py +experiment=s_cerebro_finetune
```

---

## 🔗 Sources

- **Code:** https://github.com/pulp-bio/BioFoundation  
- **Paper:** S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring (arxiv:2607.27913)  

---

## 📜 Citation

If you use S-CEReBrO, please cite:

```bibtex
@misc{bucagu2026scerebrobreakingmemorybarrier,
      title={S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring}, 
      author={Glenn Anta Bucagu and Thorir Mar Ingolfsson and Yawei Li and Luca Benini},
      year={2026},
      eprint={2607.27913},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.27913}, 
}
```

---

## 🛠️ Maintenance & Contact

- **Issues & support:** please open a GitHub issue in the BioFoundation repository.

---

## 🔗 Related Models

- **[LUNA](https://huggingface.co/PulpBio/LUNA)** — Transformer-based topology-agnostic EEG foundation model (NeurIPS 2025). Source of the channel-unification cross-attention module that LuMamba reuses.
- **[FEMBA](https://huggingface.co/PulpBio/FEMBA)** — Bidirectional Mamba foundation model for EEG. Source of the linear-complexity temporal backbone that LuMamba reuses.
- **[TinyMyo](https://huggingface.co/PulpBio/TinyMyo)** — Tiny foundation model for flexible EMG signal processing at the edge.
  
## 🗒️ Changelog

- **v1.0:** Initial release of LuMamba model card with task-specific checkpoints and instructions.