| --- |
| license: cc-by-nd-4.0 |
| language: |
| - en |
| library_name: pytorch |
| tags: |
| - eeg |
| - ppg |
| - ecg |
| - biosignals |
| - multimodal model |
| - time-series |
| - cross-modal |
| - foundation model |
| - self-supervised |
| - masked modeling |
| - transformer |
| - single encoder |
| datasets: |
| - TUEG |
| - Siena |
| - MIMIC-IV-ECG |
| - PulseDB |
| - CODE-15 |
| - TUAB |
| - PTB-XL |
| - CSN |
| - HMC |
| metrics: |
| - balanced_accuracy |
| - roc_auc |
| - pr_auc |
| - weighted_f1 |
| - cohen_kappa |
| model-index: |
| - name: PanLUNA |
| results: |
| - task: |
| finetuning: Full |
| type: time-series-classification |
| name: EEG Abnormality Detection |
| dataset: |
| type: TUAB |
| name: TUH EEG Abnormal Corpus (TUAB) |
| metrics: |
| - type: balanced_accuracy |
| value: 81.22 |
| name: Balanced Accuracy (%) |
| - type: roc_auc |
| value: 0.893 |
| name: AUROC |
| - type: pr_auc |
| value: 0.899 |
| name: AUC-PR |
| - task: |
| finetuing: Full |
| type: time-series-classification |
| name: EEG Sleep Stage Classification |
| dataset: |
| type: HMC |
| name: Haaglanden Medisch Centrum sleep staging database |
| metrics: |
| - type: balanced_accuracy |
| value: 0.742 |
| name: Balanced Accuracy (%) |
| - type: cohen_kappa |
| value: 0.695 |
| name: Cohen's Kappa |
| - type: weighted_f1 |
| value: 0.766 |
| name: Weighted F1 |
| - task: |
| finetuning: Low-Rank Adaptation (LoRA) |
| type: time-series-classification |
| name: ECG PTB-XL Super Class |
| dataset: |
| type: PTB-XL |
| name: PTB-XL, a large publicly available electrocardiography dataset |
| metrics: |
| - type: roc_auc |
| value: 0.908 |
| name: AUROC |
| - task: |
| finetuning: Low-Rank Adaptation (LoRA) |
| type: time-series-classification |
| name: ECG PTB-XL Sub Class |
| dataset: |
| type: PTB-XL |
| name: PTB-XL, a large publicly available electrocardiography dataset |
| metrics: |
| - type: roc_auc |
| value: 0.888 |
| name: AUROC |
| - task: |
| finetuning: Low-Rank Adaptation (LoRA) |
| type: time-series-classification |
| name: ECG PTB-XL Form |
| dataset: |
| type: PTB-XL |
| name: PTB-XL, a large publicly available electrocardiography dataset |
| metrics: |
| - type: roc_auc |
| value: 0.833 |
| name: AUROC |
| - task: |
| finetuning: Low-Rank Adaptation (LoRA) |
| type: time-series-classification |
| name: ECG PTB-XL Rhythm |
| dataset: |
| type: PTB-XL |
| name: PTB-XL, a large publicly available electrocardiography dataset |
| metrics: |
| - type: roc_auc |
| value: 0.964 |
| name: AUROC |
| - task: |
| finetuning: Low-Rank Adaptation (LoRA) |
| type: time-series-classification |
| name: ECG CSN |
| dataset: |
| type: CSN |
| name: Chapman-Shaoxing-Ningbo |
| metrics: |
| - type: roc_auc |
| value: 0.950 |
| name: AUROC |
| |
| --- |
| |
| <div align="center"> |
| <img src="https://raw.githubusercontent.com/masazelic/panluna/main/docs/model/logo/PanLUNA_logo.svg" alt="PanLUNA Logo" width="800"/> |
| <h1>PanLUNA: An Efficient and Robust Query-Unified Multimodal Model for Edge Biosignal Intelligence</h1> |
| </div> |
| <p align="center"> |
| <a href="https://github.com/pulp-bio/BioFoundation"> |
| <img src ="https://img.shields.io/github/stars/pulp-bio/BioFoundation?color=ccf" alt="Github"> |
| </a> |
| <a href="https://creativecommons.org/licenses/by-nd/4.0/"> |
| <img src="https://img.shields.io/badge/License-CC_BY--ND_4.0-lightgrey.svg" alt="License"> |
| </a> |
| <a href="https://arxiv.org/pdf/2604.04297"> |
| <img src="https://img.shields.io/badge/arXiv-2604.04297-b31b1b.svg" alt="Paper"> |
| </a> |
| </p> |
| |
| **PanLUNA** extends LUNA's channel-unification mechanism from topology invariance to **cross-modal fusion**, jointly processing EEG, ECG and PPG |
| within a single shared encoder via **sensor-type embeddings** -- no modality specific backbones, no paired multimodal data required during pretraining. |
|
|
| --- |
|
|
| ## 🔒 License & Usage Policy (Weights) |
| **Weights license:** The released model weights are licensed under **Creative Commons Attribution–NoDerivatives 4.0 (CC BY-ND 4.0)**. |
| This section summarizes the practical implications for users. *This is not legal advice; please read the full license text.* |
|
|
| ### ✅ You may |
| - **Use** and **redistribute** the **unmodified** LUNA weights (including in commercial settings) **with proper attribution** to the LUNA authors. |
| - **Fine-tune / adapt** the weights **for your internal use** (research or production) **without redistributing** the modified weights. |
| - **Publish your code, configs, logs, and papers** describing experiments with LUNA (please cite the paper). |
|
|
| ### 🚫 You may not |
| - **Share, host, or redistribute any modified weights** (including LoRA/adapter/delta checkpoints or pruned/quantized variants). Any parameter set that encodes an adaptation is considered a derivative and cannot be shared under CC BY-ND 4.0. |
| - **Imply endorsement** by the LUNA authors for any derivative or evaluation without our written permission. |
| - **Use the LUNA name** in a way that suggests your modified model is an official LUNA release. |
|
|
| ### 🤝 How to contribute improvements (PR-gated releases) |
| We welcome community improvements via a **pull-request (PR)** workflow. If you believe your improvements should become an **official LUNA release**: |
| 1. **Open a PR** in the [BioFoundation repository](https://github.com/pulp-bio/BioFoundation) describing the change (architecture/head/training recipe, datasets, preprocessing, compute). |
| 2. Include **reproducibility artifacts**: configs, seeds, scripts, environment details, training/validation logs, and the **evaluation protocol** (e.g., TUAB/TUAR/TUSL) with exact splits. |
| 3. Provide **comprehensive results** (AUROC/AUPR/BA, FLOPs, memory) vs. the baselines reported in the PanLUNA paper. |
| 4. After **maintainer review**, approved changes will be **retrained/validated** and, if accepted, **released by the maintainers** as a new **official PanLUNA** checkpoint under **CC BY-ND 4.0**. |
|
|
| > Rationale: CC BY-ND protects users from fragmented, lower-quality “PanLUNA variants,” while still enabling internal fine-tuning and a path for the community to upstream improvements through review. |
|
|
| --- |
|
|
| ## 🔎 Model Summary |
|
|
| - **Goal:** Compact pan-modal biosignal foundation model for EEG, ECG, and PPG within one shared encoder, with robustnes |
| to missing modalities and feasibility for ultra-low-power edge deployment. |
| - **Core idea:** PanLUNA extends LUNA’s Channel-Unification Module from EEG channel/topology unification |
| to cross-modal fusion: EEG, ECG, and PPG channels are treated as entries in a unified query set, augmented with sensor-type embeddings, and fused through cross-attention with learned latent queries inside a single encoder. |
| - **Pre-training data:** Approximately 40,000 hours of heterogeneous biosignal data from five public datasets: TUEG and Siena for EEG, MIMIC-IV and CODE-15% for ECG, and PulseDB for synchronized ECG+PPG. |
| - **Downstream tasks:** TUAB EEG abnormality detection; PTB-XL ECG Superclass (5-class), Subclass (23-class), Form (19-class), and Rhythm (12-class) classification; CSN ECG (38-class) classification; and HMC five-class sleep staging using EEG+ECG to evaluate cross-modal fusion and missing-modality robustness. |
|
|
| --- |
|
|
| ## 🚀 Model Variants |
|
|
| Aimed for feasible ultra-low-power edge deployment, PanLUNA exists in a Tiny Variant, with the following parameters: |
|
|
| | Variant | Parameters |PanLUNA parameters | |
| |-----------------|----------------|--------------------------------------| |
| | PanLUNA | 5.4M |(`num_queries` = 4, `embed_dim` = 64) | |
|
|
| To obtain the model of different size, consider scaling these parameters accordingly (e.g. num_queries=6 leads to 12M parameters). |
| |
| --- |
| |
| ## 📊 Results |
| |
| - **TUAB (Abnormal EEG Detection, bipolar montage, 22-channels):** 81.21% Balanced Accuracy, 0.899 AUROC, 0.893 AUPR. |
| - **HMC (EEG Sleep Staging Detection, 4-channels):** 74.16% Balanced Accuracy, 0.695 Cohen's Kappa, 0.765 Weighted F1. |
| - **ECG PTB-XL Super Class:** 0.908 AUROC |
| - **ECG PTB-XL Sub Class:** 0.888 AUROC |
| - **ECG PTB-XL Form:** 0.833 AUROC |
| - **ECG PTB-XL Rhythm:** 0.964 AUROC |
| - **ECG CSN:** 0.964 AUROC |
| |
| --- |
| |
| ## 🧠 Intended Use & Limitations |
| |
| **Intended use.** Research on biosignal (EEG, ECG, PPG) representation learning & classification (abnormality, sleep stages, waveform and rhythm irregularities in ECG), especially when aspiring for robustness in available channels and modalities. |
| |
| **Limitations.** |
| - **Not a medical device.** Do **not** use for clinical decisions without proper validation & regulatory clearance. |
| - **Unseen topologies:** Zero-shot transfer to **very different/dense** layouts (e.g., SEED-V) can underperform SOTA despite positive scaling; consider augmenting pre-training montage diversity and spatial encodings. |
| - **Distribution shifts:** Performance varies across cohorts, devices, and label protocols; validate locally and consider domain adaptation. |
| |
| --- |
| |
| ## 🏗️ Architecture & Training |
| |
| **PanLUNA Tokenizer & features:** Biosignals (EEG, ECG, PPG) are patch-segmented using shared convolution-based feature extractor; temporal features via 1D conv w/ GroupNorm+GELU; **frequency features** (FFT mag/phase → MLP) are added; 3D electrode coordinates encoded via **NeRF-style sinusoids → MLP** (positional enc). |
| |
| **PanLUNA Channel-Modality-Unification Module:** **Q learned queries** cross-attend to **channel-wise patch features** from different modalities to produce a **fixed Q×E latent** per patch; FFN + Transformer layers refine the query tokens. Complexity is **O(Q·C)** (linear in channels). |
| |
| **Temporal encoder:** **Patch-wise Transformer** with **RoPE** operates on the latent sequence (length = #patches), **not** on channels×patches, reducing sequence length and cost substantially. |
| |
| **No Modality Specific Encoders:** All modalities are processed within a single shared encoder, embedded only with modality-type embedding. |
| |
| **Pre-training objective:** **Masked-patch reconstruction** with Smooth-L1; decoder uses **channel-indexed queries** to reconstruct masked tokens. **Query specialization loss** encourages diverse query–channel affinities. |
| |
| --- |
| |
| ## 🔧 How to Use |
| |
| We provide `PanLUNA.safetensors`, weights for the PanLUNA model pretrained on the 40,000 hours of heterogenous biosignal data. |
| |
| PanLUNA experiments can be viewed through two Hydra configurations in `BioFoundation/config/experiments`: |
| - **`PanLUNA_finetune.yaml`** → configuration for fine-tuning experiments. |
| - **`PanLUNA_pretrain.yaml`** → configuration for pre-training experiments. |
| |
| --- |
| |
| ## 🔧 Fine-tuning — General Checklist |
| |
| 0. **Install & read data prep**: clone the [BioFoundation repo](https://github.com/pulp-bio/BioFoundation), set up the environment as described there, then open `make_datasets/README.md` for dataset-specific notes (naming, expected folder layout, and common pitfalls). |
| 1. **Point to weights**: set `pretrained_safetensors_path: /path/to/PanLUNA.safetensors` in the experiment YAML. |
| 2. **Preprocess data**: acquire fine-tuning dataset and follow preprocessing protocol (see guide in `/make_datasets/README.md`) to generate `train/test/val.h5` files. |
| 3. **Update data module of `PanLUNA_finetune.yaml` config**: |
| - **Unimodal Experiments:** |
| - Change `override /data_module` to `finetune_data_module_unimodal_PanLUNA`. |
| - Check out `config/data_module/dataset_types.yaml` to learn about parameters for each dataset. |
| - Keep `/data_module: _target_` to `datasets.finetuning_unimodal_datasets_PanLUNA.FinetuningUnimodal_Dataset`. |
| - **HDF5 file location** → change `/data_module:hdf5_file` for `train`, `test`, and `val` with the path to the corresponding HDF5 data split file. |
| - Change `channels`, `location_fn` and `sensor_type` for the intended dataset. |
| - **Multimodal Experiments:** |
| - Change `override /data_module` to `finetune_data_module_multimodal_PanLUNA`. |
| - Keep `/data_module: _target_` to `datasets.finetuning_multimodal_datasets_PanLUNA.FinetuningMultimodal_Dataset`. |
| - **HDF5 file location** → change `/data_module:hdf5_file` for `train`, `test`, and `val` with the path to the corresponding HDF5 data split file. |
| - Follow the example and instructions in the `config/data_module/finetune_data_module_multimodal_PanLUNA.yaml` to adjust channels and slicing for each multimodal dataset. |
| 4. **Task settings**: |
| - **Fine-tuning strategy:** override `finetuning: mode` with `full`, `freeze_encoder` or `lora`. Use `full` if you want to fully update PanLUNA's weights after pre-training. For training only classification head use `frozen_encoder`. For Low-Rank Adapation on selected layers use `lora`. |
| - **Classification type**: set `classification_type` (`bc`, `mcc` or `mlp`) and `model.num_classes` to match your downstream task. Change `model.num_classes` to describe the number of features in the output. |
| - Configuration file includes further `#CHANGEME` tags and instructions for a working example. |
| 5. **Env vars**: export `DATA_PATH` (dataset root) and `CHECKPOINT_DIR` (artifacts). |
| 6. **Trainer/optimizer**: adjust `gpus/devices`, `batch_size`, `max_epochs`, LR/scheduler if needed. |
| 7. **I/O**: set `io.base_output_path` and confirm `io.checkpoint_dirpath` exists. |
|
|
| To launch fine-tuning (Hydra): |
|
|
| ```bash |
| python -u run_train.py +experiment=PanLUNA_finetune |
| ``` |
|
|
| --- |
|
|
| ## ⚖️ Responsible AI, Risks & Biases |
|
|
| - **Clinical safety:** research-only; human oversight required. |
| - **Bias & drift:** montage/device/population differences can induce shifts; validate and monitor. |
| - **Artifacts & rare events:** robustness varies; use QC and task-appropriate preprocessing. |
|
|
| --- |
|
|
| ## 🔗 Sources |
|
|
| - **Code:** https://github.com/pulp-bio/BioFoundation |
| - **Paper:** PanLUNA: An Efficient and Robust Query-Unified Multimodal Model for Edge Biosignal Intelligence (arxiv:2604.04297) |
|
|
| --- |
|
|
| ## 📜 Citation |
|
|
| If you use PanLUNA, please cite: |
|
|
| ```bibtex |
| @misc{zelic2026panluna, |
| title={PanLUNA: An Efficient and Robust Query-Unified Multimodal Model for Edge Biosignal Intelligence}, |
| author={Marija Zelic and Anna Tegon and Yawei Li and Thorir Mar Ingolfsson}, |
| year={2026}, |
| eprint={2604.04297}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.AI}, |
| url={https://arxiv.org/abs/2604.04297}, |
| } |
| ``` |
|
|
| --- |
|
|
| ## 🛠️ Maintenance & Contact |
|
|
| - **Issues & support:** please open a GitHub issue in the BioFoundation repository. |
|
|
| --- |
| --- |
|
|
| ## 🔗 Related Models |
|
|
| - **[LUNA](https://huggingface.co/PulpBio/LUNA)** — Transformer-based topology-agnostic EEG foundation model (NeurIPS 2025). Source of the channel-unification cross-attention module that LuMamba reuses. |
| - **[FEMBA](https://huggingface.co/PulpBio/FEMBA)** — Bidirectional Mamba foundation model for EEG. Source of the linear-complexity temporal backbone that LuMamba reuses. |
| - **[TinyMyo](https://huggingface.co/PulpBio/TinyMyo)** — Tiny foundation model for flexible EMG signal processing at the edge. |
| - **[LuMamba](https://huggingface.co/PulpBio/LuMamba)** - Extends LUNA's channel-unification to a linear-complexity Mamba backbone, with systematic analysis of LeJEPA for biosignal SSL. |
| |
| ## 🗒️ Changelog |
|
|
| - **v1.0:** Initial release of PanLUNA model card with pretrained checkpoint and instructions. |
|
|
