Card: list the training data (datasets:, base_model:) and add a Training data section

6d736b4 verified 1 day ago

7.56 kB

---
license: mit
library_name: braindecode
tags:
- braindecode
- pytorch
- safetensors
- meg
- bbl
- self-supervised
- foundation-model
datasets:
- CamCAN
---
# BBL (pretrained, CamCAN)

Weights of the MEG encoder of *The Brain's Bitter Lesson* (Jayalath et al., ICML 2025),
pretrained on CamCAN with the band, phase and amplitude pretext tasks, for
[`braindecode.models.BBL`](https://braindecode.org/stable/generated/braindecode.models.BBL.html),
converted from the authors' checkpoint to braindecode's parameter names. The class is
added by [braindecode#1553](https://github.com/braindecode/braindecode/pull/1553)
(braindecode 2.1).

```python
from braindecode.models import BBL
model = BBL.from_pretrained("braindecode/bbl-pretrained", n_outputs=2)
features = model(x, return_features=True)["features"]  # (batch, 528)
```

The linear head (`final_layer`, `n_outputs=2` in `config.json`) is not part of the
release: it is PyTorch's default init under `torch.manual_seed(0)`. Passing `n_outputs` to
`from_pretrained` replaces it. Fine-tune or linear-probe before use.

<!-- braindecode:training-data -->
## Training data

The data that shaped these weights. The ids are the `datasets:` field of this card,
which braindecode reads into `model.card_data.datasets` when you call
`from_pretrained` (MOABB class names for MOABB datasets, OpenNeuro / NEMAR
accessions, Hub ids otherwise). `nm`/`on` ids are datasets on [NEMAR](https://nemar.org)
(`on` = NEMAR's copy of the OpenNeuro dataset with the same number). Rows marked
`private:` are data the authors did not release; they are not in `datasets:`.

| `datasets:` id | Dataset | Stage | Modality |
|---|---|---|---|
| `CamCAN` | [Cambridge Centre for Ageing and Neuroscience (Cam-CAN) MEG data, Shafto et al. 2014 / Taylor et al. 2017](https://cam-can.mrc-cbu.cam.ac.uk/dataset/) | pretraining | MEG |

- self-supervised band/phase/amplitude pretext tasks; 583 Cam-CAN subject rows trained (train split; rows with non-zero Adam state)
- Sources: https://github.com/neural-processing-lab/the-brains-bitter-lesson/blob/3965a2be11677d9c2b9cecb88109a0af6bf19d1c/README.md; checkpoint hyper_parameters (inspected 2026-11-06, meg-ports-20261106/bbl/weights/CHECKPOINT.md and evidence/parity.log); arXiv:2406.04328 Sec. 4 Datasets.
## Intended input
- MEG, `(batch, n_chans, 125)`: CamCAN's 306 Elekta `MEG*` channels in FIF order (the
  projector is indexed by position; inputs with fewer channels are zero-padded to 306, as
  the reference collator does). `dataset_index` defaults to slot 0 (CamCAN).
- Preprocessed as in the authors' `data/preprocess.py` and `data/utils.py`: band-pass
  0.1-125 Hz with 50 and 100 Hz notches, resampled to 250 Hz, 0.5 s windows (125 samples);
  each channel scaled by the median and interquartile range fitted on its recording,
  clipped to `[-5, 5]`, and the mean of the first 0.5 s subtracted. `forward` only does
  the zero-padding.
- `subject_index` row `d * 1024 + s` for subject `s` of dataset slot `d`. Only CamCAN
  subject rows are trained; the `trained_subjects` buffer flags those 583 rows and drives
  the mean-subject fallback used when no `subject_index` is given.

## Source

- Paper: D. Jayalath, G. Landau, B. Shillingford, M. Woolrich, O. Parker Jones. *The
  Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning.* ICML 2025.
  [arXiv:2406.04328](https://arxiv.org/abs/2406.04328)
- Code: [neural-processing-lab/the-brains-bitter-lesson @ `3965a2be11677d9c2b9cecb88109a0af6bf19d1c`](https://github.com/neural-processing-lab/the-brains-bitter-lesson/tree/3965a2be11677d9c2b9cecb88109a0af6bf19d1c) (MIT)
- Weights: the checkpoint "pretrained with CamCAN" linked from the README of that
  repository ("Pre-trained checkpoints"), hosted on Google Drive (file id
  `1cYdKDYoIupjoMfq5cEWgB-W74XhztWEw`), downloaded 2026-11-06: 509,226,678 bytes, sha256
  `94a35ca9bb8ca00941ce355d1b5cd97e4984b689209317a4eb18a3560a8e940b`. A PyTorch Lightning
  2.5.0.post0 `Pretrainer` checkpoint (original name
  `camcan_preproc_full_cont-best-epoch=02-val_loss=1.26.ckpt`; epoch 2, global step 60507;
  `input_dim=306, model_dim=512, subject_embedding_dim=16, projector_dim=1024`, tasks band,
  phase, amp).

## Conversion

`convert.py` (in this repository) was run with braindecode at
`bruAristimunha/braindecode@388ac6cb96361c2df3d5b530172ed6662b9e6a65` (branch `feat/meg-bbl`, PR #1553):

```
python convert.py bbl.ckpt <out_dir>
```

It keeps the `backbone.*` tensors, renames them to braindecode's keys (the old-style
`weight_g`/`weight_v` weight norm maps to `parametrizations.weight.original0/1`, which
computes the same `torch._weight_norm`), fills the `trained_subjects` buffer with the
subject rows the optimizer updated (non-zero Adam second moment; the original keeps that set
in a Python attribute that is not saved), loads everything into
`BBL(n_chans=306, n_outputs=2, n_times=125, sfreq=250.0)` and calls `save_pretrained`.
Tensor values are copied unchanged.

Dropped (training only): the SSL projector, the band/amplitude/phase pretext classifiers,
and the Lightning/optimizer state.

| file | sha256 |
|---|---|
| `model.safetensors` | `3f5022d2564762917d5b9d0df35060b8f0dd4f09128cc9cb076d5fd6aed193c3` |
| `pytorch_model.bin` | `7d0da29ad7d4404917bfa70cf6b4b3c692dcbc10501eec9ec100fb5c785679bb` |
| `config.json` | `71ba8c6efac12d7595f0723522c090400017a9b41255a8f620847a1313699f7f` |
| `convert.py` | `c3388e146017e55aa439dfba995ea8e5a06efe3e4d77acd3f97b98112b950581` |

## Verification

`BBL.from_pretrained(<this repo>, strict=True)` (loaded tensors equal `convert.py` applied
to the original checkpoint) against the original `Backbone` with the same weights and
inputs, CPU float32, eval mode, 4 windows of 0.5 s: features, logits, loss, input
gradients and the gradients of all 55 backbone parameters and of the head are equal
(max-abs difference 0.0) with 306 sensors; with 204 sensors (zero-padded to 306) the
features are equal. The mean-subject fallback (batch 1) differs by 3.7e-8 in the
16 subject-embedding features (the mean is taken in another order); the encoder part is
equal.

## License and provenance

`license: mit` follows the repository that distributes the checkpoint. The code of
[the-brains-bitter-lesson](https://github.com/neural-processing-lab/the-brains-bitter-lesson)
is MIT ([licence](https://github.com/neural-processing-lab/the-brains-bitter-lesson/blob/main/LICENSE),
Copyright (c) 2025 Dulhan Jayalath). The weights are distributed by the authors through
that repository's README (Google Drive link) without a separate weight licence. They are
re-hosted here unmodified in converted form (renamed keys, training-only parts dropped),
with attribution to the authors. Authors or rights holders who want this copy changed or
removed: please open an issue at https://github.com/braindecode/braindecode/issues and it
will be taken down.

## Citation

```bibtex
@inproceedings{jayalath2025bbl,
  title     = {{The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning}},
  author    = {Jayalath, Dulhan and Landau, Gilad and Shillingford, Brendan and Woolrich, Mark and Parker Jones, Oiwi},
  booktitle = {Forty-second International Conference on Machine Learning (ICML)},
  year      = {2025},
  organization = {PMLR}
}
@article{aristimunha2025braindecode,
  title   = {Braindecode: a deep learning library for raw electrophysiological data},
  author  = {Aristimunha, Bruno and others},
  journal = {Zenodo},
  year    = {2025},
  doi     = {10.5281/zenodo.17699192},
}
```

	---
	license: mit
	library_name: braindecode
	tags:
	- braindecode
	- pytorch
	- safetensors
	- meg
	- bbl
	- self-supervised
	- foundation-model
	datasets:
	- CamCAN
	---

	# BBL (pretrained, CamCAN)

	Weights of the MEG encoder of The Brain's Bitter Lesson (Jayalath et al., ICML 2025),
	pretrained on CamCAN with the band, phase and amplitude pretext tasks, for
	[`braindecode.models.BBL`](https://braindecode.org/stable/generated/braindecode.models.BBL.html),
	converted from the authors' checkpoint to braindecode's parameter names. The class is
	added by [braindecode#1553](https://github.com/braindecode/braindecode/pull/1553)
	(braindecode 2.1).

	```python
	from braindecode.models import BBL

	model = BBL.from_pretrained("braindecode/bbl-pretrained", n_outputs=2)
	features = model(x, return_features=True)["features"] # (batch, 528)
	```

	The linear head (`final_layer`, `n_outputs=2` in `config.json`) is not part of the
	release: it is PyTorch's default init under `torch.manual_seed(0)`. Passing `n_outputs` to
	`from_pretrained` replaces it. Fine-tune or linear-probe before use.

	<!-- braindecode:training-data -->
	## Training data

	The data that shaped these weights. The ids are the `datasets:` field of this card,
	which braindecode reads into `model.card_data.datasets` when you call
	`from_pretrained` (MOABB class names for MOABB datasets, OpenNeuro / NEMAR
	accessions, Hub ids otherwise). `nm`/`on` ids are datasets on [NEMAR](https://nemar.org)
	(`on` = NEMAR's copy of the OpenNeuro dataset with the same number). Rows marked
	`private:` are data the authors did not release; they are not in `datasets:`.

	\| `datasets:` id \| Dataset \| Stage \| Modality \|
	\|---\|---\|---\|---\|
	\| `CamCAN` \| [Cambridge Centre for Ageing and Neuroscience (Cam-CAN) MEG data, Shafto et al. 2014 / Taylor et al. 2017](https://cam-can.mrc-cbu.cam.ac.uk/dataset/) \| pretraining \| MEG \|

	- self-supervised band/phase/amplitude pretext tasks; 583 Cam-CAN subject rows trained (train split; rows with non-zero Adam state)
	- Sources: https://github.com/neural-processing-lab/the-brains-bitter-lesson/blob/3965a2be11677d9c2b9cecb88109a0af6bf19d1c/README.md; checkpoint hyper_parameters (inspected 2026-11-06, meg-ports-20261106/bbl/weights/CHECKPOINT.md and evidence/parity.log); arXiv:2406.04328 Sec. 4 Datasets.


	## Intended input

	- MEG, `(batch, n_chans, 125)`: CamCAN's 306 Elekta `MEG*` channels in FIF order (the
	projector is indexed by position; inputs with fewer channels are zero-padded to 306, as
	the reference collator does). `dataset_index` defaults to slot 0 (CamCAN).
	- Preprocessed as in the authors' `data/preprocess.py` and `data/utils.py`: band-pass
	0.1-125 Hz with 50 and 100 Hz notches, resampled to 250 Hz, 0.5 s windows (125 samples);
	each channel scaled by the median and interquartile range fitted on its recording,
	clipped to `[-5, 5]`, and the mean of the first 0.5 s subtracted. `forward` only does
	the zero-padding.
	- `subject_index` row `d * 1024 + s` for subject `s` of dataset slot `d`. Only CamCAN
	subject rows are trained; the `trained_subjects` buffer flags those 583 rows and drives
	the mean-subject fallback used when no `subject_index` is given.

	## Source

	- Paper: D. Jayalath, G. Landau, B. Shillingford, M. Woolrich, O. Parker Jones. *The
	Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning.* ICML 2025.
	[arXiv:2406.04328](https://arxiv.org/abs/2406.04328)
	- Code: [neural-processing-lab/the-brains-bitter-lesson @ `3965a2be11677d9c2b9cecb88109a0af6bf19d1c`](https://github.com/neural-processing-lab/the-brains-bitter-lesson/tree/3965a2be11677d9c2b9cecb88109a0af6bf19d1c) (MIT)
	- Weights: the checkpoint "pretrained with CamCAN" linked from the README of that
	repository ("Pre-trained checkpoints"), hosted on Google Drive (file id
	`1cYdKDYoIupjoMfq5cEWgB-W74XhztWEw`), downloaded 2026-11-06: 509,226,678 bytes, sha256
	`94a35ca9bb8ca00941ce355d1b5cd97e4984b689209317a4eb18a3560a8e940b`. A PyTorch Lightning
	2.5.0.post0 `Pretrainer` checkpoint (original name
	`camcan_preproc_full_cont-best-epoch=02-val_loss=1.26.ckpt`; epoch 2, global step 60507;
	`input_dim=306, model_dim=512, subject_embedding_dim=16, projector_dim=1024`, tasks band,
	phase, amp).

	## Conversion

	`convert.py` (in this repository) was run with braindecode at
	`bruAristimunha/braindecode@388ac6cb96361c2df3d5b530172ed6662b9e6a65` (branch `feat/meg-bbl`, PR #1553):

	```
	python convert.py bbl.ckpt <out_dir>
	```

	It keeps the `backbone.*` tensors, renames them to braindecode's keys (the old-style
	`weight_g`/`weight_v` weight norm maps to `parametrizations.weight.original0/1`, which
	computes the same `torch._weight_norm`), fills the `trained_subjects` buffer with the
	subject rows the optimizer updated (non-zero Adam second moment; the original keeps that set
	in a Python attribute that is not saved), loads everything into
	`BBL(n_chans=306, n_outputs=2, n_times=125, sfreq=250.0)` and calls `save_pretrained`.
	Tensor values are copied unchanged.

	Dropped (training only): the SSL projector, the band/amplitude/phase pretext classifiers,
	and the Lightning/optimizer state.

	\| file \| sha256 \|
	\|---\|---\|
	\| `model.safetensors` \| `3f5022d2564762917d5b9d0df35060b8f0dd4f09128cc9cb076d5fd6aed193c3` \|
	\| `pytorch_model.bin` \| `7d0da29ad7d4404917bfa70cf6b4b3c692dcbc10501eec9ec100fb5c785679bb` \|
	\| `config.json` \| `71ba8c6efac12d7595f0723522c090400017a9b41255a8f620847a1313699f7f` \|
	\| `convert.py` \| `c3388e146017e55aa439dfba995ea8e5a06efe3e4d77acd3f97b98112b950581` \|

	## Verification

	`BBL.from_pretrained(<this repo>, strict=True)` (loaded tensors equal `convert.py` applied
	to the original checkpoint) against the original `Backbone` with the same weights and
	inputs, CPU float32, eval mode, 4 windows of 0.5 s: features, logits, loss, input
	gradients and the gradients of all 55 backbone parameters and of the head are equal
	(max-abs difference 0.0) with 306 sensors; with 204 sensors (zero-padded to 306) the
	features are equal. The mean-subject fallback (batch 1) differs by 3.7e-8 in the
	16 subject-embedding features (the mean is taken in another order); the encoder part is
	equal.

	## License and provenance

	`license: mit` follows the repository that distributes the checkpoint. The code of
	[the-brains-bitter-lesson](https://github.com/neural-processing-lab/the-brains-bitter-lesson)
	is MIT ([licence](https://github.com/neural-processing-lab/the-brains-bitter-lesson/blob/main/LICENSE),
	Copyright (c) 2025 Dulhan Jayalath). The weights are distributed by the authors through
	that repository's README (Google Drive link) without a separate weight licence. They are
	re-hosted here unmodified in converted form (renamed keys, training-only parts dropped),
	with attribution to the authors. Authors or rights holders who want this copy changed or
	removed: please open an issue at https://github.com/braindecode/braindecode/issues and it
	will be taken down.

	## Citation

	```bibtex
	@inproceedings{jayalath2025bbl,
	title = {{The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning}},
	author = {Jayalath, Dulhan and Landau, Gilad and Shillingford, Brendan and Woolrich, Mark and Parker Jones, Oiwi},
	booktitle = {Forty-second International Conference on Machine Learning (ICML)},
	year = {2025},
	organization = {PMLR}
	}

	@article{aristimunha2025braindecode,
	title = {Braindecode: a deep learning library for raw electrophysiological data},
	author = {Aristimunha, Bruno and others},
	journal = {Zenodo},
	year = {2025},
	doi = {10.5281/zenodo.17699192},
	}
	```