File size: 5,674 Bytes
b0c3041 cfffb30 b0c3041 cfffb30 b0c3041 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | ---
license: mit
language:
- en
tags:
- biosignals
- sleep
- multimodal
- contrastive-learning
- eeg
- ecg
- polysomnography
library_name: open_clip
---
# SleepLM: Natural-Language Intelligence for Human Sleep
[](https://arxiv.org/abs/2602.23605)
[](https://yang-ai-lab.github.io/SleepLM/)
[](LICENSE)
[](#installation)
SleepLM is, to our knowledge, the first sleep-language foundation model family that enables targeted natural language generation from multimodal polysomnography (PSG) while also learning a shared signal–text embedding space for retrieval and open vocabulary sleep understanding. It is trained on the largest paired sleep–text corpus to date, built from five NSRR cohorts totaling 100K+ hours of PSG from 10,000+ individuals.
SleepLM supports controllable, domain-specific generation (brain, cardiac, respiration, somatic) as well as holistic summaries, moving beyond fixed label spaces like sleep stages and events. The model combines contrastive alignment, captioning, and signal reconstruction to preserve physiological fidelity while learning strong cross-modal semantics. Across a broad benchmark, SleepLM enables sleep-text retrieval, zero-shot and few-shot generalization, and robust transfer to unseen concepts.
---
## 📰 News
- **[2026-02-23]** Code released on GitHub!
- **[2026-02-23]** Project website is live!
---
## ✨ What you can do with this repo
- **Targeted caption generation** for 30-second sleep epochs using modality tokens (brain / cardiac / respiration / somatic).
- **Cross modal retrieval** by encoding signals and text into a shared embedding space and computing cosine similarity.
- Run an interactive demo in **`demo.ipynb`**.
---
## 🚀 Quickstart
### 1) Install
```bash
git clone https://github.com/yang-ai-lab/SleepLM
cd SleepLM
pip install -r requirements.txt
```
### 2) Download checkpoint
The model checkpoint is hosted on Hugging Face Hub:
```python
from huggingface_hub import hf_hub_download
checkpoint_path = hf_hub_download(repo_id="yang-ai-lab/SleepLM-Base", filename="model_checkpoint.pt")
```
Or via the CLI:
```bash
huggingface-cli download yang-ai-lab/SleepLM-Base model_checkpoint.pt
```
The checkpoint will be cached locally by `huggingface_hub` and the returned path can be passed directly to `load_checkpoint()` in `demo.ipynb`.
### 3) Prepare your data
Preprocess your PSG recordings into a float32 PyTorch tensor of shape `[N, 10, 1920]`
(N epochs × 10 channels × 1920 samples) following the channel order and signal requirements
in [Using your own signals](#-using-your-own-signals) below.
Save it as a `.pt` file and update the path in `demo.ipynb`.
### 4) Run the demo
Open and run:
- `demo.ipynb`
The notebook includes:
- similarity calculation between signal and text embeddings
- targeted caption generation with per-modality conditioning
---
## 📦 Repository contents
- `demo.ipynb` — interactive inference + visualization
- `requirements.txt` — dependencies
---
## 🧾 Input format
SleepLM expects a **30-second epoch**, sampled at **64 Hz** → **1920 samples/channel**, with **10 channels** in the order below.
### Channel order
| Index | Channel | Description |
|------:|---------|-------------|
| 0 | ECG | Electrocardiogram |
| 1 | ABD | Abdominal respiratory effort |
| 2 | THX | Thoracic respiratory effort |
| 3 | AF | Airflow |
| 4 | EOG_Left | Left eye movement |
| 5 | EOG_Right | Right eye movement |
| 6 | EEG_C3_A2 | Left central EEG |
| 7 | EEG_C4_A1 | Right central EEG |
| 8 | EMG_Chin | Chin muscle tone |
| 9 | POS | Body position |
### Body position encoding (POS channel)
```python
POSITION_ENCODING = {
0: "Right",
1: "Left",
2: "Supine",
3: "Prone",
4: "Upright",
-1: "Other/Unknown", # Use for missing data
}
```
---
## 🧪 Using your own signals
You can generate captions for your own sleep recordings by loading **preprocessed** epochs directly in `demo.ipynb`.
**Signal requirements**
- Resample to **64 Hz**
- Normalize each channel (**z-score**)
- If a channel is missing, **zero-pad** it
- POS must follow the integer encoding above
- Each epoch must be exactly **30 seconds** (**1920 samples @ 64 Hz**)
- Pack epochs into a float32 PyTorch tensor of shape `[N, 10, 1920]`
---
## 🔁 Reproducibility notes
This repo is intentionally lightweight and focuses on **inference**. If you plan to:
- reproduce paper benchmarks,
- train on NSRR cohorts,
- or evaluate cross-cohort generalization,
We are planning to opensource our training pipeline upon the acceptance of the paper.
Note that the training data will not be opensourced due credential issue. If you wish to use the same NSRR datasets, [please apply here](https://sleepdata.org/).
---
## 📝 Citation
If you use SleepLM in your research, please cite the paper:
```bibtex
@article{xu2026sleeplm,
title={SleepLM: Natural-Language Intelligence for Human Sleep},
author={Xu, Zongzhe and Shuai, Zitao and Mozaffari, Eideen and Aysola, Ravi Shankar and Kumar, Rajesh and Yang, Yuzhe},
journal={arXiv preprint arXiv:2602.23605},
year={2026}
}
```
---
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---
## 🙏 Acknowledgments
- Data sources and cohort infrastructure: **NSRR** (if applicable to your paper/training pipeline)
- Model architecture inspiration: OpenCLIP (https://github.com/mlfoundations/open_clip)
|