README.md

# 🧪 Lab Physiological Signal Dataset Hub

> **Organization**: `MultiModalSensing` on HuggingFace  
> **Maintainer**: `@bc121381`  
> **Last Updated**: 2026-04  
> **Contact**: m.r.cui@utwente.nl

This repository defines the **unified data standard** for all physiological signal datasets managed by our lab. Every dataset uploaded to the organization must follow the schema and conventions described here.

---

## 📁 Repository Index

| Dataset | Modality | Sampling Rate | Task | Owner | Access |
|---------|----------|---------------|------|-------|--------|
| e.g. [mimic-waveforms](https://huggingface.co/datasets/your-lab-org/mimic-waveforms) | ECG / ABP / SpO₂ | 125 Hz | Clinical monitoring | @member-name | 🔒 Gated |
| e.g. [pamap2-activity](https://huggingface.co/datasets/your-lab-org/pamap2-activity) | IMU / HR / Temp | 100 Hz | Activity recognition | @member-name | ✅ Public |
| *(add your dataset here after you submit to this space)* | | | | | |

> 🔒 **Gated**: Requires PhysioNet credentialing or lab approval.  
> ✅ **Public**: Freely accessible within the organization.

---

## 📐 Unified Schema

All datasets share the following field definitions. Every sample (row) represents **one time-window segment** of a recording.

### Field Specification

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `signal` | `Sequence(Sequence(float32))` | ✅ | Raw signal, shape `[n_channels, n_samples]` |
| `sampling_rate` | `float32` | ✅ | Sampling frequency in Hz |
| `modality` | `string` | ✅ | Signal type, e.g. `"EEG"`, `"ECG+PPG"`, `"IMU"` |
| `channel_names` | `Sequence(string)` | ✅ | List of channel names, e.g. `["ECG_II", "ABP"]` |
| `n_channels` | `int32` | ✅ | Number of channels |
| `n_samples` | `int32` | ✅ | Number of time points per channel |
| `unit` | `string` | ✅ | Physical unit(s), e.g. `"uV"`, `"mV+mmHg+%"` |
| `subject_id` | `string` | ✅ | Anonymized subject identifier |
| `session_id` | `string` | ✅ | Recording session identifier |
| `trial_id` | `string` | ✅ | Segment identifier within session |
| `start_time_sec` | `float64` | ✅ | Segment start time relative to recording onset (seconds) |
| `label` | `string` | ✅ | Task label (see per-dataset definition) |
| `label_scheme` | `string` | ✅ | Label semantics, e.g. `"sleep_stage"`, `"physical_activity"` |
| `extra` | `string` (JSON) | ✅ | Additional metadata as JSON string (use `{}` if none) |

### HuggingFace Feature Definition

Copy this into your `convert.py` as the canonical feature spec:

```python
from datasets import Features, Sequence, Value

LAB_FEATURES = Features({
    "signal":          Sequence(Sequence(Value("float32"))),
    "sampling_rate":   Value("float32"),
    "modality":        Value("string"),
    "channel_names":   Sequence(Value("string")),
    "n_channels":      Value("int32"),
    "n_samples":       Value("int32"),
    "unit":            Value("string"),
    "subject_id":      Value("string"),
    "session_id":      Value("string"),
    "trial_id":        Value("string"),
    "start_time_sec":  Value("float64"),
    "label":           Value("string"),
    "label_scheme":    Value("string"),
    "extra":           Value("string"),
})
```

---

## 🔑 Key Design Principles

### 1. Original Sampling Rate is Always Preserved

Do **not** resample during dataset creation. Store signals at their native sampling rate. Resampling is done at load time, per experiment requirements.

```
✅ Store at native rate → resample in DataLoader
❌ Do not permanently downsample/upsample the stored data
```

### 2. Channel Names, Not Positions

Always use `channel_names` for indexing. Never assume a channel exists at a fixed index position across datasets.

```python
# ✅ Correct
idx = sample["channel_names"].index("ECG_II")
ecg = np.array(sample["signal"])[idx]

# ❌ Wrong
ecg = np.array(sample["signal"])[0]  # position may differ across datasets
```

### 3. Label Semantics Are Dataset-Specific

The same `label` value (e.g., `"1"`) may mean different things across datasets. Always use `label` together with `label_scheme`.

```python
{"label": "1", "label_scheme": "sleep_stage"}     # → N1 sleep
{"label": "1", "label_scheme": "physical_activity"} # → sitting
{"label": "1", "label_scheme": "arrhythmia"}        # → AFib
```

### 4. Extend with `extra`, Don't Break the Schema

Dataset-specific metadata that doesn't fit the common fields goes into the `extra` JSON string field. This keeps the schema stable while allowing flexibility.

```python
import json

extra = json.dumps({
    "age": 67,
    "gender": "M",
    "icu_type": "MICU",
    "filter_applied": "0.5-40Hz bandpass",
})
```

---

## 📂 Directory Structure

Each dataset repository follows this structure:

```
MultiModalSensing/<dataset-name>/
├── train/
│   ├── data-00000-of-00002.parquet
│   └── data-00001-of-00002.parquet
├── validation/
│   └── data-00000-of-00001.parquet
├── test/
│   └── data-00000-of-00001.parquet
├── README.md             ← Dataset Card (required)
└── convert.py            ← Conversion script (required, for reproducibility)
```

---

## 🚀 How to Load Datasets

### Basic Loading

```python
from datasets import load_dataset
import numpy as np

# Load any lab dataset with the same interface
ds = load_dataset("your-lab-org/pamap2-activity", split="train")

sample = ds[0]
signal = np.array(sample["signal"])          # shape: (n_channels, n_samples)
sr     = sample["sampling_rate"]             # e.g. 100.0
chs    = sample["channel_names"]             # e.g. ["acc_x", "acc_y", ...]
label  = sample["label"]                     # e.g. "running"
```

### Using the Shared Loader (Recommended)

Install the lab utility package and use the unified loader, which handles resampling and channel selection automatically:

```python
# pip install git+https://github.com/MultiModalSensing/lab-utils.git
from lab_utils import load_lab_dataset

# Load with optional resampling and channel selection
ds = load_lab_dataset(
    name       = "pamap2-activity",
    split      = "train",
    channels   = ["acc_x", "acc_y", "acc_z"],     # select channels by name
)

# Returns {"x": np.ndarray, "y": str, "sr": float}
sample = ds[0]
print(sample["x"].shape)   # (3, n_samples)
```

### Accessing Gated Datasets (e.g., MIMIC)

```bash
# Step 1: Log in to HuggingFace
huggingface-cli login

# Step 2: Request access via the dataset page (one-time)
# https://huggingface.co/datasets/your-lab-org/mimic-waveforms
```

```python
# Step 3: Load as normal (authentication handled automatically)
ds = load_dataset("your-lab-org/mimic-waveforms", split="train")
```

---

## ➕ How to Contribute a New Dataset

Follow these 4 steps to add your dataset to the organization.

### Step 1: Write `convert.py`

Use the template below and adapt to your raw data format:

```python
"""
convert.py — Template for lab dataset conversion
Replace all <PLACEHOLDERS> with your dataset's specifics.
"""

import json
import numpy as np
import pandas as pd
from datasets import Dataset
from lab_schema import LAB_FEATURES   # import from lab-utils

DATASET_NAME   = "<your-dataset-name>"
MODALITY       = "<EEG|ECG|IMU|PPG|...>"
CHANNEL_NAMES  = ["<ch1>", "<ch2>"]    # full list of channels
SAMPLING_RATE  = <Hz>                  # native sampling rate (float)
UNIT           = "<uV|mV|m/s2|...>"
LABEL_SCHEME   = "<sleep_stage|physical_activity|arrhythmia|...>"
WINDOW_SEC     = 5.0                   # window length in seconds
OVERLAP_SEC    = 2.5                   # overlap between windows

LABEL_MAP = {
    <raw_label_value>: "<semantic_label>",
    # e.g. 1: "walking", 2: "running", ...
}

def segment(signal_2d, sr, window_sec, overlap_sec):
    """Sliding window segmentation. signal_2d: (n_channels, n_total_samples)"""
    win   = int(window_sec  * sr)
    step  = int((window_sec - overlap_sec) * sr)
    segs  = []
    for start in range(0, signal_2d.shape[1] - win + 1, step):
        segs.append(signal_2d[:, start:start + win])
    return segs  # list of (n_channels, win) arrays

def convert_subject(raw_path, subject_id):
    """Load one subject's raw file and return a list of sample dicts."""
    # ── load raw data ──────────────────────────────────────────────
    df = pd.read_csv(raw_path)   # adapt as needed

    signal_2d = df[CHANNEL_NAMES].values.T.astype(np.float32)
    labels_ts = df["label"].values   # one label per time step

    segments = segment(signal_2d, SAMPLING_RATE, WINDOW_SEC, OVERLAP_SEC)
    win = int(WINDOW_SEC * SAMPLING_RATE)

    samples = []
    for i, seg in enumerate(segments):
        start_sample = i * int((WINDOW_SEC - OVERLAP_SEC) * SAMPLING_RATE)
        window_labels = labels_ts[start_sample : start_sample + win]
        majority_label = LABEL_MAP.get(
            pd.Series(window_labels).mode()[0], "unknown"
        )

        samples.append({
            "signal":          seg.tolist(),
            "sampling_rate":   float(SAMPLING_RATE),
            "modality":        MODALITY,
            "channel_names":   CHANNEL_NAMES,
            "n_channels":      seg.shape[0],
            "n_samples":       seg.shape[1],
            "unit":            UNIT,
            "subject_id":      str(subject_id),
            "session_id":      "session_1",
            "trial_id":        f"segment_{i:06d}",
            "start_time_sec":  round(start_sample / SAMPLING_RATE, 6),
            "label":           majority_label,
            "label_scheme":    LABEL_SCHEME,
            "extra":           json.dumps({}),   # add subject metadata here
        })
    return samples

if __name__ == "__main__":
    all_samples = []
    for subject_id in range(1, 10):   # adapt to your subject list
        all_samples += convert_subject(f"data/subject{subject_id}.csv", subject_id)

    ds = Dataset.from_list(all_samples, features=LAB_FEATURES)
    # Optional: train/val/test split
    ds = ds.train_test_split(test_size=0.2, seed=42)
    ds.push_to_hub(f"your-lab-org/{DATASET_NAME}")
    print(f"✅ Uploaded {len(all_samples)} samples to your-lab-org/{DATASET_NAME}")
```

### Step 2: Create a Dataset Card (`README.md`)

Every dataset repo must have a `README.md` using the template in the section below.

### Step 3: Push to Hub

```bash
huggingface-cli login

# Public dataset
python convert.py

# Private / gated dataset (e.g., MIMIC, clinical data)
python convert.py --private   # or set via HF web UI after upload
```

### Step 4: Register in the Index

Open a PR to this README and add your dataset to the **Repository Index** table at the top.

---

## 📋 Dataset Card Template

Copy this into your dataset's `README.md`:

```markdown
---
dataset_name: <your-dataset-name>
modality: <EEG|ECG|IMU|PPG|...>
sampling_rate: <Hz>
n_channels: <N>
task: <sleep_staging|activity_recognition|arrhythmia_detection|...>
label_scheme: <scheme_name>
n_subjects: <N>
version: 1.0.0
license: <cc-by-4.0|physionet-1.5|internal>
---

## Overview

Brief description of the dataset: source, purpose, key characteristics.

## Source

- **Original dataset**: [Name](URL)
- **Paper**: citation
- **Access**: Public / Gated / Internal

## Modality & Channels

| Index | Channel Name | Unit | Description |
|-------|-------------|------|-------------|
| 0     | `<ch_name>` | uV   | <description> |

## Label Definition

| label | meaning |
|-------|---------|
| `"0"` | <class 0> |
| `"1"` | <class 1> |

**label_scheme**: `<scheme_name>`

## Splits

| Split      | Subjects | Segments |
|------------|----------|----------|
| train      | X        | X        |
| validation | X        | X        |
| test       | X        | X        |

## Preprocessing

Describe all transformations applied before storage:
- Filtering: (e.g., 0.5–40 Hz bandpass, 50 Hz notch)
- Normalization: (e.g., z-score per channel per recording)
- Windowing: X-second windows, Y-second overlap
- Resampling: original rate → stored rate

> ⚠️ Raw data is stored at its **native sampling rate**. Resample in your DataLoader as needed.

## Known Issues / Limitations

- List any known data quality issues
- Missing subjects, artifact channels, etc.

## Citation

\`\`\`bibtex
@dataset{...}
\`\`\`
```

---

## ⚙️ Naming Conventions

| Entity | Convention | Example |
|--------|-----------|---------|
| Dataset repo name | `<source>-<modality>` (lowercase, hyphen) | `mimic-waveforms`, `pamap2-activity` |
| `subject_id` | `subject<NNN>` or original ID string | `subject101`, `p10032` |
| `session_id` | `session_<N>` or protocol name | `session_1`, `protocol_run2` |
| `trial_id` | `segment_<NNNNNN>` (zero-padded) | `segment_000042` |
| `modality` | Uppercase, `+` separator for multi-modal | `EEG`, `ECG+PPG`, `IMU+HR` |
| `label_scheme` | lowercase, underscore | `sleep_stage`, `physical_activity` |

---

## 🔐 Access Control Guidelines

| Data Type | HuggingFace Setting | Notes |
|-----------|---------------------|-------|
| Public benchmark (PAMAP2, etc.) | `public` | Fine to share openly |
| PhysioNet data (MIMIC, etc.) | `gated` | Require PhysioNet credentialing |
| Proprietary / in-house data | `private` | Lab members only; get PI approval |
| De-identified clinical data | `private` or `gated` | Check IRB terms before uploading |

> ⚠️ **Never upload identifiable patient data.** When in doubt, contact the lab PI before uploading.

---

## 🛠️ Common Recipes

### Resample a loaded dataset to a target frequency

```python
import scipy.signal as ss
import numpy as np

def resample_sample(sample, target_sr):
    sig = np.array(sample["signal"])
    orig_sr = sample["sampling_rate"]
    if orig_sr == target_sr:
        return sample
    n_target = int(sig.shape[-1] * target_sr / orig_sr)
    sig_resampled = ss.resample(sig, n_target, axis=-1)
    return {**sample, "signal": sig_resampled.tolist(),
            "sampling_rate": target_sr, "n_samples": n_target}

ds = ds.map(lambda x: resample_sample(x, target_sr=100))
```

### Select specific channels by name

```python
def select_channels(sample, channel_list):
    chs = sample["channel_names"]
    idx = [chs.index(c) for c in channel_list]
    sig = np.array(sample["signal"])[idx]
    return {**sample, "signal": sig.tolist(),
            "channel_names": channel_list, "n_channels": len(channel_list)}

ds = ds.map(lambda x: select_channels(x, ["acc_x", "acc_y", "acc_z"]))
```

### Parse `extra` metadata

```python
import json

extras = [json.loads(s["extra"]) for s in ds]
ages   = [e.get("age") for e in extras]
```

---

## 📬 Questions & Contributions

- **Bug / schema issue**: Open an issue in `MultiModalSensing/lab-utils`
- **New dataset**: Follow the contribution steps above and open a PR to this README
- **Access request**: Email `m.r.cui@utwente.nl` with your HuggingFace username