🧪 Lab Physiological Signal Dataset Hub

Organization: MultiModalSensing on HuggingFace
Maintainer: @bc121381
Last Updated: 2026-04
Contact: m.r.cui@utwente.nl

This repository defines the unified data standard for all physiological signal datasets managed by our lab. Every dataset uploaded to the organization must follow the schema and conventions described here.

---

📁 Repository Index

🔒 Gated: Requires PhysioNet credentialing or lab approval.
✅ Public: Freely accessible within the organization.

---

📐 Unified Schema

All datasets share the following field definitions. Every sample (row) represents one time-window segment of a recording.

Field Specification

HuggingFace Feature Definition

Copy this into your convert.py as the canonical feature spec:

from datasets import Features, Sequence, Value

LAB_FEATURES = Features({
    "signal":          Sequence(Sequence(Value("float32"))),
    "sampling_rate":   Value("float32"),
    "modality":        Value("string"),
    "channel_names":   Sequence(Value("string")),
    "n_channels":      Value("int32"),
    "n_samples":       Value("int32"),
    "unit":            Value("string"),
    "subject_id":      Value("string"),
    "session_id":      Value("string"),
    "trial_id":        Value("string"),
    "start_time_sec":  Value("float64"),
    "label":           Value("string"),
    "label_scheme":    Value("string"),
    "extra":           Value("string"),
})

---

🔑 Key Design Principles

1. Original Sampling Rate is Always Preserved

Do not resample during dataset creation. Store signals at their native sampling rate. Resampling is done at load time, per experiment requirements.

✅ Store at native rate → resample in DataLoader
❌ Do not permanently downsample/upsample the stored data

2. Channel Names, Not Positions

Always use channel_names for indexing. Never assume a channel exists at a fixed index position across datasets.

# ✅ Correct
idx = sample["channel_names"].index("ECG_II")
ecg = np.array(sample["signal"])[idx]

# ❌ Wrong
ecg = np.array(sample["signal"])[0]  # position may differ across datasets

3. Label Semantics Are Dataset-Specific

The same label value (e.g., "1") may mean different things across datasets. Always use label together with label_scheme.

{"label": "1", "label_scheme": "sleep_stage"}     # → N1 sleep
{"label": "1", "label_scheme": "physical_activity"} # → sitting
{"label": "1", "label_scheme": "arrhythmia"}        # → AFib

4. Extend with extra, Don't Break the Schema

Dataset-specific metadata that doesn't fit the common fields goes into the extra JSON string field. This keeps the schema stable while allowing flexibility.

import json

extra = json.dumps({
    "age": 67,
    "gender": "M",
    "icu_type": "MICU",
    "filter_applied": "0.5-40Hz bandpass",
})

---

📂 Directory Structure

Each dataset repository follows this structure:

MultiModalSensing/<dataset-name>/
├── train/
│   ├── data-00000-of-00002.parquet
│   └── data-00001-of-00002.parquet
├── validation/
│   └── data-00000-of-00001.parquet
├── test/
│   └── data-00000-of-00001.parquet
├── README.md             ← Dataset Card (required)
└── convert.py            ← Conversion script (required, for reproducibility)

---

🚀 How to Load Datasets

Basic Loading

from datasets import load_dataset
import numpy as np

# Load any lab dataset with the same interface
ds = load_dataset("your-lab-org/pamap2-activity", split="train")

sample = ds[0]
signal = np.array(sample["signal"])          # shape: (n_channels, n_samples)
sr     = sample["sampling_rate"]             # e.g. 100.0
chs    = sample["channel_names"]             # e.g. ["acc_x", "acc_y", ...]
label  = sample["label"]                     # e.g. "running"

Using the Shared Loader (Recommended)

Install the lab utility package and use the unified loader, which handles resampling and channel selection automatically:

# pip install git+https://github.com/MultiModalSensing/lab-utils.git
from lab_utils import load_lab_dataset

# Load with optional resampling and channel selection
ds = load_lab_dataset(
    name       = "pamap2-activity",
    split      = "train",
    channels   = ["acc_x", "acc_y", "acc_z"],     # select channels by name
)

# Returns {"x": np.ndarray, "y": str, "sr": float}
sample = ds[0]
print(sample["x"].shape)   # (3, n_samples)

Accessing Gated Datasets (e.g., MIMIC)

# Step 1: Log in to HuggingFace
huggingface-cli login

# Step 2: Request access via the dataset page (one-time)
# https://huggingface.co/datasets/your-lab-org/mimic-waveforms

# Step 3: Load as normal (authentication handled automatically)
ds = load_dataset("your-lab-org/mimic-waveforms", split="train")

---

➕ How to Contribute a New Dataset

Follow these 4 steps to add your dataset to the organization.

Step 1: Write convert.py

Use the template below and adapt to your raw data format:

"""
convert.py — Template for lab dataset conversion
Replace all <PLACEHOLDERS> with your dataset's specifics.
"""

import json
import numpy as np
import pandas as pd
from datasets import Dataset
from lab_schema import LAB_FEATURES   # import from lab-utils

DATASET_NAME   = "<your-dataset-name>"
MODALITY       = "<EEG|ECG|IMU|PPG|...>"
CHANNEL_NAMES  = ["<ch1>", "<ch2>"]    # full list of channels
SAMPLING_RATE  = <Hz>                  # native sampling rate (float)
UNIT           = "<uV|mV|m/s2|...>"
LABEL_SCHEME   = "<sleep_stage|physical_activity|arrhythmia|...>"
WINDOW_SEC     = 5.0                   # window length in seconds
OVERLAP_SEC    = 2.5                   # overlap between windows

LABEL_MAP = {
    <raw_label_value>: "<semantic_label>",
    # e.g. 1: "walking", 2: "running", ...
}

def segment(signal_2d, sr, window_sec, overlap_sec):
    """Sliding window segmentation. signal_2d: (n_channels, n_total_samples)"""
    win   = int(window_sec  * sr)
    step  = int((window_sec - overlap_sec) * sr)
    segs  = []
    for start in range(0, signal_2d.shape[1] - win + 1, step):
        segs.append(signal_2d[:, start:start + win])
    return segs  # list of (n_channels, win) arrays

def convert_subject(raw_path, subject_id):
    """Load one subject's raw file and return a list of sample dicts."""
    # ── load raw data ──────────────────────────────────────────────
    df = pd.read_csv(raw_path)   # adapt as needed

    signal_2d = df[CHANNEL_NAMES].values.T.astype(np.float32)
    labels_ts = df["label"].values   # one label per time step

    segments = segment(signal_2d, SAMPLING_RATE, WINDOW_SEC, OVERLAP_SEC)
    win = int(WINDOW_SEC * SAMPLING_RATE)

    samples = []
    for i, seg in enumerate(segments):
        start_sample = i * int((WINDOW_SEC - OVERLAP_SEC) * SAMPLING_RATE)
        window_labels = labels_ts[start_sample : start_sample + win]
        majority_label = LABEL_MAP.get(
            pd.Series(window_labels).mode()[0], "unknown"
        )

        samples.append({
            "signal":          seg.tolist(),
            "sampling_rate":   float(SAMPLING_RATE),
            "modality":        MODALITY,
            "channel_names":   CHANNEL_NAMES,
            "n_channels":      seg.shape[0],
            "n_samples":       seg.shape[1],
            "unit":            UNIT,
            "subject_id":      str(subject_id),
            "session_id":      "session_1",
            "trial_id":        f"segment_{i:06d}",
            "start_time_sec":  round(start_sample / SAMPLING_RATE, 6),
            "label":           majority_label,
            "label_scheme":    LABEL_SCHEME,
            "extra":           json.dumps({}),   # add subject metadata here
        })
    return samples

if __name__ == "__main__":
    all_samples = []
    for subject_id in range(1, 10):   # adapt to your subject list
        all_samples += convert_subject(f"data/subject{subject_id}.csv", subject_id)

    ds = Dataset.from_list(all_samples, features=LAB_FEATURES)
    # Optional: train/val/test split
    ds = ds.train_test_split(test_size=0.2, seed=42)
    ds.push_to_hub(f"your-lab-org/{DATASET_NAME}")
    print(f"✅ Uploaded {len(all_samples)} samples to your-lab-org/{DATASET_NAME}")

Step 2: Create a Dataset Card (README.md)

Every dataset repo must have a README.md using the template in the section below.

Step 3: Push to Hub

huggingface-cli login

# Public dataset
python convert.py

# Private / gated dataset (e.g., MIMIC, clinical data)
python convert.py --private   # or set via HF web UI after upload

Step 4: Register in the Index

Open a PR to this README and add your dataset to the Repository Index table at the top.

---

📋 Dataset Card Template

Copy this into your dataset's README.md:

---
dataset_name: <your-dataset-name>
modality: <EEG|ECG|IMU|PPG|...>
sampling_rate: <Hz>
n_channels: <N>
task: <sleep_staging|activity_recognition|arrhythmia_detection|...>
label_scheme: <scheme_name>
n_subjects: <N>
version: 1.0.0
license: <cc-by-4.0|physionet-1.5|internal>
---

## Overview

Brief description of the dataset: source, purpose, key characteristics.

## Source

- **Original dataset**: [Name](URL)
- **Paper**: citation
- **Access**: Public / Gated / Internal

## Modality & Channels

| Index | Channel Name | Unit | Description |
|-------|-------------|------|-------------|
| 0     | `<ch_name>` | uV   | <description> |

## Label Definition

| label | meaning |
|-------|---------|
| `"0"` | <class 0> |
| `"1"` | <class 1> |

**label_scheme**: `<scheme_name>`

## Splits

| Split      | Subjects | Segments |
|------------|----------|----------|
| train      | X        | X        |
| validation | X        | X        |
| test       | X        | X        |

## Preprocessing

Describe all transformations applied before storage:
- Filtering: (e.g., 0.5–40 Hz bandpass, 50 Hz notch)
- Normalization: (e.g., z-score per channel per recording)
- Windowing: X-second windows, Y-second overlap
- Resampling: original rate → stored rate

> ⚠️ Raw data is stored at its **native sampling rate**. Resample in your DataLoader as needed.

## Known Issues / Limitations

- List any known data quality issues
- Missing subjects, artifact channels, etc.

## Citation

\`\`\`bibtex
@dataset{...}
\`\`\`

---

⚙️ Naming Conventions

---

🔐 Access Control Guidelines

⚠️ Never upload identifiable patient data. When in doubt, contact the lab PI before uploading.

---

🛠️ Common Recipes

Resample a loaded dataset to a target frequency

import scipy.signal as ss
import numpy as np

def resample_sample(sample, target_sr):
    sig = np.array(sample["signal"])
    orig_sr = sample["sampling_rate"]
    if orig_sr == target_sr:
        return sample
    n_target = int(sig.shape[-1] * target_sr / orig_sr)
    sig_resampled = ss.resample(sig, n_target, axis=-1)
    return {**sample, "signal": sig_resampled.tolist(),
            "sampling_rate": target_sr, "n_samples": n_target}

ds = ds.map(lambda x: resample_sample(x, target_sr=100))

Select specific channels by name

def select_channels(sample, channel_list):
    chs = sample["channel_names"]
    idx = [chs.index(c) for c in channel_list]
    sig = np.array(sample["signal"])[idx]
    return {**sample, "signal": sig.tolist(),
            "channel_names": channel_list, "n_channels": len(channel_list)}

ds = ds.map(lambda x: select_channels(x, ["acc_x", "acc_y", "acc_z"]))

Parse extra metadata

import json

extras = [json.loads(s["extra"]) for s in ds]
ages   = [e.get("age") for e in extras]

---

📬 Questions & Contributions

• Bug / schema issue: Open an issue in MultiModalSensing/lab-utils
• New dataset: Follow the contribution steps above and open a PR to this README
• Access request: Email m.r.cui@utwente.nl with your HuggingFace username