Update README.md

README.md

@@ -1,10 +1,463 @@
+# 🧪 Lab Physiological Signal Dataset Hub
+
+> **Organization**: `MultiModalSensing` on HuggingFace  
+> **Maintainer**: `@bc121381`  
+> **Last Updated**: 2026-04  
+> **Contact**: m.r.cui@utwente.nl
+
+This repository defines the **unified data standard** for all physiological signal datasets managed by our lab. Every dataset uploaded to the organization must follow the schema and conventions described here.
+
 ---
-title: README
-emoji: 📈
-colorFrom: purple
-colorTo: green
-sdk: static
-pinned: false
+
+## 📁 Repository Index
+
+| Dataset | Modality | Sampling Rate | Task | Owner | Access |
+|---------|----------|---------------|------|-------|--------|
+| e.g. [mimic-waveforms](https://huggingface.co/datasets/your-lab-org/mimic-waveforms) | ECG / ABP / SpO₂ | 125 Hz | Clinical monitoring | @member-name | 🔒 Gated |
+| e.g. [pamap2-activity](https://huggingface.co/datasets/your-lab-org/pamap2-activity) | IMU / HR / Temp | 100 Hz | Activity recognition | @member-name | ✅ Public |
+| *(add your dataset here after you submit to this space)* | | | | | |
+
+> 🔒 **Gated**: Requires PhysioNet credentialing or lab approval.  
+> ✅ **Public**: Freely accessible within the organization.
+
+---
+
+## 📐 Unified Schema
+
+All datasets share the following field definitions. Every sample (row) represents **one time-window segment** of a recording.
+
+### Field Specification
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `signal` | `Sequence(Sequence(float32))` | ✅ | Raw signal, shape `[n_channels, n_samples]` |
+| `sampling_rate` | `float32` | ✅ | Sampling frequency in Hz |
+| `modality` | `string` | ✅ | Signal type, e.g. `"EEG"`, `"ECG+PPG"`, `"IMU"` |
+| `channel_names` | `Sequence(string)` | ✅ | List of channel names, e.g. `["ECG_II", "ABP"]` |
+| `n_channels` | `int32` | ✅ | Number of channels |
+| `n_samples` | `int32` | ✅ | Number of time points per channel |
+| `unit` | `string` | ✅ | Physical unit(s), e.g. `"uV"`, `"mV+mmHg+%"` |
+| `subject_id` | `string` | ✅ | Anonymized subject identifier |
+| `session_id` | `string` | ✅ | Recording session identifier |
+| `trial_id` | `string` | ✅ | Segment identifier within session |
+| `start_time_sec` | `float64` | ✅ | Segment start time relative to recording onset (seconds) |
+| `label` | `string` | ✅ | Task label (see per-dataset definition) |
+| `label_scheme` | `string` | ✅ | Label semantics, e.g. `"sleep_stage"`, `"physical_activity"` |
+| `extra` | `string` (JSON) | ✅ | Additional metadata as JSON string (use `{}` if none) |
+
+### HuggingFace Feature Definition
+
+Copy this into your `convert.py` as the canonical feature spec:
+
+```python
+from datasets import Features, Sequence, Value
+
+LAB_FEATURES = Features({
+    "signal":          Sequence(Sequence(Value("float32"))),
+    "sampling_rate":   Value("float32"),
+    "modality":        Value("string"),
+    "channel_names":   Sequence(Value("string")),
+    "n_channels":      Value("int32"),
+    "n_samples":       Value("int32"),
+    "unit":            Value("string"),
+    "subject_id":      Value("string"),
+    "session_id":      Value("string"),
+    "trial_id":        Value("string"),
+    "start_time_sec":  Value("float64"),
+    "label":           Value("string"),
+    "label_scheme":    Value("string"),
+    "extra":           Value("string"),
+})
+```
+
 ---
 
-Edit this `README.md` markdown file to author your organization card.
+## 🔑 Key Design Principles
+
+### 1. Original Sampling Rate is Always Preserved
+
+Do **not** resample during dataset creation. Store signals at their native sampling rate. Resampling is done at load time, per experiment requirements.
+
+```
+✅ Store at native rate → resample in DataLoader
+❌ Do not permanently downsample/upsample the stored data
+```
+
+### 2. Channel Names, Not Positions
+
+Always use `channel_names` for indexing. Never assume a channel exists at a fixed index position across datasets.
+
+```python
+# ✅ Correct
+idx = sample["channel_names"].index("ECG_II")
+ecg = np.array(sample["signal"])[idx]
+
+# ❌ Wrong
+ecg = np.array(sample["signal"])[0]  # position may differ across datasets
+```
+
+### 3. Label Semantics Are Dataset-Specific
+
+The same `label` value (e.g., `"1"`) may mean different things across datasets. Always use `label` together with `label_scheme`.
+
+```python
+{"label": "1", "label_scheme": "sleep_stage"}     # → N1 sleep
+{"label": "1", "label_scheme": "physical_activity"} # → sitting
+{"label": "1", "label_scheme": "arrhythmia"}        # → AFib
+```
+
+### 4. Extend with `extra`, Don't Break the Schema
+
+Dataset-specific metadata that doesn't fit the common fields goes into the `extra` JSON string field. This keeps the schema stable while allowing flexibility.
+
+```python
+import json
+
+extra = json.dumps({
+    "age": 67,
+    "gender": "M",
+    "icu_type": "MICU",
+    "filter_applied": "0.5-40Hz bandpass",
+})
+```
+
+---
+
+## 📂 Directory Structure
+
+Each dataset repository follows this structure:
+
+```
+MultiModalSensing/<dataset-name>/
+├── train/
+│   ├── data-00000-of-00002.parquet
+│   └── data-00001-of-00002.parquet
+├── validation/
+│   └── data-00000-of-00001.parquet
+├── test/
+│   └── data-00000-of-00001.parquet
+├── README.md             ← Dataset Card (required)
+└── convert.py            ← Conversion script (required, for reproducibility)
+```
+
+---
+
+## 🚀 How to Load Datasets
+
+### Basic Loading
+
+```python
+from datasets import load_dataset
+import numpy as np
+
+# Load any lab dataset with the same interface
+ds = load_dataset("your-lab-org/pamap2-activity", split="train")
+
+sample = ds[0]
+signal = np.array(sample["signal"])          # shape: (n_channels, n_samples)
+sr     = sample["sampling_rate"]             # e.g. 100.0
+chs    = sample["channel_names"]             # e.g. ["acc_x", "acc_y", ...]
+label  = sample["label"]                     # e.g. "running"
+```
+
+### Using the Shared Loader (Recommended)
+
+Install the lab utility package and use the unified loader, which handles resampling and channel selection automatically:
+
+```python
+# pip install git+https://github.com/MultiModalSensing/lab-utils.git
+from lab_utils import load_lab_dataset
+
+# Load with optional resampling and channel selection
+ds = load_lab_dataset(
+    name       = "pamap2-activity",
+    split      = "train",
+    channels   = ["acc_x", "acc_y", "acc_z"],     # select channels by name
+)
+
+# Returns {"x": np.ndarray, "y": str, "sr": float}
+sample = ds[0]
+print(sample["x"].shape)   # (3, n_samples)
+```
+
+### Accessing Gated Datasets (e.g., MIMIC)
+
+```bash
+# Step 1: Log in to HuggingFace
+huggingface-cli login
+
+# Step 2: Request access via the dataset page (one-time)
+# https://huggingface.co/datasets/your-lab-org/mimic-waveforms
+```
+
+```python
+# Step 3: Load as normal (authentication handled automatically)
+ds = load_dataset("your-lab-org/mimic-waveforms", split="train")
+```
+
+---
+
+## ➕ How to Contribute a New Dataset
+
+Follow these 4 steps to add your dataset to the organization.
+
+### Step 1: Write `convert.py`
+
+Use the template below and adapt to your raw data format:
+
+```python
+"""
+convert.py — Template for lab dataset conversion
+Replace all <PLACEHOLDERS> with your dataset's specifics.
+"""
+
+import json
+import numpy as np
+import pandas as pd
+from datasets import Dataset
+from lab_schema import LAB_FEATURES   # import from lab-utils
+
+DATASET_NAME   = "<your-dataset-name>"
+MODALITY       = "<EEG|ECG|IMU|PPG|...>"
+CHANNEL_NAMES  = ["<ch1>", "<ch2>"]    # full list of channels
+SAMPLING_RATE  = <Hz>                  # native sampling rate (float)
+UNIT           = "<uV|mV|m/s2|...>"
+LABEL_SCHEME   = "<sleep_stage|physical_activity|arrhythmia|...>"
+WINDOW_SEC     = 5.0                   # window length in seconds
+OVERLAP_SEC    = 2.5                   # overlap between windows
+
+LABEL_MAP = {
+    <raw_label_value>: "<semantic_label>",
+    # e.g. 1: "walking", 2: "running", ...
+}
+
+def segment(signal_2d, sr, window_sec, overlap_sec):
+    """Sliding window segmentation. signal_2d: (n_channels, n_total_samples)"""
+    win   = int(window_sec  * sr)
+    step  = int((window_sec - overlap_sec) * sr)
+    segs  = []
+    for start in range(0, signal_2d.shape[1] - win + 1, step):
+        segs.append(signal_2d[:, start:start + win])
+    return segs  # list of (n_channels, win) arrays
+
+def convert_subject(raw_path, subject_id):
+    """Load one subject's raw file and return a list of sample dicts."""
+    # ── load raw data ──────────────────────────────────────────────
+    df = pd.read_csv(raw_path)   # adapt as needed
+
+    signal_2d = df[CHANNEL_NAMES].values.T.astype(np.float32)
+    labels_ts = df["label"].values   # one label per time step
+
+    segments = segment(signal_2d, SAMPLING_RATE, WINDOW_SEC, OVERLAP_SEC)
+    win = int(WINDOW_SEC * SAMPLING_RATE)
+
+    samples = []
+    for i, seg in enumerate(segments):
+        start_sample = i * int((WINDOW_SEC - OVERLAP_SEC) * SAMPLING_RATE)
+        window_labels = labels_ts[start_sample : start_sample + win]
+        majority_label = LABEL_MAP.get(
+            pd.Series(window_labels).mode()[0], "unknown"
+        )
+
+        samples.append({
+            "signal":          seg.tolist(),
+            "sampling_rate":   float(SAMPLING_RATE),
+            "modality":        MODALITY,
+            "channel_names":   CHANNEL_NAMES,
+            "n_channels":      seg.shape[0],
+            "n_samples":       seg.shape[1],
+            "unit":            UNIT,
+            "subject_id":      str(subject_id),
+            "session_id":      "session_1",
+            "trial_id":        f"segment_{i:06d}",
+            "start_time_sec":  round(start_sample / SAMPLING_RATE, 6),
+            "label":           majority_label,
+            "label_scheme":    LABEL_SCHEME,
+            "extra":           json.dumps({}),   # add subject metadata here
+        })
+    return samples
+
+if __name__ == "__main__":
+    all_samples = []
+    for subject_id in range(1, 10):   # adapt to your subject list
+        all_samples += convert_subject(f"data/subject{subject_id}.csv", subject_id)
+
+    ds = Dataset.from_list(all_samples, features=LAB_FEATURES)
+    # Optional: train/val/test split
+    ds = ds.train_test_split(test_size=0.2, seed=42)
+    ds.push_to_hub(f"your-lab-org/{DATASET_NAME}")
+    print(f"✅ Uploaded {len(all_samples)} samples to your-lab-org/{DATASET_NAME}")
+```
+
+### Step 2: Create a Dataset Card (`README.md`)
+
+Every dataset repo must have a `README.md` using the template in the section below.
+
+### Step 3: Push to Hub
+
+```bash
+huggingface-cli login
+
+# Public dataset
+python convert.py
+
+# Private / gated dataset (e.g., MIMIC, clinical data)
+python convert.py --private   # or set via HF web UI after upload
+```
+
+### Step 4: Register in the Index
+
+Open a PR to this README and add your dataset to the **Repository Index** table at the top.
+
+---
+
+## 📋 Dataset Card Template
+
+Copy this into your dataset's `README.md`:
+
+```markdown
+---
+dataset_name: <your-dataset-name>
+modality: <EEG|ECG|IMU|PPG|...>
+sampling_rate: <Hz>
+n_channels: <N>
+task: <sleep_staging|activity_recognition|arrhythmia_detection|...>
+label_scheme: <scheme_name>
+n_subjects: <N>
+version: 1.0.0
+license: <cc-by-4.0|physionet-1.5|internal>
+---
+
+## Overview
+
+Brief description of the dataset: source, purpose, key characteristics.
+
+## Source
+
+- **Original dataset**: [Name](URL)
+- **Paper**: citation
+- **Access**: Public / Gated / Internal
+
+## Modality & Channels
+
+| Index | Channel Name | Unit | Description |
+|-------|-------------|------|-------------|
+| 0     | `<ch_name>` | uV   | <description> |
+
+## Label Definition
+
+| label | meaning |
+|-------|---------|
+| `"0"` | <class 0> |
+| `"1"` | <class 1> |
+
+**label_scheme**: `<scheme_name>`
+
+## Splits
+
+| Split      | Subjects | Segments |
+|------------|----------|----------|
+| train      | X        | X        |
+| validation | X        | X        |
+| test       | X        | X        |
+
+## Preprocessing
+
+Describe all transformations applied before storage:
+- Filtering: (e.g., 0.5–40 Hz bandpass, 50 Hz notch)
+- Normalization: (e.g., z-score per channel per recording)
+- Windowing: X-second windows, Y-second overlap
+- Resampling: original rate → stored rate
+
+> ⚠️ Raw data is stored at its **native sampling rate**. Resample in your DataLoader as needed.
+
+## Known Issues / Limitations
+
+- List any known data quality issues
+- Missing subjects, artifact channels, etc.
+
+## Citation
+
+\`\`\`bibtex
+@dataset{...}
+\`\`\`
+```
+
+---
+
+## ⚙️ Naming Conventions
+
+| Entity | Convention | Example |
+|--------|-----------|---------|
+| Dataset repo name | `<source>-<modality>` (lowercase, hyphen) | `mimic-waveforms`, `pamap2-activity` |
+| `subject_id` | `subject<NNN>` or original ID string | `subject101`, `p10032` |
+| `session_id` | `session_<N>` or protocol name | `session_1`, `protocol_run2` |
+| `trial_id` | `segment_<NNNNNN>` (zero-padded) | `segment_000042` |
+| `modality` | Uppercase, `+` separator for multi-modal | `EEG`, `ECG+PPG`, `IMU+HR` |
+| `label_scheme` | lowercase, underscore | `sleep_stage`, `physical_activity` |
+
+---
+
+## 🔐 Access Control Guidelines
+
+| Data Type | HuggingFace Setting | Notes |
+|-----------|---------------------|-------|
+| Public benchmark (PAMAP2, etc.) | `public` | Fine to share openly |
+| PhysioNet data (MIMIC, etc.) | `gated` | Require PhysioNet credentialing |
+| Proprietary / in-house data | `private` | Lab members only; get PI approval |
+| De-identified clinical data | `private` or `gated` | Check IRB terms before uploading |
+
+> ⚠️ **Never upload identifiable patient data.** When in doubt, contact the lab PI before uploading.
+
+---
+
+## 🛠️ Common Recipes
+
+### Resample a loaded dataset to a target frequency
+
+```python
+import scipy.signal as ss
+import numpy as np
+
+def resample_sample(sample, target_sr):
+    sig = np.array(sample["signal"])
+    orig_sr = sample["sampling_rate"]
+    if orig_sr == target_sr:
+        return sample
+    n_target = int(sig.shape[-1] * target_sr / orig_sr)
+    sig_resampled = ss.resample(sig, n_target, axis=-1)
+    return {**sample, "signal": sig_resampled.tolist(),
+            "sampling_rate": target_sr, "n_samples": n_target}
+
+ds = ds.map(lambda x: resample_sample(x, target_sr=100))
+```
+
+### Select specific channels by name
+
+```python
+def select_channels(sample, channel_list):
+    chs = sample["channel_names"]
+    idx = [chs.index(c) for c in channel_list]
+    sig = np.array(sample["signal"])[idx]
+    return {**sample, "signal": sig.tolist(),
+            "channel_names": channel_list, "n_channels": len(channel_list)}
+
+ds = ds.map(lambda x: select_channels(x, ["acc_x", "acc_y", "acc_z"]))
+```
+
+### Parse `extra` metadata
+
+```python
+import json
+
+extras = [json.loads(s["extra"]) for s in ds]
+ages   = [e.get("age") for e in extras]
+```
+
+---
+
+## 📬 Questions & Contributions
+
+- **Bug / schema issue**: Open an issue in `MultiModalSensing/lab-utils`
+- **New dataset**: Follow the contribution steps above and open a PR to this README
+- **Access request**: Email `m.r.cui@utwente.nl` with your HuggingFace username