SynheartFocus_MultitaskModel/README.md

![img.png](img.png)
# Multi-Task Learning for Emotional and Cognitive State Detection

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![TensorFlow 2.x](https://img.shields.io/badge/tensorflow-2.x-orange.svg)](https://www.tensorflow.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

**Simultaneously predict emotional state (stress) and cognitive state (mental effort) from physiological signals using a unified deep learning architecture.**

> 🎯 **Key Innovation:** Multi-task framework achieving **70.7% accuracy** with multimodal fusion (+8.3% over IBI-only) and principled data preparation (+3.7% over naive training).

---

## Why Multi-Task Learning?

Traditional approaches train separate models for stress and effort detection. Our unified multi-task framework offers:

- **Joint representation learning:** Shared encoders extract features relevant to both emotional and cognitive states
- **Efficient architecture:** Single model (~87k parameters) predicts both dimensions simultaneously
- **Task-specific training:** Principled data preparation (masking ambiguous samples) improves generalization
- **Real-time inference:** One forward pass → both stress and effort predictions

---

## Quick Start

```bash
# Install
git clone https://github.com/yisakSyn/SynheartFocus_MultitaskModel.git
cd SynheartFocus_MultitaskModel
pip install -r requirements.txt

# Prepare data (SWELL-KW dataset)
python prepare_multimodal_data_v5.py

# Train with LOSO cross-validation
python trainer_v8_masked_effortMultimodal.py --loso --data_dir ./prepared_data_v5_multimodal

# Single holdout evaluation
python trainer_v8_masked_effortMultimodal.py --holdout pp09 pp17 pp25
```

---

## Architecture

```
Physiological Inputs (4 streams)
├─ IBI timeseries (120 samples @ 2Hz = 60s) ──┐
├─ HRV features (14 features)                 ├─→ IBI Encoder (CNN-LSTM) → Dense(32)
├─ EDA timeseries (120 samples @ 2Hz = 60s) ──┤                                 ↓
└─ EDA features (12 features)                 └─→ EDA Encoder (CNN-LSTM) → Dense(32)
                                                                                  ↓
                                                           Fusion: Concat(32+16+32+16) = 96d
                                                                                  ↓
                                                                    Dense(64) → Dense(32)
                                                                                  ↓
                                                           ┌──────────────────────┴──────────────────────┐
                                                      Stress Head                                   Effort Head
                                                     Dense(2, softmax)                            Dense(2, softmax)
                                                  (trained on all samples)                    (trained on valid only)
```

### Model Details

**IBI Encoder (CNN-LSTM):**
- Conv1D blocks: 24→32→48 filters (kernels 7→5→3)
- LSTM(32) with attention pooling
- HRV feature branch: Dense(24→16)
- Output: 48 dimensions (32+16)

**EDA Encoder (CNN-LSTM):**
- Conv1D blocks: 24→32→48 filters (kernels 9→7→5, larger for EDA)
- LSTM(32) with attention pooling
- EDA feature branch: Dense(24→16)
- Output: 48 dimensions (32+16)

**Fusion & Classification:**
- Concatenate: [IBI(32) + HRV(16) + EDA(32) + EDA_feat(16)] = 96 dims
- Dense layers: 64 → 32 with dropout (0.5, 0.25)
- Two task-specific output heads (softmax)

**Joint Optimization:**
```python
# Loss computation
loss_stress = focal_loss(y_true_stress, y_pred_stress)  # All samples
loss_effort = masked_focal_loss(y_true_effort, y_pred_effort, mask)  # Valid only
total_loss = loss_stress + loss_effort
```

**Parameters:** ~87,000 total

---

## Results

### LOSO Cross-Validation (18 subjects)

| Metric | Accuracy | Std Dev |
|--------|----------|---------|
| **Stress Detection** | 68.9% | ±12.9% |
| **Effort Detection** | 72.5% | ±14.1% |
| **Average** | **70.7%** | **±13.2%** |

### Ablation Studies

**1. Multimodal Fusion:**

| Configuration | Stress | Effort | Average |
|---------------|--------|--------|---------|
| IBI only (Run 5) | 60.2% | 64.6% | 62.4% |
| **IBI + EDA (ours, Run 3)** | **68.9%** | **72.5%** | **70.7%** |
| **Improvement** | **+8.7%** | **+7.9%** | **+8.3%** |

**Key Finding:** Multimodal fusion (IBI + EDA) provides substantial improvement over single-modality IBI-only approach.

**2. Task-Specific Data Preparation:**

| Training Strategy | Stress | Effort | Average |
|-------------------|--------|--------|---------|
| All conditions (c1, c2, c3) | 68.7% | 65.2% | 67.0% |
| **Task-specific masking (ours)** | **68.9%** | **72.5%** | **70.7%** |
| **Improvement** | +0.2% | **+7.3%*** | **+3.7%** |

*p < 0.01 (McNemar's test)

**Key Finding:** Masking ambiguous effort labels (c2 interruption condition) improves effort detection by 7.3% while maintaining stress accuracy.

### Key Contributions Validated

Our experimental results validate two independent contributions:

1. **Multimodal Fusion (+8.3%):** Combining IBI and EDA signals outperforms IBI-only approach
2. **Task-Specific Data Preparation (+3.7%):** Masking ambiguous samples improves effort detection
3. **Combined Framework (70.7%):** Multi-task architecture with principled data curation

---

## Methodology

### Multi-Task Learning Framework

**Core Design:** Unified architecture with shared encoders and task-specific output heads enables joint optimization of both emotional and cognitive state prediction.

**Loss Function:**
```python
L_total = L_stress + L_effort

where:
  L_stress = FocalLoss(y_stress, pred_stress)  # All samples
  L_effort = MaskedFocalLoss(y_effort, pred_effort, mask)  # Valid samples only
```

### Task-Specific Data Preparation

| Condition | Stress Label | Effort Label | Training Strategy |
|-----------|--------------|--------------|-------------------|
| **c1** (Neutral) | 0 (Low) | 0 (Low) | Both tasks trained |
| **c2** (Interruption) | 1 (High) | **MASKED** | Stress only |
| **c3** (Time Pressure) | 1 (High) | 1 (High) | Both tasks trained |

**Rationale:** Interruption-based tasks (c2) produce ambiguous effort labels due to individual differences in multitasking ability. NASA-TLX analysis shows 2.2× higher variance in effort ratings for c2 (CV=0.44) vs. c3 (CV=0.20).

**Implementation:**
- Stress head: Trained on all 11,332 samples
- Effort head: Trained on 7,998 valid samples (c1 + c3 only)
- Masking applied during loss computation, not data filtering

### Signal Processing

**IBI Extraction:**
1. R-peak detection from ECG (2048 Hz)
2. Artifact correction (physiological constraints: 300-2000ms)
3. Cubic spline interpolation to 2 Hz
4. 120-sample windows (60s at 2 Hz) with 75% overlap

**HRV Features (14):**
- Time-domain (7): Mean IBI, SDNN, RMSSD, pNN50, CV, Mean HR, Std HR
- Frequency-domain (4): LF, HF, LF/HF, Total power
- Nonlinear (3): SD1, SD2, SD1/SD2

**EDA Processing:**
1. Detrending and lowpass filter (1 Hz cutoff)
2. Downsample to 2 Hz
3. 120-sample windows with 75% overlap

**EDA Features (12):**
- Raw statistics (4): Mean, SD, Min, Max
- Tonic component (4): Mean, SD, slope, range
- Phasic component (4): Mean amplitude, SD, SCR count, Mean SCR height

---

## Project Structure

```
multitask-stress-effort/
├── prepare_multimodal_data_v5.py    # Raw signals → ML features
├── trainer_v8_masked_effortMultimodal.py  # Multi-task training
├── Questionnaire.xlsx               # Labels from SWELL-KW
├── Processed_IBI_HR_EDA/           # MATLAB-processed signals
│   └── ppXX_date_cY_IBI_HR_EDA.mat
├── prepared_data_v5_multimodal/    # ML-ready dataset
│   ├── X_ibi.npy                   # (N, 120, 1)
│   ├── X_hrv.npy                   # (N, 14)
│   ├── X_eda.npy                   # (N, 120, 1)
│   ├── X_eda_features.npy          # (N, 12)
│   ├── y_stress.npy                # (N, 2) one-hot
│   ├── y_effort.npy                # (N, 2) one-hot, masked=[0,0]
│   ├── effort_mask.npy             # (N,) 1=valid, 0=masked
│   └── ...
└── runs/                            # Training results
    └── synheart_v8_multimodal_*/
```

---

## Training Configuration

| Parameter | Value |
|-----------|-------|
| Optimizer | AdamW |
| Learning Rate | 2×10⁻⁴ |
| Weight Decay | 1×10⁻³ |
| Batch Size | 64 |
| Max Epochs | 200 |
| Early Stopping | 25 epochs patience |
| Loss Function | Focal Loss (γ=1.5) + Label Smoothing (ε=0.05) |
| Dropout | 0.35 (encoders), 0.50 (fusion) |
| Augmentation | Time Masking (p=0.5, width=15) |

---

## Citation

```bibtex
@article{author2025multitask,
  title={Simultaneous Detection of Emotional and Cognitive States: 
         A Multi-Task Deep Learning Approach},
  author={[Your Name]},
  journal={IEEE Transactions on Affective Computing},
  year={2025},
  note={Under review}
}
```

---

## Requirements

```
tensorflow>=2.10.0
numpy>=1.21.0
scipy>=1.7.0
scikit-learn>=1.0.0
matplotlib>=3.5.0
pandas>=1.3.0
openpyxl>=3.0.0  # For Excel label files
```

Full list: [requirements.txt](requirements.txt)

---

## Key References

**Multi-Task Learning:**
- Caruana (1997) - "Multitask Learning"
- Ruder (2017) - "Multi-Task Learning in Deep Neural Networks"

**Dataset & Cognitive Load:**
- Koldijk et al. (2014) - "SWELL Knowledge Work Dataset"
- Monsell (2003) - "Task Switching"
- Hockey (1997) - "Compensatory Control Theory"

---

## License

MIT License - see [LICENSE](LICENSE) for details.

---

## Contact

**[Yisak T]** - [yisak@synheart.ai] 
PhD

---

<p align="center">
  <i>Multi-task framework for joint emotional and cognitive state detection from physiological signals</i><br>
  <b>70.7% LOSO accuracy | Multimodal IBI+EDA fusion | Task-specific training</b>
</p>