| --- |
| license: mit |
| language: |
| - en |
| library_name: pytorch |
| tags: |
| - multi-modal |
| - daily-activity |
| - wearable-sensors |
| - benchmark |
| --- |
| |
| # PULSE — Code Repository |
|
|
| Reference implementation, training scripts, and benchmark baselines for the |
| **PULSE** dataset paper (under double-blind review at NeurIPS 2026 Evaluations & |
| Datasets Track). |
|
|
| > **Dataset:** [`velvet-pine-22/PULSE`](https://huggingface.co/datasets/velvet-pine-22/PULSE) |
| > · **Sample subset (≈285 MB):** [`velvet-pine-22/PULSE-sample`](https://huggingface.co/datasets/velvet-pine-22/PULSE-sample) |
|
|
| ## Repository layout |
|
|
| ``` |
| PULSE-code/ |
| ├── experiments/ |
| │ ├── data/ # PyTorch Dataset wrappers |
| │ │ ├── dataset.py # core multi-modal dataset (T1, T2) |
| │ │ ├── dataset_seqpred.py # T2 fine-grained action recognition |
| │ │ ├── dataset_grasp_state.py # T3 grasp onset anticipation |
| │ │ ├── dataset_forecast.py # auxiliary forecasting heads |
| │ │ └── dataset_signal_forecast.py # T5 tactile-driven motion forecast |
| │ │ |
| │ ├── nets/ # Model architectures |
| │ │ ├── models.py # backbone networks (Transformer / LSTM / 1D-CNN) |
| │ │ ├── models_seqpred.py # DailyActFormer (DAF) — multi-modal Transformer |
| │ │ ├── models_forecast.py # forecasting heads |
| │ │ ├── models_forecast_priv.py # privileged-tactile variants for T5 |
| │ │ ├── published_models.py # third-party model implementations |
| │ │ └── baselines_published/ # 7 published baselines (re-implementation) |
| │ │ ├── baselines.py # DeepConvLSTM / InceptionTime / MS-TCN / etc. |
| │ │ └── syncfuse.py # under-pressure-style multi-modal fusion |
| │ │ |
| │ ├── tasks/ # Training + evaluation entry points |
| │ │ ├── train_exp1.py # T1 — scene recognition |
| │ │ ├── train_seqpred.py # T2 — action recognition (DAF + ablations) |
| │ │ ├── train_grasp_state.py # T3 — grasp onset anticipation |
| │ │ ├── train_pred_cls.py # T3 alt classification head |
| │ │ ├── train_exp_missing.py # T4 — missing-modality robustness |
| │ │ ├── train_signal_forecast.py # T5 — tactile-driven motion forecasting |
| │ │ ├── train_signal_forecast_priv.py # T5 privileged variants |
| │ │ ├── train_baselines_t1.py # baselines for T1 |
| │ │ ├── train_exp{2,3,4}.py # ablation experiments |
| │ │ ├── train_exp_{anticipate,grip,pose,retrieval,zeroshot}.py # auxiliary |
| │ │ ├── train_pred.py / train_forecast.py |
| │ │ ├── eval_baselines.py / eval_combined.py |
| │ │ └── published_baselines.py # baseline registry |
| │ │ |
| │ ├── analysis/ # Case study, figures, data prep utilities |
| │ │ ├── grasp_phase_analysis.py # case study (gaze→EMG→hand→contact cascade) |
| │ │ ├── modality_viz.py / analysis_figures.py / data_statistics_figure.py |
| │ │ ├── extract_video_features.py / extract_videomae_features.py |
| │ │ ├── build_taxonomy.py / generate_action_labels.py / generate_coarse_annotations.py |
| │ │ ├── reannotate_actions.py / gen_val_comparison.py |
| │ │ ├── exp_per_subject.py / check_seg_lengths.py |
| │ │ └── aggregate_*.py # collate run results |
| │ │ |
| │ ├── slurm/ # 60+ SLURM launch scripts (one per main experiment) |
| │ │ └── run_*.sh |
| │ │ |
| │ ├── taxonomy.py # shared 18-primitive taxonomy |
| │ ├── s9_primitives.json |
| │ └── taxonomy_v3.json |
| │ |
| ├── scripts/ # Top-level utilities (not task-specific) |
| │ ├── build_paper_tables.py # collates results JSONs into LaTeX tables |
| │ ├── eval_macrof1.py / eval_subset.py / eval_topk_v3.py |
| │ └── dispatch_eval.sh # batch dispatcher |
| │ |
| ├── LICENSE # MIT |
| ├── requirements.txt # Python deps |
| └── README.md |
| ``` |
|
|
| ## Quick start |
|
|
| ```bash |
| # 1. Set up Python environment |
| python -m venv .venv && source .venv/bin/activate |
| pip install -r requirements.txt |
| |
| # 2. Point at the PULSE dataset (download from HuggingFace first) |
| export PULSE_ROOT=/path/to/PULSE # the dataset root (not this code repo) |
| |
| # 3. Run a training entry point as a module (from the experiments/ directory) |
| cd experiments |
| python -m tasks.train_seqpred \ |
| --root $PULSE_ROOT \ |
| --modalities mocap emg eyetrack imu pressure \ |
| --output_dir runs/t2_daf |
| |
| # 4. Reproduce paper tables (after training all benchmarks) |
| cd .. |
| python scripts/build_paper_tables.py \ |
| --results_root experiments/runs/ \ |
| --out tables/ |
| ``` |
|
|
| > **Why `python -m tasks.train_seqpred` and not `python tasks/train_seqpred.py`?** |
| > The training scripts import sibling modules (`from data.dataset import …`, |
| > `from nets.models import …`). Running with `-m` from the `experiments/` |
| > directory makes Python treat `data/`, `nets/`, `tasks/`, and `analysis/` as |
| > top-level packages so the imports resolve cleanly. |
|
|
| ## Reproducing the benchmark tasks |
|
|
| | Task | Entry point | Output | |
| |---|---|---| |
| | T1 — Scene recognition (8-way) | `tasks.train_exp1` | scene-classification metrics | |
| | T2 — Fine-grained action recognition | `tasks.train_seqpred` | verb / noun / hand top-k accuracy | |
| | T3 — Grasp onset anticipation | `tasks.train_grasp_state` / `tasks.train_pred_cls` | anticipation F1 / time-to-contact | |
| | T4 — Missing-modality robustness | `tasks.train_exp_missing` + `tasks.eval_combined` | per-modality ablation table | |
| | T5 — Tactile-driven grasp-state recognition | `tasks.train_signal_forecast` (+ `_priv` variants) | sub-second grasp-state metrics | |
| | T6 — Cross-modal pressure prediction | `tasks.train_forecast` / `tasks.train_signal_forecast` | pressure reconstruction metrics | |
|
|
| The exact command lines (with hyperparameters, seeds, GPU configs) used for |
| every paper table are checked in under `experiments/slurm/run_*.sh`, one |
| SLURM script per paper experiment. Output JSON files from these runs are |
| collated into LaTeX tables by `scripts/build_paper_tables.py`. |
|
|
| ## Hardware |
|
|
| Headline experiments were run on **NVIDIA A800 (80 GB)** GPUs. A single seed of |
| DailyActFormer T2 trains in ~6 hours on one A800. Most baselines fit on a |
| single 24 GB consumer GPU. |
|
|
| ## License & attribution |
|
|
| Code is released under **MIT** (see `LICENSE`). The PULSE dataset itself is |
| released under **CC BY-NC 4.0** (see the dataset repository). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{anonymous2026pulse, |
| title = {PULSE: A Synchronized Five-Modality Dataset for Multi-Modal Daily Activity Understanding}, |
| author = {Anonymous Authors}, |
| booktitle = {Submitted to NeurIPS 2026 Evaluations and Datasets Track}, |
| year = {2026}, |
| note = {Under double-blind review} |
| } |
| ``` |
|
|
