knowledge/nb_00_data_processing.md

---
name: Notebook 00 - Data Processing
description: Cell-by-cell index of 00_data_processing.ipynb — processes calcium imaging datasets into standardized summary tables (responses, ROI attributes, pairwise distances)
type: reference
---

**Path:** `python/playground/Maedeh/slap2_paper/analysis/00_data_processing.ipynb`

## Purpose
Processes calcium imaging data from multiple SLAP2 datasets to generate standardized summary tables: neural responses, ROI attributes (spatial coordinates, cable distances), and pairwise distance metrics. Prepares data for downstream correlation and distance analysis.

## Cell Log
- **Cell 0**: Imports (numpy, pandas, h5py, matplotlib, seaborn, scipy, scbc modules)
- **Cell 1**: Matplotlib config (Arial, 8pt)
- **Cell 2**: Load dataset metadata from Excel (`slap2_datasets.xlsx`, sheet "raster")
- **Cell 3**: Markdown: "run and save individual datasets"
- **Cell 4**: Main loop: iterate datasets, generate summary tables (responses, ROI attrs, pairs) at 200 Hz, save per-experiment
- **Cell 5**: Load/verify summary tables from first dataset
- **Cell 6**: Markdown: "load and concat for all datasets and save"
- **Cell 7**: Concatenate all datasets, save `roi_attr_all_datasets.parquet` and `roi_pairs_all_datasets.parquet`
- **Cells 10-17**: Compute dF/F0 for soma (hull_window=4s, denoise_window=1s), artifact detection, F0 baseline visualization
- **Cells 20-27**: Playground — single dataset processing, inspect tables
- **Cells 28-37**: Build stimulus response tables (3s stim + 1s post), trial-averaged responses, synapse attribute tables, pairwise distance tables, dendrogram coordinate scatter plot

## Data Loaded
- Excel: `slap2_datasets.xlsx` (sheet "raster")
- Per-dataset experiment summary files via `initialize_exp()`

## Outputs Saved
- `roi_attr_all_datasets.parquet` (concatenated ROI attributes)
- `roi_pairs_all_datasets.parquet` (concatenated pairwise metrics)
- Per-experiment summary tables in `[dataset_dir]/ExperimentSummary/`

## Figures
- Calcium trace artifact detection (raw, derivatives, cleaned)
- F0 baseline + dF/F response
- Dendrogram scatter (x vs z coordinates, soma in red)