knowledge/nb_00_data_processing.md

metadata

name: Notebook 00 - Data Processing
description: >-
  Cell-by-cell index of 00_data_processing.ipynb — processes calcium imaging
  datasets into standardized summary tables (responses, ROI attributes, pairwise
  distances)
type: reference

Path: python/playground/Maedeh/slap2_paper/analysis/00_data_processing.ipynb

Purpose

Processes calcium imaging data from multiple SLAP2 datasets to generate standardized summary tables: neural responses, ROI attributes (spatial coordinates, cable distances), and pairwise distance metrics. Prepares data for downstream correlation and distance analysis.

Cell Log

• Cell 0: Imports (numpy, pandas, h5py, matplotlib, seaborn, scipy, scbc modules)
• Cell 1: Matplotlib config (Arial, 8pt)
• Cell 2: Load dataset metadata from Excel (slap2_datasets.xlsx, sheet "raster")
• Cell 3: Markdown: "run and save individual datasets"
• Cell 4: Main loop: iterate datasets, generate summary tables (responses, ROI attrs, pairs) at 200 Hz, save per-experiment
• Cell 5: Load/verify summary tables from first dataset
• Cell 6: Markdown: "load and concat for all datasets and save"
• Cell 7: Concatenate all datasets, save roi_attr_all_datasets.parquet and roi_pairs_all_datasets.parquet
• Cells 10-17: Compute dF/F0 for soma (hull_window=4s, denoise_window=1s), artifact detection, F0 baseline visualization
• Cells 20-27: Playground — single dataset processing, inspect tables
• Cells 28-37: Build stimulus response tables (3s stim + 1s post), trial-averaged responses, synapse attribute tables, pairwise distance tables, dendrogram coordinate scatter plot

Data Loaded

• Excel: slap2_datasets.xlsx (sheet "raster")
• Per-dataset experiment summary files via initialize_exp()

Outputs Saved

• roi_attr_all_datasets.parquet (concatenated ROI attributes)
• roi_pairs_all_datasets.parquet (concatenated pairwise metrics)
• Per-experiment summary tables in [dataset_dir]/ExperimentSummary/

Figures

• Calcium trace artifact detection (raw, derivatives, cleaned)
• F0 baseline + dF/F response
• Dendrogram scatter (x vs z coordinates, soma in red)