docs/data_preprocessing_notebook.md

# `notebooks/data_preprocessing.ipynb` — documentation

This note describes what the notebook does, cell by cell.

---

## Cell 0 — Load DeepLoc and inspect it

| Code | What it does |
|------|----------------|
| `import pandas as pd` | Uses pandas for tabular data. |
| `df = pd.read_csv("../data/raw/deeploc.csv")` | Reads the raw CSV from the project’s `data/raw` folder (path is relative to the notebook under `notebooks/`). |
| `print(df.shape)` | Prints row and column counts (example run: **28,303** rows × **16** columns). |
| `print(df.columns.tolist())` | Lists all column names: identifiers, metadata, label columns, and the sequence column. |
| `df.head()` | Shows the first five rows as a preview. |

**Takeaway:** The file includes columns such as `ACC`, `Kingdom`, `Partition`, eleven subcellular-location columns (numeric), `Sequence`, and an index-like `Unnamed: 0` column.

---

## Cell 1 — Shared constants for cleaning and modeling

| Code | What it does |
|------|----------------|
| `from pathlib import Path` | Used later for path handling when saving files (`save_processed_data`). |
| `VALID_AMINO_ACIDS` | Allowed one-letter codes: standard 20 amino acids plus common extensions (**X** unknown, **B** Asx, **Z** Glx, **U** selenocysteine, **O** pyrrolysine). Rows are kept only if every character in the sequence is in this set (after uppercasing). |
| `LABEL_COLUMNS` | The **11** DeepLoc subcellular compartments treated as **binary multilabel** targets. |
| `SEQUENCE_COLUMN = "Sequence"` | Name of the protein sequence column. |
| `ID_COLUMN = "ACC"` | Accession / protein ID used as the row identifier in the final table. |

**Purpose:** Single place to define valid amino acid alphabets and which columns are labels.

---

## Cell 2 — Cleaning pipeline and saving processed data

This cell defines helper functions and runs a full pipeline inside `if __name__ == "__main__":`. In Jupyter, `__name__` is usually `"__main__"`, so that block **does** run when you execute the cell.

### `is_valid_sequence(seq)`

- Requires a **string**; strips whitespace and uppercases.
- Rejects empty sequences.
- Returns **True** only if **every** character is in `VALID_AMINO_ACIDS`.

### `load_dataset(file_path)`

- Reads the CSV with `pd.read_csv`.
- Prints confirmation, shape, and column list; returns the dataframe.

### `clean_sequences(df)`

- Works on a **copy** of the dataframe.
- `dropna(subset=[SEQUENCE_COLUMN])` — removes rows with missing sequences.
- Converts `Sequence` to string, **strip + upper**.
- Keeps rows where `is_valid_sequence` is true.
- `drop_duplicates(subset=[SEQUENCE_COLUMN])` — drops duplicate sequences (first row kept).
- Prints row counts before and after (example: **28,303** unchanged if nothing was dropped).

### `clean_labels(df)`

- For each label column: `to_numeric(..., errors="coerce")`, `fillna(0)`, `astype(int)`, then **`clip(0, 1)`** so labels are binary **0/1**.
- Adds **`label_count`**: sum of the eleven labels per row (number of “positive” compartments).
- Prints the distribution of `label_count`.

### `build_final_dataframe(df)`

- Keeps **`ACC`**, **`Sequence`**, and the **11** label columns only (drops `Unnamed: 0`, `Kingdom`, `Partition`, etc.).
- Example shape: **28,303 × 13** (1 ID + 1 sequence + 11 labels).
- Prints shape and `head()` of the result.

### `save_processed_data(df, output_path)`

- Uses `Path(output_path)`; creates parent directories if missing (`mkdir(parents=True, exist_ok=True)`).
- Writes CSV with `to_csv(..., index=False)`.
- Prints the output path.

### `if __name__ == "__main__":` block

- Sets **`input_file`** and **`output_file`** (raw `deeploc.csv` → processed `deeploc_multilabel.csv`). Uses **raw strings** `r"..."` so Windows backslashes are not interpreted as escape sequences.
- Pipeline order: **load → clean sequences → clean labels → build final dataframe → save**.

**Net effect:** A cleaned **multilabel** CSV with protein ID, normalized sequence, and eleven binary compartment columns, saved under `data/processed/` (exact path as configured in the notebook).

---

## One-line summary

The notebook **loads** `deeploc.csv`, **sanitizes** sequences and **binarizes** subcellular labels, **drops** non-essential columns, and **writes** a compact multilabel file **`deeploc_multilabel.csv`**, after an exploratory load in cell 0.