Fix downstream notebook project paths and unify tutorial links

Update the five downstream notebooks to resolve paths from the repository layout, use the bundled BioMod checkpoint paths and supported commands, and read actual task outputs. Keep training and evaluation opt-in. Standardize the main README tutorial column to Notebook links. This commit only changes these six files; model weights, paired configuration, datasets, and training source are unchanged.

BioMod/toturials/cell_annotation.ipynb

@@ -1,338 +1,331 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Cell annotation demo: Zheng68K with weight0407 on one GPU\n",
-    "\n",
-    "This notebook is a one-GPU tutorial version of the Zheng68K cell-annotation launcher. It follows the parameter style of `weight0526module1/cell_annotaion/zheng68K/start.sh`, but uses the same weight0407 backbone as the zero-shot batch-integration demo."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Privacy-safe absolute paths\n",
-    "\n",
-    "The notebook uses a public-looking alias path rather than a personal project path:\n",
-    "\n",
-    "`/data/share/biomodulefm_demo/single_cell_model`\n",
-    "\n",
-    "All paths below are absolute and resolve through that alias on the server."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## What this demo runs\n",
-    "\n",
-    "- Dataset: `/data/share/biomodulefm_demo/single_cell_model/DATA/singcell/Zheng68K.h5ad`\n",
-    "- Cell annotation script: `/data/jianqingzhu/BioMod-main/BioMod/cell_annotation/performer_module_finetune.py`\n",
-    "- Model config: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory.pkl`\n",
-    "- Model checkpoint: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth`\n",
-    "- Output directory: `/data/share/biomodulefm_demo/tutorial_runs/cell_annotation_zheng68k_weight0407_1gpu_demo_smoke`\n",
-    "\n",
-    "Run this notebook with the `sclong` Python environment/kernel. The command below is single-GPU: it sets `CUDA_VISIBLE_DEVICES=0` and runs `python`, not 4-process `torchrun`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "import json\n",
-    "import os\n",
-    "import subprocess\n",
-    "import sys\n",
-    "try:\n",
-    "    from IPython.display import display\n",
-    "except Exception:\n",
-    "    def display(obj):\n",
-    "        print(obj)\n",
-    "import pandas as pd\n",
-    "\n",
-    "BASE = Path('/data/share/biomodulefm_demo/single_cell_model')\n",
-    "MODULE_MODEL = BASE / 'module_model'\n",
-    "PROJECT_ROOT = Path('/data/jianqingzhu/BioMod-main/BioMod')\n",
-    "\n",
-    "SCRIPT = PROJECT_ROOT / 'cell_annotation' / 'performer_module_finetune.py'\n",
-    "DATA_PATH = BASE / 'DATA' / 'singcell' / 'Zheng68K.h5ad'\n",
-    "\n",
-    "# Use the same weight0407 backbone as the zero-shot batch-integration demo.\n",
-    "CONFIG_PKL = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory.pkl'\n",
-    "CKPT_PATH = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth'\n",
-    "\n",
-    "MODULE_DATA_DIR = PROJECT_ROOT / 'module_data_all'\n",
-    "GENE_LIST = MODULE_MODEL / 'selected_genes_27k.txt'\n",
-    "GENE_MAP = MODULE_MODEL / 'gene2ens.csv'\n",
-    "GENE2VEC = MODULE_MODEL / 'selected_gene2vec_27k.npy'\n",
-    "\n",
-    "OUTPUT_PARENT = Path('/data/share/biomodulefm_demo/tutorial_runs')\n",
-    "RUN_NAME = 'cell_annotation_zheng68k_weight0407_1gpu_demo_smoke'\n",
-    "RUN_DIR = OUTPUT_PARENT / RUN_NAME\n",
-    "RUN_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "print('Python:', sys.executable)\n",
-    "print('Run directory:', RUN_DIR)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "required_files = [\n",
-    "    SCRIPT,\n",
-    "    DATA_PATH,\n",
-    "    CONFIG_PKL,\n",
-    "    CKPT_PATH,\n",
-    "    GENE_LIST,\n",
-    "    GENE_MAP,\n",
-    "    GENE2VEC,\n",
-    "]\n",
-    "required_dirs = [PROJECT_ROOT, MODULE_DATA_DIR]\n",
-    "\n",
-    "missing = [str(p) for p in required_files if not p.is_file()] + [str(p) for p in required_dirs if not p.is_dir()]\n",
-    "if missing:\n",
-    "    raise FileNotFoundError('Missing required paths:\\n' + '\\n'.join(missing))\n",
-    "\n",
-    "for p in required_files:\n",
-    "    print('[FOUND FILE]', p)\n",
-    "for p in required_dirs:\n",
-    "    print('[FOUND DIR ]', p)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Single-GPU runtime settings.\n",
-    "os.environ['PYTHONUNBUFFERED'] = '1'\n",
-    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'\n",
-    "os.environ.setdefault('OMP_NUM_THREADS', '8')\n",
-    "os.environ.setdefault('NCCL_DEBUG', 'WARN')\n",
-    "os.environ.setdefault('NUMBA_CACHE_DIR', f'/tmp/numba_cache_cell_annotation_demo_{os.environ.get(\"USER\", \"unknown\")}_{os.getpid()}')\n",
-    "Path(os.environ['NUMBA_CACHE_DIR']).mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "pythonpath_parts = [str(MODULE_MODEL), str(PROJECT_ROOT), str(PROJECT_ROOT.parent / 'model_dependencies')]\n",
-    "if os.environ.get('PYTHONPATH'):\n",
-    "    pythonpath_parts.append(os.environ['PYTHONPATH'])\n",
-    "os.environ['PYTHONPATH'] = ':'.join(pythonpath_parts)\n",
-    "\n",
-    "print('CUDA_VISIBLE_DEVICES =', os.environ['CUDA_VISIBLE_DEVICES'])\n",
-    "print('OMP_NUM_THREADS      =', os.environ['OMP_NUM_THREADS'])\n",
-    "print('NUMBA_CACHE_DIR     =', os.environ['NUMBA_CACHE_DIR'])\n",
-    "print('PYTHONPATH          =', os.environ['PYTHONPATH'])"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Training settings\n",
-    "\n",
-    "The original shell command used four GPUs with per-device `batch_size=24`, so the effective batch size was about 96. For this one-GPU demo, `batch_size=24` and `grad_accum_steps=4` keeps the effective training batch similar.\n",
-    "\n",
-    "For a quick smoke test, set `EPOCHS = 1` and optionally set `MAX_CELLS` to a small number such as `5000`. For a full run matching the training recipe, use `EPOCHS = 300` and `MAX_CELLS = None`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "SEED = 2021\n",
-    "# Tutorial default: a one-GPU smoke run that completes quickly.\n",
-    "# For a full run, set EPOCHS = 300, PATIENCE = 10, and MAX_CELLS = None.\n",
-    "EPOCHS = 1\n",
-    "PATIENCE = 1\n",
-    "MAX_CELLS = 5000\n",
-    "\n",
-    "BATCH_SIZE = 24\n",
-    "EVAL_BATCH_SIZE = 48\n",
-    "GRAD_ACCUM_STEPS = 4\n",
-    "\n",
-    "LR_HEAD = '1e-4'\n",
-    "LR_BACKBONE = '1e-4'\n",
-    "WEIGHT_DECAY = '1e-2'\n",
-    "DROPOUT = '0.2'\n",
-    "\n",
-    "print({\n",
-    "    'seed': SEED,\n",
-    "    'epochs': EPOCHS,\n",
-    "    'patience': PATIENCE,\n",
-    "    'max_cells': MAX_CELLS,\n",
-    "    'batch_size': BATCH_SIZE,\n",
-    "    'eval_batch_size': EVAL_BATCH_SIZE,\n",
-    "    'grad_accum_steps': GRAD_ACCUM_STEPS,\n",
-    "})"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "cmd = [\n",
-    "    sys.executable,\n",
-    "    str(SCRIPT),\n",
-    "    '--device', 'cuda',\n",
-    "    '--seed', str(SEED),\n",
-    "    '--data_path', str(DATA_PATH),\n",
-    "    '--label_column', 'celltype',\n",
-    "    '--gene_name_column', 'index',\n",
-    "    '--val_ratio', '0.1',\n",
-    "    '--test_ratio', '0.2',\n",
-    "    '--min_genes', '10',\n",
-    "    '--min_cells', '10',\n",
-    "    '--config_pkl', str(CONFIG_PKL),\n",
-    "    '--performer_checkpoint', str(CKPT_PATH),\n",
-    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
-    "    '--gene_list_path', str(GENE_LIST),\n",
-    "    '--gene_mapping_path', str(GENE_MAP),\n",
-    "    '--gene2vec_data_path', str(GENE2VEC),\n",
-    "    '--gene2ens_file', str(GENE_MAP),\n",
-    "    '--gene2ens_column', 'Ensembl gene ID',\n",
-    "    '--head_type', 'attn_pool',\n",
-    "    '--finetune_mode', 'full',\n",
-    "    '--cls_decoder_layers', '3',\n",
-    "    '--lr_head', LR_HEAD,\n",
-    "    '--lr_backbone', LR_BACKBONE,\n",
-    "    '--batch_size', str(BATCH_SIZE),\n",
-    "    '--eval_batch_size', str(EVAL_BATCH_SIZE),\n",
-    "    '--grad_accum_steps', str(GRAD_ACCUM_STEPS),\n",
-    "    '--epochs', str(EPOCHS),\n",
-    "    '--patience', str(PATIENCE),\n",
-    "    '--weight_decay', WEIGHT_DECAY,\n",
-    "    '--label_smoothing', '0',\n",
-    "    '--dropout', DROPOUT,\n",
-    "    '--output_dir', str(OUTPUT_PARENT),\n",
-    "    '--run_name', RUN_NAME,\n",
-    "    '--save_test_artifacts',\n",
-    "]\n",
-    "\n",
-    "if MAX_CELLS is not None:\n",
-    "    cmd.extend(['--max_cells', str(MAX_CELLS)])\n",
-    "\n",
-    "print('Command:')\n",
-    "print(' '.join(cmd))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Run the one-GPU fine-tuning job.\n",
-    "# The tutorial default is a smoke run. A full EPOCHS=300 run on all Zheng68K cells can take days in this environment.\n",
-    "log_path = RUN_DIR / 'manual_launch_1gpu.log'\n",
-    "print('Writing log to:', log_path)\n",
-    "\n",
-    "with log_path.open('w') as log:\n",
-    "    process = subprocess.Popen(\n",
-    "        cmd,\n",
-    "        stdout=subprocess.PIPE,\n",
-    "        stderr=subprocess.STDOUT,\n",
-    "        text=True,\n",
-    "        bufsize=1,\n",
-    "        cwd=str(PROJECT_ROOT.parent),\n",
-    "        env=os.environ.copy(),\n",
-    "    )\n",
-    "    for line in process.stdout:\n",
-    "        print(line, end='')\n",
-    "        log.write(line)\n",
-    "    return_code = process.wait()\n",
-    "\n",
-    "if return_code != 0:\n",
-    "    raise RuntimeError(f'cell annotation demo failed with return code {return_code}; see {log_path}')\n",
-    "\n",
-    "print('Done.')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Inspect outputs\n",
-    "\n",
-    "When `--save_test_artifacts` is enabled, the script saves `best_model.pt`, `run_config.json`, `metrics_summary.json`, a classification report, and confusion matrices in the run directory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for path in sorted(RUN_DIR.glob('*')):\n",
-    "    size_mb = path.stat().st_size / 1024 / 1024 if path.is_file() else 0\n",
-    "    print(f'{path.name:45s} {size_mb:8.2f} MB')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "metrics_path = RUN_DIR / 'metrics_summary.json'\n",
-    "with metrics_path.open('r', encoding='utf-8') as handle:\n",
-    "    metrics_summary = json.load(handle)\n",
-    "\n",
-    "metrics_summary"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "report_path = RUN_DIR / 'test_classification_report.txt'\n",
-    "print(report_path.read_text(encoding='utf-8'))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "conf_path = RUN_DIR / 'test_confusion_matrix_normalized.csv'\n",
-    "conf_norm = pd.read_csv(conf_path, index_col=0)\n",
-    "display(conf_norm)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Cell annotation with BioMod model weights\n",
+    "\n",
+    "Fine-tune a cell-type classifier on the local Zheng68K dataset using the BioMod model weights. This notebook prepares the paths and command, launches a single-GPU job when explicitly enabled, and reads the classification results."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Project paths\n",
+    "\n",
+    "Start Jupyter from `BioMod/toturials/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inputs and workflow\n",
+    "\n",
+    "- Entry point: `BioMod/cell_annotation/performer_module_finetune.py`.\n",
+    "- Dataset: `data/downstream/final_data/Zheng68K.h5ad`.\n",
+    "- Model weights and matching configuration: the `.pth` / `.pkl` pair under `weight/0810model/`.\n",
+    "- Gene vocabulary, mapping, embeddings, and module definitions: `model_dependencies/`.\n",
+    "- Results: `tutorial_runs/cell_annotation_zheng68k_biomod/`.\n",
+    "\n",
+    "Select a kernel with the project's cell annotation dependencies installed. The script filters cells and genes, aligns input genes to the model vocabulary, and creates stratified training, validation, and test splits. It selects the checkpoint using validation macro-F1, then evaluates that checkpoint on the test split. The input expression values must already use the preprocessing intended for your experiment; this script does not normalize or log-transform them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import json\n",
+    "import os\n",
+    "import shlex\n",
+    "import subprocess\n",
+    "import sys\n",
+    "import tempfile\n",
+    "import pandas as pd\n",
+    "try:\n",
+    "    from IPython.display import display\n",
+    "except ImportError:\n",
+    "    def display(obj):\n",
+    "        print(obj)\n",
+    "\n",
+    "PROJECT_ROOT = Path('../..').resolve()\n",
+    "if not (PROJECT_ROOT / 'BioMod' / 'models_performer_module.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
+    "    raise FileNotFoundError(\n",
+    "        'Project root not found. Start Jupyter from BioMod/toturials, or set '\n",
+    "        'PROJECT_ROOT to your repository root (use Path(\".\").resolve() if '\n",
+    "        'Jupyter was started at the repository root).'\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
+    "ASSET_ROOT = PROJECT_ROOT / 'model_dependencies'\n",
+    "SCRIPT = SOURCE_DIR / 'cell_annotation' / 'performer_module_finetune.py'\n",
+    "DATA_PATH = PROJECT_ROOT / 'data' / 'downstream' / 'final_data' / 'Zheng68K.h5ad'\n",
+    "\n",
+    "# Keep the BioMod model weights and their matching configuration together.\n",
+    "WEIGHT_DIR = PROJECT_ROOT / 'weight' / '0810model'\n",
+    "CONFIG_PKL = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2.pkl'\n",
+    "CKPT_PATH = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2_epoch0.pth'\n",
+    "MODULE_DATA_DIR = ASSET_ROOT / 'module_data_all'\n",
+    "GENE_LIST = ASSET_ROOT / 'selected_genes_27k.txt'\n",
+    "GENE_MAP = ASSET_ROOT / 'gene2ens.csv'\n",
+    "GENE2VEC = ASSET_ROOT / 'selected_gene2vec_27k.npy'\n",
+    "\n",
+    "OUTPUT_PARENT = PROJECT_ROOT / 'tutorial_runs'\n",
+    "RUN_NAME = 'cell_annotation_zheng68k_biomod'\n",
+    "RUN_DIR = OUTPUT_PARENT / RUN_NAME\n",
+    "RUN_TRAINING = False\n",
+    "\n",
+    "print('Python:', sys.executable)\n",
+    "print('Repository:', PROJECT_ROOT)\n",
+    "print('Dataset:', DATA_PATH)\n",
+    "print('Run directory:', RUN_DIR)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "required_files = [\n",
+    "    SCRIPT, DATA_PATH, CONFIG_PKL, CKPT_PATH, GENE_LIST, GENE_MAP, GENE2VEC,\n",
+    "    SOURCE_DIR / 'models_performer_module.py',\n",
+    "    MODULE_DATA_DIR / 'merged_modules.json',\n",
+    "]\n",
+    "required_dirs = [SOURCE_DIR, ASSET_ROOT]\n",
+    "missing = [str(p) for p in required_files if not p.is_file()]\n",
+    "missing += [str(p) for p in required_dirs if not p.is_dir()]\n",
+    "if missing:\n",
+    "    raise FileNotFoundError('Missing required paths:\\n' + '\\n'.join(missing))\n",
+    "for path in required_files:\n",
+    "    print('[FOUND]', path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Configure only the child process; leave the notebook kernel environment unchanged.\n",
+    "run_env = os.environ.copy()\n",
+    "run_env['PYTHONUNBUFFERED'] = '1'\n",
+    "run_env['CUDA_VISIBLE_DEVICES'] = '0'\n",
+    "run_env.setdefault('OMP_NUM_THREADS', '8')\n",
+    "run_env.setdefault('NCCL_DEBUG', 'WARN')\n",
+    "run_env.setdefault('NUMBA_CACHE_DIR', str(Path(tempfile.gettempdir()) / f'biomod_annotation_{os.getpid()}'))\n",
+    "pythonpath_parts = [str(PROJECT_ROOT), str(SOURCE_DIR), str(ASSET_ROOT)]\n",
+    "if run_env.get('PYTHONPATH'):\n",
+    "    pythonpath_parts.append(run_env['PYTHONPATH'])\n",
+    "run_env['PYTHONPATH'] = os.pathsep.join(pythonpath_parts)\n",
+    "print('CUDA_VISIBLE_DEVICES =', run_env['CUDA_VISIBLE_DEVICES'])\n",
+    "print('PYTHONPATH =', run_env['PYTHONPATH'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training settings\n",
+    "\n",
+    "Review the hyperparameters below before setting `RUN_TRAINING = True` in the setup cell. Adjust batch sizes to your GPU memory. `MAX_CELLS = None` uses all cells that pass filtering; setting an integer requests a subset. The values below configure an example training run and do not establish a benchmark result or a runtime estimate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "SEED = 2021\n",
+    "EPOCHS = 1\n",
+    "PATIENCE = 1\n",
+    "MAX_CELLS = 5000\n",
+    "\n",
+    "BATCH_SIZE = 24\n",
+    "EVAL_BATCH_SIZE = 48\n",
+    "GRAD_ACCUM_STEPS = 4\n",
+    "LR_HEAD = '1e-4'\n",
+    "LR_BACKBONE = '1e-4'\n",
+    "WEIGHT_DECAY = '1e-2'\n",
+    "DROPOUT = '0.2'\n",
+    "LABEL_COLUMN = 'celltype'\n",
+    "GENE_NAME_COLUMN = 'index'\n",
+    "\n",
+    "print({\n",
+    "    'seed': SEED, 'epochs': EPOCHS, 'patience': PATIENCE,\n",
+    "    'max_cells': MAX_CELLS, 'batch_size': BATCH_SIZE,\n",
+    "    'eval_batch_size': EVAL_BATCH_SIZE, 'grad_accum_steps': GRAD_ACCUM_STEPS,\n",
+    "    'label_column': LABEL_COLUMN, 'gene_name_column': GENE_NAME_COLUMN,\n",
+    "})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cmd = [\n",
+    "    sys.executable, str(SCRIPT),\n",
+    "    '--device', 'cuda',\n",
+    "    '--seed', str(SEED),\n",
+    "    '--data_path', str(DATA_PATH),\n",
+    "    '--label_column', LABEL_COLUMN,\n",
+    "    '--gene_name_column', GENE_NAME_COLUMN,\n",
+    "    '--val_ratio', '0.1',\n",
+    "    '--test_ratio', '0.2',\n",
+    "    '--min_genes', '10',\n",
+    "    '--min_cells', '10',\n",
+    "    '--config_pkl', str(CONFIG_PKL),\n",
+    "    '--performer_checkpoint', str(CKPT_PATH),\n",
+    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
+    "    '--gene_list_path', str(GENE_LIST),\n",
+    "    '--gene_mapping_path', str(GENE_MAP),\n",
+    "    '--gene2vec_data_path', str(GENE2VEC),\n",
+    "    '--gene2ens_file', str(GENE_MAP),\n",
+    "    '--gene2ens_column', 'Ensembl gene ID',\n",
+    "    '--head_type', 'attn_pool',\n",
+    "    '--finetune_mode', 'full',\n",
+    "    '--cls_decoder_layers', '3',\n",
+    "    '--lr_head', LR_HEAD,\n",
+    "    '--lr_backbone', LR_BACKBONE,\n",
+    "    '--batch_size', str(BATCH_SIZE),\n",
+    "    '--eval_batch_size', str(EVAL_BATCH_SIZE),\n",
+    "    '--grad_accum_steps', str(GRAD_ACCUM_STEPS),\n",
+    "    '--epochs', str(EPOCHS),\n",
+    "    '--patience', str(PATIENCE),\n",
+    "    '--weight_decay', WEIGHT_DECAY,\n",
+    "    '--label_smoothing', '0',\n",
+    "    '--dropout', DROPOUT,\n",
+    "    '--output_dir', str(OUTPUT_PARENT),\n",
+    "    '--run_name', RUN_NAME,\n",
+    "    '--save_test_artifacts',\n",
+    "]\n",
+    "if MAX_CELLS is not None:\n",
+    "    cmd.extend(['--max_cells', str(MAX_CELLS)])\n",
+    "print('Command:')\n",
+    "print(subprocess.list2cmdline(cmd) if os.name == 'nt' else shlex.join(cmd))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log_path = RUN_DIR / 'training.log'\n",
+    "if not RUN_TRAINING:\n",
+    "    print('Training is disabled. Review the paths and settings, then set RUN_TRAINING = True to launch.')\n",
+    "else:\n",
+    "    RUN_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    Path(run_env['NUMBA_CACHE_DIR']).mkdir(parents=True, exist_ok=True)\n",
+    "    print('Writing log to:', log_path)\n",
+    "    with log_path.open('w', encoding='utf-8') as log:\n",
+    "        process = subprocess.Popen(\n",
+    "            cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n",
+    "            text=True, encoding='utf-8', errors='replace', bufsize=1,\n",
+    "            cwd=str(PROJECT_ROOT), env=run_env,\n",
+    "        )\n",
+    "        for line in process.stdout:\n",
+    "            print(line, end='')\n",
+    "            log.write(line)\n",
+    "        return_code = process.wait()\n",
+    "    if return_code != 0:\n",
+    "        raise RuntimeError(f'Cell annotation failed with return code {return_code}; see {log_path}')\n",
+    "    print('Training finished.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inspect outputs\n",
+    "\n",
+    "A successful run saves `best_model.pt` and `run_config.json`. With `--save_test_artifacts`, it also writes `metrics_summary.json`, `test_classification_report.txt`, and confusion matrices in TXT, NPY, and CSV formats. The notebook captures the training console output in `training.log`. These files are created in the run directory; the input BioMod model weights are not overwritten. Result cells also work when training is disabled and previously generated outputs are available."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if not RUN_DIR.is_dir():\n",
+    "    print('No run directory yet:', RUN_DIR)\n",
+    "else:\n",
+    "    for path in sorted(RUN_DIR.iterdir()):\n",
+    "        size_mb = path.stat().st_size / 1024 / 1024 if path.is_file() else 0\n",
+    "        print(f'{path.name:45s} {size_mb:8.2f} MB')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics_path = RUN_DIR / 'metrics_summary.json'\n",
+    "if metrics_path.is_file():\n",
+    "    with metrics_path.open('r', encoding='utf-8') as handle:\n",
+    "        metrics_summary = json.load(handle)\n",
+    "    display(metrics_summary)\n",
+    "else:\n",
+    "    print('Metrics are not available yet:', metrics_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "report_path = RUN_DIR / 'test_classification_report.txt'\n",
+    "if report_path.is_file():\n",
+    "    print(report_path.read_text(encoding='utf-8'))\n",
+    "else:\n",
+    "    print('Classification report is not available yet:', report_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "conf_path = RUN_DIR / 'test_confusion_matrix_normalized.csv'\n",
+    "if conf_path.is_file():\n",
+    "    display(pd.read_csv(conf_path, index_col=0))\n",
+    "else:\n",
+    "    print('Confusion matrix is not available yet:', conf_path)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

BioMod/toturials/drug_combination.ipynb

@@ -1,754 +1,673 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Drug combination synergy prediction demo (PerformerModule weight0407)\n",
-    "\n",
-    "This notebook prepares a runnable demo command for `drug_combination/train_drug_combination_performer_module.py` using configurable absolute paths.\n",
-    "\n",
-    "The drug-combination data are not present in the current project yet, so `RUN_TRAINING` is set to `False` by default. After placing `data_summary/` under the task directory, switch `RUN_TRAINING = True` in the command cell to start training.\n",
-    "\n",
-    "The validation set is used for checkpoint selection. Reported metrics describe\n",
-    "validation performance; this workflow does not produce a separate final test evaluation.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Expected data layout\n",
-    "\n",
-    "Place the drug-combination dataset here:\n",
-    "\n",
-    "```text\n",
-    "$PERFORMER_MODULE_ROOT/drug_combination/data_summary/\n",
-    "|-- new_labels_0_10.csv\n",
-    "|-- new_cell_features_954.csv\n",
-    "|-- smiles_0_10.csv\n",
-    "`-- test_data/\n",
-    "    |-- independent_input.csv\n",
-    "    `-- independent_cell_features_954.csv\n",
-    "```\n",
-    "\n",
-    "The default demo follows the reported single-fold/domain-generalization setting: `epochs=500`, `lr=1e-4`, `batch_size=256`, `n_folds=1`, and `pool_method=max`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "import importlib.util\n",
-    "import os\n",
-    "import shlex\n",
-    "import subprocess\n",
-    "import sys\n",
-    "\n",
-    "import pandas as pd\n",
-    "import numpy as np\n",
-    "\n",
-    "try:\n",
-    "    from IPython.display import Image, display\n",
-    "except Exception:\n",
-    "    Image = None\n",
-    "    def display(obj):\n",
-    "        print(obj)\n",
-    "\n",
-    "PROJECT_ROOT = Path(os.environ.get('PERFORMER_MODULE_ROOT', Path.cwd().parent if Path.cwd().name == 'toturials' else Path.cwd())).resolve()\n",
-    "TASK_ROOT = PROJECT_ROOT / 'drug_combination'\n",
-    "TRAIN_SCRIPT = TASK_ROOT / 'train_drug_combination_performer_module.py'\n",
-    "\n",
-    "# Shared pretrained assets. Set BIOMODULEFM_ASSET_ROOT to the directory that contains\n",
-    "# weight/0407weight and module_model/.\n",
-    "ASSET_ROOT = Path(os.environ.get('BIOMODULEFM_ASSET_ROOT', '/data/share/biomodulefm_demo/single_cell_model'))\n",
-    "LEGACY_MODULE_MODEL = ASSET_ROOT / 'module_model'\n",
-    "WEIGHT_DIR = ASSET_ROOT / 'weight' / '0407weight'\n",
-    "\n",
-    "CHECKPOINT_PATH = WEIGHT_DIR / 'mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth'\n",
-    "HYPER_PARAMS_PATH = WEIGHT_DIR / 'mae_tost_48M_cross_attn_only_pure_memory.pkl'\n",
-    "MODULE_DATA_DIR = PROJECT_ROOT / 'module_data_all'\n",
-    "GENE_LIST_PATH = LEGACY_MODULE_MODEL / 'selected_genes_27k.txt'\n",
-    "GENE_MAPPING_PATH = LEGACY_MODULE_MODEL / 'gene2ens.csv'\n",
-    "GENE2VEC_PATH = LEGACY_MODULE_MODEL / 'selected_gene2vec_27k.npy'\n",
-    "SYMBOL2ENS_PATH = LEGACY_MODULE_MODEL / 'gene2ens.csv'\n",
-    "PERFORMER_CONT_ROOT = LEGACY_MODULE_MODEL / 'performer_pytorch_cont'\n",
-    "\n",
-    "DATA_DIR = TASK_ROOT / 'data_summary'\n",
-    "RUN_ROOT = Path('/data/share/biomodulefm_demo/tutorial_runs/drug_combination_weight0407')\n",
-    "OUTPUT_DIR = RUN_ROOT / 'results'\n",
-    "\n",
-    "TRAIN_PYTHON = Path('/opt/conda/envs/sclong/bin/python')\n",
-    "if not TRAIN_PYTHON.exists():\n",
-    "    TRAIN_PYTHON = Path(sys.executable)\n",
-    "\n",
-    "for path in [PROJECT_ROOT, LEGACY_MODULE_MODEL, PERFORMER_CONT_ROOT]:\n",
-    "    if path.exists() and str(path) not in sys.path:\n",
-    "        sys.path.insert(0, str(path))\n",
-    "\n",
-    "pythonpath_parts = [\n",
-    "    str(PROJECT_ROOT),\n",
-    "    str(LEGACY_MODULE_MODEL),\n",
-    "    str(PERFORMER_CONT_ROOT),\n",
-    "    os.environ.get('PYTHONPATH', ''),\n",
-    "]\n",
-    "PYTHONPATH = ':'.join([part for part in pythonpath_parts if part])\n",
-    "\n",
-    "print('project root :', PROJECT_ROOT)\n",
-    "print('task root    :', TASK_ROOT)\n",
-    "print('train script :', TRAIN_SCRIPT)\n",
-    "print('python       :', TRAIN_PYTHON)\n",
-    "print('data dir     :', DATA_DIR)\n",
-    "print('output dir   :', OUTPUT_DIR)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "required_code_and_weight_files = {\n",
-    "    'train_script': TRAIN_SCRIPT,\n",
-    "    'checkpoint_path': CHECKPOINT_PATH,\n",
-    "    'hyper_params_path': HYPER_PARAMS_PATH,\n",
-    "    'module_data_dir': MODULE_DATA_DIR,\n",
-    "    'gene_list_path': GENE_LIST_PATH,\n",
-    "    'gene_mapping_path': GENE_MAPPING_PATH,\n",
-    "    'gene2vec_path': GENE2VEC_PATH,\n",
-    "    'symbol2ens_path': SYMBOL2ENS_PATH,\n",
-    "    'performer_pytorch_cont': PERFORMER_CONT_ROOT,\n",
-    "}\n",
-    "\n",
-    "expected_data_files = {\n",
-    "    'train_labels': DATA_DIR / 'new_labels_0_10.csv',\n",
-    "    'train_cell_features': DATA_DIR / 'new_cell_features_954.csv',\n",
-    "    'drug_smiles': DATA_DIR / 'smiles_0_10.csv',\n",
-    "    'validation_labels': DATA_DIR / 'test_data' / 'independent_input.csv',\n",
-    "    'validation_cell_features': DATA_DIR / 'test_data' / 'independent_cell_features_954.csv',\n",
-    "}\n",
-    "\n",
-    "def status_table(paths):\n",
-    "    return pd.DataFrame(\n",
-    "        [\n",
-    "            {'name': name, 'path': str(path), 'exists': path.exists()}\n",
-    "            for name, path in paths.items()\n",
-    "        ]\n",
-    "    )\n",
-    "\n",
-    "code_weight_status = status_table(required_code_and_weight_files)\n",
-    "data_status = status_table(expected_data_files)\n",
-    "DATA_READY = bool(data_status['exists'].all())\n",
-    "\n",
-    "display(code_weight_status)\n",
-    "display(data_status)\n",
-    "print('DATA_READY =', DATA_READY)\n",
-    "\n",
-    "missing_required = code_weight_status.loc[~code_weight_status['exists'], 'path'].tolist()\n",
-    "if missing_required:\n",
-    "    raise FileNotFoundError('Missing required code/weight assets:\\n' + '\\n'.join(missing_required))\n",
-    "\n",
-    "if not DATA_READY:\n",
-    "    print('Data files are not complete yet. This notebook will not launch training until RUN_TRAINING=True and DATA_READY=True.')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "packages = {\n",
-    "    'torch': 'torch',\n",
-    "    'torch_geometric': 'torch_geometric',\n",
-    "    'rdkit': 'rdkit',\n",
-    "    'networkx': 'networkx',\n",
-    "    'sklearn': 'scikit-learn',\n",
-    "    'pandas': 'pandas',\n",
-    "    'numpy': 'numpy',\n",
-    "}\n",
-    "\n",
-    "dependency_rows = []\n",
-    "for module_name, package_name in packages.items():\n",
-    "    dependency_rows.append(\n",
-    "        {\n",
-    "            'module': module_name,\n",
-    "            'package': package_name,\n",
-    "            'available': importlib.util.find_spec(module_name) is not None,\n",
-    "        }\n",
-    "    )\n",
-    "\n",
-    "dependency_status = pd.DataFrame(dependency_rows)\n",
-    "display(dependency_status)\n",
-    "\n",
-    "missing_packages = dependency_status.loc[~dependency_status['available'], 'package'].tolist()\n",
-    "if missing_packages:\n",
-    "    print('Install missing packages in the sclong environment, for example:')\n",
-    "    print('python -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple ' + ' '.join(missing_packages))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Training parameters matching the previous reported command.\n",
-    "EPOCHS = 500\n",
-    "BATCH_SIZE = 256\n",
-    "LR = 1e-4\n",
-    "N_FOLDS = 1\n",
-    "POOL_METHOD = 'max'\n",
-    "EVAL_INTERVAL = 5\n",
-    "SEED = 42\n",
-    "\n",
-    "# Domain-generalization mode is the script default. Set to False for k-fold CV.\n",
-    "DOMAIN_GENERALIZATION = True\n",
-    "FORCE_PREPROCESS = False\n",
-    "\n",
-    "# Keep False while data_summary/ is absent.\n",
-    "RUN_TRAINING = False\n",
-    "\n",
-    "cmd = [\n",
-    "    str(TRAIN_PYTHON), str(TRAIN_SCRIPT),\n",
-    "    '--checkpoint_path', str(CHECKPOINT_PATH),\n",
-    "    '--hyper_params_path', str(HYPER_PARAMS_PATH),\n",
-    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
-    "    '--gene_list_path', str(GENE_LIST_PATH),\n",
-    "    '--gene_mapping_path', str(GENE_MAPPING_PATH),\n",
-    "    '--gene2vec_path', str(GENE2VEC_PATH),\n",
-    "    '--symbol2ens_path', str(SYMBOL2ENS_PATH),\n",
-    "    '--data_dir', str(DATA_DIR),\n",
-    "    '--output_dir', str(OUTPUT_DIR),\n",
-    "    '--epochs', str(EPOCHS),\n",
-    "    '--batch_size', str(BATCH_SIZE),\n",
-    "    '--lr', str(LR),\n",
-    "    '--n_folds', str(N_FOLDS),\n",
-    "    '--pool_method', POOL_METHOD,\n",
-    "    '--eval_interval', str(EVAL_INTERVAL),\n",
-    "    '--seed', str(SEED),\n",
-    "]\n",
-    "\n",
-    "if not DOMAIN_GENERALIZATION:\n",
-    "    cmd.append('--no_domain_gen')\n",
-    "if FORCE_PREPROCESS:\n",
-    "    cmd.append('--force_preprocess')\n",
-    "\n",
-    "print(' '.join(shlex.quote(x) for x in cmd))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "env = os.environ.copy()\n",
-    "env['PYTHONPATH'] = PYTHONPATH\n",
-    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "if RUN_TRAINING:\n",
-    "    if not DATA_READY:\n",
-    "        missing_data = data_status.loc[~data_status['exists'], 'path'].tolist()\n",
-    "        raise FileNotFoundError('Data are incomplete:\\n' + '\\n'.join(missing_data))\n",
-    "    result = subprocess.run(\n",
-    "        cmd,\n",
-    "        cwd=str(TASK_ROOT),\n",
-    "        env=env,\n",
-    "        text=True,\n",
-    "        stdout=subprocess.PIPE,\n",
-    "        stderr=subprocess.STDOUT,\n",
-    "        check=False,\n",
-    "    )\n",
-    "    print(result.stdout)\n",
-    "    if result.returncode != 0:\n",
-    "        raise RuntimeError(f'Training failed with return code {result.returncode}')\n",
-    "else:\n",
-    "    print('RUN_TRAINING is False, so no training/smokecheck was launched.')\n",
-    "    print('Set RUN_TRAINING=True after data_summary/ is available.')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "if OUTPUT_DIR.exists():\n",
-    "    output_files = sorted(OUTPUT_DIR.glob('*'))\n",
-    "    display(pd.DataFrame(\n",
-    "        [\n",
-    "            {\n",
-    "                'file': f.name,\n",
-    "                'size_mb': round(f.stat().st_size / 1024 / 1024, 3),\n",
-    "                'path': str(f),\n",
-    "            }\n",
-    "            for f in output_files\n",
-    "        ]\n",
-    "    ))\n",
-    "else:\n",
-    "    print('No output directory yet:', OUTPUT_DIR)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Output files\n",
-    "\n",
-    "When training finishes, the script writes results under `OUTPUT_DIR`, including:\n",
-    "\n",
-    "- `best_model_fold{i}.pt`\n",
-    "- `loss_fold{i}.csv`\n",
-    "- `loss_curve_fold{i}.png`\n",
-    "- `metrics_fold{i}.csv`\n",
-    "- `roc_curve_fold{i}.png`\n",
-    "- `fpr_tpr_fold{i}.pkl`\n",
-    "- `cell_emb_cache_{pool_method}.npy`\n",
-    "\n",
-    "The plotting cells below regenerate publication-friendly PDF/PNG/SVG summaries from these files and also support older DeepDDS-style prediction CSV files when they are present. The README notes a previous reference setting of `lr=1e-4`, `epochs=500`, `pool_method=max`, `n_folds=1`, with reported AUROC around `0.6824` for the old run.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Data overview plots\n",
-    "\n",
-    "These plots summarize the drug-combination dataset before training. They are useful for quick notebook demos and GitHub documentation because they do not require a completed model run.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import matplotlib\n",
-    "matplotlib.use('Agg')\n",
-    "import matplotlib.pyplot as plt\n",
-    "from matplotlib.ticker import MaxNLocator\n",
-    "\n",
-    "PLOT_DIR = OUTPUT_DIR / 'notebook_plots'\n",
-    "PLOT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "plt.rcParams.update({\n",
-    "    'font.family': 'sans-serif',\n",
-    "    'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
-    "    'font.size': 7,\n",
-    "    'axes.labelsize': 7,\n",
-    "    'axes.titlesize': 8,\n",
-    "    'xtick.labelsize': 6,\n",
-    "    'ytick.labelsize': 6,\n",
-    "    'legend.fontsize': 6,\n",
-    "    'axes.linewidth': 0.7,\n",
-    "    'pdf.fonttype': 42,\n",
-    "    'ps.fonttype': 42,\n",
-    "    'svg.fonttype': 'none',\n",
-    "})\n",
-    "\n",
-    "\n",
-    "def save_fig(fig, stem: str, dpi: int = 600, show: bool = True):\n",
-    "    png_path = PLOT_DIR / f'{stem}.png'\n",
-    "    pdf_path = PLOT_DIR / f'{stem}.pdf'\n",
-    "    fig.savefig(png_path, dpi=dpi, bbox_inches='tight')\n",
-    "    fig.savefig(pdf_path, bbox_inches='tight')\n",
-    "    plt.close(fig)\n",
-    "    print('Saved:', png_path)\n",
-    "    print('Saved:', pdf_path)\n",
-    "    if show and Image is not None:\n",
-    "        display(Image(filename=str(png_path)))\n",
-    "    return png_path, pdf_path\n",
-    "\n",
-    "\n",
-    "def despine(ax):\n",
-    "    ax.spines['top'].set_visible(False)\n",
-    "    ax.spines['right'].set_visible(False)\n",
-    "    ax.tick_params(width=0.7, length=2.5)\n",
-    "\n",
-    "\n",
-    "def read_label_table(path: Path) -> pd.DataFrame:\n",
-    "    df = pd.read_csv(path)\n",
-    "    lower_map = {str(col).lower(): col for col in df.columns}\n",
-    "    for required in ['drug1', 'drug2', 'cell', 'label']:\n",
-    "        if required not in lower_map:\n",
-    "            raise ValueError(f'{path} is missing a required column similar to {required!r}. Columns: {list(df.columns)}')\n",
-    "    df = df.rename(columns={lower_map['drug1']: 'drug1', lower_map['drug2']: 'drug2', lower_map['cell']: 'cell', lower_map['label']: 'label'})\n",
-    "    return df[['drug1', 'drug2', 'cell', 'label']].copy()\n",
-    "\n",
-    "\n",
-    "data_overview = []\n",
-    "train_labels = validation_labels = None\n",
-    "if expected_data_files['train_labels'].exists():\n",
-    "    train_labels = read_label_table(expected_data_files['train_labels'])\n",
-    "    data_overview.append({\n",
-    "        'split': 'train',\n",
-    "        'rows': len(train_labels),\n",
-    "        'positive': int((train_labels['label'] == 1).sum()),\n",
-    "        'negative': int((train_labels['label'] == 0).sum()),\n",
-    "        'unique_cells': train_labels['cell'].nunique(),\n",
-    "        'unique_drug1': train_labels['drug1'].nunique(),\n",
-    "        'unique_drug2': train_labels['drug2'].nunique(),\n",
-    "        'unique_pairs': train_labels[['drug1', 'drug2']].drop_duplicates().shape[0],\n",
-    "    })\n",
-    "if expected_data_files['validation_labels'].exists():\n",
-    "    validation_labels = read_label_table(expected_data_files['validation_labels'])\n",
-    "    data_overview.append({\n",
-    "        'split': 'validation',\n",
-    "        'rows': len(validation_labels),\n",
-    "        'positive': int((validation_labels['label'] == 1).sum()),\n",
-    "        'negative': int((validation_labels['label'] == 0).sum()),\n",
-    "        'unique_cells': validation_labels['cell'].nunique(),\n",
-    "        'unique_drug1': validation_labels['drug1'].nunique(),\n",
-    "        'unique_drug2': validation_labels['drug2'].nunique(),\n",
-    "        'unique_pairs': validation_labels[['drug1', 'drug2']].drop_duplicates().shape[0],\n",
-    "    })\n",
-    "\n",
-    "data_overview_df = pd.DataFrame(data_overview)\n",
-    "if data_overview_df.empty:\n",
-    "    print('No label files found yet, so data overview plots were skipped.')\n",
-    "else:\n",
-    "    data_overview_df.to_csv(PLOT_DIR / 'drug_combination_data_overview.csv', index=False)\n",
-    "    display(data_overview_df)\n",
-    "\n",
-    "    label_rows = []\n",
-    "    if train_labels is not None:\n",
-    "        label_rows.extend([{'split': 'train', 'label': int(k), 'count': int(v)} for k, v in train_labels['label'].value_counts().sort_index().items()])\n",
-    "    if validation_labels is not None:\n",
-    "        label_rows.extend([{'split': 'validation', 'label': int(k), 'count': int(v)} for k, v in validation_labels['label'].value_counts().sort_index().items()])\n",
-    "    label_df = pd.DataFrame(label_rows)\n",
-    "    label_df.to_csv(PLOT_DIR / 'drug_combination_label_counts.csv', index=False)\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(3.5, 2.4))\n",
-    "    splits = list(label_df['split'].drop_duplicates())\n",
-    "    x = np.arange(len(splits))\n",
-    "    width = 0.34\n",
-    "    colors = {0: '#4E79A7', 1: '#E15759'}\n",
-    "    for offset, label in zip([-width / 2, width / 2], [0, 1]):\n",
-    "        values = [int(label_df[(label_df['split'] == split) & (label_df['label'] == label)]['count'].sum()) for split in splits]\n",
-    "        ax.bar(x + offset, values, width=width, color=colors[label], edgecolor='#333333', linewidth=0.4, label=f'label {label}')\n",
-    "        for xpos, value in zip(x + offset, values):\n",
-    "            ax.text(xpos, value, str(value), ha='center', va='bottom', fontsize=6)\n",
-    "    ax.set_xticks(x)\n",
-    "    ax.set_xticklabels(splits, rotation=15, ha='right')\n",
-    "    ax.set_ylabel('Number of combinations')\n",
-    "    ax.set_title('Drug-combination label balance')\n",
-    "    ax.legend(frameon=False)\n",
-    "    ax.yaxis.set_major_locator(MaxNLocator(integer=True))\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_label_balance')\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(4.1, 2.5))\n",
-    "    overview_long = data_overview_df.melt(id_vars='split', value_vars=['unique_cells', 'unique_drug1', 'unique_drug2', 'unique_pairs'], var_name='feature', value_name='count')\n",
-    "    features = ['unique_cells', 'unique_drug1', 'unique_drug2', 'unique_pairs']\n",
-    "    x = np.arange(len(features))\n",
-    "    width = 0.36 if len(splits) > 1 else 0.5\n",
-    "    palette = ['#59A14F', '#F28E2B']\n",
-    "    for i, split in enumerate(splits):\n",
-    "        values = [int(overview_long[(overview_long['split'] == split) & (overview_long['feature'] == feature)]['count'].sum()) for feature in features]\n",
-    "        ax.bar(x + (i - (len(splits) - 1) / 2) * width, values, width=width, color=palette[i % len(palette)], edgecolor='#333333', linewidth=0.4, label=split)\n",
-    "    ax.set_xticks(x)\n",
-    "    ax.set_xticklabels(['cells', 'drug1', 'drug2', 'drug pairs'], rotation=20, ha='right')\n",
-    "    ax.set_ylabel('Unique count')\n",
-    "    ax.set_title('Dataset coverage')\n",
-    "    ax.legend(frameon=False)\n",
-    "    ax.yaxis.set_major_locator(MaxNLocator(integer=True))\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_dataset_coverage')\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Result plots from training outputs\n",
-    "\n",
-    "This section reuses the evaluation signals from the training script and the older DeepDDS/scLong plotting convention: AUROC, PR-AUC, accuracy, balanced accuracy, precision, TPR, kappa, recall, ROC curves, and confusion matrices when prediction CSV files are available.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import pickle\n",
-    "from sklearn import metrics\n",
-    "from sklearn.metrics import confusion_matrix\n",
-    "\n",
-    "\n",
-    "def find_fold_files(pattern: str):\n",
-    "    return sorted(OUTPUT_DIR.glob(pattern))\n",
-    "\n",
-    "\n",
-    "def load_metrics_files():\n",
-    "    frames = []\n",
-    "    for path in find_fold_files('metrics_fold*.csv'):\n",
-    "        df = pd.read_csv(path)\n",
-    "        stem = path.stem.replace('metrics_fold', '')\n",
-    "        try:\n",
-    "            fold = int(stem)\n",
-    "        except ValueError:\n",
-    "            fold = len(frames)\n",
-    "        df['fold'] = fold\n",
-    "        df['source_file'] = str(path)\n",
-    "        frames.append(df)\n",
-    "    return pd.concat(frames, ignore_index=True) if frames else pd.DataFrame()\n",
-    "\n",
-    "\n",
-    "def load_loss_files():\n",
-    "    frames = []\n",
-    "    for path in find_fold_files('loss_fold*.csv'):\n",
-    "        df = pd.read_csv(path)\n",
-    "        stem = path.stem.replace('loss_fold', '')\n",
-    "        try:\n",
-    "            fold = int(stem)\n",
-    "        except ValueError:\n",
-    "            fold = len(frames)\n",
-    "        df['fold'] = fold\n",
-    "        frames.append(df)\n",
-    "    return pd.concat(frames, ignore_index=True) if frames else pd.DataFrame()\n",
-    "\n",
-    "\n",
-    "metrics_df = load_metrics_files()\n",
-    "loss_df = load_loss_files()\n",
-    "\n",
-    "if metrics_df.empty:\n",
-    "    print('No metrics_fold*.csv files found yet:', OUTPUT_DIR)\n",
-    "else:\n",
-    "    metric_cols = [col for col in ['AUC', 'PR_AUC', 'ACC', 'BACC', 'PREC', 'TPR', 'KAPPA', 'RECALL'] if col in metrics_df.columns]\n",
-    "    best_rows = metrics_df.sort_values(['fold', 'AUC'], ascending=[True, False]).groupby('fold', as_index=False).head(1)\n",
-    "    summary_rows = []\n",
-    "    for metric in metric_cols:\n",
-    "        values = best_rows[metric].astype(float).to_numpy()\n",
-    "        summary_rows.append({'metric': metric, 'mean': float(np.mean(values)), 'std': float(np.std(values)), 'n_folds': int(len(values))})\n",
-    "    summary_df = pd.DataFrame(summary_rows)\n",
-    "    best_rows.to_csv(PLOT_DIR / 'drug_combination_best_metrics_by_fold.csv', index=False)\n",
-    "    summary_df.to_csv(PLOT_DIR / 'drug_combination_metric_summary.csv', index=False)\n",
-    "    display(best_rows)\n",
-    "    display(summary_df)\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(4.1, 2.6))\n",
-    "    for metric, color in [('AUC', '#4E79A7'), ('PR_AUC', '#E15759'), ('ACC', '#59A14F'), ('BACC', '#F28E2B')]:\n",
-    "        if metric in metrics_df.columns:\n",
-    "            for fold, frame in metrics_df.groupby('fold'):\n",
-    "                ax.plot(frame['epoch'], frame[metric], linewidth=0.8, alpha=0.65, color=color)\n",
-    "            mean_by_epoch = metrics_df.groupby('epoch')[metric].mean().reset_index()\n",
-    "            ax.plot(mean_by_epoch['epoch'], mean_by_epoch[metric], linewidth=1.8, color=color, label=metric)\n",
-    "    ax.set_xlabel('Epoch')\n",
-    "    ax.set_ylabel('Score')\n",
-    "    ax.set_ylim(0, 1.02)\n",
-    "    ax.set_title('Validation metrics over training')\n",
-    "    ax.legend(frameon=False, ncol=2)\n",
-    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
-    "    ax.set_axisbelow(True)\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_metric_curves')\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(3.8, 2.5))\n",
-    "    plot_metrics = [m for m in ['AUC', 'PR_AUC', 'ACC', 'BACC', 'PREC', 'TPR', 'KAPPA', 'RECALL'] if m in best_rows.columns]\n",
-    "    values = [float(best_rows[m].mean()) for m in plot_metrics]\n",
-    "    colors = ['#4E79A7', '#E15759', '#59A14F', '#F28E2B', '#76B7B2', '#B07AA1', '#9C755F', '#BAB0AC'][:len(plot_metrics)]\n",
-    "    bars = ax.bar(np.arange(len(plot_metrics)), values, color=colors, edgecolor='#333333', linewidth=0.4)\n",
-    "    for bar, value in zip(bars, values):\n",
-    "        ax.text(bar.get_x() + bar.get_width() / 2, value + 0.015, f'{value:.3f}', ha='center', va='bottom', fontsize=6)\n",
-    "    ax.set_xticks(np.arange(len(plot_metrics)))\n",
-    "    ax.set_xticklabels(plot_metrics, rotation=25, ha='right')\n",
-    "    ax.set_ylim(0, 1.05)\n",
-    "    ax.set_ylabel('Best-fold score')\n",
-    "    ax.set_title('Drug-combination prediction summary')\n",
-    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
-    "    ax.set_axisbelow(True)\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_best_metric_summary')\n",
-    "\n",
-    "if loss_df.empty:\n",
-    "    print('No loss_fold*.csv files found yet:', OUTPUT_DIR)\n",
-    "else:\n",
-    "    fig, ax = plt.subplots(figsize=(4.1, 2.5))\n",
-    "    for fold, frame in loss_df.groupby('fold'):\n",
-    "        ax.plot(frame['epoch'], frame['avg_loss'], linewidth=0.8, alpha=0.65, label=f'fold {fold}')\n",
-    "    mean_loss = loss_df.groupby('epoch')['avg_loss'].mean().reset_index()\n",
-    "    ax.plot(mean_loss['epoch'], mean_loss['avg_loss'], linewidth=1.8, color='#222222', label='mean')\n",
-    "    ax.set_xlabel('Epoch')\n",
-    "    ax.set_ylabel('Average loss')\n",
-    "    ax.set_title('Training loss')\n",
-    "    ax.legend(frameon=False, ncol=2)\n",
-    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
-    "    ax.set_axisbelow(True)\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_loss_curves')\n",
-    "\n",
-    "roc_files = find_fold_files('fpr_tpr_fold*.pkl')\n",
-    "if not roc_files:\n",
-    "    print('No fpr_tpr_fold*.pkl files found yet:', OUTPUT_DIR)\n",
-    "else:\n",
-    "    fig, ax = plt.subplots(figsize=(3.2, 3.2))\n",
-    "    roc_rows = []\n",
-    "    for path in roc_files:\n",
-    "        with path.open('rb') as handle:\n",
-    "            fpr, tpr = pickle.load(handle)\n",
-    "        stem = path.stem.replace('fpr_tpr_fold', '')\n",
-    "        try:\n",
-    "            fold = int(stem)\n",
-    "        except ValueError:\n",
-    "            fold = len(roc_rows)\n",
-    "        auc_value = float(metrics.auc(fpr, tpr))\n",
-    "        roc_rows.append({'fold': fold, 'auc': auc_value})\n",
-    "        ax.plot(fpr, tpr, linewidth=1.0, label=f'fold {fold}, AUROC={auc_value:.3f}')\n",
-    "    ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
-    "    ax.set_xlabel('False positive rate')\n",
-    "    ax.set_ylabel('True positive rate')\n",
-    "    ax.set_xlim(0, 1)\n",
-    "    ax.set_ylim(0, 1.02)\n",
-    "    ax.set_aspect('equal', adjustable='box')\n",
-    "    ax.legend(frameon=False, loc='lower right')\n",
-    "    ax.set_title('ROC curve')\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_roc_curves')\n",
-    "    pd.DataFrame(roc_rows).to_csv(PLOT_DIR / 'drug_combination_roc_auc_from_curves.csv', index=False)\n",
-    "\n",
-    "\n",
-    "def load_legacy_prediction_csv(path: Path):\n",
-    "    raw = pd.read_csv(path, header=None)\n",
-    "    arr = raw.to_numpy()\n",
-    "    candidates = []\n",
-    "    # Old DeepDDS files are often saved as rows: labels, predicted labels, scores or index, labels, predicted labels, scores.\n",
-    "    if arr.shape[0] >= 3:\n",
-    "        candidates.append(('row_0_2', arr[0], None, arr[2]))\n",
-    "    if arr.shape[0] >= 4:\n",
-    "        candidates.append(('row_1_3', arr[1], arr[2], arr[3]))\n",
-    "    if arr.shape[1] >= 3:\n",
-    "        candidates.append(('col_0_2', arr[:, 0], None, arr[:, 2]))\n",
-    "    if arr.shape[1] >= 4:\n",
-    "        candidates.append(('col_1_3', arr[:, 1], arr[:, 2], arr[:, 3]))\n",
-    "    for mode, labels, pred_labels, scores in candidates:\n",
-    "        labels = pd.to_numeric(pd.Series(labels), errors='coerce').dropna().to_numpy(dtype=float)\n",
-    "        scores = pd.to_numeric(pd.Series(scores), errors='coerce').dropna().to_numpy(dtype=float)\n",
-    "        if len(labels) != len(scores) or len(labels) < 2:\n",
-    "            continue\n",
-    "        unique_labels = set(np.unique(labels).astype(int).tolist())\n",
-    "        if unique_labels.issubset({0, 1}) and len(unique_labels) == 2:\n",
-    "            if pred_labels is None or len(pred_labels) != len(labels):\n",
-    "                pred_labels = (scores >= 0.5).astype(int)\n",
-    "            else:\n",
-    "                pred_labels = pd.to_numeric(pd.Series(pred_labels), errors='coerce').fillna(0).to_numpy(dtype=float).astype(int)\n",
-    "            return labels.astype(int), scores.astype(float), pred_labels.astype(int), mode\n",
-    "    return None\n",
-    "\n",
-    "legacy_files = sorted(OUTPUT_DIR.glob('*result*.csv')) + sorted(OUTPUT_DIR.glob('*ROC*.csv'))\n",
-    "legacy_records = []\n",
-    "for path in legacy_files:\n",
-    "    loaded = load_legacy_prediction_csv(path)\n",
-    "    if loaded is None:\n",
-    "        continue\n",
-    "    labels, scores, pred_labels, mode = loaded\n",
-    "    fpr, tpr, _ = metrics.roc_curve(labels, scores)\n",
-    "    precision, recall, _ = metrics.precision_recall_curve(labels, scores)\n",
-    "    auc_value = float(metrics.auc(fpr, tpr))\n",
-    "    pr_auc = float(metrics.auc(recall, precision))\n",
-    "    cm = confusion_matrix(labels, pred_labels)\n",
-    "    legacy_records.append({'path': path, 'mode': mode, 'labels': labels, 'scores': scores, 'pred_labels': pred_labels, 'fpr': fpr, 'tpr': tpr, 'precision': precision, 'recall': recall, 'auc': auc_value, 'pr_auc': pr_auc, 'cm': cm})\n",
-    "\n",
-    "if not legacy_records:\n",
-    "    print('No legacy prediction CSV with labels/scores found. PR and confusion-matrix plots will be available after prediction CSV files are saved.')\n",
-    "else:\n",
-    "    legacy_summary = pd.DataFrame([{'file': r['path'].name, 'mode': r['mode'], 'auc': r['auc'], 'pr_auc': r['pr_auc'], 'n': len(r['labels'])} for r in legacy_records])\n",
-    "    legacy_summary.to_csv(PLOT_DIR / 'drug_combination_legacy_prediction_summary.csv', index=False)\n",
-    "    display(legacy_summary)\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(3.2, 3.2))\n",
-    "    for r in legacy_records:\n",
-    "        ax.plot(r['fpr'], r['tpr'], linewidth=1.0, label=f'{r[\"path\"].stem}, AUROC={r[\"auc\"]:.3f}')\n",
-    "    ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
-    "    ax.set_xlabel('False positive rate')\n",
-    "    ax.set_ylabel('True positive rate')\n",
-    "    ax.set_xlim(0, 1)\n",
-    "    ax.set_ylim(0, 1.02)\n",
-    "    ax.set_aspect('equal', adjustable='box')\n",
-    "    ax.legend(frameon=False, loc='lower right')\n",
-    "    ax.set_title('ROC from prediction CSV')\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_legacy_prediction_roc')\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(3.2, 3.0))\n",
-    "    for r in legacy_records:\n",
-    "        ax.plot(r['recall'], r['precision'], linewidth=1.0, label=f'{r[\"path\"].stem}, AUPR={r[\"pr_auc\"]:.3f}')\n",
-    "    ax.set_xlabel('Recall')\n",
-    "    ax.set_ylabel('Precision')\n",
-    "    ax.set_xlim(0, 1)\n",
-    "    ax.set_ylim(0, 1.02)\n",
-    "    ax.legend(frameon=False, loc='lower left')\n",
-    "    ax.set_title('Precision-recall curve')\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_legacy_prediction_pr_curve')\n",
-    "\n",
-    "    first = legacy_records[0]\n",
-    "    fig, ax = plt.subplots(figsize=(2.4, 2.3))\n",
-    "    im = ax.imshow(first['cm'], cmap='Blues')\n",
-    "    for row in range(first['cm'].shape[0]):\n",
-    "        for col in range(first['cm'].shape[1]):\n",
-    "            ax.text(col, row, int(first['cm'][row, col]), ha='center', va='center', fontsize=7)\n",
-    "    ax.set_xticks([0, 1])\n",
-    "    ax.set_yticks([0, 1])\n",
-    "    ax.set_xticklabels(['0', '1'])\n",
-    "    ax.set_yticklabels(['0', '1'])\n",
-    "    ax.set_xlabel('Predicted label')\n",
-    "    ax.set_ylabel('True label')\n",
-    "    ax.set_title('Confusion matrix')\n",
-    "    fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n",
-    "    fig.tight_layout()\n",
-    "    save_fig(fig, 'drug_combination_legacy_confusion_matrix')\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Generated plot files\n",
-    "\n",
-    "This cell lists the figures and source tables generated by the notebook plotting sections.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "plot_files = sorted(PLOT_DIR.glob('*'))\n",
-    "if plot_files:\n",
-    "    display(pd.DataFrame([\n",
-    "        {'file': path.name, 'size_kb': round(path.stat().st_size / 1024, 2), 'path': str(path)}\n",
-    "        for path in plot_files\n",
-    "    ]))\n",
-    "else:\n",
-    "    print('No notebook plot files were generated yet.')\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3 (sclong)",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "name": "python",
-   "pygments_lexer": "ipython3"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Drug combination synergy prediction with BioMod\n",
+    "\n",
+    "This notebook checks the bundled drug-combination inputs, prepares a command for\n",
+    "`BioMod/drug_combination/train_drug_combination_performer_module.py`, and plots\n",
+    "available training outputs. It uses the BioMod model weights and configuration.\n",
+    "\n",
+    "`RUN_TRAINING = False` by default. Set it to `True` only when you are ready to train\n",
+    "in the active notebook environment. The validation set selects the checkpoint;\n",
+    "this workflow does not produce a separate final test evaluation.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Project paths and dataset layout\n",
+    "\n",
+    "Start Jupyter from `BioMod/toturials/`. The path cell uses `Path('../..')` for the\n",
+    "repository root and resolves project-relative paths for the training subprocess.\n",
+    "If Jupyter starts at the repository root, use `Path('.')` instead. The dataset is\n",
+    "already included in the project:\n",
+    "\n",
+    "```text\n",
+    "data/downstream/final_data/drug_combination_synergy/data_summary/\n",
+    "|-- new_labels_0_10.csv\n",
+    "|-- new_cell_features_954.csv\n",
+    "|-- smiles_0_10.csv\n",
+    "`-- test_data/\n",
+    "    |-- independent_input.csv\n",
+    "    `-- independent_cell_features_954.csv\n",
+    "```\n",
+    "\n",
+    "The historical folder name `test_data` is retained; these files are used as the\n",
+    "validation set. The example uses the domain-generalization mode with one run,\n",
+    "500 epochs, a batch size of 256, and max pooling. These are example settings,\n",
+    "not a verified performance claim for the current weights.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import importlib.util\n",
+    "import os\n",
+    "import shlex\n",
+    "import subprocess\n",
+    "import sys\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "try:\n",
+    "    from IPython.display import Image, display\n",
+    "except ImportError:\n",
+    "    Image = None\n",
+    "    def display(obj):\n",
+    "        print(obj)\n",
+    "\n",
+    "# Start Jupyter with BioMod/toturials as the working directory.\n",
+    "# If Jupyter starts at the repository root, change this line to Path('.').resolve().\n",
+    "PROJECT_ROOT = Path('../..').resolve()\n",
+    "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
+    "if not (SOURCE_DIR / 'models_performer_module.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
+    "    raise FileNotFoundError(\n",
+    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        \"(use Path('.').resolve() if Jupyter was launched at that root).\"\n",
+    "    )\n",
+    "REPO_ROOT = PROJECT_ROOT\n",
+    "\n",
+    "DEPENDENCY_ROOT = REPO_ROOT / 'model_dependencies'\n",
+    "WEIGHT_DIR = REPO_ROOT / 'weight' / '0810model'\n",
+    "MODULE_DATA_DIR = DEPENDENCY_ROOT / 'module_data_all'\n",
+    "TRAIN_PYTHON = Path(sys.executable).resolve()\n",
+    "PYTHONPATH = os.pathsep.join(\n",
+    "    [str(SOURCE_DIR), str(DEPENDENCY_ROOT)]\n",
+    "    + ([os.environ['PYTHONPATH']] if os.environ.get('PYTHONPATH') else [])\n",
+    ")\n",
+    "\n",
+    "TASK_ROOT = SOURCE_DIR / 'drug_combination'\n",
+    "TRAIN_SCRIPT = TASK_ROOT / 'train_drug_combination_performer_module.py'\n",
+    "CHECKPOINT_PATH = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2_epoch0.pth'\n",
+    "HYPER_PARAMS_PATH = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2.pkl'\n",
+    "GENE_LIST_PATH = DEPENDENCY_ROOT / 'selected_genes_27k.txt'\n",
+    "GENE_MAPPING_PATH = DEPENDENCY_ROOT / 'gene2ens.csv'\n",
+    "GENE2VEC_PATH = DEPENDENCY_ROOT / 'selected_gene2vec_27k.npy'\n",
+    "SYMBOL2ENS_PATH = GENE_MAPPING_PATH\n",
+    "PERFORMER_CONT_ROOT = DEPENDENCY_ROOT / 'performer_pytorch_cont'\n",
+    "DATA_DIR = REPO_ROOT / 'data/downstream/final_data/drug_combination_synergy/data_summary'\n",
+    "RUN_ROOT = REPO_ROOT / 'tutorial_runs/drug_combination'\n",
+    "OUTPUT_DIR = RUN_ROOT / 'results'\n",
+    "\n",
+    "print('repository   :', REPO_ROOT)\n",
+    "print('train script :', TRAIN_SCRIPT)\n",
+    "print('python       :', TRAIN_PYTHON)\n",
+    "print('data dir     :', DATA_DIR)\n",
+    "print('output dir   :', OUTPUT_DIR)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "required_code_and_weight_files = {\n",
+    "    'train_script': TRAIN_SCRIPT,\n",
+    "    'checkpoint_path': CHECKPOINT_PATH,\n",
+    "    'hyper_params_path': HYPER_PARAMS_PATH,\n",
+    "    'module_data_dir': MODULE_DATA_DIR,\n",
+    "    'gene_list_path': GENE_LIST_PATH,\n",
+    "    'gene_mapping_path': GENE_MAPPING_PATH,\n",
+    "    'gene2vec_path': GENE2VEC_PATH,\n",
+    "    'symbol2ens_path': SYMBOL2ENS_PATH,\n",
+    "    'performer_pytorch_cont': PERFORMER_CONT_ROOT,\n",
+    "}\n",
+    "\n",
+    "expected_data_files = {\n",
+    "    'train_labels': DATA_DIR / 'new_labels_0_10.csv',\n",
+    "    'train_cell_features': DATA_DIR / 'new_cell_features_954.csv',\n",
+    "    'drug_smiles': DATA_DIR / 'smiles_0_10.csv',\n",
+    "    'validation_labels': DATA_DIR / 'test_data' / 'independent_input.csv',\n",
+    "    'validation_cell_features': DATA_DIR / 'test_data' / 'independent_cell_features_954.csv',\n",
+    "}\n",
+    "\n",
+    "def status_table(paths):\n",
+    "    return pd.DataFrame(\n",
+    "        [\n",
+    "            {'name': name, 'path': str(path), 'exists': path.exists()}\n",
+    "            for name, path in paths.items()\n",
+    "        ]\n",
+    "    )\n",
+    "\n",
+    "code_weight_status = status_table(required_code_and_weight_files)\n",
+    "data_status = status_table(expected_data_files)\n",
+    "DATA_READY = bool(data_status['exists'].all())\n",
+    "\n",
+    "display(code_weight_status)\n",
+    "display(data_status)\n",
+    "print('DATA_READY =', DATA_READY)\n",
+    "\n",
+    "missing_required = code_weight_status.loc[~code_weight_status['exists'], 'path'].tolist()\n",
+    "if missing_required:\n",
+    "    raise FileNotFoundError('Missing required code/weight assets:\\n' + '\\n'.join(missing_required))\n",
+    "\n",
+    "if not DATA_READY:\n",
+    "    print('Some dataset files are missing. This notebook will not launch training until RUN_TRAINING=True and DATA_READY=True.')\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "packages = {\n",
+    "    'torch': 'torch',\n",
+    "    'torch_geometric': 'torch_geometric',\n",
+    "    'rdkit': 'rdkit',\n",
+    "    'networkx': 'networkx',\n",
+    "    'sklearn': 'scikit-learn',\n",
+    "    'pandas': 'pandas',\n",
+    "    'numpy': 'numpy',\n",
+    "    'matplotlib': 'matplotlib',\n",
+    "    'scipy': 'scipy',\n",
+    "    'einops': 'einops',\n",
+    "    'local_attention': 'local-attention',\n",
+    "}\n",
+    "\n",
+    "dependency_rows = []\n",
+    "for module_name, package_name in packages.items():\n",
+    "    dependency_rows.append(\n",
+    "        {\n",
+    "            'module': module_name,\n",
+    "            'package': package_name,\n",
+    "            'available': importlib.util.find_spec(module_name) is not None,\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "dependency_status = pd.DataFrame(dependency_rows)\n",
+    "display(dependency_status)\n",
+    "\n",
+    "missing_packages = dependency_status.loc[~dependency_status['available'], 'package'].tolist()\n",
+    "if missing_packages:\n",
+    "    print('Install missing packages in the active notebook environment:')\n",
+    "    install_cmd = [str(TRAIN_PYTHON), '-m', 'pip', 'install', *missing_packages]\n",
+    "    print(subprocess.list2cmdline(install_cmd) if os.name == 'nt' else shlex.join(install_cmd))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example training parameters.\n",
+    "EPOCHS = 500\n",
+    "BATCH_SIZE = 256\n",
+    "LR = 1e-4\n",
+    "N_FOLDS = 1\n",
+    "POOL_METHOD = 'max'\n",
+    "EVAL_INTERVAL = 5\n",
+    "SEED = 42\n",
+    "\n",
+    "# Domain-generalization mode is the script default. Set to False for k-fold CV.\n",
+    "DOMAIN_GENERALIZATION = True\n",
+    "FORCE_PREPROCESS = False\n",
+    "\n",
+    "# Keep False to inspect inputs and commands without starting training.\n",
+    "RUN_TRAINING = False\n",
+    "\n",
+    "cmd = [\n",
+    "    str(TRAIN_PYTHON), str(TRAIN_SCRIPT),\n",
+    "    '--checkpoint_path', str(CHECKPOINT_PATH),\n",
+    "    '--hyper_params_path', str(HYPER_PARAMS_PATH),\n",
+    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
+    "    '--gene_list_path', str(GENE_LIST_PATH),\n",
+    "    '--gene_mapping_path', str(GENE_MAPPING_PATH),\n",
+    "    '--gene2vec_path', str(GENE2VEC_PATH),\n",
+    "    '--symbol2ens_path', str(SYMBOL2ENS_PATH),\n",
+    "    '--data_dir', str(DATA_DIR),\n",
+    "    '--output_dir', str(OUTPUT_DIR),\n",
+    "    '--epochs', str(EPOCHS),\n",
+    "    '--batch_size', str(BATCH_SIZE),\n",
+    "    '--lr', str(LR),\n",
+    "    '--n_folds', str(N_FOLDS),\n",
+    "    '--pool_method', POOL_METHOD,\n",
+    "    '--eval_interval', str(EVAL_INTERVAL),\n",
+    "    '--seed', str(SEED),\n",
+    "]\n",
+    "\n",
+    "if not DOMAIN_GENERALIZATION:\n",
+    "    cmd.append('--no_domain_gen')\n",
+    "if FORCE_PREPROCESS:\n",
+    "    cmd.append('--force_preprocess')\n",
+    "\n",
+    "if EVAL_INTERVAL < 1 or EPOCHS < EVAL_INTERVAL:\n",
+    "    raise ValueError('Use at least one validation interval: EPOCHS >= EVAL_INTERVAL >= 1.')\n",
+    "if not DOMAIN_GENERALIZATION and N_FOLDS < 2:\n",
+    "    raise ValueError('Cross-validation requires N_FOLDS >= 2.')\n",
+    "print(subprocess.list2cmdline(cmd) if os.name == 'nt' else shlex.join(cmd))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "env = os.environ.copy()\n",
+    "env['PYTHONPATH'] = PYTHONPATH\n",
+    "\n",
+    "if RUN_TRAINING:\n",
+    "    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    if not DATA_READY:\n",
+    "        missing_data = data_status.loc[~data_status['exists'], 'path'].tolist()\n",
+    "        raise FileNotFoundError('Data are incomplete:\\n' + '\\n'.join(missing_data))\n",
+    "    result = subprocess.run(\n",
+    "        cmd,\n",
+    "        cwd=str(TASK_ROOT),\n",
+    "        env=env,\n",
+    "        text=True,\n",
+    "        stdout=subprocess.PIPE,\n",
+    "        stderr=subprocess.STDOUT,\n",
+    "        check=False,\n",
+    "    )\n",
+    "    print(result.stdout)\n",
+    "    if result.returncode != 0:\n",
+    "        raise RuntimeError(f'Training failed with return code {result.returncode}')\n",
+    "else:\n",
+    "    print('RUN_TRAINING is False, so no training/smokecheck was launched.')\n",
+    "    print('Set RUN_TRAINING=True after reviewing the inputs, dependencies, and command.')\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if OUTPUT_DIR.exists():\n",
+    "    output_files = sorted(OUTPUT_DIR.glob('*'))\n",
+    "    display(pd.DataFrame(\n",
+    "        [\n",
+    "            {\n",
+    "                'file': f.name,\n",
+    "                'size_mb': round(f.stat().st_size / 1024 / 1024, 3),\n",
+    "                'path': str(f),\n",
+    "            }\n",
+    "            for f in output_files\n",
+    "        ]\n",
+    "    ))\n",
+    "else:\n",
+    "    print('No output directory yet:', OUTPUT_DIR)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Output files\n",
+    "\n",
+    "The training script writes the following files under `OUTPUT_DIR`:\n",
+    "\n",
+    "- `best_model_fold{i}.pt`: the selected downstream model.\n",
+    "- `loss_fold{i}.csv` and `loss_curve_fold{i}.png`: training loss (`epoch`, `avg_loss`).\n",
+    "- `metrics_fold{i}.csv`: validation metrics (`epoch`, `AUC`, `PR_AUC`, `ACC`,\n",
+    "  `BACC`, `PREC`, `TPR`, `KAPPA`, `RECALL`).\n",
+    "- `roc_curve_fold{i}.png` and `fpr_tpr_fold{i}.pkl`: the selected validation ROC.\n",
+    "- `cell_emb_cache_{pool_method}.npy` and `cell_emb_dim_{pool_method}.txt`: cached\n",
+    "  cell features when caching is enabled.\n",
+    "\n",
+    "The following cells use only available local outputs. Missing results are skipped.\n",
+    "After changing the weights or input data, use a fresh output directory and set\n",
+    "`FORCE_PREPROCESS = True` to rebuild the script's processed-data cache.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Data overview plots\n",
+    "\n",
+    "These plots summarize the drug-combination dataset before training. They are useful for quick notebook demos and GitHub documentation because they do not require a completed model run.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib\n",
+    "matplotlib.use('Agg')\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.ticker import MaxNLocator\n",
+    "\n",
+    "PLOT_DIR = OUTPUT_DIR / 'notebook_plots'\n",
+    "PLOT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "plt.rcParams.update({\n",
+    "    'font.family': 'sans-serif',\n",
+    "    'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
+    "    'font.size': 7,\n",
+    "    'axes.labelsize': 7,\n",
+    "    'axes.titlesize': 8,\n",
+    "    'xtick.labelsize': 6,\n",
+    "    'ytick.labelsize': 6,\n",
+    "    'legend.fontsize': 6,\n",
+    "    'axes.linewidth': 0.7,\n",
+    "    'pdf.fonttype': 42,\n",
+    "    'ps.fonttype': 42,\n",
+    "    'svg.fonttype': 'none',\n",
+    "})\n",
+    "\n",
+    "\n",
+    "def save_fig(fig, stem: str, dpi: int = 600, show: bool = True):\n",
+    "    png_path = PLOT_DIR / f'{stem}.png'\n",
+    "    pdf_path = PLOT_DIR / f'{stem}.pdf'\n",
+    "    fig.savefig(png_path, dpi=dpi, bbox_inches='tight')\n",
+    "    fig.savefig(pdf_path, bbox_inches='tight')\n",
+    "    plt.close(fig)\n",
+    "    print('Saved:', png_path)\n",
+    "    print('Saved:', pdf_path)\n",
+    "    if show and Image is not None:\n",
+    "        display(Image(filename=str(png_path)))\n",
+    "    return png_path, pdf_path\n",
+    "\n",
+    "\n",
+    "def despine(ax):\n",
+    "    ax.spines['top'].set_visible(False)\n",
+    "    ax.spines['right'].set_visible(False)\n",
+    "    ax.tick_params(width=0.7, length=2.5)\n",
+    "\n",
+    "\n",
+    "def read_label_table(path: Path) -> pd.DataFrame:\n",
+    "    df = pd.read_csv(path)\n",
+    "    lower_map = {str(col).lower(): col for col in df.columns}\n",
+    "    for required in ['drug1', 'drug2', 'cell', 'label']:\n",
+    "        if required not in lower_map:\n",
+    "            raise ValueError(f'{path} is missing a required column similar to {required!r}. Columns: {list(df.columns)}')\n",
+    "    df = df.rename(columns={lower_map['drug1']: 'drug1', lower_map['drug2']: 'drug2', lower_map['cell']: 'cell', lower_map['label']: 'label'})\n",
+    "    return df[['drug1', 'drug2', 'cell', 'label']].copy()\n",
+    "\n",
+    "\n",
+    "data_overview = []\n",
+    "train_labels = validation_labels = None\n",
+    "if expected_data_files['train_labels'].exists():\n",
+    "    train_labels = read_label_table(expected_data_files['train_labels'])\n",
+    "    data_overview.append({\n",
+    "        'split': 'train',\n",
+    "        'rows': len(train_labels),\n",
+    "        'positive': int((train_labels['label'] == 1).sum()),\n",
+    "        'negative': int((train_labels['label'] == 0).sum()),\n",
+    "        'unique_cells': train_labels['cell'].nunique(),\n",
+    "        'unique_drug1': train_labels['drug1'].nunique(),\n",
+    "        'unique_drug2': train_labels['drug2'].nunique(),\n",
+    "        'unique_pairs': train_labels[['drug1', 'drug2']].drop_duplicates().shape[0],\n",
+    "    })\n",
+    "if expected_data_files['validation_labels'].exists():\n",
+    "    validation_labels = read_label_table(expected_data_files['validation_labels'])\n",
+    "    data_overview.append({\n",
+    "        'split': 'validation',\n",
+    "        'rows': len(validation_labels),\n",
+    "        'positive': int((validation_labels['label'] == 1).sum()),\n",
+    "        'negative': int((validation_labels['label'] == 0).sum()),\n",
+    "        'unique_cells': validation_labels['cell'].nunique(),\n",
+    "        'unique_drug1': validation_labels['drug1'].nunique(),\n",
+    "        'unique_drug2': validation_labels['drug2'].nunique(),\n",
+    "        'unique_pairs': validation_labels[['drug1', 'drug2']].drop_duplicates().shape[0],\n",
+    "    })\n",
+    "\n",
+    "data_overview_df = pd.DataFrame(data_overview)\n",
+    "if data_overview_df.empty:\n",
+    "    print('No label files found yet, so data overview plots were skipped.')\n",
+    "else:\n",
+    "    data_overview_df.to_csv(PLOT_DIR / 'drug_combination_data_overview.csv', index=False)\n",
+    "    display(data_overview_df)\n",
+    "\n",
+    "    label_rows = []\n",
+    "    if train_labels is not None:\n",
+    "        label_rows.extend([{'split': 'train', 'label': int(k), 'count': int(v)} for k, v in train_labels['label'].value_counts().sort_index().items()])\n",
+    "    if validation_labels is not None:\n",
+    "        label_rows.extend([{'split': 'validation', 'label': int(k), 'count': int(v)} for k, v in validation_labels['label'].value_counts().sort_index().items()])\n",
+    "    label_df = pd.DataFrame(label_rows)\n",
+    "    label_df.to_csv(PLOT_DIR / 'drug_combination_label_counts.csv', index=False)\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(3.5, 2.4))\n",
+    "    splits = list(label_df['split'].drop_duplicates())\n",
+    "    x = np.arange(len(splits))\n",
+    "    width = 0.34\n",
+    "    colors = {0: '#4E79A7', 1: '#E15759'}\n",
+    "    for offset, label in zip([-width / 2, width / 2], [0, 1]):\n",
+    "        values = [int(label_df[(label_df['split'] == split) & (label_df['label'] == label)]['count'].sum()) for split in splits]\n",
+    "        ax.bar(x + offset, values, width=width, color=colors[label], edgecolor='#333333', linewidth=0.4, label=f'label {label}')\n",
+    "        for xpos, value in zip(x + offset, values):\n",
+    "            ax.text(xpos, value, str(value), ha='center', va='bottom', fontsize=6)\n",
+    "    ax.set_xticks(x)\n",
+    "    ax.set_xticklabels(splits, rotation=15, ha='right')\n",
+    "    ax.set_ylabel('Number of combinations')\n",
+    "    ax.set_title('Drug-combination label balance')\n",
+    "    ax.legend(frameon=False)\n",
+    "    ax.yaxis.set_major_locator(MaxNLocator(integer=True))\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_label_balance')\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(4.1, 2.5))\n",
+    "    overview_long = data_overview_df.melt(id_vars='split', value_vars=['unique_cells', 'unique_drug1', 'unique_drug2', 'unique_pairs'], var_name='feature', value_name='count')\n",
+    "    features = ['unique_cells', 'unique_drug1', 'unique_drug2', 'unique_pairs']\n",
+    "    x = np.arange(len(features))\n",
+    "    width = 0.36 if len(splits) > 1 else 0.5\n",
+    "    palette = ['#59A14F', '#F28E2B']\n",
+    "    for i, split in enumerate(splits):\n",
+    "        values = [int(overview_long[(overview_long['split'] == split) & (overview_long['feature'] == feature)]['count'].sum()) for feature in features]\n",
+    "        ax.bar(x + (i - (len(splits) - 1) / 2) * width, values, width=width, color=palette[i % len(palette)], edgecolor='#333333', linewidth=0.4, label=split)\n",
+    "    ax.set_xticks(x)\n",
+    "    ax.set_xticklabels(['cells', 'drug1', 'drug2', 'drug pairs'], rotation=20, ha='right')\n",
+    "    ax.set_ylabel('Unique count')\n",
+    "    ax.set_title('Dataset coverage')\n",
+    "    ax.legend(frameon=False)\n",
+    "    ax.yaxis.set_major_locator(MaxNLocator(integer=True))\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_dataset_coverage')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Result plots from training outputs\n",
+    "\n",
+    "Plot the saved training loss, validation metric history, and ROC curves. The\n",
+    "current script does not export per-example prediction CSV files, so this section\n",
+    "does not assume that precision-recall or confusion-matrix inputs are available.\n",
+    "Only open ROC pickle files produced by your own trusted training run.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pickle\n",
+    "from sklearn import metrics\n",
+    "\n",
+    "\n",
+    "def find_fold_files(pattern: str):\n",
+    "    return sorted(OUTPUT_DIR.glob(pattern))\n",
+    "\n",
+    "\n",
+    "def load_metrics_files():\n",
+    "    frames = []\n",
+    "    for path in find_fold_files('metrics_fold*.csv'):\n",
+    "        df = pd.read_csv(path)\n",
+    "        stem = path.stem.replace('metrics_fold', '')\n",
+    "        try:\n",
+    "            fold = int(stem)\n",
+    "        except ValueError:\n",
+    "            fold = len(frames)\n",
+    "        df['fold'] = fold\n",
+    "        df['source_file'] = str(path)\n",
+    "        frames.append(df)\n",
+    "    return pd.concat(frames, ignore_index=True) if frames else pd.DataFrame()\n",
+    "\n",
+    "\n",
+    "def load_loss_files():\n",
+    "    frames = []\n",
+    "    for path in find_fold_files('loss_fold*.csv'):\n",
+    "        df = pd.read_csv(path)\n",
+    "        stem = path.stem.replace('loss_fold', '')\n",
+    "        try:\n",
+    "            fold = int(stem)\n",
+    "        except ValueError:\n",
+    "            fold = len(frames)\n",
+    "        df['fold'] = fold\n",
+    "        frames.append(df)\n",
+    "    return pd.concat(frames, ignore_index=True) if frames else pd.DataFrame()\n",
+    "\n",
+    "\n",
+    "metrics_df = load_metrics_files()\n",
+    "loss_df = load_loss_files()\n",
+    "\n",
+    "if metrics_df.empty:\n",
+    "    print('No metrics_fold*.csv files found yet:', OUTPUT_DIR)\n",
+    "else:\n",
+    "    metric_cols = [col for col in ['AUC', 'PR_AUC', 'ACC', 'BACC', 'PREC', 'TPR', 'KAPPA', 'RECALL'] if col in metrics_df.columns]\n",
+    "    best_rows = metrics_df.sort_values(['fold', 'AUC'], ascending=[True, False]).groupby('fold', as_index=False).head(1)\n",
+    "    summary_rows = []\n",
+    "    for metric in metric_cols:\n",
+    "        values = best_rows[metric].astype(float).to_numpy()\n",
+    "        summary_rows.append({'metric': metric, 'mean': float(np.mean(values)), 'std': float(np.std(values)), 'n_folds': int(len(values))})\n",
+    "    summary_df = pd.DataFrame(summary_rows)\n",
+    "    best_rows.to_csv(PLOT_DIR / 'drug_combination_best_metrics_by_fold.csv', index=False)\n",
+    "    summary_df.to_csv(PLOT_DIR / 'drug_combination_metric_summary.csv', index=False)\n",
+    "    display(best_rows)\n",
+    "    display(summary_df)\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(4.1, 2.6))\n",
+    "    for metric, color in [('AUC', '#4E79A7'), ('PR_AUC', '#E15759'), ('ACC', '#59A14F'), ('BACC', '#F28E2B')]:\n",
+    "        if metric in metrics_df.columns:\n",
+    "            for fold, frame in metrics_df.groupby('fold'):\n",
+    "                ax.plot(frame['epoch'], frame[metric], linewidth=0.8, alpha=0.65, color=color)\n",
+    "            mean_by_epoch = metrics_df.groupby('epoch')[metric].mean().reset_index()\n",
+    "            ax.plot(mean_by_epoch['epoch'], mean_by_epoch[metric], linewidth=1.8, color=color, label=metric)\n",
+    "    ax.set_xlabel('Epoch')\n",
+    "    ax.set_ylabel('Score')\n",
+    "    ax.set_ylim(0, 1.02)\n",
+    "    ax.set_title('Validation metrics over training')\n",
+    "    ax.legend(frameon=False, ncol=2)\n",
+    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
+    "    ax.set_axisbelow(True)\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_metric_curves')\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(3.8, 2.5))\n",
+    "    plot_metrics = [m for m in ['AUC', 'PR_AUC', 'ACC', 'BACC', 'PREC', 'TPR', 'KAPPA', 'RECALL'] if m in best_rows.columns]\n",
+    "    values = [float(best_rows[m].mean()) for m in plot_metrics]\n",
+    "    colors = ['#4E79A7', '#E15759', '#59A14F', '#F28E2B', '#76B7B2', '#B07AA1', '#9C755F', '#BAB0AC'][:len(plot_metrics)]\n",
+    "    bars = ax.bar(np.arange(len(plot_metrics)), values, color=colors, edgecolor='#333333', linewidth=0.4)\n",
+    "    for bar, value in zip(bars, values):\n",
+    "        ax.text(bar.get_x() + bar.get_width() / 2, value + 0.015, f'{value:.3f}', ha='center', va='bottom', fontsize=6)\n",
+    "    ax.set_xticks(np.arange(len(plot_metrics)))\n",
+    "    ax.set_xticklabels(plot_metrics, rotation=25, ha='right')\n",
+    "    ax.set_ylim(0, 1.05)\n",
+    "    ax.set_ylabel('Best-fold score')\n",
+    "    ax.set_title('Drug-combination prediction summary')\n",
+    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
+    "    ax.set_axisbelow(True)\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_best_metric_summary')\n",
+    "\n",
+    "if loss_df.empty:\n",
+    "    print('No loss_fold*.csv files found yet:', OUTPUT_DIR)\n",
+    "else:\n",
+    "    fig, ax = plt.subplots(figsize=(4.1, 2.5))\n",
+    "    for fold, frame in loss_df.groupby('fold'):\n",
+    "        ax.plot(frame['epoch'], frame['avg_loss'], linewidth=0.8, alpha=0.65, label=f'fold {fold}')\n",
+    "    mean_loss = loss_df.groupby('epoch')['avg_loss'].mean().reset_index()\n",
+    "    ax.plot(mean_loss['epoch'], mean_loss['avg_loss'], linewidth=1.8, color='#222222', label='mean')\n",
+    "    ax.set_xlabel('Epoch')\n",
+    "    ax.set_ylabel('Average loss')\n",
+    "    ax.set_title('Training loss')\n",
+    "    ax.legend(frameon=False, ncol=2)\n",
+    "    ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.3)\n",
+    "    ax.set_axisbelow(True)\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_loss_curves')\n",
+    "\n",
+    "roc_files = find_fold_files('fpr_tpr_fold*.pkl')\n",
+    "if not roc_files:\n",
+    "    print('No fpr_tpr_fold*.pkl files found yet:', OUTPUT_DIR)\n",
+    "else:\n",
+    "    fig, ax = plt.subplots(figsize=(3.2, 3.2))\n",
+    "    roc_rows = []\n",
+    "    for path in roc_files:\n",
+    "        with path.open('rb') as handle:\n",
+    "            fpr, tpr = pickle.load(handle)\n",
+    "        stem = path.stem.replace('fpr_tpr_fold', '')\n",
+    "        try:\n",
+    "            fold = int(stem)\n",
+    "        except ValueError:\n",
+    "            fold = len(roc_rows)\n",
+    "        auc_value = float(metrics.auc(fpr, tpr))\n",
+    "        roc_rows.append({'fold': fold, 'auc': auc_value})\n",
+    "        ax.plot(fpr, tpr, linewidth=1.0, label=f'fold {fold}, AUROC={auc_value:.3f}')\n",
+    "    ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
+    "    ax.set_xlabel('False positive rate')\n",
+    "    ax.set_ylabel('True positive rate')\n",
+    "    ax.set_xlim(0, 1)\n",
+    "    ax.set_ylim(0, 1.02)\n",
+    "    ax.set_aspect('equal', adjustable='box')\n",
+    "    ax.legend(frameon=False, loc='lower right')\n",
+    "    ax.set_title('ROC curve')\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    save_fig(fig, 'drug_combination_roc_curves')\n",
+    "    pd.DataFrame(roc_rows).to_csv(PLOT_DIR / 'drug_combination_roc_auc_from_curves.csv', index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generated plot files\n",
+    "\n",
+    "This cell lists the figures and source tables generated by the notebook plotting sections.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plot_files = sorted(PLOT_DIR.glob('*'))\n",
+    "if plot_files:\n",
+    "    display(pd.DataFrame([\n",
+    "        {'file': path.name, 'size_kb': round(path.stat().st_size / 1024, 2), 'path': str(path)}\n",
+    "        for path in plot_files\n",
+    "    ]))\n",
+    "else:\n",
+    "    print('No notebook plot files were generated yet.')\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (sclong)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

BioMod/toturials/drug_response_prediction.ipynb

@@ -1,906 +1,871 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Drug response prediction demo: GDSC/CCLE with weight0407 on one GPU\n",
-    "\n",
-    "This notebook is a tutorial version of the drug-response launcher based on `train_drug_performer_module.py`. It uses the weight0407 PerformerModule backbone, drug graph features, and CCLE/GDSC response data.\n",
-    "\n",
-    "By default, the notebook does not relaunch the full 300-epoch training job. It checks the inputs, builds the exact command, reads the existing `weight0407/drug` outputs, computes metrics, and writes a small tutorial scatter plot. To retrain, set `RUN_TRAINING = True` in the training cell.\n",
-    "\n",
-    "The validation set is used for checkpoint selection. Reported metrics describe\n",
-    "validation performance; this workflow does not produce a separate final test evaluation.\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Privacy-safe absolute paths\n",
-    "\n",
-    "The notebook uses this alias path instead of a personal project path:\n",
-    "\n",
-    "`/data/share/biomodulefm_demo/single_cell_model`\n",
-    "\n",
-    "All paths below are absolute and resolve through that alias on the server."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## What this demo runs\n",
-    "\n",
-    "- Task code: `/data/jianqingzhu/BioMod-main/BioMod/drug_response/train_drug_performer_module.py`\n",
-    "- Drug-response data: `/data/share/biomodulefm_demo/single_cell_model/DATA/drug_data`\n",
-    "- Weight0407 checkpoint: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth`\n",
-    "- Weight0407 config: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory.pkl`\n",
-    "- Default tutorial output: `/data/share/biomodulefm_demo/tutorial_runs/drug_response_prediction_weight0407_1gpu_demo`\n",
-    "- Existing full-run output used for display when tutorial output is absent: `/data/share/biomodulefm_demo/single_cell_model/module_model/result/weight0407/drug`"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "import os\n",
-    "import shlex\n",
-    "import subprocess\n",
-    "import sys\n",
-    "\n",
-    "try:\n",
-    "    from IPython.display import Image, display\n",
-    "except Exception:\n",
-    "    Image = None\n",
-    "    def display(obj):\n",
-    "        print(obj)\n",
-    "import numpy as np\n",
-    "import pandas as pd\n",
-    "from scipy.stats import pearsonr\n",
-    "\n",
-    "BASE = Path('/data/share/biomodulefm_demo/single_cell_model')\n",
-    "MODULE_MODEL = BASE / 'module_model'\n",
-    "PROJECT_ROOT = Path('/data/jianqingzhu/BioMod-main/BioMod')\n",
-    "TASK_ROOT = PROJECT_ROOT / 'drug_response'\n",
-    "SCRIPT = TASK_ROOT / 'train_drug_performer_module.py'\n",
-    "\n",
-    "DATA_ROOT = BASE / 'DATA' / 'drug_data'\n",
-    "WEIGHT_DIR = BASE / 'weight' / '0407weight'\n",
-    "CKPT_PATH = WEIGHT_DIR / 'mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth'\n",
-    "CONFIG_PKL = WEIGHT_DIR / 'mae_tost_48M_cross_attn_only_pure_memory.pkl'\n",
-    "\n",
-    "MODULE_DATA_DIR = PROJECT_ROOT / 'module_data_all'\n",
-    "GENE2VEC = MODULE_MODEL / 'selected_gene2vec_27k.npy'\n",
-    "GENE_MAP = MODULE_MODEL / 'gene2ens.csv'\n",
-    "GENE_LIST = MODULE_MODEL / 'selected_genes_27k.txt'\n",
-    "\n",
-    "TUTORIAL_RUN_ROOT = Path('/data/share/biomodulefm_demo/tutorial_runs')\n",
-    "RESULT_DIR = TUTORIAL_RUN_ROOT / 'drug_response_prediction_weight0407_1gpu_demo'\n",
-    "REFERENCE_RESULT_DIR = MODULE_MODEL / 'result' / 'weight0407' / 'drug'\n",
-    "RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "paths = pd.DataFrame([\n",
-    "    {'name': 'project_alias', 'path': BASE},\n",
-    "    {'name': 'script', 'path': SCRIPT},\n",
-    "    {'name': 'data_root', 'path': DATA_ROOT},\n",
-    "    {'name': 'checkpoint', 'path': CKPT_PATH},\n",
-    "    {'name': 'config_pkl', 'path': CONFIG_PKL},\n",
-    "    {'name': 'module_data_dir', 'path': MODULE_DATA_DIR},\n",
-    "    {'name': 'gene_list', 'path': GENE_LIST},\n",
-    "    {'name': 'gene_map', 'path': GENE_MAP},\n",
-    "    {'name': 'gene2vec', 'path': GENE2VEC},\n",
-    "    {'name': 'tutorial_result_dir', 'path': RESULT_DIR},\n",
-    "    {'name': 'reference_result_dir', 'path': REFERENCE_RESULT_DIR},\n",
-    "])\n",
-    "display(paths)\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "required_files = [\n",
-    "    SCRIPT,\n",
-    "    CKPT_PATH,\n",
-    "    CONFIG_PKL,\n",
-    "    GENE_LIST,\n",
-    "    GENE_MAP,\n",
-    "    GENE2VEC,\n",
-    "    DATA_ROOT / 'CCLE' / 'GDSC_IC50.csv',\n",
-    "    DATA_ROOT / 'CCLE' / 'genomic_expression_561celllines_697genes_demap_features.csv',\n",
-    "    DATA_ROOT / 'CCLE' / 'genomic_mutation_34673_demap_features.csv',\n",
-    "    DATA_ROOT / 'CCLE' / 'genomic_copynumber_561celllines_710genes_demap_features.csv',\n",
-    "    DATA_ROOT / 'CCLE' / 'genomic_methylation_561celllines_808genes_demap_features.csv',\n",
-    "    DATA_ROOT / 'GDSC' / 'drug_graph_feat.zip',\n",
-    "]\n",
-    "required_dirs = [PROJECT_ROOT, TASK_ROOT, DATA_ROOT, MODULE_DATA_DIR]\n",
-    "\n",
-    "missing = [str(p) for p in required_files if not p.exists()]\n",
-    "missing += [str(p) for p in required_dirs if not p.is_dir()]\n",
-    "if missing:\n",
-    "    raise FileNotFoundError('Missing required inputs:\\n' + '\\n'.join(missing))\n",
-    "\n",
-    "print('All required files and directories are present.')\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Runtime setup\n",
-    "\n",
-    "The full training script was originally run in the `sclong` conda environment. If this notebook kernel is different, the command cell below still points the subprocess to `/opt/conda/envs/sclong/bin/python` when that interpreter exists."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'\n",
-    "os.environ['PYTHONUNBUFFERED'] = '1'\n",
-    "os.environ.setdefault('OMP_NUM_THREADS', '8')\n",
-    "os.environ.setdefault('OPENBLAS_NUM_THREADS', '8')\n",
-    "os.environ.setdefault('MKL_NUM_THREADS', '8')\n",
-    "\n",
-    "SCLONG_PYTHON = Path('/opt/conda/envs/sclong/bin/python')\n",
-    "TRAIN_PYTHON = SCLONG_PYTHON if SCLONG_PYTHON.exists() else Path(sys.executable)\n",
-    "print('Notebook Python:', sys.executable)\n",
-    "print('Training Python:', TRAIN_PYTHON)\n",
-    "\n",
-    "try:\n",
-    "    import torch\n",
-    "    print('torch:', torch.__version__)\n",
-    "    print('cuda available:', torch.cuda.is_available())\n",
-    "    if torch.cuda.is_available():\n",
-    "        print('cuda device:', torch.cuda.get_device_name(0))\n",
-    "except Exception as exc:\n",
-    "    print('Torch check skipped:', repr(exc))\n",
-    "\n",
-    "try:\n",
-    "    import hickle\n",
-    "    print('hickle:', getattr(hickle, '__version__', 'ok'))\n",
-    "except Exception:\n",
-    "    subprocess.check_call([\n",
-    "        str(TRAIN_PYTHON), '-m', 'pip', 'install',\n",
-    "        '-i', 'https://pypi.tuna.tsinghua.edu.cn/simple',\n",
-    "        'hickle',\n",
-    "    ])\n",
-    "    import hickle\n",
-    "    print('hickle:', getattr(hickle, '__version__', 'ok'))\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Inspect input files\n",
-    "\n",
-    "This cell gives a quick sanity check for the drug-response files without loading the large full matrices into memory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def size_mb(path: Path) -> float:\n",
-    "    return path.stat().st_size / 1024 / 1024\n",
-    "\n",
-    "input_summary = []\n",
-    "for path in sorted((DATA_ROOT / 'CCLE').glob('*')):\n",
-    "    if path.is_file():\n",
-    "        input_summary.append({'file': str(path.relative_to(DATA_ROOT)), 'size_mb': round(size_mb(path), 2)})\n",
-    "for path in sorted((DATA_ROOT / 'GDSC').glob('*')):\n",
-    "    if path.is_file():\n",
-    "        input_summary.append({'file': str(path.relative_to(DATA_ROOT)), 'size_mb': round(size_mb(path), 2)})\n",
-    "\n",
-    "display(pd.DataFrame(input_summary))\n",
-    "\n",
-    "ic50_path = DATA_ROOT / 'CCLE' / 'GDSC_IC50.csv'\n",
-    "ic50_preview = pd.read_csv(ic50_path, nrows=5)\n",
-    "display(ic50_preview)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Training command\n",
-    "\n",
-    "This is the weight0407 absolute-path version of the older launch command. The tutorial result directory is separate from the existing full-run result directory so reruns do not overwrite paper-facing outputs by accident."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "EPOCHS = 300\n",
-    "BATCH_SIZE = 32\n",
-    "POOL_METHOD = 'max'\n",
-    "\n",
-    "cmd = [\n",
-    "    str(TRAIN_PYTHON), str(SCRIPT),\n",
-    "    '--gpu_id', '0',\n",
-    "    '--data_path', str(DATA_ROOT),\n",
-    "    '--checkpoint_path', str(CKPT_PATH),\n",
-    "    '--hyper_params_path', str(CONFIG_PKL),\n",
-    "    '--epochs', str(EPOCHS),\n",
-    "    '--batch_size', str(BATCH_SIZE),\n",
-    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
-    "    '--gene_list_path', str(GENE_LIST),\n",
-    "    '--gene_mapping_path', str(GENE_MAP),\n",
-    "    '--symbol2ens_path', str(GENE_MAP),\n",
-    "    '--gene2vec_path', str(GENE2VEC),\n",
-    "    '--save_dir', str(RESULT_DIR),\n",
-    "    '--pool_method', POOL_METHOD,\n",
-    "]\n",
-    "\n",
-    "print('cd', shlex.quote(str(TASK_ROOT)))\n",
-    "print(' '.join(shlex.quote(part) for part in cmd))\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "RUN_TRAINING = False\n",
-    "\n",
-    "if RUN_TRAINING:\n",
-    "    RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "    log_path = RESULT_DIR / 'train.log'\n",
-    "    print('Writing log to:', log_path)\n",
-    "    with log_path.open('w') as log_file:\n",
-    "        proc = subprocess.Popen(\n",
-    "            cmd,\n",
-    "            cwd=str(TASK_ROOT),\n",
-    "            stdout=subprocess.PIPE,\n",
-    "            stderr=subprocess.STDOUT,\n",
-    "            text=True,\n",
-    "            bufsize=1,\n",
-    "        )\n",
-    "        assert proc.stdout is not None\n",
-    "        for line in proc.stdout:\n",
-    "            print(line, end='')\n",
-    "            log_file.write(line)\n",
-    "        return_code = proc.wait()\n",
-    "    if return_code != 0:\n",
-    "        raise RuntimeError(f'Training failed with exit code {return_code}')\n",
-    "else:\n",
-    "    print('Training was not launched. Set RUN_TRAINING = True to run the full job.')\n",
-    "    print('The following cells will use tutorial outputs if present; otherwise they use the existing weight0407/drug outputs.')\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Inspect outputs\n",
-    "\n",
-    "The active result directory is chosen in this order: first the tutorial run directory, then the existing `weight0407/drug` full-run directory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "candidate_dirs = [RESULT_DIR, REFERENCE_RESULT_DIR]\n",
-    "ACTIVE_RESULT_DIR = None\n",
-    "for candidate in candidate_dirs:\n",
-    "    if (candidate / 'predictions.npz').exists():\n",
-    "        ACTIVE_RESULT_DIR = candidate\n",
-    "        break\n",
-    "\n",
-    "if ACTIVE_RESULT_DIR is None:\n",
-    "    raise FileNotFoundError('No predictions.npz found. Run training first or check REFERENCE_RESULT_DIR.')\n",
-    "\n",
-    "print('Active result directory:', ACTIVE_RESULT_DIR)\n",
-    "rows = []\n",
-    "for path in sorted(ACTIVE_RESULT_DIR.rglob('*')):\n",
-    "    if path.is_file():\n",
-    "        rows.append({'file': str(path.relative_to(ACTIVE_RESULT_DIR)), 'size_mb': round(size_mb(path), 3)})\n",
-    "display(pd.DataFrame(rows))\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "pred_path = ACTIVE_RESULT_DIR / 'predictions.npz'\n",
-    "pred_npz = np.load(pred_path, allow_pickle=True)\n",
-    "print('prediction keys:', pred_npz.files)\n",
-    "\n",
-    "pred = pred_npz['pred'] if 'pred' in pred_npz.files else pred_npz['preds']\n",
-    "# Prefer the validation key; accept historical output files as well.\n",
-    "if 'validation' in pred_npz.files:\n",
-    "    true = pred_npz['validation']\n",
-    "elif 'test' in pred_npz.files:\n",
-    "    true = pred_npz['test']\n",
-    "else:\n",
-    "    true = pred_npz['true']\n",
-    "\n",
-    "pcc = pearsonr(pred, true)[0]\n",
-    "mae = np.mean(np.abs(pred - true))\n",
-    "rmse = np.sqrt(np.mean((pred - true) ** 2))\n",
-    "summary = pd.DataFrame([{\n",
-    "    'n_validation': len(pred),\n",
-    "    'pcc': pcc,\n",
-    "    'mae': mae,\n",
-    "    'rmse': rmse,\n",
-    "}])\n",
-    "display(summary)\n",
-    "\n",
-    "meta = pd.DataFrame({\n",
-    "    'pred': pred,\n",
-    "    'true': true,\n",
-    "    'drug_id': pred_npz['drug_id'] if 'drug_id' in pred_npz.files else ['NA'] * len(pred),\n",
-    "    'cell_line_id': pred_npz['cell_line_id'] if 'cell_line_id' in pred_npz.files else ['NA'] * len(pred),\n",
-    "    'cancer_type': pred_npz['cancer_type'] if 'cancer_type' in pred_npz.files else ['NA'] * len(pred),\n",
-    "})\n",
-    "display(meta.head())\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import random\n",
-    "from scipy.stats import spearmanr\n",
-    "\n",
-    "TCGA_LABEL_SET = [\n",
-    "    'ALL', 'BLCA', 'BRCA', 'CESC', 'DLBC', 'LIHC', 'LUAD', 'ESCA', 'GBM',\n",
-    "    'HNSC', 'KIRC', 'LAML', 'LCML', 'LGG', 'LUSC', 'MESO', 'MM', 'NB',\n",
-    "    'OV', 'PAAD', 'SCLC', 'SKCM', 'STAD', 'THCA', 'COAD/READ',\n",
-    "]\n",
-    "\n",
-    "\n",
-    "def pearson_corr(x, y):\n",
-    "    x = np.asarray(x, dtype=float).reshape(-1)\n",
-    "    y = np.asarray(y, dtype=float).reshape(-1)\n",
-    "    mask = np.isfinite(x) & np.isfinite(y)\n",
-    "    x = x[mask]\n",
-    "    y = y[mask]\n",
-    "    if len(x) < 2 or np.std(x) == 0 or np.std(y) == 0:\n",
-    "        return np.nan\n",
-    "    return float(np.corrcoef(x, y)[0, 1])\n",
-    "\n",
-    "\n",
-    "def regression_metrics(frame: pd.DataFrame) -> dict:\n",
-    "    y_true = frame['true'].astype(float).to_numpy()\n",
-    "    y_pred = frame['pred'].astype(float).to_numpy()\n",
-    "    return {\n",
-    "        'n': int(len(frame)),\n",
-    "        'pcc': pearson_corr(y_true, y_pred),\n",
-    "        'spearman': float(spearmanr(y_true, y_pred).correlation) if len(frame) >= 2 else np.nan,\n",
-    "        'mae': float(np.mean(np.abs(y_pred - y_true))),\n",
-    "        'mse': float(np.mean((y_pred - y_true) ** 2)),\n",
-    "        'rmse': float(np.sqrt(np.mean((y_pred - y_true) ** 2))),\n",
-    "    }\n",
-    "\n",
-    "\n",
-    "def reconstruct_validation_metadata(data_root: Path, n_expected: int, ratio: float = 0.95, seed: int = 0) -> pd.DataFrame:\n",
-    "    \"\"\"Rebuild the validation split metadata using the same deterministic split as the training script.\"\"\"\n",
-    "    drug_info_file = data_root / 'GDSC' / '1.Drug_listMon Jun 24 09_00_55 2019.csv'\n",
-    "    cell_line_info_file = data_root / 'CCLE' / 'Cell_lines_annotations_20181226.txt'\n",
-    "    drug_feature_dir = data_root / 'GDSC' / 'drug_graph_feat'\n",
-    "    gene_expression_file = data_root / 'CCLE' / 'genomic_expression_561celllines_697genes_demap_features.csv'\n",
-    "    ic50_file = data_root / 'CCLE' / 'GDSC_IC50.csv'\n",
-    "\n",
-    "    for required in [drug_info_file, cell_line_info_file, drug_feature_dir, gene_expression_file, ic50_file]:\n",
-    "        if not required.exists():\n",
-    "            raise FileNotFoundError(required)\n",
-    "\n",
-    "    drug_info = pd.read_csv(drug_info_file)\n",
-    "    drugid2pubchemid = {\n",
-    "        str(row.iloc[0]): str(row.iloc[5])\n",
-    "        for _, row in drug_info.iterrows()\n",
-    "        if str(row.iloc[5]).isdigit()\n",
-    "    }\n",
-    "\n",
-    "    cellline2cancertype = {}\n",
-    "    with cell_line_info_file.open() as handle:\n",
-    "        for line in handle.readlines()[1:]:\n",
-    "            parts = line.rstrip('\\n').split('\\t')\n",
-    "            if len(parts) >= 2:\n",
-    "                cellline2cancertype[parts[1]] = parts[-1]\n",
-    "\n",
-    "    gexpr_index = set(pd.read_csv(gene_expression_file, usecols=[0]).iloc[:, 0].astype(str))\n",
-    "    drug_pubchem_ids = {path.stem for path in drug_feature_dir.glob('*.hkl')}\n",
-    "\n",
-    "    ic50 = pd.read_csv(ic50_file, header=0, index_col=0)\n",
-    "    drug_match_list = [idx for idx in ic50.index if idx.split(':')[1] in drugid2pubchemid]\n",
-    "    ic50 = ic50.loc[drug_match_list]\n",
-    "\n",
-    "    data_idx = []\n",
-    "    for each_drug in ic50.index:\n",
-    "        pubchem_id = drugid2pubchemid[each_drug.split(':')[-1]]\n",
-    "        if pubchem_id not in drug_pubchem_ids:\n",
-    "            continue\n",
-    "        for each_cellline in ic50.columns:\n",
-    "            if each_cellline not in gexpr_index or each_cellline not in cellline2cancertype:\n",
-    "                continue\n",
-    "            value = ic50.loc[each_drug, each_cellline]\n",
-    "            if pd.isna(value):\n",
-    "                continue\n",
-    "            data_idx.append((each_cellline, pubchem_id, float(value), cellline2cancertype[each_cellline]))\n",
-    "\n",
-    "    rng = random.Random(seed)\n",
-    "    validation_idx = []\n",
-    "    for each_type in TCGA_LABEL_SET:\n",
-    "        subtype = [item for item in data_idx if item[-1] == each_type]\n",
-    "        if not subtype:\n",
-    "            continue\n",
-    "        k = max(1, int(ratio * len(subtype)))\n",
-    "        train_list = rng.sample(subtype, k)\n",
-    "        validation_idx.extend([item for item in subtype if item not in train_list])\n",
-    "\n",
-    "    reconstructed = pd.DataFrame(validation_idx, columns=['cell_line_id', 'drug_id', 'true_from_metadata', 'cancer_type'])\n",
-    "    if len(reconstructed) != n_expected:\n",
-    "        raise ValueError(f'Reconstructed {len(reconstructed)} rows, but predictions contain {n_expected} rows.')\n",
-    "    return reconstructed[['drug_id', 'cell_line_id', 'cancer_type']]\n",
-    "\n",
-    "\n",
-    "analysis_df = meta.copy()\n",
-    "analysis_df['pred'] = pd.to_numeric(analysis_df['pred'], errors='coerce')\n",
-    "analysis_df['true'] = pd.to_numeric(analysis_df['true'], errors='coerce')\n",
-    "analysis_df = analysis_df.replace([np.inf, -np.inf], np.nan).dropna(subset=['pred', 'true']).reset_index(drop=True)\n",
-    "\n",
-    "metadata_cols = ['drug_id', 'cell_line_id', 'cancer_type']\n",
-    "metadata_missing = any(col not in analysis_df.columns for col in metadata_cols) or all(\n",
-    "    analysis_df[col].astype(str).isin(['NA', 'nan', 'None', '']).all() for col in metadata_cols if col in analysis_df.columns\n",
-    ")\n",
-    "\n",
-    "if metadata_missing:\n",
-    "    try:\n",
-    "        recovered_meta = reconstruct_validation_metadata(DATA_ROOT, n_expected=len(analysis_df), ratio=0.95, seed=0)\n",
-    "        for col in metadata_cols:\n",
-    "            analysis_df[col] = recovered_meta[col].to_numpy()\n",
-    "        print('Recovered validation metadata from the original GDSC/CCLE split.')\n",
-    "    except Exception as exc:\n",
-    "        print('Metadata recovery skipped:', repr(exc))\n",
-    "        for col in metadata_cols:\n",
-    "            if col not in analysis_df.columns:\n",
-    "                analysis_df[col] = 'NA'\n",
-    "else:\n",
-    "    print('Using drug/cell/cancer metadata stored in predictions.npz.')\n",
-    "\n",
-    "overall_metrics = pd.DataFrame([regression_metrics(analysis_df)])\n",
-    "display(overall_metrics)\n",
-    "display(analysis_df.head())\n",
-    "\n",
-    "metric_columns = ['pred', 'true']\n",
-    "by_cancer = (\n",
-    "    analysis_df.groupby('cancer_type', dropna=False)[metric_columns]\n",
-    "    .apply(lambda frame: pd.Series(regression_metrics(frame)))\n",
-    "    .reset_index()\n",
-    "    .sort_values('n', ascending=False)\n",
-    ")\n",
-    "by_drug = (\n",
-    "    analysis_df.groupby('drug_id', dropna=False)[metric_columns]\n",
-    "    .apply(lambda frame: pd.Series(regression_metrics(frame)))\n",
-    "    .reset_index()\n",
-    "    .sort_values('pcc', ascending=False)\n",
-    ")\n",
-    "\n",
-    "display(by_cancer.head(12))\n",
-    "display(by_drug.head(10))\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Tutorial scatter plot\n",
-    "\n",
-    "This cell creates a compact PDF and PNG summary of predicted versus measured drug response. The output is written to the tutorial result directory even when metrics are read from the existing full-run directory."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import matplotlib.pyplot as plt\n",
-    "\n",
-    "plt.rcParams.update({\n",
-    "    'font.family': 'sans-serif',\n",
-    "    'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
-    "    'font.size': 7,\n",
-    "    'pdf.fonttype': 42,\n",
-    "    'ps.fonttype': 42,\n",
-    "})\n",
-    "\n",
-    "scatter_png = RESULT_DIR / 'drug_response_prediction_scatter.png'\n",
-    "scatter_pdf = RESULT_DIR / 'drug_response_prediction_scatter.pdf'\n",
-    "\n",
-    "low = float(min(np.min(true), np.min(pred)))\n",
-    "high = float(max(np.max(true), np.max(pred)))\n",
-    "pad = 0.05 * (high - low)\n",
-    "lims = (low - pad, high + pad)\n",
-    "\n",
-    "fig, ax = plt.subplots(figsize=(3.35, 3.1))\n",
-    "ax.scatter(true, pred, s=7, alpha=0.45, linewidths=0, color='#2f6fb0')\n",
-    "ax.plot(lims, lims, color='#444444', linewidth=0.8, linestyle='--')\n",
-    "ax.set_xlim(lims)\n",
-    "ax.set_ylim(lims)\n",
-    "ax.set_xlabel('Measured drug response')\n",
-    "ax.set_ylabel('Predicted drug response')\n",
-    "ax.set_title('Drug response prediction')\n",
-    "ax.text(0.03, 0.97, f'PCC = {pcc:.3f}\\nN = {len(pred)}', transform=ax.transAxes, va='top')\n",
-    "ax.spines['top'].set_visible(False)\n",
-    "ax.spines['right'].set_visible(False)\n",
-    "fig.tight_layout()\n",
-    "fig.savefig(scatter_png, dpi=300, bbox_inches='tight')\n",
-    "fig.savefig(scatter_pdf, bbox_inches='tight')\n",
-    "plt.close(fig)\n",
-    "\n",
-    "print('Saved:', scatter_png)\n",
-    "print('Saved:', scatter_pdf)\n",
-    "if Image is not None:\n",
-    "    display(Image(filename=str(scatter_png)))\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Paper-style drug-response plots\n",
-    "\n",
-    "The original plotting scripts under `module_model/draw` use the same core outputs: `predictions.npz`, Pearson correlation, per-drug grouping, and a median-threshold ROC. The cells below bring those plotting routines into the notebook and save editable PDF plus high-resolution PNG copies under the tutorial output directory.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from matplotlib.ticker import FormatStrFormatter\n",
-    "\n",
-    "try:\n",
-    "    from sklearn.metrics import roc_curve, roc_auc_score\n",
-    "except Exception as exc:\n",
-    "    roc_curve = None\n",
-    "    roc_auc_score = None\n",
-    "    print('ROC dependencies unavailable:', repr(exc))\n",
-    "\n",
-    "PAPER_PLOT_DIR = RESULT_DIR / 'drug_response_paper_plots'\n",
-    "PAPER_PLOT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "plt.rcParams.update({\n",
-    "    'font.family': 'sans-serif',\n",
-    "    'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
-    "    'font.size': 7,\n",
-    "    'axes.labelsize': 7,\n",
-    "    'axes.titlesize': 8,\n",
-    "    'xtick.labelsize': 6,\n",
-    "    'ytick.labelsize': 6,\n",
-    "    'legend.fontsize': 6,\n",
-    "    'axes.linewidth': 0.7,\n",
-    "    'pdf.fonttype': 42,\n",
-    "    'ps.fonttype': 42,\n",
-    "    'svg.fonttype': 'none',\n",
-    "})\n",
-    "\n",
-    "\n",
-    "def save_figure(fig, stem: str, dpi: int = 600):\n",
-    "    png_path = PAPER_PLOT_DIR / f'{stem}.png'\n",
-    "    pdf_path = PAPER_PLOT_DIR / f'{stem}.pdf'\n",
-    "    fig.savefig(png_path, dpi=dpi, bbox_inches='tight')\n",
-    "    fig.savefig(pdf_path, bbox_inches='tight')\n",
-    "    plt.close(fig)\n",
-    "    print('Saved:', png_path)\n",
-    "    print('Saved:', pdf_path)\n",
-    "    return png_path, pdf_path\n",
-    "\n",
-    "\n",
-    "def despine(ax):\n",
-    "    ax.spines['top'].set_visible(False)\n",
-    "    ax.spines['right'].set_visible(False)\n",
-    "    ax.tick_params(width=0.7, length=2.5)\n",
-    "\n",
-    "\n",
-    "def finite_xy(frame: pd.DataFrame):\n",
-    "    y_true = pd.to_numeric(frame['true'], errors='coerce').to_numpy(dtype=float)\n",
-    "    y_pred = pd.to_numeric(frame['pred'], errors='coerce').to_numpy(dtype=float)\n",
-    "    mask = np.isfinite(y_true) & np.isfinite(y_pred)\n",
-    "    return y_true[mask], y_pred[mask]\n",
-    "\n",
-    "\n",
-    "# 1) Overall measured-vs-predicted scatter.\n",
-    "y_true, y_pred = finite_xy(analysis_df)\n",
-    "low = float(min(y_true.min(), y_pred.min()))\n",
-    "high = float(max(y_true.max(), y_pred.max()))\n",
-    "pad = 0.05 * (high - low)\n",
-    "lims = (low - pad, high + pad)\n",
-    "metrics = regression_metrics(analysis_df)\n",
-    "\n",
-    "fig, ax = plt.subplots(figsize=(3.35, 3.15))\n",
-    "ax.scatter(y_true, y_pred, s=5, alpha=0.35, linewidths=0, color='#2F6FB0', rasterized=True)\n",
-    "ax.plot(lims, lims, color='#333333', linewidth=0.8, linestyle='--')\n",
-    "ax.set_xlim(lims)\n",
-    "ax.set_ylim(lims)\n",
-    "ax.set_xlabel('Measured drug response')\n",
-    "ax.set_ylabel('Predicted drug response')\n",
-    "ax.set_title('BioMod drug response')\n",
-    "ax.text(\n",
-    "    0.03, 0.97,\n",
-    "    f'PCC = {metrics[\"pcc\"]:.3f}\\nRMSE = {metrics[\"rmse\"]:.3f}\\nN = {metrics[\"n\"]}',\n",
-    "    transform=ax.transAxes,\n",
-    "    va='top',\n",
-    "    ha='left',\n",
-    ")\n",
-    "despine(ax)\n",
-    "fig.tight_layout()\n",
-    "overall_scatter_png, overall_scatter_pdf = save_figure(fig, 'drug_response_overall_scatter')\n",
-    "\n",
-    "# 2) Residual distribution.\n",
-    "residual = y_pred - y_true\n",
-    "fig, ax = plt.subplots(figsize=(3.35, 2.35))\n",
-    "ax.hist(residual, bins=45, color='#76B7B2', edgecolor='white', linewidth=0.3)\n",
-    "ax.axvline(0, color='#333333', linestyle='--', linewidth=0.8)\n",
-    "ax.set_xlabel('Prediction residual')\n",
-    "ax.set_ylabel('Number of samples')\n",
-    "ax.set_title('Residual distribution')\n",
-    "despine(ax)\n",
-    "fig.tight_layout()\n",
-    "residual_png, residual_pdf = save_figure(fig, 'drug_response_residual_distribution')\n",
-    "\n",
-    "# 3) Top-5 most precise drugs, matching the logic of plot_drug_response_panel_d.py.\n",
-    "MIN_SAMPLES_PER_DRUG = 3\n",
-    "valid_drug_stats = by_drug[(by_drug['drug_id'].astype(str) != 'NA') & (by_drug['n'] >= MIN_SAMPLES_PER_DRUG)].dropna(subset=['pcc'])\n",
-    "if len(valid_drug_stats) > 0:\n",
-    "    top5_drugs = valid_drug_stats.sort_values('pcc', ascending=False).head(5)['drug_id'].astype(str).tolist()\n",
-    "    top5_summary = valid_drug_stats[valid_drug_stats['drug_id'].astype(str).isin(top5_drugs)].copy()\n",
-    "    top5_points = analysis_df[analysis_df['drug_id'].astype(str).isin(top5_drugs)].copy()\n",
-    "    top5_summary.to_csv(PAPER_PLOT_DIR / 'top5_drug_pcc_summary.csv', index=False)\n",
-    "    top5_points.to_csv(PAPER_PLOT_DIR / 'top5_drug_selected_points.csv', index=False)\n",
-    "\n",
-    "    fig, ax = plt.subplots(figsize=(3.65, 3.25))\n",
-    "    palette = plt.get_cmap('tab10')\n",
-    "    for idx, drug_id in enumerate(top5_drugs):\n",
-    "        sub = top5_points[top5_points['drug_id'].astype(str) == drug_id]\n",
-    "        ax.scatter(\n",
-    "            sub['true'], sub['pred'],\n",
-    "            s=10, alpha=0.75, linewidths=0,\n",
-    "            color=palette(idx), label=str(drug_id), rasterized=True,\n",
-    "        )\n",
-    "    t_true, t_pred = finite_xy(top5_points)\n",
-    "    low = float(min(t_true.min(), t_pred.min()))\n",
-    "    high = float(max(t_true.max(), t_pred.max()))\n",
-    "    pad = 0.05 * (high - low)\n",
-    "    lims = (low - pad, high + pad)\n",
-    "    ax.plot(lims, lims, color='#333333', linewidth=0.8, linestyle='--')\n",
-    "    ax.set_xlim(lims)\n",
-    "    ax.set_ylim(lims)\n",
-    "    ax.set_xlabel('Measured drug response')\n",
-    "    ax.set_ylabel('Predicted drug response')\n",
-    "    ax.set_title('Most precise 5 drugs')\n",
-    "    ax.legend(title='PubChem ID', frameon=False, loc='best', handletextpad=0.3, borderpad=0.2)\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    top5_png, top5_pdf = save_figure(fig, 'drug_response_top5_drugs')\n",
-    "else:\n",
-    "    top5_png = top5_pdf = None\n",
-    "    print('Top-5 drug panel skipped: drug_id metadata is unavailable or too sparse.')\n",
-    "\n",
-    "# 4) Per-cancer-type PCC bar plot.\n",
-    "valid_cancer = by_cancer[(by_cancer['cancer_type'].astype(str) != 'NA') & (by_cancer['n'] >= 2)].dropna(subset=['pcc']).copy()\n",
-    "if len(valid_cancer) > 0:\n",
-    "    valid_cancer = valid_cancer.sort_values('pcc', ascending=True)\n",
-    "    valid_cancer.to_csv(PAPER_PLOT_DIR / 'per_cancer_type_metrics.csv', index=False)\n",
-    "    fig_height = max(2.2, 0.16 * len(valid_cancer) + 0.7)\n",
-    "    fig, ax = plt.subplots(figsize=(3.7, fig_height))\n",
-    "    colors = ['#4E79A7' if value < valid_cancer['pcc'].median() else '#59A14F' for value in valid_cancer['pcc']]\n",
-    "    ax.barh(valid_cancer['cancer_type'].astype(str), valid_cancer['pcc'], color=colors, edgecolor='#333333', linewidth=0.35)\n",
-    "    ax.set_xlabel('Pearson correlation')\n",
-    "    ax.set_ylabel('Cancer type')\n",
-    "    ax.set_xlim(min(0, float(valid_cancer['pcc'].min()) - 0.05), 1.0)\n",
-    "    ax.xaxis.set_major_formatter(FormatStrFormatter('%.2f'))\n",
-    "    ax.grid(axis='x', linestyle='--', linewidth=0.4, alpha=0.35)\n",
-    "    ax.set_axisbelow(True)\n",
-    "    ax.set_title('Performance by cancer type')\n",
-    "    despine(ax)\n",
-    "    fig.tight_layout()\n",
-    "    cancer_png, cancer_pdf = save_figure(fig, 'drug_response_per_cancer_type_pcc')\n",
-    "else:\n",
-    "    cancer_png = cancer_pdf = None\n",
-    "    print('Cancer-type PCC plot skipped: cancer_type metadata is unavailable or too sparse.')\n",
-    "\n",
-    "# 5) Median-threshold ROC, adapted from plot_drug_response_roc_rewrite.py.\n",
-    "if roc_curve is not None and roc_auc_score is not None:\n",
-    "    threshold = float(np.median(y_true))\n",
-    "    y_binary = (y_true <= threshold).astype(int)\n",
-    "    roc_score = -y_pred\n",
-    "    if len(np.unique(y_binary)) == 2:\n",
-    "        fpr, tpr, roc_thresholds = roc_curve(y_binary, roc_score)\n",
-    "        auc_value = float(roc_auc_score(y_binary, roc_score))\n",
-    "        roc_table = pd.DataFrame({'fpr': fpr, 'tpr': tpr, 'threshold': roc_thresholds})\n",
-    "        roc_table.to_csv(PAPER_PLOT_DIR / 'drug_response_roc_curve.csv', index=False)\n",
-    "\n",
-    "        fig, ax = plt.subplots(figsize=(3.1, 3.1))\n",
-    "        ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
-    "        ax.plot(fpr, tpr, color='#9C6ADE', linewidth=1.1, label=f'AUROC = {auc_value:.3f}')\n",
-    "        ax.set_xlabel('False positive rate')\n",
-    "        ax.set_ylabel('True positive rate')\n",
-    "        ax.set_xlim(0, 1)\n",
-    "        ax.set_ylim(0, 1)\n",
-    "        ax.set_aspect('equal', adjustable='box')\n",
-    "        ax.legend(frameon=False, loc='lower right')\n",
-    "        ax.set_title('Median-threshold ROC')\n",
-    "        despine(ax)\n",
-    "        fig.tight_layout()\n",
-    "        roc_png, roc_pdf = save_figure(fig, 'drug_response_median_threshold_roc')\n",
-    "        fig_svg = PAPER_PLOT_DIR / 'drug_response_median_threshold_roc.svg'\n",
-    "        fig, ax = plt.subplots(figsize=(3.1, 3.1))\n",
-    "        ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
-    "        ax.plot(fpr, tpr, color='#9C6ADE', linewidth=1.1, label=f'AUROC = {auc_value:.3f}')\n",
-    "        ax.set_xlabel('False positive rate')\n",
-    "        ax.set_ylabel('True positive rate')\n",
-    "        ax.set_xlim(0, 1)\n",
-    "        ax.set_ylim(0, 1)\n",
-    "        ax.set_aspect('equal', adjustable='box')\n",
-    "        ax.legend(frameon=False, loc='lower right')\n",
-    "        ax.set_title('Median-threshold ROC')\n",
-    "        despine(ax)\n",
-    "        fig.tight_layout()\n",
-    "        fig.savefig(fig_svg, bbox_inches='tight')\n",
-    "        plt.close(fig)\n",
-    "        print('Saved:', fig_svg)\n",
-    "    else:\n",
-    "        roc_png = roc_pdf = None\n",
-    "        print('ROC plot skipped: median split produced a single class.')\n",
-    "else:\n",
-    "    roc_png = roc_pdf = None\n",
-    "\n",
-    "# 6) Reported model-comparison bar from plot_cancer_drug_response_nature_color.py.\n",
-    "reported_scores = pd.DataFrame({\n",
-    "    'model': ['scBERT', 'scGPT', 'Geneformer', 'scFoundation', 'scLong', 'BioMod'],\n",
-    "    'score': [0.85, 0.89, 0.89, 0.87, 0.88, 0.93],\n",
-    "})\n",
-    "reported_scores.to_csv(PAPER_PLOT_DIR / 'reported_model_comparison_scores.csv', index=False)\n",
-    "fig, ax = plt.subplots(figsize=(4.2, 2.55))\n",
-    "colors = ['#4E79A7', '#59A14F', '#F28E2B', '#76B7B2', '#B07AA1', '#E15759']\n",
-    "x = np.arange(len(reported_scores))\n",
-    "bars = ax.bar(x, reported_scores['score'], width=0.62, color=colors, edgecolor='#333333', linewidth=0.45)\n",
-    "for bar, value, model_name in zip(bars, reported_scores['score'], reported_scores['model']):\n",
-    "    ax.text(\n",
-    "        bar.get_x() + bar.get_width() / 2,\n",
-    "        value + 0.003,\n",
-    "        f'{value:.2f}',\n",
-    "        ha='center',\n",
-    "        va='bottom',\n",
-    "        fontsize=6,\n",
-    "        fontweight='bold' if model_name == 'BioMod' else 'normal',\n",
-    "    )\n",
-    "ax.set_ylabel('Drug-response score')\n",
-    "ax.set_xticks(x)\n",
-    "ax.set_xticklabels(reported_scores['model'], rotation=25, ha='right')\n",
-    "ax.set_ylim(0.80, 0.95)\n",
-    "ax.set_yticks(np.arange(0.80, 0.951, 0.03))\n",
-    "ax.yaxis.set_major_formatter(FormatStrFormatter('%.2f'))\n",
-    "for tick in ax.get_xticklabels():\n",
-    "    if tick.get_text() == 'BioMod':\n",
-    "        tick.set_fontweight('bold')\n",
-    "ax.grid(axis='y', linestyle='--', linewidth=0.4, alpha=0.30)\n",
-    "ax.set_axisbelow(True)\n",
-    "despine(ax)\n",
-    "fig.tight_layout()\n",
-    "comparison_png, comparison_pdf = save_figure(fig, 'drug_response_reported_model_comparison')\n",
-    "\n",
-    "plot_outputs = [\n",
-    "    overall_scatter_png, residual_png, top5_png, cancer_png, roc_png, comparison_png,\n",
-    "]\n",
-    "plot_outputs = [path for path in plot_outputs if path is not None]\n",
-    "display(pd.DataFrame({'plot': plot_outputs}))\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Existing figure outputs\n",
-    "\n",
-    "If the full-run figure files are present, this cell displays the available PNG panels. The corresponding PDF files remain in the same folders."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "figure_candidates = [\n",
-    "    scatter_png,\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_overall_scatter.png',\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_residual_distribution.png',\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_top5_drugs.png',\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_per_cancer_type_pcc.png',\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_median_threshold_roc.png',\n",
-    "    RESULT_DIR / 'drug_response_paper_plots' / 'drug_response_reported_model_comparison.png',\n",
-    "    ACTIVE_RESULT_DIR / 'panel_d' / 'panel_d_most_precise_5_drugs.png',\n",
-    "    ACTIVE_RESULT_DIR / 'roc_plot' / 'drug_response_roc.png',\n",
-    "    ACTIVE_RESULT_DIR / 'roc_plot' / 'predictions_roc.png',\n",
-    "]\n",
-    "\n",
-    "for fig_path in figure_candidates:\n",
-    "    if fig_path.exists():\n",
-    "        print(fig_path)\n",
-    "        if Image is not None:\n",
-    "            display(Image(filename=str(fig_path)))\n",
-    "    else:\n",
-    "        print('Missing:', fig_path)\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Related plotting scripts\n",
-    "\n",
-    "The full project also contains dedicated plotting scripts for paper panels. This notebook reproduces the self-contained `predictions.npz` plots directly. The leave-drug blind and GSEA-style panels require additional CSV/GMT inputs and should be run only after those files are prepared.\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "DRAW_DIR = MODULE_MODEL / 'draw'\n",
-    "draw_scripts = [\n",
-    "    'plot_cancer_drug_response_nature_color.py',\n",
-    "    'plot_drug_response_panel_d.py',\n",
-    "    'plot_drug_response_panel_e.py',\n",
-    "    'plot_drug_response_panel_f.py',\n",
-    "    'plot_drug_response_roc_rewrite.py',\n",
-    "]\n",
-    "display(pd.DataFrame([\n",
-    "    {'script': name, 'path': DRAW_DIR / name, 'exists': (DRAW_DIR / name).exists()}\n",
-    "    for name in draw_scripts\n",
-    "]))\n"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Drug response prediction with BioMod\n",
+    "\n",
+    "This notebook uses the BioMod model weights, drug graph features, and the bundled\n",
+    "GDSC/CCLE response data with `train_drug_performer_module.py`.\n",
+    "\n",
+    "By default, `RUN_TRAINING = False`: the notebook checks inputs and prepares the\n",
+    "training command. Result and plotting cells run only when `predictions.npz`\n",
+    "exists in the configured output directory. Set `RUN_TRAINING = True` to train.\n",
+    "\n",
+    "The validation set selects the checkpoint. Reported metrics describe validation\n",
+    "performance; this workflow does not produce a separate final test evaluation.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Project paths\n",
+    "\n",
+    "Start Jupyter from `BioMod/toturials/`. The next cell uses `Path('../..')` for the\n",
+    "repository root and resolves project-relative paths for the training subprocess.\n",
+    "If Jupyter starts at the repository root, use `Path('.')` instead. All inputs come\n",
+    "from this project; no private server alias or external reference run is required.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inputs and outputs\n",
+    "\n",
+    "- Training script: `BioMod/drug_response/train_drug_performer_module.py`\n",
+    "- Dataset: `data/downstream/final_data/single_drug_response/data/`\n",
+    "- BioMod model weights and configuration: `weight/0810model/`\n",
+    "- Shared model assets: `model_dependencies/`\n",
+    "- Tutorial output: `tutorial_runs/drug_response_prediction/`\n",
+    "\n",
+    "These paths are relative to the repository in this overview. The command cell\n",
+    "prints their resolved absolute paths. The drug graph files are already extracted\n",
+    "under `GDSC/drug_graph_feat/`; no ZIP archive or additional omics matrices are\n",
+    "required by this training script.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import importlib.util\n",
+    "import os\n",
+    "import shlex\n",
+    "import subprocess\n",
+    "import sys\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "\n",
+    "try:\n",
+    "    from IPython.display import Image, display\n",
+    "except ImportError:\n",
+    "    Image = None\n",
+    "    def display(obj):\n",
+    "        print(obj)\n",
+    "\n",
+    "from scipy.stats import pearsonr\n",
+    "\n",
+    "# Start Jupyter with BioMod/toturials as the working directory.\n",
+    "# If Jupyter starts at the repository root, change this line to Path('.').resolve().\n",
+    "PROJECT_ROOT = Path('../..').resolve()\n",
+    "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
+    "if not (SOURCE_DIR / 'models_performer_module.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
+    "    raise FileNotFoundError(\n",
+    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        \"(use Path('.').resolve() if Jupyter was launched at that root).\"\n",
+    "    )\n",
+    "REPO_ROOT = PROJECT_ROOT\n",
+    "\n",
+    "DEPENDENCY_ROOT = REPO_ROOT / 'model_dependencies'\n",
+    "WEIGHT_DIR = REPO_ROOT / 'weight' / '0810model'\n",
+    "MODULE_DATA_DIR = DEPENDENCY_ROOT / 'module_data_all'\n",
+    "TRAIN_PYTHON = Path(sys.executable).resolve()\n",
+    "PYTHONPATH = os.pathsep.join(\n",
+    "    [str(SOURCE_DIR), str(DEPENDENCY_ROOT)]\n",
+    "    + ([os.environ['PYTHONPATH']] if os.environ.get('PYTHONPATH') else [])\n",
+    ")\n",
+    "\n",
+    "TASK_ROOT = SOURCE_DIR / 'drug_response'\n",
+    "SCRIPT = TASK_ROOT / 'train_drug_performer_module.py'\n",
+    "DATA_ROOT = REPO_ROOT / 'data/downstream/final_data/single_drug_response/data'\n",
+    "CKPT_PATH = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2_epoch0.pth'\n",
+    "CONFIG_PKL = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2.pkl'\n",
+    "GENE2VEC = DEPENDENCY_ROOT / 'selected_gene2vec_27k.npy'\n",
+    "GENE_MAP = DEPENDENCY_ROOT / 'gene2ens.csv'\n",
+    "GENE_LIST = DEPENDENCY_ROOT / 'selected_genes_27k.txt'\n",
+    "RESULT_DIR = REPO_ROOT / 'tutorial_runs/drug_response_prediction'\n",
+    "\n",
+    "paths = pd.DataFrame([\n",
+    "    {'name': name, 'path': str(path), 'exists': path.exists()}\n",
+    "    for name, path in {\n",
+    "        'repository': REPO_ROOT, 'script': SCRIPT, 'data_root': DATA_ROOT,\n",
+    "        'checkpoint': CKPT_PATH, 'config_pkl': CONFIG_PKL,\n",
+    "        'module_data_dir': MODULE_DATA_DIR, 'gene_list': GENE_LIST,\n",
+    "        'gene_map': GENE_MAP, 'gene2vec': GENE2VEC,\n",
+    "        'tutorial_result_dir': RESULT_DIR,\n",
+    "    }.items()\n",
+    "])\n",
+    "display(paths)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "required_files = [\n",
+    "    SCRIPT, CKPT_PATH, CONFIG_PKL, GENE_LIST, GENE_MAP, GENE2VEC,\n",
+    "    DATA_ROOT / 'CCLE/GDSC_IC50.csv',\n",
+    "    DATA_ROOT / 'CCLE/Cell_lines_annotations_20181226.txt',\n",
+    "    DATA_ROOT / 'CCLE/genomic_expression_561celllines_697genes_demap_features.csv',\n",
+    "    DATA_ROOT / 'GDSC/1.Drug_listMon Jun 24 09_00_55 2019.csv',\n",
+    "]\n",
+    "required_dirs = [PROJECT_ROOT, TASK_ROOT, MODULE_DATA_DIR,\n",
+    "                 DEPENDENCY_ROOT / 'performer_pytorch_cont', DATA_ROOT / 'GDSC/drug_graph_feat']\n",
+    "missing = [str(path) for path in required_files if not path.is_file()]\n",
+    "missing += [str(path) for path in required_dirs if not path.is_dir()]\n",
+    "if missing:\n",
+    "    raise FileNotFoundError('Missing required inputs:\\n' + '\\n'.join(missing))\n",
+    "if not any((DATA_ROOT / 'GDSC/drug_graph_feat').glob('*.hkl')):\n",
+    "    raise FileNotFoundError('No .hkl drug graph files found under GDSC/drug_graph_feat.')\n",
+    "print('All required inputs are present. Weight/configuration files have not been loaded.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Runtime setup\n",
+    "\n",
+    "Training uses the active notebook kernel's Python interpreter (`sys.executable`).\n",
+    "The dependency check reports missing packages without installing them automatically.\n",
+    "Install them in this environment before enabling training.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('Notebook and training Python:', TRAIN_PYTHON)\n",
+    "packages = {\n",
+    "    'torch': 'torch', 'hickle': 'hickle', 'scipy': 'scipy',\n",
+    "    'numpy': 'numpy', 'pandas': 'pandas', 'matplotlib': 'matplotlib',\n",
+    "    'sklearn': 'scikit-learn', 'einops': 'einops', 'local_attention': 'local-attention',\n",
+    "}\n",
+    "dependency_status = pd.DataFrame([\n",
+    "    {'module': module, 'package': package,\n",
+    "     'available': importlib.util.find_spec(module) is not None}\n",
+    "    for module, package in packages.items()\n",
+    "])\n",
+    "display(dependency_status)\n",
+    "missing_packages = dependency_status.loc[~dependency_status['available'], 'package'].tolist()\n",
+    "if missing_packages:\n",
+    "    install_cmd = [str(TRAIN_PYTHON), '-m', 'pip', 'install', *missing_packages]\n",
+    "    print(subprocess.list2cmdline(install_cmd) if os.name == 'nt' else shlex.join(install_cmd))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inspect input files\n",
+    "\n",
+    "This cell gives a quick sanity check for the drug-response files without loading the large full matrices into memory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def size_mb(path: Path) -> float:\n",
+    "    return path.stat().st_size / 1024 / 1024\n",
+    "\n",
+    "input_summary = []\n",
+    "for path in sorted((DATA_ROOT / 'CCLE').glob('*')):\n",
+    "    if path.is_file():\n",
+    "        input_summary.append({'file': str(path.relative_to(DATA_ROOT)), 'size_mb': round(size_mb(path), 2)})\n",
+    "for path in sorted((DATA_ROOT / 'GDSC').glob('*')):\n",
+    "    if path.is_file():\n",
+    "        input_summary.append({'file': str(path.relative_to(DATA_ROOT)), 'size_mb': round(size_mb(path), 2)})\n",
+    "\n",
+    "display(pd.DataFrame(input_summary))\n",
+    "\n",
+    "ic50_path = DATA_ROOT / 'CCLE' / 'GDSC_IC50.csv'\n",
+    "ic50_preview = pd.read_csv(ic50_path, nrows=5)\n",
+    "display(ic50_preview)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training command\n",
+    "\n",
+    "Review the resolved input and output paths below. Outputs are written to this\n",
+    "project's tutorial run directory. Change `RESULT_DIR` to a different run directory\n",
+    "before launching an experiment whose outputs you want to keep separate.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "EPOCHS = 300\n",
+    "BATCH_SIZE = 32\n",
+    "POOL_METHOD = 'max'\n",
+    "\n",
+    "cmd = [\n",
+    "    str(TRAIN_PYTHON), str(SCRIPT),\n",
+    "    '--gpu_id', '0',\n",
+    "    '--data_path', str(DATA_ROOT),\n",
+    "    '--checkpoint_path', str(CKPT_PATH),\n",
+    "    '--hyper_params_path', str(CONFIG_PKL),\n",
+    "    '--epochs', str(EPOCHS),\n",
+    "    '--batch_size', str(BATCH_SIZE),\n",
+    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
+    "    '--gene_list_path', str(GENE_LIST),\n",
+    "    '--gene_mapping_path', str(GENE_MAP),\n",
+    "    '--symbol2ens_path', str(GENE_MAP),\n",
+    "    '--gene2vec_path', str(GENE2VEC),\n",
+    "    '--save_dir', str(RESULT_DIR),\n",
+    "    '--pool_method', POOL_METHOD,\n",
+    "]\n",
+    "\n",
+    "print('Working directory:', TASK_ROOT)\n",
+    "print(subprocess.list2cmdline(cmd) if os.name == 'nt' else shlex.join(cmd))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "RUN_TRAINING = False\n",
+    "env = os.environ.copy()\n",
+    "env['PYTHONPATH'] = PYTHONPATH\n",
+    "env['PYTHONUNBUFFERED'] = '1'\n",
+    "\n",
+    "if RUN_TRAINING:\n",
+    "    RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    log_path = RESULT_DIR / 'train.log'\n",
+    "    print('Writing log to:', log_path)\n",
+    "    with log_path.open('w') as log_file:\n",
+    "        proc = subprocess.Popen(\n",
+    "            cmd,\n",
+    "            cwd=str(TASK_ROOT),\n",
+    "            env=env,\n",
+    "            stdout=subprocess.PIPE,\n",
+    "            stderr=subprocess.STDOUT,\n",
+    "            text=True,\n",
+    "            bufsize=1,\n",
+    "        )\n",
+    "        assert proc.stdout is not None\n",
+    "        for line in proc.stdout:\n",
+    "            print(line, end='')\n",
+    "            log_file.write(line)\n",
+    "        return_code = proc.wait()\n",
+    "    if return_code != 0:\n",
+    "        raise RuntimeError(f'Training failed with exit code {return_code}')\n",
+    "else:\n",
+    "    print('Training was not launched. Set RUN_TRAINING = True to run the full job.')\n",
+    "    print('The following cells inspect results only if predictions.npz exists in RESULT_DIR.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inspect outputs\n",
+    "\n",
+    "Only the configured `RESULT_DIR` is used. The current script saves numerical arrays\n",
+    "under the `pred` and `validation` keys in `predictions.npz`. The reader also accepts\n",
+    "the historical `preds`, `test`, and `true` key names. Missing results are skipped.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ACTIVE_RESULT_DIR = RESULT_DIR\n",
+    "pred_path = ACTIVE_RESULT_DIR / 'predictions.npz'\n",
+    "RESULTS_READY = pred_path.is_file()\n",
+    "if RESULTS_READY:\n",
+    "    print('Active result directory:', ACTIVE_RESULT_DIR)\n",
+    "    display(pd.DataFrame([\n",
+    "        {'file': str(path.relative_to(ACTIVE_RESULT_DIR)), 'size_mb': round(size_mb(path), 3)}\n",
+    "        for path in sorted(ACTIVE_RESULT_DIR.rglob('*')) if path.is_file()\n",
+    "    ]))\n",
+    "else:\n",
+    "    print('No predictions.npz found; result analysis and plots will be skipped:', pred_path)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "meta = pd.DataFrame(columns=['pred', 'true', 'drug_id', 'cell_line_id', 'cancer_type'])\n",
+    "pred = true = np.array([], dtype=float)\n",
+    "if RESULTS_READY:\n",
+    "    with np.load(pred_path, allow_pickle=True) as pred_npz:\n",
+    "        print('Prediction keys:', pred_npz.files)\n",
+    "        pred_key = next((key for key in ('pred', 'preds') if key in pred_npz.files), None)\n",
+    "        true_key = next((key for key in ('validation', 'test', 'true') if key in pred_npz.files), None)\n",
+    "        if pred_key is None or true_key is None:\n",
+    "            raise ValueError('Prediction file needs pred/preds and validation/test/true arrays.')\n",
+    "        pred = np.asarray(pred_npz[pred_key], dtype=float).reshape(-1)\n",
+    "        true = np.asarray(pred_npz[true_key], dtype=float).reshape(-1)\n",
+    "        if len(pred) != len(true):\n",
+    "            raise ValueError('Prediction and target lengths do not match.')\n",
+    "        meta = pd.DataFrame({'pred': pred, 'true': true})\n",
+    "        for column in ['drug_id', 'cell_line_id', 'cancer_type']:\n",
+    "            meta[column] = np.asarray(pred_npz[column]).reshape(-1) if column in pred_npz.files else 'NA'\n",
+    "    finite = np.isfinite(pred) & np.isfinite(true)\n",
+    "    pred, true = pred[finite], true[finite]\n",
+    "    RESULTS_READY = bool(len(pred))\n",
+    "    if RESULTS_READY:\n",
+    "        pcc = float(pearsonr(pred, true)[0]) if len(pred) >= 2 and np.std(pred) > 0 and np.std(true) > 0 else np.nan\n",
+    "        summary = pd.DataFrame([{'n_validation': len(pred), 'pcc': pcc,\n",
+    "                                 'mae': float(np.mean(np.abs(pred - true))),\n",
+    "                                 'rmse': float(np.sqrt(np.mean((pred - true) ** 2)))}])\n",
+    "        display(summary)\n",
+    "        display(meta.head())\n",
+    "    else:\n",
+    "        print('No finite prediction/target pairs; result plots will be skipped.')\n",
+    "else:\n",
+    "    print('Prediction metrics skipped: no output file is available.')\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RESULTS_READY:\n",
+    "    import random\n",
+    "    from scipy.stats import spearmanr\n",
+    "\n",
+    "    TCGA_LABEL_SET = [\n",
+    "        'ALL', 'BLCA', 'BRCA', 'CESC', 'DLBC', 'LIHC', 'LUAD', 'ESCA', 'GBM',\n",
+    "        'HNSC', 'KIRC', 'LAML', 'LCML', 'LGG', 'LUSC', 'MESO', 'MM', 'NB',\n",
+    "        'OV', 'PAAD', 'SCLC', 'SKCM', 'STAD', 'THCA', 'COAD/READ',\n",
+    "    ]\n",
+    "\n",
+    "\n",
+    "    def pearson_corr(x, y):\n",
+    "        x = np.asarray(x, dtype=float).reshape(-1)\n",
+    "        y = np.asarray(y, dtype=float).reshape(-1)\n",
+    "        mask = np.isfinite(x) & np.isfinite(y)\n",
+    "        x = x[mask]\n",
+    "        y = y[mask]\n",
+    "        if len(x) < 2 or np.std(x) == 0 or np.std(y) == 0:\n",
+    "            return np.nan\n",
+    "        return float(np.corrcoef(x, y)[0, 1])\n",
+    "\n",
+    "\n",
+    "    def regression_metrics(frame: pd.DataFrame) -> dict:\n",
+    "        y_true = frame['true'].astype(float).to_numpy()\n",
+    "        y_pred = frame['pred'].astype(float).to_numpy()\n",
+    "        return {\n",
+    "            'n': int(len(frame)),\n",
+    "            'pcc': pearson_corr(y_true, y_pred),\n",
+    "            'spearman': float(spearmanr(y_true, y_pred).correlation) if len(frame) >= 2 else np.nan,\n",
+    "            'mae': float(np.mean(np.abs(y_pred - y_true))),\n",
+    "            'mse': float(np.mean((y_pred - y_true) ** 2)),\n",
+    "            'rmse': float(np.sqrt(np.mean((y_pred - y_true) ** 2))),\n",
+    "        }\n",
+    "\n",
+    "\n",
+    "    def reconstruct_validation_metadata(data_root: Path, n_expected: int, ratio: float = 0.95, seed: int = 0) -> pd.DataFrame:\n",
+    "        \"\"\"Rebuild the validation split metadata using the same deterministic split as the training script.\"\"\"\n",
+    "        drug_info_file = data_root / 'GDSC' / '1.Drug_listMon Jun 24 09_00_55 2019.csv'\n",
+    "        cell_line_info_file = data_root / 'CCLE' / 'Cell_lines_annotations_20181226.txt'\n",
+    "        drug_feature_dir = data_root / 'GDSC' / 'drug_graph_feat'\n",
+    "        gene_expression_file = data_root / 'CCLE' / 'genomic_expression_561celllines_697genes_demap_features.csv'\n",
+    "        ic50_file = data_root / 'CCLE' / 'GDSC_IC50.csv'\n",
+    "\n",
+    "        for required in [drug_info_file, cell_line_info_file, drug_feature_dir, gene_expression_file, ic50_file]:\n",
+    "            if not required.exists():\n",
+    "                raise FileNotFoundError(required)\n",
+    "\n",
+    "        drug_info = pd.read_csv(drug_info_file)\n",
+    "        drugid2pubchemid = {\n",
+    "            str(row.iloc[0]): str(row.iloc[5])\n",
+    "            for _, row in drug_info.iterrows()\n",
+    "            if str(row.iloc[5]).isdigit()\n",
+    "        }\n",
+    "\n",
+    "        cellline2cancertype = {}\n",
+    "        with cell_line_info_file.open() as handle:\n",
+    "            for line in handle.readlines()[1:]:\n",
+    "                parts = line.rstrip('\\n').split('\\t')\n",
+    "                if len(parts) >= 2:\n",
+    "                    cellline2cancertype[parts[1]] = parts[-1]\n",
+    "\n",
+    "        gexpr_index = set(pd.read_csv(gene_expression_file, usecols=[0]).iloc[:, 0].astype(str))\n",
+    "        drug_pubchem_ids = {path.stem for path in drug_feature_dir.glob('*.hkl')}\n",
+    "\n",
+    "        ic50 = pd.read_csv(ic50_file, header=0, index_col=0)\n",
+    "        drug_match_list = [idx for idx in ic50.index if idx.split(':')[1] in drugid2pubchemid]\n",
+    "        ic50 = ic50.loc[drug_match_list]\n",
+    "\n",
+    "        data_idx = []\n",
+    "        for each_drug in ic50.index:\n",
+    "            pubchem_id = drugid2pubchemid[each_drug.split(':')[-1]]\n",
+    "            if pubchem_id not in drug_pubchem_ids:\n",
+    "                continue\n",
+    "            for each_cellline in ic50.columns:\n",
+    "                if each_cellline not in gexpr_index or each_cellline not in cellline2cancertype:\n",
+    "                    continue\n",
+    "                value = ic50.loc[each_drug, each_cellline]\n",
+    "                if pd.isna(value):\n",
+    "                    continue\n",
+    "                data_idx.append((each_cellline, pubchem_id, float(value), cellline2cancertype[each_cellline]))\n",
+    "\n",
+    "        rng = random.Random(seed)\n",
+    "        validation_idx = []\n",
+    "        for each_type in TCGA_LABEL_SET:\n",
+    "            subtype = [item for item in data_idx if item[-1] == each_type]\n",
+    "            if not subtype:\n",
+    "                continue\n",
+    "            k = max(1, int(ratio * len(subtype)))\n",
+    "            train_list = rng.sample(subtype, k)\n",
+    "            validation_idx.extend([item for item in subtype if item not in train_list])\n",
+    "\n",
+    "        reconstructed = pd.DataFrame(validation_idx, columns=['cell_line_id', 'drug_id', 'true_from_metadata', 'cancer_type'])\n",
+    "        if len(reconstructed) != n_expected:\n",
+    "            raise ValueError(f'Reconstructed {len(reconstructed)} rows, but predictions contain {n_expected} rows.')\n",
+    "        return reconstructed\n",
+    "\n",
+    "\n",
+    "    analysis_df = meta.copy()\n",
+    "    analysis_df['pred'] = pd.to_numeric(analysis_df['pred'], errors='coerce')\n",
+    "    analysis_df['true'] = pd.to_numeric(analysis_df['true'], errors='coerce')\n",
+    "\n",
+    "    metadata_cols = ['drug_id', 'cell_line_id', 'cancer_type']\n",
+    "    metadata_missing = any(col not in analysis_df.columns for col in metadata_cols) or all(\n",
+    "        analysis_df[col].astype(str).isin(['NA', 'nan', 'None', '']).all() for col in metadata_cols if col in analysis_df.columns\n",
+    "    )\n",
+    "\n",
+    "    if metadata_missing:\n",
+    "        try:\n",
+    "            recovered_meta = reconstruct_validation_metadata(DATA_ROOT, n_expected=len(analysis_df), ratio=0.95, seed=0)\n",
+    "            if not np.allclose(analysis_df['true'].to_numpy(dtype=float), recovered_meta['true_from_metadata'].to_numpy(dtype=float), rtol=1e-5, atol=1e-6, equal_nan=True):\n",
+    "                raise ValueError('Reconstructed targets do not match the prediction rows; metadata will not be assigned.')\n",
+    "            for col in metadata_cols:\n",
+    "                analysis_df[col] = recovered_meta[col].to_numpy()\n",
+    "            print('Recovered validation metadata from the original GDSC/CCLE split.')\n",
+    "        except Exception as exc:\n",
+    "            print('Metadata recovery skipped:', repr(exc))\n",
+    "            for col in metadata_cols:\n",
+    "                if col not in analysis_df.columns:\n",
+    "                    analysis_df[col] = 'NA'\n",
+    "    else:\n",
+    "        print('Using drug/cell/cancer metadata stored in predictions.npz.')\n",
+    "\n",
+    "    analysis_df = analysis_df.replace([np.inf, -np.inf], np.nan).dropna(subset=['pred', 'true']).reset_index(drop=True)\n",
+    "\n",
+    "    overall_metrics = pd.DataFrame([regression_metrics(analysis_df)])\n",
+    "    display(overall_metrics)\n",
+    "    display(analysis_df.head())\n",
+    "\n",
+    "    metric_columns = ['pred', 'true']\n",
+    "    by_cancer = (\n",
+    "        analysis_df.groupby('cancer_type', dropna=False)[metric_columns]\n",
+    "        .apply(lambda frame: pd.Series(regression_metrics(frame)))\n",
+    "        .reset_index()\n",
+    "        .sort_values('n', ascending=False)\n",
+    "    )\n",
+    "    by_drug = (\n",
+    "        analysis_df.groupby('drug_id', dropna=False)[metric_columns]\n",
+    "        .apply(lambda frame: pd.Series(regression_metrics(frame)))\n",
+    "        .reset_index()\n",
+    "        .sort_values('pcc', ascending=False)\n",
+    "    )\n",
+    "\n",
+    "    display(by_cancer.head(12))\n",
+    "    display(by_drug.head(10))\n",
+    "else:\n",
+    "    print('Grouped analysis skipped: no usable predictions are available.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Validation scatter plot\n",
+    "\n",
+    "When predictions exist, this cell saves a PDF and PNG of predicted versus measured\n",
+    "drug response under `RESULT_DIR`. No external reference-run directory is used.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RESULTS_READY:\n",
+    "    import matplotlib.pyplot as plt\n",
+    "\n",
+    "    plt.rcParams.update({\n",
+    "        'font.family': 'sans-serif',\n",
+    "        'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
+    "        'font.size': 7,\n",
+    "        'pdf.fonttype': 42,\n",
+    "        'ps.fonttype': 42,\n",
+    "    })\n",
+    "\n",
+    "    RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    scatter_png = RESULT_DIR / 'drug_response_prediction_scatter.png'\n",
+    "    scatter_pdf = RESULT_DIR / 'drug_response_prediction_scatter.pdf'\n",
+    "\n",
+    "    low = float(min(np.min(true), np.min(pred)))\n",
+    "    high = float(max(np.max(true), np.max(pred)))\n",
+    "    pad = max(0.05 * (high - low), 0.05)\n",
+    "    lims = (low - pad, high + pad)\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(3.35, 3.1))\n",
+    "    ax.scatter(true, pred, s=7, alpha=0.45, linewidths=0, color='#2f6fb0')\n",
+    "    ax.plot(lims, lims, color='#444444', linewidth=0.8, linestyle='--')\n",
+    "    ax.set_xlim(lims)\n",
+    "    ax.set_ylim(lims)\n",
+    "    ax.set_xlabel('Measured drug response')\n",
+    "    ax.set_ylabel('Predicted drug response')\n",
+    "    ax.set_title('Drug response prediction')\n",
+    "    ax.text(0.03, 0.97, f'PCC = {pcc:.3f}\\nN = {len(pred)}', transform=ax.transAxes, va='top')\n",
+    "    ax.spines['top'].set_visible(False)\n",
+    "    ax.spines['right'].set_visible(False)\n",
+    "    fig.tight_layout()\n",
+    "    fig.savefig(scatter_png, dpi=300, bbox_inches='tight')\n",
+    "    fig.savefig(scatter_pdf, bbox_inches='tight')\n",
+    "    plt.close(fig)\n",
+    "\n",
+    "    print('Saved:', scatter_png)\n",
+    "    print('Saved:', scatter_pdf)\n",
+    "    if Image is not None:\n",
+    "        display(Image(filename=str(scatter_png)))\n",
+    "else:\n",
+    "    print('Scatter plot skipped: no usable predictions are available.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Additional validation plots\n",
+    "\n",
+    "These self-contained plots use the current `predictions.npz`: overall scatter,\n",
+    "residuals, per-drug and per-cancer summaries when matching metadata is available,\n",
+    "and an exploratory median-threshold ROC. The median-threshold classification is a\n",
+    "derived visualization of the regression outputs, not an independent test result.\n",
+    "No hardcoded model-comparison scores or external plotting scripts are used.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if RESULTS_READY:\n",
+    "    from matplotlib.ticker import FormatStrFormatter\n",
+    "\n",
+    "    try:\n",
+    "        from sklearn.metrics import roc_curve, roc_auc_score\n",
+    "    except Exception as exc:\n",
+    "        roc_curve = None\n",
+    "        roc_auc_score = None\n",
+    "        print('ROC dependencies unavailable:', repr(exc))\n",
+    "\n",
+    "    PAPER_PLOT_DIR = RESULT_DIR / 'drug_response_paper_plots'\n",
+    "    PAPER_PLOT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "    plt.rcParams.update({\n",
+    "        'font.family': 'sans-serif',\n",
+    "        'font.sans-serif': ['Arial', 'DejaVu Sans'],\n",
+    "        'font.size': 7,\n",
+    "        'axes.labelsize': 7,\n",
+    "        'axes.titlesize': 8,\n",
+    "        'xtick.labelsize': 6,\n",
+    "        'ytick.labelsize': 6,\n",
+    "        'legend.fontsize': 6,\n",
+    "        'axes.linewidth': 0.7,\n",
+    "        'pdf.fonttype': 42,\n",
+    "        'ps.fonttype': 42,\n",
+    "        'svg.fonttype': 'none',\n",
+    "    })\n",
+    "\n",
+    "\n",
+    "    def save_figure(fig, stem: str, dpi: int = 600):\n",
+    "        png_path = PAPER_PLOT_DIR / f'{stem}.png'\n",
+    "        pdf_path = PAPER_PLOT_DIR / f'{stem}.pdf'\n",
+    "        fig.savefig(png_path, dpi=dpi, bbox_inches='tight')\n",
+    "        fig.savefig(pdf_path, bbox_inches='tight')\n",
+    "        plt.close(fig)\n",
+    "        print('Saved:', png_path)\n",
+    "        print('Saved:', pdf_path)\n",
+    "        return png_path, pdf_path\n",
+    "\n",
+    "\n",
+    "    def despine(ax):\n",
+    "        ax.spines['top'].set_visible(False)\n",
+    "        ax.spines['right'].set_visible(False)\n",
+    "        ax.tick_params(width=0.7, length=2.5)\n",
+    "\n",
+    "\n",
+    "    def finite_xy(frame: pd.DataFrame):\n",
+    "        y_true = pd.to_numeric(frame['true'], errors='coerce').to_numpy(dtype=float)\n",
+    "        y_pred = pd.to_numeric(frame['pred'], errors='coerce').to_numpy(dtype=float)\n",
+    "        mask = np.isfinite(y_true) & np.isfinite(y_pred)\n",
+    "        return y_true[mask], y_pred[mask]\n",
+    "\n",
+    "\n",
+    "    # 1) Overall measured-vs-predicted scatter.\n",
+    "    y_true, y_pred = finite_xy(analysis_df)\n",
+    "    low = float(min(y_true.min(), y_pred.min()))\n",
+    "    high = float(max(y_true.max(), y_pred.max()))\n",
+    "    pad = max(0.05 * (high - low), 0.05)\n",
+    "    lims = (low - pad, high + pad)\n",
+    "    metrics = regression_metrics(analysis_df)\n",
+    "\n",
+    "    fig, ax = plt.subplots(figsize=(3.35, 3.15))\n",
+    "    ax.scatter(y_true, y_pred, s=5, alpha=0.35, linewidths=0, color='#2F6FB0', rasterized=True)\n",
+    "    ax.plot(lims, lims, color='#333333', linewidth=0.8, linestyle='--')\n",
+    "    ax.set_xlim(lims)\n",
+    "    ax.set_ylim(lims)\n",
+    "    ax.set_xlabel('Measured drug response')\n",
+    "    ax.set_ylabel('Predicted drug response')\n",
+    "    ax.set_title('BioMod drug response')\n",
+    "    ax.text(\n",
+    "        0.03, 0.97,\n",
+    "        f'PCC = {metrics[\"pcc\"]:.3f}\\nRMSE = {metrics[\"rmse\"]:.3f}\\nN = {metrics[\"n\"]}',\n",
+    "        transform=ax.transAxes,\n",
+    "        va='top',\n",
+    "        ha='left',\n",
+    "    )\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    overall_scatter_png, overall_scatter_pdf = save_figure(fig, 'drug_response_overall_scatter')\n",
+    "\n",
+    "    # 2) Residual distribution.\n",
+    "    residual = y_pred - y_true\n",
+    "    fig, ax = plt.subplots(figsize=(3.35, 2.35))\n",
+    "    ax.hist(residual, bins=45, color='#76B7B2', edgecolor='white', linewidth=0.3)\n",
+    "    ax.axvline(0, color='#333333', linestyle='--', linewidth=0.8)\n",
+    "    ax.set_xlabel('Prediction residual')\n",
+    "    ax.set_ylabel('Number of samples')\n",
+    "    ax.set_title('Residual distribution')\n",
+    "    despine(ax)\n",
+    "    fig.tight_layout()\n",
+    "    residual_png, residual_pdf = save_figure(fig, 'drug_response_residual_distribution')\n",
+    "\n",
+    "    # 3) Top-5 most precise drugs, among drugs with sufficient validation samples.\n",
+    "    MIN_SAMPLES_PER_DRUG = 3\n",
+    "    valid_drug_stats = by_drug[(by_drug['drug_id'].astype(str) != 'NA') & (by_drug['n'] >= MIN_SAMPLES_PER_DRUG)].dropna(subset=['pcc'])\n",
+    "    if len(valid_drug_stats) > 0:\n",
+    "        top5_drugs = valid_drug_stats.sort_values('pcc', ascending=False).head(5)['drug_id'].astype(str).tolist()\n",
+    "        top5_summary = valid_drug_stats[valid_drug_stats['drug_id'].astype(str).isin(top5_drugs)].copy()\n",
+    "        top5_points = analysis_df[analysis_df['drug_id'].astype(str).isin(top5_drugs)].copy()\n",
+    "        top5_summary.to_csv(PAPER_PLOT_DIR / 'top5_drug_pcc_summary.csv', index=False)\n",
+    "        top5_points.to_csv(PAPER_PLOT_DIR / 'top5_drug_selected_points.csv', index=False)\n",
+    "\n",
+    "        fig, ax = plt.subplots(figsize=(3.65, 3.25))\n",
+    "        palette = plt.get_cmap('tab10')\n",
+    "        for idx, drug_id in enumerate(top5_drugs):\n",
+    "            sub = top5_points[top5_points['drug_id'].astype(str) == drug_id]\n",
+    "            ax.scatter(\n",
+    "                sub['true'], sub['pred'],\n",
+    "                s=10, alpha=0.75, linewidths=0,\n",
+    "                color=palette(idx), label=str(drug_id), rasterized=True,\n",
+    "            )\n",
+    "        t_true, t_pred = finite_xy(top5_points)\n",
+    "        low = float(min(t_true.min(), t_pred.min()))\n",
+    "        high = float(max(t_true.max(), t_pred.max()))\n",
+    "        pad = max(0.05 * (high - low), 0.05)\n",
+    "        lims = (low - pad, high + pad)\n",
+    "        ax.plot(lims, lims, color='#333333', linewidth=0.8, linestyle='--')\n",
+    "        ax.set_xlim(lims)\n",
+    "        ax.set_ylim(lims)\n",
+    "        ax.set_xlabel('Measured drug response')\n",
+    "        ax.set_ylabel('Predicted drug response')\n",
+    "        ax.set_title('Most precise 5 drugs')\n",
+    "        ax.legend(title='PubChem ID', frameon=False, loc='best', handletextpad=0.3, borderpad=0.2)\n",
+    "        despine(ax)\n",
+    "        fig.tight_layout()\n",
+    "        top5_png, top5_pdf = save_figure(fig, 'drug_response_top5_drugs')\n",
+    "    else:\n",
+    "        top5_png = top5_pdf = None\n",
+    "        print('Top-5 drug panel skipped: drug_id metadata is unavailable or too sparse.')\n",
+    "\n",
+    "    # 4) Per-cancer-type PCC bar plot.\n",
+    "    valid_cancer = by_cancer[(by_cancer['cancer_type'].astype(str) != 'NA') & (by_cancer['n'] >= 2)].dropna(subset=['pcc']).copy()\n",
+    "    if len(valid_cancer) > 0:\n",
+    "        valid_cancer = valid_cancer.sort_values('pcc', ascending=True)\n",
+    "        valid_cancer.to_csv(PAPER_PLOT_DIR / 'per_cancer_type_metrics.csv', index=False)\n",
+    "        fig_height = max(2.2, 0.16 * len(valid_cancer) + 0.7)\n",
+    "        fig, ax = plt.subplots(figsize=(3.7, fig_height))\n",
+    "        colors = ['#4E79A7' if value < valid_cancer['pcc'].median() else '#59A14F' for value in valid_cancer['pcc']]\n",
+    "        ax.barh(valid_cancer['cancer_type'].astype(str), valid_cancer['pcc'], color=colors, edgecolor='#333333', linewidth=0.35)\n",
+    "        ax.set_xlabel('Pearson correlation')\n",
+    "        ax.set_ylabel('Cancer type')\n",
+    "        ax.set_xlim(min(0, float(valid_cancer['pcc'].min()) - 0.05), 1.0)\n",
+    "        ax.xaxis.set_major_formatter(FormatStrFormatter('%.2f'))\n",
+    "        ax.grid(axis='x', linestyle='--', linewidth=0.4, alpha=0.35)\n",
+    "        ax.set_axisbelow(True)\n",
+    "        ax.set_title('Performance by cancer type')\n",
+    "        despine(ax)\n",
+    "        fig.tight_layout()\n",
+    "        cancer_png, cancer_pdf = save_figure(fig, 'drug_response_per_cancer_type_pcc')\n",
+    "    else:\n",
+    "        cancer_png = cancer_pdf = None\n",
+    "        print('Cancer-type PCC plot skipped: cancer_type metadata is unavailable or too sparse.')\n",
+    "\n",
+    "    # 5) Median-threshold ROC, an exploratory view of the regression outputs.\n",
+    "    if roc_curve is not None and roc_auc_score is not None:\n",
+    "        threshold = float(np.median(y_true))\n",
+    "        y_binary = (y_true <= threshold).astype(int)\n",
+    "        roc_score = -y_pred\n",
+    "        if len(np.unique(y_binary)) == 2:\n",
+    "            fpr, tpr, roc_thresholds = roc_curve(y_binary, roc_score)\n",
+    "            auc_value = float(roc_auc_score(y_binary, roc_score))\n",
+    "            roc_table = pd.DataFrame({'fpr': fpr, 'tpr': tpr, 'threshold': roc_thresholds})\n",
+    "            roc_table.to_csv(PAPER_PLOT_DIR / 'drug_response_roc_curve.csv', index=False)\n",
+    "\n",
+    "            fig, ax = plt.subplots(figsize=(3.1, 3.1))\n",
+    "            ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
+    "            ax.plot(fpr, tpr, color='#9C6ADE', linewidth=1.1, label=f'AUROC = {auc_value:.3f}')\n",
+    "            ax.set_xlabel('False positive rate')\n",
+    "            ax.set_ylabel('True positive rate')\n",
+    "            ax.set_xlim(0, 1)\n",
+    "            ax.set_ylim(0, 1)\n",
+    "            ax.set_aspect('equal', adjustable='box')\n",
+    "            ax.legend(frameon=False, loc='lower right')\n",
+    "            ax.set_title('Median-threshold ROC')\n",
+    "            despine(ax)\n",
+    "            fig.tight_layout()\n",
+    "            roc_png, roc_pdf = save_figure(fig, 'drug_response_median_threshold_roc')\n",
+    "            fig_svg = PAPER_PLOT_DIR / 'drug_response_median_threshold_roc.svg'\n",
+    "            fig, ax = plt.subplots(figsize=(3.1, 3.1))\n",
+    "            ax.plot([0, 1], [0, 1], linestyle='--', color='#333333', linewidth=0.8, alpha=0.7)\n",
+    "            ax.plot(fpr, tpr, color='#9C6ADE', linewidth=1.1, label=f'AUROC = {auc_value:.3f}')\n",
+    "            ax.set_xlabel('False positive rate')\n",
+    "            ax.set_ylabel('True positive rate')\n",
+    "            ax.set_xlim(0, 1)\n",
+    "            ax.set_ylim(0, 1)\n",
+    "            ax.set_aspect('equal', adjustable='box')\n",
+    "            ax.legend(frameon=False, loc='lower right')\n",
+    "            ax.set_title('Median-threshold ROC')\n",
+    "            despine(ax)\n",
+    "            fig.tight_layout()\n",
+    "            fig.savefig(fig_svg, bbox_inches='tight')\n",
+    "            plt.close(fig)\n",
+    "            print('Saved:', fig_svg)\n",
+    "        else:\n",
+    "            roc_png = roc_pdf = None\n",
+    "            print('ROC plot skipped: median split produced a single class.')\n",
+    "    else:\n",
+    "        roc_png = roc_pdf = None\n",
+    "\n",
+    "\n",
+    "    plot_outputs = [overall_scatter_png, residual_png, top5_png, cancer_png, roc_png]\n",
+    "    plot_outputs = [path for path in plot_outputs if path is not None]\n",
+    "    display(pd.DataFrame({'plot': plot_outputs}))\n",
+    "else:\n",
+    "    print('Additional plots skipped: no usable predictions are available.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Available figure outputs\n",
+    "\n",
+    "Display figures that exist under the configured tutorial output directory.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "figure_candidates = [RESULT_DIR / 'drug_response_prediction_scatter.png']\n",
+    "figure_candidates += sorted((RESULT_DIR / 'drug_response_paper_plots').glob('*.png'))\n",
+    "available_figures = [path for path in figure_candidates if path.is_file()]\n",
+    "if available_figures:\n",
+    "    for fig_path in available_figures:\n",
+    "        print(fig_path)\n",
+    "        if Image is not None:\n",
+    "            display(Image(filename=str(fig_path)))\n",
+    "else:\n",
+    "    print('No generated figures are available in:', RESULT_DIR)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generated files\n",
+    "\n",
+    "List local files produced by training or the notebook's plotting cells.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "generated_files = [path for path in sorted(RESULT_DIR.rglob('*')) if path.is_file()]\n",
+    "if generated_files:\n",
+    "    display(pd.DataFrame([\n",
+    "        {'file': str(path.relative_to(RESULT_DIR)), 'path': str(path)}\n",
+    "        for path in generated_files\n",
+    "    ]))\n",
+    "else:\n",
+    "    print('No outputs have been generated yet.')\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

BioMod/toturials/gene_perturbation.ipynb

@@ -1,359 +1,318 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Gene perturbation demo: Norman smoke with weight0407 on one GPU\n",
-    "\n",
-    "This notebook is a tutorial version of the Norman smoke-test launcher for the gene perturbation task. It keeps the single-GPU smoke-test recipe from `gene_perturbations/smoke.sh`, uses the weight0407 backbone, and writes outputs under the tutorials result folder."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Privacy-safe absolute paths\n",
-    "\n",
-    "The notebook uses this alias path instead of a personal project path:\n",
-    "\n",
-    "`/data/share/biomodulefm_demo/single_cell_model`\n",
-    "\n",
-    "All paths below are absolute and resolve through that alias on the server."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## What this demo runs\n",
-    "\n",
-    "- Task code: `/data/jianqingzhu/BioMod-main/BioMod/gene_perturbation/train_module.py`\n",
-    "- Dataset root: `/data/share/biomodulefm_demo/single_cell_model/DATA/GEARS_data`\n",
-    "- Dataset name: `norman`\n",
-    "- Split: `simulation`\n",
-    "- Split seed: `1`\n",
-    "- Backbone config: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory.pkl`\n",
-    "- Backbone checkpoint: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth`\n",
-    "- Output directory: `/data/share/biomodulefm_demo/tutorial_runs/gene_perturbation_norman_smoke_weight0407_1gpu_demo`\n",
-    "\n",
-    "Run this notebook with the `sclong_h200` Python environment/kernel, or another environment that can run the original gene perturbation smoke test."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "import importlib\n",
-    "import os\n",
-    "import shutil\n",
-    "import subprocess\n",
-    "import sys\n",
-    "import pandas as pd\n",
-    "try:\n",
-    "    from IPython.display import display\n",
-    "except Exception:\n",
-    "    def display(obj):\n",
-    "        print(obj)\n",
-    "\n",
-    "BASE = Path('/data/share/biomodulefm_demo/single_cell_model')\n",
-    "MODULE_MODEL = BASE / 'module_model'\n",
-    "\n",
-    "# Run the current module-named source tree.\n",
-    "PROJECT_ROOT = Path('/data/jianqingzhu/BioMod-main/BioMod')\n",
-    "TASK_ROOT = PROJECT_ROOT / 'gene_perturbation'\n",
-    "TRAIN_SCRIPT = TASK_ROOT / 'train_module.py'\n",
-    "\n",
-    "DATA_DIR = BASE / 'DATA' / 'GEARS_data'\n",
-    "DATA_NAME = 'norman'\n",
-    "SPLIT = 'simulation'\n",
-    "SEED = 1\n",
-    "TRAIN_GENE_SET_SIZE = 0.75\n",
-    "\n",
-    "# Use the same weight0407 backbone as the zero-shot and cell-annotation tutorial notebooks.\n",
-    "SCFM_HYPER = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory.pkl'\n",
-    "SCFM_CKPT = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth'\n",
-    "SCFM_GENES = MODULE_MODEL / 'selected_genes_27k.txt'\n",
-    "SCFM_GENE2VEC = MODULE_MODEL / 'selected_gene2vec_27k.npy'\n",
-    "GENE_MAP = MODULE_MODEL / 'gene2ens.csv'\n",
-    "MODULE_DATA_DIR = PROJECT_ROOT / 'module_data_all'\n",
-    "PERFORMER_CONT = PROJECT_ROOT.parent / 'model_dependencies' / 'performer_pytorch_cont'\n",
-    "\n",
-    "OUTPUT_PARENT = Path('/data/share/biomodulefm_demo/tutorial_runs')\n",
-    "RUN_NAME = 'gene_perturbation_norman_smoke_weight0407_1gpu_demo'\n",
-    "RESULT_DIR = OUTPUT_PARENT / RUN_NAME\n",
-    "RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "VERBOSE_OUTPUT = RESULT_DIR / 'norman_verbose.txt'\n",
-    "\n",
-    "SPLIT_FILE = DATA_DIR / DATA_NAME / 'splits' / f'{DATA_NAME}_{SPLIT}_{SEED}_{TRAIN_GENE_SET_SIZE}.pkl'\n",
-    "TARGET_H5AD = DATA_DIR / DATA_NAME / 'perturb_processed.h5ad'\n",
-    "\n",
-    "print('Python:', sys.executable)\n",
-    "print('Task root:', TASK_ROOT)\n",
-    "print('Run directory:', RESULT_DIR)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "required_files = [\n",
-    "    TRAIN_SCRIPT,\n",
-    "    TARGET_H5AD,\n",
-    "    SPLIT_FILE,\n",
-    "    SCFM_HYPER,\n",
-    "    SCFM_CKPT,\n",
-    "    SCFM_GENES,\n",
-    "    SCFM_GENE2VEC,\n",
-    "    GENE_MAP,\n",
-    "    ROOT / 'models_performer_module.py',\n",
-    "]\n",
-    "required_dirs = [TASK_ROOT, MODULE_DATA_DIR]\n",
-    "\n",
-    "missing = [str(p) for p in required_files if not p.is_file()] + [str(p) for p in required_dirs if not p.is_dir()]\n",
-    "if missing:\n",
-    "    raise FileNotFoundError('Missing required paths:\\n' + '\\n'.join(missing))\n",
-    "\n",
-    "for p in required_files:\n",
-    "    print('[FOUND FILE]', p)\n",
-    "for p in required_dirs:\n",
-    "    print('[FOUND DIR ]', p)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Runtime setup\n",
-    "\n",
-    "The original smoke script creates a temporary `performer_module` package shim so that `performer_module.models_performer_module` resolves to the 0418 code snapshot. This avoids import conflicts with other project copies of similarly named modules."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'\n",
-    "os.environ['PYTHONUNBUFFERED'] = '1'\n",
-    "os.environ['OMP_NUM_THREADS'] = '1'\n",
-    "os.environ['NUMBA_NUM_THREADS'] = '1'\n",
-    "os.environ['NUMBA_DISABLE_JIT'] = '1'\n",
-    "os.environ['NUMBA_CACHE_DIR'] = f'/tmp/numba_cache_gene_perturbation_demo_{os.environ.get(\"USER\", \"unknown\")}_{os.getpid()}'\n",
-    "os.environ['WANDB_MODE'] = 'offline'\n",
-    "os.environ['TORCH_NCCL_ASYNC_ERROR_HANDLING'] = '1'\n",
-    "os.environ.pop('NCCL_ASYNC_ERROR_HANDLING', None)\n",
-    "os.environ['NCCL_DEBUG'] = 'WARN'\n",
-    "os.environ['TORCH_CPP_LOG_LEVEL'] = 'WARNING'\n",
-    "os.environ['MASTER_ADDR'] = '127.0.0.1'\n",
-    "os.environ['MASTER_PORT'] = os.environ.get('MASTER_PORT', '29621')\n",
-    "os.environ['NCCL_SOCKET_IFNAME'] = 'eth0'\n",
-    "os.environ['GLOO_SOCKET_IFNAME'] = 'eth0'\n",
-    "Path(os.environ['NUMBA_CACHE_DIR']).mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "PKG_SHIM = Path(f'/tmp/performer_module_pkg_gene_perturbation_demo_{os.getpid()}')\n",
-    "if PKG_SHIM.exists() or PKG_SHIM.is_symlink():\n",
-    "    shutil.rmtree(PKG_SHIM, ignore_errors=True)\n",
-    "(PKG_SHIM / 'performer_module').mkdir(parents=True, exist_ok=True)\n",
-    "os.symlink(ROOT / 'models_performer_module.py', PKG_SHIM / 'performer_module' / 'models_performer_module.py')\n",
-    "(PKG_SHIM / 'performer_module' / '__init__.py').write_text('', encoding='utf-8')\n",
-    "\n",
-    "pythonpath_parts = [\n",
-    "    str(PKG_SHIM),\n",
-    "    str(TASK_ROOT),\n",
-    "    str(PERFORMER_CONT),\n",
-    "    str(PROJECT_ROOT),\n",
-    "]\n",
-    "if os.environ.get('PYTHONPATH'):\n",
-    "    pythonpath_parts.append(os.environ['PYTHONPATH'])\n",
-    "os.environ['PYTHONPATH'] = ':'.join(pythonpath_parts)\n",
-    "for path_part in reversed(pythonpath_parts):\n",
-    "    if path_part and path_part not in sys.path:\n",
-    "        sys.path.insert(0, path_part)\n",
-    "\n",
-    "print('CUDA_VISIBLE_DEVICES =', os.environ['CUDA_VISIBLE_DEVICES'])\n",
-    "print('NUMBA_CACHE_DIR     =', os.environ['NUMBA_CACHE_DIR'])\n",
-    "print('PKG_SHIM            =', PKG_SHIM)\n",
-    "print('PYTHONPATH          =', os.environ['PYTHONPATH'])\n",
-    "print('sys.path head      =', sys.path[:4])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Pre-check that the current module source is importable.\n",
-    "module = importlib.import_module('models_performer_module')\n",
-    "print('[CHECK] module file:', module.__file__)\n",
-    "print('[CHECK] has sanitize_performer_module_config:', hasattr(module, 'sanitize_performer_module_config'))\n",
-    "if not hasattr(module, 'sanitize_performer_module_config'):\n",
-    "    raise RuntimeError('sanitize_performer_module_config not found in imported models_performer_module')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Smoke-test training command\n",
-    "\n",
-    "`train_module.py` initializes PyTorch distributed training unconditionally, so even the one-GPU version should be launched with one process. This notebook uses `python -m torch.distributed.run --nproc_per_node=1`, equivalent to a single-process `torchrun`."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "EPOCHS = 1\n",
-    "BATCH_SIZE = 16\n",
-    "TEST_BATCH_SIZE = 32\n",
-    "HIDDEN_SIZE = 1024\n",
-    "ACCUMULATION_STEPS = 8\n",
-    "LR = '1e-3'\n",
-    "MASTER_PORT = os.environ['MASTER_PORT']\n",
-    "\n",
-    "cmd = [\n",
-    "    sys.executable,\n",
-    "    '-m', 'torch.distributed.run',\n",
-    "    '--standalone',\n",
-    "    '--nnodes=1',\n",
-    "    '--nproc_per_node=1',\n",
-    "    f'--master_port={MASTER_PORT}',\n",
-    "    str(TRAIN_SCRIPT),\n",
-    "    '--data_dir', str(DATA_DIR),\n",
-    "    '--data_name', DATA_NAME,\n",
-    "    '--split', SPLIT,\n",
-    "    '--result_dir', str(RESULT_DIR),\n",
-    "    '--seed', str(SEED),\n",
-    "    '--epochs', str(EPOCHS),\n",
-    "    '--batch_size', str(BATCH_SIZE),\n",
-    "    '--test_batch_size', str(TEST_BATCH_SIZE),\n",
-    "    '--train_gene_set_size', str(TRAIN_GENE_SET_SIZE),\n",
-    "    '--hidden_size', str(HIDDEN_SIZE),\n",
-    "    '--run_name', 'weight0407_norman_smoke_1gpu_demo',\n",
-    "    '--scfm_hyper_params_path', str(SCFM_HYPER),\n",
-    "    '--scfm_ckpt_path', str(SCFM_CKPT),\n",
-    "    '--scfm_genes_list_path', str(SCFM_GENES),\n",
-    "    '--gene_mapping_path', str(GENE_MAP),\n",
-    "    '--scfm_gene2vec_file', str(SCFM_GENE2VEC),\n",
-    "    '--accumulation_steps', str(ACCUMULATION_STEPS),\n",
-    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
-    "    '--verbose_output_path', str(VERBOSE_OUTPUT),\n",
-    "    '--lr', LR,\n",
-    "]\n",
-    "\n",
-    "print('Command:')\n",
-    "print(' '.join(cmd))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Run the one-GPU smoke test. It should be much shorter than a full gene perturbation training run.\n",
-    "log_path = RESULT_DIR / 'manual_launch_1gpu.log'\n",
-    "print('Writing log to:', log_path)\n",
-    "\n",
-    "with log_path.open('w') as log:\n",
-    "    process = subprocess.Popen(\n",
-    "        cmd,\n",
-    "        stdout=subprocess.PIPE,\n",
-    "        stderr=subprocess.STDOUT,\n",
-    "        text=True,\n",
-    "        bufsize=1,\n",
-    "        cwd=str(TASK_ROOT),\n",
-    "        env=os.environ.copy(),\n",
-    "    )\n",
-    "    for line in process.stdout:\n",
-    "        print(line, end='')\n",
-    "        log.write(line)\n",
-    "    return_code = process.wait()\n",
-    "\n",
-    "if return_code != 0:\n",
-    "    raise RuntimeError(f'gene perturbation smoke demo failed with return code {return_code}; see {log_path}')\n",
-    "\n",
-    "print('Done.')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Inspect outputs\n",
-    "\n",
-    "The smoke run usually writes `config.pkl`, `model.pt`, `params.csv`, the notebook log, and `norman_verbose.txt` when `--verbose_output_path` is enabled."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for path in sorted(RESULT_DIR.glob('*')):\n",
-    "    size_mb = path.stat().st_size / 1024 / 1024 if path.is_file() else 0\n",
-    "    print(f'{path.name:45s} {size_mb:8.2f} MB')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "params_path = RESULT_DIR / 'params.csv'\n",
-    "if params_path.exists():\n",
-    "    params = pd.read_csv(params_path, index_col=0)\n",
-    "    display(params)\n",
-    "else:\n",
-    "    print('params.csv not found yet:', params_path)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "if VERBOSE_OUTPUT.exists():\n",
-    "    with VERBOSE_OUTPUT.open('r', encoding='utf-8', errors='replace') as handle:\n",
-    "        for _, line in zip(range(20), handle):\n",
-    "            print(line.rstrip())\n",
-    "else:\n",
-    "    print('Verbose output not found yet:', VERBOSE_OUTPUT)"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Gene perturbation prediction with BioMod model weights\n",
+    "\n",
+    "Train the gene perturbation model on the local Norman dataset with a BioMod backbone. This notebook prepares the project paths, configures the simulation split, launches training when explicitly enabled, and inspects the saved outputs."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Project paths\n",
+    "\n",
+    "Start Jupyter from `BioMod/toturials/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inputs and workflow\n",
+    "\n",
+    "- Entry point: `BioMod/gene_perturbation/train_module.py`.\n",
+    "- Dataset parent: `data/downstream/final_data/`; the loader appends `norman/`.\n",
+    "- Processed expression: `norman/perturb_processed.h5ad` under that parent.\n",
+    "- Split: `simulation`, seed `1`, training-gene fraction `0.75`.\n",
+    "- Model weights and matching configuration: the `.pth` / `.pkl` pair under `weight/0810model/`.\n",
+    "- Gene vocabulary, mapping, embeddings, and module definitions: `model_dependencies/`.\n",
+    "- Results: `tutorial_runs/gene_perturbation_norman_biomod/`.\n",
+    "\n",
+    "Use a kernel with the project's gene perturbation dependencies installed. The current training entry point initializes NCCL and requires a Linux CUDA environment, including a suitably configured WSL environment. The notebook can prepare paths on Windows, but native Windows cannot run this NCCL training entry point. Use paths valid inside the selected training environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import json\n",
+    "import os\n",
+    "import shlex\n",
+    "import subprocess\n",
+    "import sys\n",
+    "import tempfile\n",
+    "import pandas as pd\n",
+    "try:\n",
+    "    from IPython.display import display\n",
+    "except ImportError:\n",
+    "    def display(obj):\n",
+    "        print(obj)\n",
+    "\n",
+    "PROJECT_ROOT = Path('../..').resolve()\n",
+    "if not (PROJECT_ROOT / 'BioMod' / 'models_performer_module.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
+    "    raise FileNotFoundError(\n",
+    "        'Project root not found. Start Jupyter from BioMod/toturials, or set '\n",
+    "        'PROJECT_ROOT to your repository root (use Path(\".\").resolve() if '\n",
+    "        'Jupyter was started at the repository root).'\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
+    "ASSET_ROOT = PROJECT_ROOT / 'model_dependencies'\n",
+    "TASK_ROOT = SOURCE_DIR / 'gene_perturbation'\n",
+    "TRAIN_SCRIPT = TASK_ROOT / 'train_module.py'\n",
+    "DATA_DIR = PROJECT_ROOT / 'data' / 'downstream' / 'final_data'\n",
+    "DATA_NAME = 'norman'\n",
+    "SPLIT = 'simulation'\n",
+    "SEED = 1\n",
+    "TRAIN_GENE_SET_SIZE = 0.75\n",
+    "\n",
+    "# Keep the BioMod model weights and their matching configuration together.\n",
+    "WEIGHT_DIR = PROJECT_ROOT / 'weight' / '0810model'\n",
+    "SCFM_HYPER = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2.pkl'\n",
+    "SCFM_CKPT = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2_epoch0.pth'\n",
+    "SCFM_GENES = ASSET_ROOT / 'selected_genes_27k.txt'\n",
+    "SCFM_GENE2VEC = ASSET_ROOT / 'selected_gene2vec_27k.npy'\n",
+    "GENE_MAP = ASSET_ROOT / 'gene2ens.csv'\n",
+    "MODULE_DATA_DIR = ASSET_ROOT / 'module_data_all'\n",
+    "\n",
+    "OUTPUT_PARENT = PROJECT_ROOT / 'tutorial_runs'\n",
+    "RUN_NAME = 'gene_perturbation_norman_biomod'\n",
+    "RESULT_DIR = OUTPUT_PARENT / RUN_NAME\n",
+    "RUN_TRAINING = False\n",
+    "SPLIT_FILE = DATA_DIR / DATA_NAME / 'splits' / f'{DATA_NAME}_{SPLIT}_{SEED}_{TRAIN_GENE_SET_SIZE}.pkl'\n",
+    "TARGET_H5AD = DATA_DIR / DATA_NAME / 'perturb_processed.h5ad'\n",
+    "\n",
+    "print('Python:', sys.executable)\n",
+    "print('Repository:', PROJECT_ROOT)\n",
+    "print('Dataset:', TARGET_H5AD)\n",
+    "print('Run directory:', RESULT_DIR)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "required_files = [\n",
+    "    TRAIN_SCRIPT, TARGET_H5AD, SPLIT_FILE, SCFM_HYPER, SCFM_CKPT,\n",
+    "    SCFM_GENES, SCFM_GENE2VEC, GENE_MAP,\n",
+    "    SOURCE_DIR / 'models_performer_module.py',\n",
+    "    MODULE_DATA_DIR / 'merged_modules.json',\n",
+    "]\n",
+    "required_dirs = [TASK_ROOT, ASSET_ROOT, DATA_DIR / DATA_NAME / 'data_pyg']\n",
+    "missing = [str(p) for p in required_files if not p.is_file()]\n",
+    "missing += [str(p) for p in required_dirs if not p.is_dir()]\n",
+    "if missing:\n",
+    "    raise FileNotFoundError('Missing required paths:\\n' + '\\n'.join(missing))\n",
+    "for path in required_files:\n",
+    "    print('[FOUND]', path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Runtime setup\n",
+    "\n",
+    "The current source imports `BioMod.models_performer_module` directly. The child process uses the repository and bundled dependency directories on `PYTHONPATH`; no temporary package or symlink is needed. The notebook leaves the kernel environment unchanged."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "run_env = os.environ.copy()\n",
+    "run_env['CUDA_VISIBLE_DEVICES'] = '0'\n",
+    "run_env['PYTHONUNBUFFERED'] = '1'\n",
+    "run_env.setdefault('OMP_NUM_THREADS', '1')\n",
+    "run_env.setdefault('NUMBA_NUM_THREADS', '1')\n",
+    "run_env['NUMBA_DISABLE_JIT'] = '1'\n",
+    "run_env.setdefault('NUMBA_CACHE_DIR', str(Path(tempfile.gettempdir()) / f'biomod_perturbation_{os.getpid()}'))\n",
+    "run_env['WANDB_MODE'] = 'offline'\n",
+    "run_env.setdefault('TORCH_NCCL_ASYNC_ERROR_HANDLING', '1')\n",
+    "run_env.pop('NCCL_ASYNC_ERROR_HANDLING', None)\n",
+    "run_env.setdefault('NCCL_DEBUG', 'WARN')\n",
+    "pythonpath_parts = [str(TASK_ROOT), str(PROJECT_ROOT), str(SOURCE_DIR), str(ASSET_ROOT)]\n",
+    "if run_env.get('PYTHONPATH'):\n",
+    "    pythonpath_parts.append(run_env['PYTHONPATH'])\n",
+    "run_env['PYTHONPATH'] = os.pathsep.join(pythonpath_parts)\n",
+    "print('CUDA_VISIBLE_DEVICES =', run_env['CUDA_VISIBLE_DEVICES'])\n",
+    "print('PYTHONPATH =', run_env['PYTHONPATH'])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# File checks do not load the model weights or deserialize the configuration.\n",
+    "print('Backbone source:', SOURCE_DIR / 'models_performer_module.py')\n",
+    "print('Bundled Performer:', ASSET_ROOT / 'performer_pytorch_cont')\n",
+    "print('Training enabled:', RUN_TRAINING)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training command\n",
+    "\n",
+    "`train_module.py` initializes distributed training even for one GPU. The command therefore uses `python -m torch.distributed.run --standalone --nproc_per_node=1`. Review the settings below, then set `RUN_TRAINING = True` in the setup cell to launch. The training script selects a checkpoint using validation MSE on the top 20 differentially expressed genes, then evaluates the selected checkpoint on the test split. Batch sizes, epoch count, and learning rate are configurable examples, not a verified benchmark recipe."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "EPOCHS = 1\n",
+    "BATCH_SIZE = 16\n",
+    "TEST_BATCH_SIZE = 32\n",
+    "HIDDEN_SIZE = 1024\n",
+    "ACCUMULATION_STEPS = 8\n",
+    "LR = '1e-3'\n",
+    "\n",
+    "cmd = [\n",
+    "    sys.executable, '-m', 'torch.distributed.run',\n",
+    "    '--standalone', '--nnodes=1', '--nproc_per_node=1',\n",
+    "    str(TRAIN_SCRIPT),\n",
+    "    '--data_dir', str(DATA_DIR),\n",
+    "    '--data_name', DATA_NAME,\n",
+    "    '--split', SPLIT,\n",
+    "    '--result_dir', str(RESULT_DIR),\n",
+    "    '--seed', str(SEED),\n",
+    "    '--epochs', str(EPOCHS),\n",
+    "    '--batch_size', str(BATCH_SIZE),\n",
+    "    '--test_batch_size', str(TEST_BATCH_SIZE),\n",
+    "    '--train_gene_set_size', str(TRAIN_GENE_SET_SIZE),\n",
+    "    '--hidden_size', str(HIDDEN_SIZE),\n",
+    "    '--run_name', RUN_NAME,\n",
+    "    '--scfm_hyper_params_path', str(SCFM_HYPER),\n",
+    "    '--scfm_ckpt_path', str(SCFM_CKPT),\n",
+    "    '--scfm_genes_list_path', str(SCFM_GENES),\n",
+    "    '--gene_mapping_path', str(GENE_MAP),\n",
+    "    '--scfm_gene2vec_file', str(SCFM_GENE2VEC),\n",
+    "    '--accumulation_steps', str(ACCUMULATION_STEPS),\n",
+    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
+    "    '--lr', LR,\n",
+    "]\n",
+    "print('Command:')\n",
+    "print(subprocess.list2cmdline(cmd) if os.name == 'nt' else shlex.join(cmd))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log_path = RESULT_DIR / 'training.log'\n",
+    "if not RUN_TRAINING:\n",
+    "    print('Training is disabled. Review the paths and settings, then set RUN_TRAINING = True to launch.')\n",
+    "else:\n",
+    "    if sys.platform != 'linux':\n",
+    "        raise RuntimeError('The training script requires Linux with CUDA/NCCL. Use a compatible kernel and project paths.')\n",
+    "    RESULT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    Path(run_env['NUMBA_CACHE_DIR']).mkdir(parents=True, exist_ok=True)\n",
+    "    print('Writing log to:', log_path)\n",
+    "    with log_path.open('w', encoding='utf-8') as log:\n",
+    "        process = subprocess.Popen(\n",
+    "            cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n",
+    "            text=True, encoding='utf-8', errors='replace', bufsize=1,\n",
+    "            cwd=str(TASK_ROOT), env=run_env,\n",
+    "        )\n",
+    "        for line in process.stdout:\n",
+    "            print(line, end='')\n",
+    "            log.write(line)\n",
+    "        return_code = process.wait()\n",
+    "    if return_code != 0:\n",
+    "        raise RuntimeError(f'Gene perturbation training failed with return code {return_code}; see {log_path}')\n",
+    "    print('Training finished.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inspect outputs\n",
+    "\n",
+    "A successful run saves the selected model as `model.pt`, its configuration as `config.pkl`, and the command parameters as `params.csv`. The notebook captures console metrics in `training.log`; the current script does not provide a separate verbose-output argument. These outputs are written to the run directory and do not overwrite the input BioMod model weights. Result cells can also inspect a previous completed run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if not RESULT_DIR.is_dir():\n",
+    "    print('No run directory yet:', RESULT_DIR)\n",
+    "else:\n",
+    "    for path in sorted(RESULT_DIR.iterdir()):\n",
+    "        size_mb = path.stat().st_size / 1024 / 1024 if path.is_file() else 0\n",
+    "        print(f'{path.name:45s} {size_mb:8.2f} MB')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "params_path = RESULT_DIR / 'params.csv'\n",
+    "if params_path.is_file():\n",
+    "    display(pd.read_csv(params_path, index_col=0))\n",
+    "else:\n",
+    "    print('Parameters are not available yet:', params_path)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if log_path.is_file():\n",
+    "    from collections import deque\n",
+    "    with log_path.open('r', encoding='utf-8', errors='replace') as handle:\n",
+    "        print(''.join(deque(handle, maxlen=40)))\n",
+    "else:\n",
+    "    print('Training log is not available yet:', log_path)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

BioMod/toturials/zero_shot_batch_integration.ipynb

@@ -1,351 +1,340 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "# Zero-shot batch integration demo: Kim2020 Lung no malignant\n",
-    "\n",
-    "This notebook is a single-dataset tutorial version of `run_zero_shot_all_weight0407_5times.sh`. It runs the weight0407 model on the `Kim2020_Lung_noMalignant.h5ad` dataset, computes scIB metrics, and generates UMAP visualizations."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## What this demo uses\n",
-    "\n",
-    "- Dataset: `/data/share/biomodulefm_demo/single_cell_model/DATA/singcell/Kim2020_Lung_noMalignant.h5ad`\n",
-    "- Model config: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory.pkl`\n",
-    "- Model checkpoint: `/data/share/biomodulefm_demo/single_cell_model/weight/0407weight/mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth`\n",
-    "- Runner script: `/data/share/biomodulefm_demo/single_cell_model/module_model/draw/zero_shot_umap_scib.py`\n",
-    "- Output directory: `/data/share/biomodulefm_demo/tutorial_runs/zero_shot_kim_lung_no_malignant_demo`\n",
-    "\n",
-    "Run this notebook with the `sclong` Python environment/kernel."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from pathlib import Path\n",
-    "import importlib.util\n",
-    "import os\n",
-    "import subprocess\n",
-    "import sys\n",
-    "try:\n",
-    "    from IPython.display import Image, display\n",
-    "except Exception:\n",
-    "    Image = None\n",
-    "    def display(obj):\n",
-    "        print(obj)\n",
-    "import pandas as pd\n",
-    "\n",
-    "BASE = Path('/data/share/biomodulefm_demo/single_cell_model')\n",
-    "MODULE_MODEL = BASE / 'module_model'\n",
-    "PROJECT_ROOT = Path('/data/jianqingzhu/BioMod-main/BioMod')\n",
-    "\n",
-    "SCRIPT = MODULE_MODEL / 'draw' / 'zero_shot_umap_scib.py'\n",
-    "DATA_PATH = BASE / 'DATA' / 'singcell' / 'Kim2020_Lung_noMalignant.h5ad'\n",
-    "\n",
-    "CONFIG_PKL = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory.pkl'\n",
-    "CKPT_PATH = BASE / 'weight' / '0407weight' / 'mae_tost_48M_cross_attn_only_pure_memory_epoch12.pth'\n",
-    "\n",
-    "GENE2VEC = MODULE_MODEL / 'selected_gene2vec_27k.npy'\n",
-    "GENE_MAP = MODULE_MODEL / 'gene2ens.csv'\n",
-    "GENE_LIST = MODULE_MODEL / 'selected_genes_27k.txt'\n",
-    "MODULE_DATA_DIR = PROJECT_ROOT / 'module_data_all'\n",
-    "PERFORMER_CONT = MODULE_MODEL / 'performer_pytorch_cont'\n",
-    "\n",
-    "TUTORIAL_RUN_ROOT = Path('/data/share/biomodulefm_demo/tutorial_runs')\n",
-    "OUT_DIR = TUTORIAL_RUN_ROOT / 'zero_shot_kim_lung_no_malignant_demo'\n",
-    "OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "OUTPUT_PREFIX = 'kim_lung_no_malignant_zero_shot_demo_seed0'\n",
-    "\n",
-    "# Use 'all' to match the shell launcher. For a quick smoke test, change this to a smaller number such as '2000'.\n",
-    "N_CELLS = 'all'\n",
-    "BATCH_SIZE = 8\n",
-    "UMAP_RANDOM_STATE = 0\n",
-    "\n",
-    "print('Python:', sys.executable)\n",
-    "print('Output directory:', OUT_DIR)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "required_files = [\n",
-    "    SCRIPT,\n",
-    "    DATA_PATH,\n",
-    "    CONFIG_PKL,\n",
-    "    CKPT_PATH,\n",
-    "    GENE2VEC,\n",
-    "    GENE_MAP,\n",
-    "    GENE_LIST,\n",
-    "]\n",
-    "required_dirs = [PROJECT_ROOT, MODULE_DATA_DIR]\n",
-    "\n",
-    "missing = [str(p) for p in required_files if not p.is_file()] + [str(p) for p in required_dirs if not p.is_dir()]\n",
-    "if missing:\n",
-    "    raise FileNotFoundError('Missing required paths:\\n' + '\\n'.join(missing))\n",
-    "\n",
-    "for p in required_files:\n",
-    "    print('[FOUND FILE]', p)\n",
-    "for p in required_dirs:\n",
-    "    print('[FOUND DIR ]', p)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Match the runtime environment used by the shell launcher.\n",
-    "os.environ['PYTHONUNBUFFERED'] = '1'\n",
-    "os.environ.setdefault('CUDA_VISIBLE_DEVICES', '0')\n",
-    "os.environ.setdefault('OMP_NUM_THREADS', '8')\n",
-    "os.environ.setdefault('NUMBA_CACHE_DIR', f'/tmp/numba_cache_zero_shot_demo_{os.environ.get(\"USER\", \"unknown\")}_{os.getpid()}')\n",
-    "Path(os.environ['NUMBA_CACHE_DIR']).mkdir(parents=True, exist_ok=True)\n",
-    "\n",
-    "pythonpath_parts = [str(MODULE_MODEL), str(PROJECT_ROOT)]\n",
-    "if PERFORMER_CONT.is_dir():\n",
-    "    pythonpath_parts.append(str(PERFORMER_CONT))\n",
-    "if os.environ.get('PYTHONPATH'):\n",
-    "    pythonpath_parts.append(os.environ['PYTHONPATH'])\n",
-    "os.environ['PYTHONPATH'] = ':'.join(pythonpath_parts)\n",
-    "\n",
-    "print('CUDA_VISIBLE_DEVICES =', os.environ['CUDA_VISIBLE_DEVICES'])\n",
-    "print('OMP_NUM_THREADS      =', os.environ['OMP_NUM_THREADS'])\n",
-    "print('NUMBA_CACHE_DIR     =', os.environ['NUMBA_CACHE_DIR'])\n",
-    "print('PYTHONPATH          =', os.environ['PYTHONPATH'])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Install scIB from the Tsinghua PyPI mirror if the current kernel does not have it.\n",
-    "if importlib.util.find_spec('scib') is None:\n",
-    "    subprocess.check_call([\n",
-    "        sys.executable, '-m', 'pip', 'install', 'scib',\n",
-    "        '-i', 'https://pypi.tuna.tsinghua.edu.cn/simple',\n",
-    "        '--trusted-host', 'pypi.tuna.tsinghua.edu.cn',\n",
-    "        '--retries', '10',\n",
-    "        '--timeout', '60',\n",
-    "    ])\n",
-    "\n",
-    "import scib\n",
-    "print('scib import ok:', getattr(scib, '__version__', 'unknown'))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "cmd = [\n",
-    "    sys.executable,\n",
-    "    str(SCRIPT),\n",
-    "    '--data_path', str(DATA_PATH),\n",
-    "    '--gene2vec_path', str(GENE2VEC),\n",
-    "    '--scfm_hyper_params_path', str(CONFIG_PKL),\n",
-    "    '--scfm_ckpt_path', str(CKPT_PATH),\n",
-    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
-    "    '--gene_mapping_path', str(GENE_MAP),\n",
-    "    '--scfm_genes_file', str(GENE_LIST),\n",
-    "    '--project_root', str(PROJECT_ROOT),\n",
-    "    '--output_dir', str(OUT_DIR),\n",
-    "    '--output_prefix', OUTPUT_PREFIX,\n",
-    "    '--gene_col', 'gene_name',\n",
-    "    '--batch_key', 'sample',\n",
-    "    '--label_key', 'cell_type',\n",
-    "    '--output_key', 'X_sclong_module',\n",
-    "    '--pool_method', 'max',\n",
-    "    '--n_cells', N_CELLS,\n",
-    "    '--batch_size', str(BATCH_SIZE),\n",
-    "    '--umap_n_neighbors', '15',\n",
-    "    '--umap_min_dist', '0.5',\n",
-    "    '--umap_random_state', str(UMAP_RANDOM_STATE),\n",
-    "    '--umap_celltype_title', 'Kim2020 Lung no malignant | cell type',\n",
-    "    '--umap_batch_title', 'Kim2020 Lung no malignant | sample',\n",
-    "]\n",
-    "\n",
-    "print('Command:')\n",
-    "print(' '.join(cmd))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "log_path = OUT_DIR / f'{OUTPUT_PREFIX}.log'\n",
-    "print('Writing log to:', log_path)\n",
-    "\n",
-    "with log_path.open('w') as log:\n",
-    "    process = subprocess.Popen(\n",
-    "        cmd,\n",
-    "        stdout=subprocess.PIPE,\n",
-    "        stderr=subprocess.STDOUT,\n",
-    "        text=True,\n",
-    "        bufsize=1,\n",
-    "        env=os.environ.copy(),\n",
-    "    )\n",
-    "    for line in process.stdout:\n",
-    "        print(line, end='')\n",
-    "        log.write(line)\n",
-    "    return_code = process.wait()\n",
-    "\n",
-    "if return_code != 0:\n",
-    "    raise RuntimeError(f'zero-shot demo failed with return code {return_code}; see {log_path}')\n",
-    "\n",
-    "print('Done.')"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "metrics_path = OUT_DIR / f'{OUTPUT_PREFIX}_scib_metrics.csv'\n",
-    "metrics = pd.read_csv(metrics_path)\n",
-    "metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "avg_metrics = metrics[metrics['metric'].isin(['avg_batch', 'avg_bio', 'avg_score'])].copy()\n",
-    "avg_metrics"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "celltype_png = OUT_DIR / f'{OUTPUT_PREFIX}_umap_by_cell_type.png'\n",
-    "batch_png = OUT_DIR / f'{OUTPUT_PREFIX}_umap_by_batch.png'\n",
-    "\n",
-    "print(celltype_png)\n",
-    "if Image is not None:\n",
-    "    display(Image(filename=str(celltype_png)))\n",
-    "\n",
-    "print(batch_png)\n",
-    "if Image is not None:\n",
-    "    display(Image(filename=str(batch_png)))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Optional: export an A4 PDF/PNG UMAP panel\n",
-    "\n",
-    "The runner already outputs two PNG files. The following cell reads the saved UMAP `.h5ad` and combines cell-type and batch/sample UMAPs into one A4 landscape panel using 7 pt Arial-style settings. If Arial is unavailable on the server, matplotlib will fall back to the nearest available font."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b67f908e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import math\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "import scanpy as sc\n",
-    "\n",
-    "plt.rcParams.update({\n",
-    "    'font.family': ['Arial', 'DejaVu Sans'],\n",
-    "    'font.size': 7,\n",
-    "    'axes.titlesize': 7,\n",
-    "    'axes.labelsize': 7,\n",
-    "    'xtick.labelsize': 7,\n",
-    "    'ytick.labelsize': 7,\n",
-    "    'legend.fontsize': 6,\n",
-    "    'pdf.fonttype': 42,\n",
-    "    'ps.fonttype': 42,\n",
-    "})\n",
-    "\n",
-    "def plot_umap_axis(ax, adata, key, title):\n",
-    "    coords = np.asarray(adata.obsm['X_umap'])\n",
-    "    values = adata.obs[key].astype('category')\n",
-    "    categories = list(values.cat.categories)\n",
-    "    palette = []\n",
-    "    for cmap_name in ['tab20', 'tab20b', 'tab20c']:\n",
-    "        cmap = plt.get_cmap(cmap_name)\n",
-    "        palette.extend([cmap(i) for i in range(cmap.N)])\n",
-    "    for color, category in zip(palette, categories):\n",
-    "        mask = values.astype(str).to_numpy() == str(category)\n",
-    "        ax.scatter(coords[mask, 0], coords[mask, 1], s=2.2, c=[color], alpha=0.86, linewidths=0, rasterized=True, label=str(category))\n",
-    "    ax.set_title(title)\n",
-    "    ax.set_xticks([])\n",
-    "    ax.set_yticks([])\n",
-    "    for spine in ax.spines.values():\n",
-    "        spine.set_visible(False)\n",
-    "    ax.set_aspect('equal', adjustable='datalim')\n",
-    "    ncol = max(1, min(5, math.ceil(max(len(categories), 1) / 10)))\n",
-    "    ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.025), ncol=ncol, frameon=False, markerscale=2.4, handlelength=0.8, handletextpad=0.25, columnspacing=0.65, borderaxespad=0)\n",
-    "\n",
-    "umap_h5ad = OUT_DIR / f'{OUTPUT_PREFIX}_umap.h5ad'\n",
-    "adata_umap = sc.read_h5ad(umap_h5ad)\n",
-    "\n",
-    "fig, axes = plt.subplots(1, 2, figsize=(11.69, 8.27))\n",
-    "fig.subplots_adjust(left=0.025, right=0.985, bottom=0.30, top=0.90, wspace=0.04)\n",
-    "plot_umap_axis(axes[0], adata_umap, 'cell_type', 'Kim2020 Lung no malignant | cell type')\n",
-    "plot_umap_axis(axes[1], adata_umap, 'sample', 'Kim2020 Lung no malignant | sample')\n",
-    "fig.suptitle('Zero-shot batch integration demo | weight0407', fontsize=7)\n",
-    "\n",
-    "a4_pdf = OUT_DIR / f'{OUTPUT_PREFIX}_umap_a4.pdf'\n",
-    "a4_png = OUT_DIR / f'{OUTPUT_PREFIX}_umap_a4.png'\n",
-    "fig.savefig(a4_pdf)\n",
-    "fig.savefig(a4_png, dpi=300)\n",
-    "plt.close(fig)\n",
-    "\n",
-    "print('Saved:', a4_pdf)\n",
-    "print('Saved:', a4_png)\n",
-    "if Image is not None:\n",
-    "    display(Image(filename=str(a4_png)))"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.9"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Zero-shot batch integration: Kim2020 Lung\n",
+    "\n",
+    "This notebook extracts cell embeddings with the BioMod model weights, evaluates batch integration with the bundled runner, and optionally plots UMAPs. It uses the full `Kim2020_Lung_noMalignant.h5ad` dataset. Evaluation does not train or update the pretrained model.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Project paths and environment\n",
+    "\n",
+    "Start Jupyter with `BioMod/toturials` as its working directory and select the environment described in [the project README](../../README.md). Paths below are relative to this notebook directory, following the project-relative layout used in scGPT tutorials. `Path.resolve()` turns them into absolute paths for subprocesses. If Jupyter was started at the repository root, set `PROJECT_ROOT = Path('.').resolve()` instead.\n",
+    "\n",
+    "- Data: `../../data/downstream/final_data/Kim2020_Lung_noMalignant.h5ad`\n",
+    "- BioMod model weights and configuration: `../../weight/0810model/`\n",
+    "- Shared assets: `../../model_dependencies/`\n",
+    "- Runner: `../zero_shot_batch/performer_module_zero_shot.py`\n",
+    "- Results: `../../tutorial_runs/zero_shot_kim_lung_no_malignant/`\n",
+    "\n",
+    "Review the configuration, then set `RUN_EVALUATION = True` to run inference. The runner consumes `adata.X` without normalization or log transformation; check that its expression preprocessing is appropriate for the checkpoint and your experiment. It loads the input dataset into memory before batching inference.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import importlib.util\n",
+    "import os\n",
+    "import subprocess\n",
+    "import sys\n",
+    "\n",
+    "PROJECT_ROOT = Path('../..').resolve()\n",
+    "if not (PROJECT_ROOT / 'BioMod').is_dir() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
+    "    raise FileNotFoundError(\n",
+    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        \"(use Path('.').resolve() when launched at the repository root).\"\n",
+    "    )\n",
+    "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
+    "DEPENDENCIES_DIR = PROJECT_ROOT / 'model_dependencies'\n",
+    "SCRIPT = SOURCE_DIR / 'zero_shot_batch' / 'performer_module_zero_shot.py'\n",
+    "DATA_PATH = PROJECT_ROOT / 'data' / 'downstream' / 'final_data' / 'Kim2020_Lung_noMalignant.h5ad'\n",
+    "WEIGHT_DIR = PROJECT_ROOT / 'weight' / '0810model'\n",
+    "CONFIG_PKL = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2.pkl'\n",
+    "CKPT_PATH = WEIGHT_DIR / 'mae_tost_70M_75mask_ec2_epoch0.pth'\n",
+    "GENE2VEC = DEPENDENCIES_DIR / 'selected_gene2vec_27k.npy'\n",
+    "GENE_MAP = DEPENDENCIES_DIR / 'gene2ens.csv'\n",
+    "GENE_LIST = DEPENDENCIES_DIR / 'selected_genes_27k.txt'\n",
+    "MODULE_DATA_DIR = DEPENDENCIES_DIR / 'module_data_all'\n",
+    "OUT_DIR = PROJECT_ROOT / 'tutorial_runs' / 'zero_shot_kim_lung_no_malignant'\n",
+    "\n",
+    "RUN_EVALUATION = False\n",
+    "MAKE_UMAP = False\n",
+    "N_CELLS = 'all'\n",
+    "BATCH_SIZE = 8\n",
+    "POOL_METHOD = 'max'\n",
+    "METRICS_MODE = 'scgpt'\n",
+    "UMAP_RANDOM_STATE = 0\n",
+    "\n",
+    "print('Python:', sys.executable)\n",
+    "print('Repository root:', PROJECT_ROOT)\n",
+    "print('Output directory:', OUT_DIR)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "required_files = [SCRIPT, DATA_PATH, CONFIG_PKL, CKPT_PATH, GENE2VEC, GENE_MAP, GENE_LIST,\n",
+    "                  MODULE_DATA_DIR / 'merged_modules.json']\n",
+    "required_dirs = [DEPENDENCIES_DIR / 'performer_pytorch_cont']\n",
+    "missing = [str(p) for p in required_files if not p.is_file()]\n",
+    "missing += [str(p) for p in required_dirs if not p.is_dir()]\n",
+    "if missing:\n",
+    "    raise FileNotFoundError('Missing required project paths:\\n' + '\\n'.join(missing))\n",
+    "for p in required_files + required_dirs:\n",
+    "    print('[FOUND]', p)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Configure only the child process; retain the active kernel's Python environment.\n",
+    "run_env = os.environ.copy()\n",
+    "run_env['PYTHONUNBUFFERED'] = '1'\n",
+    "run_env.setdefault('OMP_NUM_THREADS', '8')\n",
+    "pythonpath_parts = [str(SOURCE_DIR), str(DEPENDENCIES_DIR)]\n",
+    "if run_env.get('PYTHONPATH'):\n",
+    "    pythonpath_parts.append(run_env['PYTHONPATH'])\n",
+    "run_env['PYTHONPATH'] = os.pathsep.join(pythonpath_parts)\n",
+    "print('PYTHONPATH:', run_env['PYTHONPATH'])\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install project dependencies in the selected kernel environment before evaluation.\n",
+    "required_modules = ['numpy', 'pandas', 'torch', 'scanpy', 'scib', 'einops']\n",
+    "missing_modules = [name for name in required_modules if importlib.util.find_spec(name) is None]\n",
+    "if missing_modules:\n",
+    "    message = ('Missing dependencies: ' + ', '.join(missing_modules)\n",
+    "               + '. Follow the installation instructions in the project README.')\n",
+    "    if RUN_EVALUATION:\n",
+    "        raise ImportError(message)\n",
+    "    print(message)\n",
+    "else:\n",
+    "    print('Core evaluation dependencies are available.')\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cmd = [\n",
+    "    sys.executable, str(SCRIPT),\n",
+    "    '--data_path', str(DATA_PATH),\n",
+    "    '--gene2vec_path', str(GENE2VEC),\n",
+    "    '--scfm_hyper_params_path', str(CONFIG_PKL),\n",
+    "    '--scfm_ckpt_path', str(CKPT_PATH),\n",
+    "    '--module_data_dir', str(MODULE_DATA_DIR),\n",
+    "    '--gene_mapping_path', str(GENE_MAP),\n",
+    "    '--scfm_genes_file', str(GENE_LIST),\n",
+    "    '--output_dir', str(OUT_DIR),\n",
+    "    '--dataset_preset', 'lung',\n",
+    "    '--gene_col', 'gene_name',\n",
+    "    '--batch_key', 'sample',\n",
+    "    '--label_key', 'cell_type',\n",
+    "    '--output_key', 'X_sclong_module',\n",
+    "    '--pool_method', POOL_METHOD,\n",
+    "    '--metrics_mode', METRICS_MODE,\n",
+    "    '--n_cells', N_CELLS,\n",
+    "    '--batch_size', str(BATCH_SIZE),\n",
+    "    '--min_genes_per_cell', '0',\n",
+    "    '--min_cells_per_gene', '0',\n",
+    "]\n",
+    "print('Command arguments:')\n",
+    "print(cmd)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "log_path = OUT_DIR / 'zero_shot.log'\n",
+    "if not RUN_EVALUATION:\n",
+    "    print('Evaluation is disabled. Set RUN_EVALUATION = True and rerun the configuration cells to start.')\n",
+    "else:\n",
+    "    OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "    with log_path.open('w', encoding='utf-8') as log:\n",
+    "        process = subprocess.Popen(\n",
+    "            cmd, cwd=SCRIPT.parent, env=run_env,\n",
+    "            stdout=subprocess.PIPE, stderr=subprocess.STDOUT,\n",
+    "            text=True, encoding='utf-8', errors='replace', bufsize=1,\n",
+    "        )\n",
+    "        for line in process.stdout:\n",
+    "            print(line, end='')\n",
+    "            log.write(line)\n",
+    "        return_code = process.wait()\n",
+    "    if return_code != 0:\n",
+    "        raise RuntimeError(f'Evaluation failed with code {return_code}; see {log_path}')\n",
+    "    print('Evaluation completed:', OUT_DIR)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "metrics_path = OUT_DIR / 'sclong_module_batch_cell_emb_mode.csv'\n",
+    "embedding_path = OUT_DIR / 'sclong_module_batch_cell_emb_mode_ce.npy'\n",
+    "metrics = None\n",
+    "if metrics_path.is_file():\n",
+    "    import pandas as pd\n",
+    "    from IPython.display import display\n",
+    "    metrics = pd.read_csv(metrics_path)\n",
+    "    required_columns = {'metric', 'label', 'value'}\n",
+    "    if not required_columns.issubset(metrics.columns):\n",
+    "        raise ValueError(f'Unexpected metrics columns: {list(metrics.columns)}')\n",
+    "    display(metrics)\n",
+    "else:\n",
+    "    print('No evaluation metrics yet:', metrics_path)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if metrics is not None:\n",
+    "    # The bundled scGPT metric mode reports these two aggregates; it does not emit avg_score.\n",
+    "    avg_metrics = metrics[metrics['metric'].isin(['avg_batch', 'avg_bio'])].copy()\n",
+    "    if avg_metrics.empty:\n",
+    "        print('No aggregate metrics in this result; inspect the individual metrics above.')\n",
+    "    else:\n",
+    "        display(avg_metrics)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('Metrics:', metrics_path)\n",
+    "print('Cell embeddings:', embedding_path)\n",
+    "print('Run log:', log_path)\n",
+    "print('The runner saves CSV/NPY results. UMAP figures are created by the optional cell below.')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Optional: plot UMAP and export a PDF/PNG panel\n",
+    "\n",
+    "After a successful evaluation, set `MAKE_UMAP = True` in the configuration cell. The following cell combines the saved cell embeddings with the input dataset's observation metadata. It requires the full dataset (`N_CELLS = 'all'`) with no cell filtering, as configured above, so the rows retain the input order. Use embeddings from this dataset and this configuration; the runner does not save cell identifiers alongside the NPY array. Subsampled or filtered embeddings require their matching observation metadata and cannot be plotted by this cell.\n",
+    "\n",
+    "The optional plot requires Scanpy and Matplotlib. It computes UMAP from the embeddings and writes `umap_a4.pdf`, `umap_a4.png`, and `umap.h5ad` to the result directory.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b67f908e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if not MAKE_UMAP:\n",
+    "    print('UMAP export is disabled. Set MAKE_UMAP = True after evaluation.')\n",
+    "elif not embedding_path.is_file():\n",
+    "    print('No cell embeddings yet:', embedding_path)\n",
+    "else:\n",
+    "    if N_CELLS != 'all':\n",
+    "        raise ValueError('UMAP export requires the full, unfiltered dataset in input row order.')\n",
+    "    import math\n",
+    "    import numpy as np\n",
+    "    import matplotlib.pyplot as plt\n",
+    "    import scanpy as sc\n",
+    "    from IPython.display import Image, display\n",
+    "\n",
+    "    source = sc.read_h5ad(DATA_PATH, backed='r')\n",
+    "    try:\n",
+    "        obs = source.obs.copy()\n",
+    "    finally:\n",
+    "        source.file.close()\n",
+    "    embeddings = np.load(embedding_path, allow_pickle=False)\n",
+    "    if embeddings.ndim != 2 or embeddings.shape[0] != len(obs):\n",
+    "        raise ValueError('Embedding rows do not match the full input dataset.')\n",
+    "    if not {'sample', 'cell_type'}.issubset(obs.columns):\n",
+    "        raise KeyError('The input metadata must contain sample and cell_type.')\n",
+    "    if len(obs) < 3 or not np.isfinite(embeddings).all():\n",
+    "        raise ValueError('UMAP requires at least three cells and finite embeddings.')\n",
+    "    adata_umap = sc.AnnData(X=embeddings, obs=obs)\n",
+    "    sc.pp.neighbors(adata_umap, use_rep='X', n_pcs=None,\n",
+    "                    n_neighbors=min(15, len(obs) - 1), random_state=UMAP_RANDOM_STATE)\n",
+    "    sc.tl.umap(adata_umap, min_dist=0.5, random_state=UMAP_RANDOM_STATE)\n",
+    "\n",
+    "    def plot_umap_axis(ax, adata, key, title):\n",
+    "        coords = np.asarray(adata.obsm['X_umap'])\n",
+    "        values = adata.obs[key].astype('string').fillna('Missing')\n",
+    "        categories = sorted(values.unique())\n",
+    "        palette = [plt.get_cmap(name)(i) for name in ['tab20', 'tab20b', 'tab20c']\n",
+    "                   for i in range(plt.get_cmap(name).N)]\n",
+    "        for index, category in enumerate(categories):\n",
+    "            mask = (values == category).to_numpy(dtype=bool)\n",
+    "            ax.scatter(coords[mask, 0], coords[mask, 1], s=2.2,\n",
+    "                       color=palette[index % len(palette)], alpha=0.86,\n",
+    "                       linewidths=0, rasterized=True, label=str(category))\n",
+    "        ax.set_title(title)\n",
+    "        ax.set_xticks([])\n",
+    "        ax.set_yticks([])\n",
+    "        for spine in ax.spines.values():\n",
+    "            spine.set_visible(False)\n",
+    "        ax.set_aspect('equal', adjustable='datalim')\n",
+    "        ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.025),\n",
+    "                  ncol=max(1, min(5, math.ceil(len(categories) / 10))),\n",
+    "                  frameon=False, markerscale=2.4, fontsize=6)\n",
+    "\n",
+    "    with plt.rc_context({'font.family': 'DejaVu Sans', 'font.size': 7, 'pdf.fonttype': 42}):\n",
+    "        fig, axes = plt.subplots(1, 2, figsize=(11.69, 8.27))\n",
+    "        fig.subplots_adjust(left=0.025, right=0.985, bottom=0.30, top=0.90, wspace=0.04)\n",
+    "        plot_umap_axis(axes[0], adata_umap, 'cell_type', 'Cell type')\n",
+    "        plot_umap_axis(axes[1], adata_umap, 'sample', 'Sample')\n",
+    "        fig.suptitle('Zero-shot batch integration | BioMod model weights')\n",
+    "        a4_pdf = OUT_DIR / 'umap_a4.pdf'\n",
+    "        a4_png = OUT_DIR / 'umap_a4.png'\n",
+    "        fig.savefig(a4_pdf)\n",
+    "        fig.savefig(a4_png, dpi=300)\n",
+    "        plt.close(fig)\n",
+    "    adata_umap.write_h5ad(OUT_DIR / 'umap.h5ad')\n",
+    "    print('Saved:', a4_pdf)\n",
+    "    print('Saved:', a4_png)\n",
+    "    display(Image(filename=str(a4_png)))\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

README.md

@@ -80,13 +80,13 @@ The following tasks have implementations in this repository. The links identify
 available source code and examples; these workflows have not been validated end
 to end with the bundled checkpoint.
 
-| Task | Implementation | Notebook or guide |
+| Task | Implementation | Notebook |
 | --- | --- | --- |
 | Cell-type annotation | [Fine-tuning script](BioMod/cell_annotation/performer_module_finetune.py) | [Notebook](BioMod/toturials/cell_annotation.ipynb) |
 | Zero-shot cell embeddings and batch-effect evaluation | [Embedding and evaluation script](BioMod/zero_shot_batch/performer_module_zero_shot.py) | [Notebook](BioMod/toturials/zero_shot_batch_integration.ipynb) |
 | Single-drug response regression (IC50) | [Training script](BioMod/drug_response/train_drug_performer_module.py) | [Notebook](BioMod/toturials/drug_response_prediction.ipynb) |
-| Drug-pair synergy classification | [Training script](BioMod/drug_combination/train_drug_combination_performer_module.py) | [Task guide](BioMod/drug_combination/README.md), [notebook](BioMod/toturials/drug_combination.ipynb) |
-| Gene perturbation response prediction | [Training script](BioMod/gene_perturbation/train_module.py) | [Task guide](BioMod/gene_perturbation/README.md), [notebook](BioMod/toturials/gene_perturbation.ipynb) |
+| Drug-pair synergy classification | [Training script](BioMod/drug_combination/train_drug_combination_performer_module.py) | [Notebook](BioMod/toturials/drug_combination.ipynb) |
+| Gene perturbation response prediction | [Training script](BioMod/gene_perturbation/train_module.py) | [Notebook](BioMod/toturials/gene_perturbation.ipynb) |
 
 ### Pretraining, evaluation, and analysis scripts