Unify downstream task layout and document dataset launch commands

Place each task notebook beside its README and train/evaluate entry point; organize helper code and shared module-source definitions. Add complete per-dataset launch commands and preserve declared zero-count cell annotation classes. Remove the unused environment template and stale ignore rules, and refresh project and data-directory README paths. Pretrained weights, paired configuration and raw data remain unchanged.

BioMod/.env.example

@@ -1,13 +0,0 @@
-# Copy this file to `.env` or export the variables in your shell.
-# Never commit real API keys.
-
-# Alibaba DashScope OpenAI-compatible API, used by Qwen embedding/chat utilities.
-DASHSCOPE_API_KEY=
-DASHSCOPE_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
-
-# Optional OpenAI-compatible DeepSeek endpoint used by archived text-quality scripts.
-DEEPSEEK_API_KEY=
-DEEPSEEK_BASE_URL=
-
-# Optional experiment tracking.
-WANDB_API_KEY=

BioMod/.gitignore

@@ -14,7 +14,6 @@ htmlcov/
 # Local environments and secrets
 .env
 .env.*
-!.env.example
 .venv/
 venv/
 env/
@@ -44,10 +43,7 @@ nohup.out
 *.hickle
 
 # Generated task outputs and caches
-gene_inverse_norman/outputs/
-toturials/execution_checks/
-stage2_alignment/archive_text_quality_eval/qwen_cache/
-stage2_alignment/best_scheme_qwen/cached_text_embeddings.pt
+**/execution_checks/
 
 # OS/editor files
 .DS_Store

BioMod/README.md

@@ -1,358 +1,371 @@
-# BioMod
-
-Source code and tutorial notebooks for BioMod single-cell pretraining and
-downstream experiments using BioMod representations.
-
-This source directory contains the pretraining entry point and shared training
-utilities, alongside downstream task runners, tutorial notebooks, environment
-files, and plotting helpers.
-
-## Highlights
-
-| Area | What is included |
-| --- | --- |
-| Representation | Module-based single-cell encoder utilities and module metadata |
-| Annotation | Supervised cell-type annotation workflows |
-| Integration | Zero-shot batch integration and UMAP/scIB-style reporting |
-| Perturbation | Norman gene-perturbation training and evaluation tutorial |
-| Drug response | Single-drug response prediction with GDSC/CCLE data and validation-based model selection |
-| Drug combinations | Drug-combination synergy prediction with validation-based model selection |
-| Figures | Notebook-level PDF/PNG/SVG plotting cells for paper and GitHub display |
-
-## Repository Layout
-
-This is the `BioMod/` source directory inside the project root. See the
-[full project layout](../README.md#repository-layout) for sibling
-`model_dependencies/` and `weight/` directories; `data/` and `.cache/` are omitted
-from these repository trees.
-
-```text
-BioMod/                                                   # Task code: training, evaluation and tutorials
-├── cell_annotation/                                      # Cell-type annotation and model fine-tuning
-│   ├── biomod_data/                                      # Hallmark and Reactome modules with loading utilities
-│   │   ├── load_biomod.py
-│   │   ├── biomod1_hallmarks.json
-│   │   └── biomod2_reactome_2022_filtered.json
-│   ├── cmd_finetune.md
-│   └── biomod_finetune.py
-├── drug_combination/                                     # Drug-combination synergy prediction
-│   ├── README.md
-│   └── train_drug_combination_biomod.py
-├── drug_response/                                        # Single-drug response prediction and training
-│   ├── cmd.txt
-│   └── train_drug_biomod.py
-├── gene_perturbation/                                    # Perturbation response prediction: processing, training and inference
-│   ├── data_utils_local.py
-│   ├── gears_reduce_local.py
-│   ├── inference_local.py
-│   ├── model_biomod.py
-│   ├── pertdata_local.py
-│   ├── README.md
-│   ├── scfm_utils_local.py
-│   ├── train_biomod.py
-│   └── utils.py
-├── tests/                                                # CPU tests for shared training utilities
-│   └── test_training_utils.py
-├── toturials/                                            # Jupyter tutorials for five downstream tasks
-│   ├── cell_annotation.ipynb
-│   ├── drug_combination.ipynb
-│   ├── drug_response_prediction.ipynb
-│   ├── gene_perturbation.ipynb
-│   └── zero_shot_batch_integration.ipynb
-├── zero_shot_batch/                                      # Zero-shot batch integration, evaluation and visualization
-│   ├── sequencing_depth/                                 # Low-depth data generation and zero-shot evaluation
-│   │   ├── cmd.txt
-│   │   ├── make_low_depth_h5ad.py
-│   │   └── run_zero_shot_low_depth.py
-│   ├── biomod_zero_shot.py
-│   ├── README.md
-│   └── visualize_results.py
-├── .env.example
-├── .gitattributes
-├── .gitignore
-├── command.txt
-├── environment.yml
-├── models_biomod.py
-├── biomod_dataset.py
-├── biomod.py
-├── README.md
-├── requirements.txt
-└── training_utils.py
-```
-
-The notebook directory is intentionally named `toturials` to preserve existing
-paths used by the project.
-
-## Quick Start
-
-From the outer repository root, enter the source directory and create a fresh
-environment:
-
-```bash
-cd BioMod
-conda env create -f environment.yml
-conda activate biomod
-```
-
-Or install Python packages into an existing CUDA/PyTorch environment:
-
-```bash
-python -m pip install -r requirements.txt
-```
-
-From this source directory, activate the supplied environment and start Jupyter
-with the tutorial directory as its working directory:
-
-```bash
-conda activate biomod
-cd toturials
-jupyter lab
-```
-
-### Training utility checks
-
-The pretraining entry point imports `training_utils.py` explicitly. This local
-module supplies random seeding, a sequential distributed validation sampler,
-cosine learning-rate scheduling with warmup and restarts, scalar metric
-reduction, and resumable checkpoint saving. It does not depend on a separate
-package named `utils` or on a downstream task's utility module.
-
-From this source directory, run the CPU utility tests and check entry-point
-imports after installing the dependencies:
-
-```bash
-python -B -m unittest discover -s tests -p test_training_utils.py -v
-python -B biomod.py --help
-```
-
-The utility tests use synthetic tensors and temporary checkpoints; they do not
-load the supplied pretrained weights. Passing these checks does not validate a
-full training run. The pretraining entry point requires CUDA and an NCCL
-process group. Supply `--data_path` explicitly: the pretraining corpus is not
-bundled. Configure module inputs with `--biomod_data_dir` or `--biomod_data_path`.
-Generated checkpoints default to `tutorial_runs/pretraining/` under the outer
-repository root, with periodic latest checkpoints under its `latest/` subdirectory.
-
-## Shared Assets
-
-The BioMod model weights and matching configuration are stored in
-the outer repository's `weight/`. Use the supplied `.pth` and `.pkl`
-together.
-
-Tutorials expect a pretrained BioMod checkpoint and shared
-gene assets. For the shell examples below, set `BIOMOD_ROOT` to the directory containing
-`weight/` and `model_dependencies/` (the outer `BioMod/` project directory).
-
-```bash
-# Run this from the inner BioMod/ source directory.
-export BIOMOD_ROOT="$(cd .. && pwd)"
-```
-
-Current local asset files:
-
-```text
-$BIOMOD_ROOT/weight/                # BioMod model weights and matching configuration
-├── BioMod.pkl
-└── BioMod.pth
-
-$BIOMOD_ROOT/model_dependencies/              # Shared gene mappings, gene lists, embeddings and model components
-├── biomod_data_all/                                     # Shared module definitions and construction tools
-│   ├── create_uncovered_biomods.py
-│   ├── merge_biomods.py
-│   └── merged_biomods.json
-├── pytorch_cont/                              # Attention and model components
-│   ├── __init__.py
-│   ├── ding_models.py
-│   ├── pytorch.py
-│   ├── pytorch_old.py
-│   └── reversible.py
-├── gene2ens.csv
-├── selected_gene2vec_27k.npy
-└── selected_genes_27k.txt
-```
-
-Module definitions and construction tools are shared through
-`$BIOMOD_ROOT/model_dependencies/biomod_data_all/`.
-
-## BioMod model weights
-
-Use `BioMod.pth` with
-`BioMod.pkl` from the shared-assets directory above. The supplied
-configuration specifies a 2-layer encoder, a 27,874-gene axis, and 660 module
-definitions. Keep this configuration paired with the weights so that task
-runners can restore the corresponding architecture settings.
-
-## Tutorials
-
-| Task | Notebook | Main entry point | Default output style |
-| --- | --- | --- | --- |
-| Cell-type annotation | `toturials/cell_annotation.ipynb` | task script under `cell_annotation/` | metrics, classification report, confusion matrix |
-| Zero-shot batch integration | `toturials/zero_shot_batch_integration.ipynb` | `zero_shot_batch/` runner | UMAP PDF/PNG, avgBatch, avgBio |
-| Gene perturbation | `toturials/gene_perturbation.ipynb` | `gene_perturbation/` runner | selected model, configuration, parameters, and training log |
-| Drug response prediction | `toturials/drug_response_prediction.ipynb` | `drug_response/train_drug_biomod.py` | scatter, residuals, ROC, per-drug and per-cancer tables |
-| Drug-combination synergy | `toturials/drug_combination.ipynb` | `drug_combination/train_drug_combination_biomod.py` | label balance, loss, validation metrics, and ROC curves |
-
-Each notebook derives paths from `PROJECT_ROOT = Path('../..').resolve()` when
-Jupyter runs from `BioMod/toturials/`. From the repository root, change this to
-`Path('.').resolve()`. Training is disabled by default with `RUN_TRAINING = False`;
-zero-shot inference uses `RUN_EVALUATION = False`. Review the configuration and
-path checks before enabling execution. The shell variable `BIOMOD_ROOT` is for
-the command examples only and does not override notebook configuration cells.
-
-## Data Layout
-
-The Hugging Face repository includes datasets under the project-root `data/`
-directory. Individual tasks may require additional inputs or a different data
-layout. Check each task's required files, then configure its data paths for the
-local environment; the presence of `data/` does not imply that every tutorial
-has all of its inputs.
-
-### Single-Drug Response
-
-Expected clean data root:
-
-```text
-data/downstream/final_data/single_drug_response/data/
-|-- CCLE/
-|   |-- Cell_lines_annotations_20181226.txt
-|   |-- GDSC_IC50.csv
-|   `-- genomic_expression_561celllines_697genes_demap_features.csv
-`-- GDSC/
-    |-- 1.Drug_listMon Jun 24 09_00_55 2019.csv
-    `-- drug_graph_feat/
-```
-
-### Drug-Combination Synergy
-
-Expected clean data root:
-
-```text
-data/downstream/final_data/drug_combination_synergy/data_summary/
-|-- new_labels_0_10.csv
-|-- new_cell_features_954.csv
-|-- smiles_0_10.csv
-`-- test_data/
-    |-- independent_input.csv
-    `-- independent_cell_features_954.csv
-```
-
-The drug-combination tutorial reads the metric tables and ROC artifacts produced
-by its training script. The script does not export per-example prediction CSVs.
-
-## Example Commands
-
-Run these commands from the inner `BioMod/` source directory after setting
-`BIOMOD_ROOT` as shown above. Drug response prediction with the BioMod model weights:
-
-```bash
-python drug_response/train_drug_biomod.py \
-  --gpu_id 0 \
-  --data_path "$BIOMOD_ROOT/data/downstream/final_data/single_drug_response/data" \
-  --checkpoint_path "$BIOMOD_ROOT/weight/BioMod.pth" \
-  --hyper_params_path "$BIOMOD_ROOT/weight/BioMod.pkl" \
-  --biomod_data_dir "$BIOMOD_ROOT/model_dependencies/biomod_data_all" \
-  --gene_list_path "$BIOMOD_ROOT/model_dependencies/selected_genes_27k.txt" \
-  --gene_mapping_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
-  --gene2vec_path "$BIOMOD_ROOT/model_dependencies/selected_gene2vec_27k.npy" \
-  --symbol2ens_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
-  --save_dir "$BIOMOD_ROOT/tutorial_runs/drug_response_prediction" \
-  --epochs 300 \
-  --batch_size 32 \
-  --pool_method max
-```
-
-Drug-combination synergy prediction:
-
-```bash
-python drug_combination/train_drug_combination_biomod.py \
-  --checkpoint_path "$BIOMOD_ROOT/weight/BioMod.pth" \
-  --hyper_params_path "$BIOMOD_ROOT/weight/BioMod.pkl" \
-  --biomod_data_dir "$BIOMOD_ROOT/model_dependencies/biomod_data_all" \
-  --gene_list_path "$BIOMOD_ROOT/model_dependencies/selected_genes_27k.txt" \
-  --gene_mapping_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
-  --gene2vec_path "$BIOMOD_ROOT/model_dependencies/selected_gene2vec_27k.npy" \
-  --symbol2ens_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
-  --data_dir "$BIOMOD_ROOT/data/downstream/final_data/drug_combination_synergy/data_summary" \
-  --output_dir "$BIOMOD_ROOT/tutorial_runs/drug_combination/results" \
-  --epochs 500 \
-  --batch_size 256 \
-  --lr 1e-4 \
-  --n_folds 1 \
-  --pool_method max
-```
-
-## Display Outputs
-
-The notebooks regenerate figure files in task output directories. Typical files
-include:
-
-| Task | Generated display files |
-| --- | --- |
-| Zero-shot integration | A4 UMAP PDF/PNG panels, combined cell-type and batch views |
-| Drug response | `drug_response_overall_scatter.pdf`, `drug_response_median_threshold_roc.svg`, per-drug and per-cancer CSV summaries |
-| Drug combination | `drug_combination_metric_curves.pdf`, `drug_combination_roc_curves.pdf`, `drug_combination_label_balance.pdf` |
-| Cell annotation | confusion matrices and classification reports |
-| Gene perturbation | training logs containing perturbation evaluation metrics |
-
-For GitHub display, use PNG files. For manuscripts, use the PDF/SVG exports.
-The plotting cells use embedded, editable PDF fonts where supported.
-
-## Evaluation Results
-
-This README does not provide verified benchmark scores for the bundled BioMod
-model weights. Run the relevant task with a recorded data split and configuration
-before reporting performance. Notebook plotting cells and historical outputs
-alone do not establish performance for the supplied weights.
-
-## Reproducibility Conventions
-
-- Record the exact command line, checkpoint, hyper-parameter pkl, data version,
-  random seed, and output directory for every final run.
-- Keep raw data, cached embeddings, model checkpoints, metric tables, and paper
-  figures in separate directories.
-- Prefer shared storage for large `.h5ad`, `.npy`, and checkpoint outputs.
-- Do not commit private data, raw model weights, API keys, or personal absolute
-  paths.
-
-## Troubleshooting
-
-### Missing data
-
-Open the relevant notebook and run the path-check cell. It prints all required
-files and shows which entries are missing.
-
-### Missing model assets
-
-Check `BIOMOD_ROOT` for shell commands or `PROJECT_ROOT` in a notebook, and verify that the BioMod model weights, matching
-`.pkl` configuration, gene2vec file, gene list, gene mapping, and
-`pytorch_cont/` directory exist at the paths listed under Shared Assets.
-
-### No space left on device
-
-Large `.h5ad`, `.npy`, checkpoint, and figure outputs can fill local disks.
-Move result directories to shared storage and remove partial failed outputs only
-after confirming they are not needed.
-
-### Matplotlib font warnings
-
-The notebooks request Arial first and fall back to DejaVu Sans when Arial is not
-installed. PDF font embedding is enabled through `pdf.fonttype = 42`.
-
-## GitHub Release Checklist
-
-- Keep `README.md`, `requirements.txt`, `environment.yml`, `.gitignore`, and
-  `.gitattributes` at the repository root.
-- Confirm tutorial notebooks open without personal paths.
-- Keep large data and checkpoints outside Git, then document download or
-  placement instructions.
-- Add final figure examples under a small `docs/` or `assets/` directory if the
-  repository needs rendered images on the GitHub landing page.
-- Add license, citation, and data-availability text before public release.
-
-## Changelog
-
-| Date | Change |
-| --- | --- |
-| 2026-06-15 | Added paper-style plotting cells for single-drug and drug-combination tutorials. |
-| 2026-06-15 | Removed `stage2_alignment/` from the active repository scope. |
-| 2026-06-15 | Added root environment files and GitHub-ready repository documentation. |
+# BioMod
+
+Source code and tutorial notebooks for BioMod single-cell pretraining and
+downstream experiments using BioMod representations.
+
+This source directory contains the pretraining entry point and shared training
+utilities, alongside downstream task runners, tutorial notebooks, environment
+files, and plotting helpers.
+
+## Highlights
+
+| Area | What is included |
+| --- | --- |
+| Representation | Module-based single-cell encoder utilities and module metadata |
+| Annotation | Supervised cell-type annotation workflows |
+| Integration | Zero-shot batch integration and UMAP/scIB-style reporting |
+| Perturbation | Norman gene-perturbation training and evaluation tutorial |
+| Drug response | Single-drug response prediction with GDSC/CCLE data and validation-based model selection |
+| Drug combinations | Drug-combination synergy prediction with validation-based model selection |
+| Figures | Notebook-level PDF/PNG/SVG plotting cells for paper and GitHub display |
+
+## Repository Layout
+
+This is the `BioMod/` source directory inside the project root. See the
+[full project layout](../README.md#repository-layout) for sibling
+`model_dependencies/` and `weight/` directories; `data/` and `.cache/` are omitted
+from these repository trees.
+
+Each downstream task provides a `README.md`, a `tutorial.ipynb`, and a `train.py`
+entry point, except zero-shot evaluation, which uses `evaluate.py`. Gene
+perturbation keeps its supporting code in `components/`; zero-shot visualization
+and sequencing-depth helpers are under `tools/`. Optional Hallmark and Reactome
+source definitions are shared under `model_dependencies/biomod_data_sources/`
+in the outer project root.
+
+```text
+BioMod/                                                        # Task code: training, evaluation and tutorials
+├── cell_annotation/                                           # Cell-type annotation and model fine-tuning
+│   ├── README.md
+│   ├── train.py
+│   └── tutorial.ipynb
+├── drug_combination/                                          # Drug-combination synergy prediction
+│   ├── README.md
+│   ├── train.py
+│   └── tutorial.ipynb
+├── drug_response/                                             # Single-drug response prediction
+│   ├── README.md
+│   ├── train.py
+│   └── tutorial.ipynb
+├── gene_perturbation/                                         # Gene perturbation response prediction
+│   ├── components/                                            # Task models, datasets, training and inference helpers
+│   │   ├── __init__.py
+│   │   ├── data_utils.py
+│   │   ├── dataset.py
+│   │   ├── inference.py
+│   │   ├── model.py
+│   │   ├── trainer.py
+│   │   ├── training_utils.py
+│   │   └── utils.py
+│   ├── README.md
+│   ├── train.py
+│   └── tutorial.ipynb
+├── tests/                                                     # CPU tests for shared training utilities
+│   └── test_training_utils.py
+├── zero_shot_batch/                                           # Zero-shot integration and evaluation
+│   ├── tools/                                                 # Optional visualization and evaluation tools
+│   │   ├── sequencing_depth/                                  # Low-depth expression generation and evaluation
+│   │   │   ├── make_low_depth_h5ad.py
+│   │   │   ├── README.md
+│   │   │   └── run_zero_shot_low_depth.py
+│   │   └── visualize_results.py
+│   ├── evaluate.py
+│   ├── README.md
+│   └── tutorial.ipynb
+├── .gitattributes
+├── .gitignore
+├── biomod.py
+├── biomod_dataset.py
+├── command.txt
+├── environment.yml
+├── models_biomod.py
+├── README.md
+├── requirements.txt
+└── training_utils.py
+```
+
+## Quick Start
+
+From the outer repository root, enter the source directory and create a fresh
+environment:
+
+```bash
+cd BioMod
+conda env create -f environment.yml
+conda activate biomod
+```
+
+Or install Python packages into an existing CUDA/PyTorch environment:
+
+```bash
+python -m pip install -r requirements.txt
+```
+
+From this source directory, activate the supplied environment and start Jupyter
+in the chosen task directory. For cell annotation:
+
+```bash
+conda activate biomod
+cd cell_annotation
+jupyter lab tutorial.ipynb
+```
+
+Use `drug_response`, `drug_combination`, `gene_perturbation`, or `zero_shot_batch`
+in place of `cell_annotation` to open another task.
+
+### Training utility checks
+
+The pretraining entry point imports `training_utils.py` explicitly. This local
+module supplies random seeding, a sequential distributed validation sampler,
+cosine learning-rate scheduling with warmup and restarts, scalar metric
+reduction, and resumable checkpoint saving. It does not depend on a separate
+package named `utils` or on a downstream task's utility module.
+
+From this source directory, run the CPU utility tests and check entry-point
+imports after installing the dependencies:
+
+```bash
+python -B -m unittest discover -s tests -p test_training_utils.py -v
+python -B biomod.py --help
+```
+
+The utility tests use synthetic tensors and temporary checkpoints; they do not
+load the supplied pretrained weights. Passing these checks does not validate a
+full training run. The pretraining entry point requires CUDA and an NCCL
+process group. Supply `--data_path` explicitly: the pretraining corpus is not
+bundled. Configure module inputs with `--biomod_data_dir` or `--biomod_data_path`.
+Generated checkpoints default to `tutorial_runs/pretraining/` under the outer
+repository root, with periodic latest checkpoints under its `latest/` subdirectory.
+
+## Shared Assets
+
+The BioMod model weights and matching configuration are stored in
+the outer repository's `weight/`. Use the supplied `.pth` and `.pkl`
+together.
+
+Tutorials expect a pretrained BioMod checkpoint and shared
+gene assets. For the shell examples below, set `BIOMOD_ROOT` to the directory containing
+`weight/` and `model_dependencies/` (the outer `BioMod/` project directory).
+
+```bash
+# Run this from the inner BioMod/ source directory.
+export BIOMOD_ROOT="$(cd .. && pwd)"
+```
+
+Current local asset files:
+
+```text
+$BIOMOD_ROOT/weight/                # BioMod model weights and matching configuration
+├── BioMod.pkl
+└── BioMod.pth
+
+$BIOMOD_ROOT/model_dependencies/              # Shared gene mappings, gene lists, embeddings and model components
+├── biomod_data_all/                                     # Shared module definitions and construction tools
+│   ├── create_uncovered_biomods.py
+│   ├── merge_biomods.py
+│   └── merged_biomods.json
+├── biomod_data_sources/                      # Optional Hallmark and Reactome source definitions
+│   ├── biomod1_hallmarks.json
+│   ├── biomod2_reactome_2022_filtered.json
+│   └── load_biomod.py
+├── pytorch_cont/                              # Attention and model components
+│   ├── __init__.py
+│   ├── ding_models.py
+│   ├── pytorch.py
+│   ├── pytorch_old.py
+│   └── reversible.py
+├── gene2ens.csv
+├── selected_gene2vec_27k.npy
+└── selected_genes_27k.txt
+```
+
+Default runtime module definitions and construction tools are shared through
+`$BIOMOD_ROOT/model_dependencies/biomod_data_all/`. The optional
+[Hallmark and Reactome source definitions](../model_dependencies/biomod_data_sources/)
+and JSON loading helper are stored in `biomod_data_sources/`. These source
+resources do not replace the default definitions paired with the supplied checkpoint.
+
+## BioMod model weights
+
+Use `BioMod.pth` with
+`BioMod.pkl` from the shared-assets directory above. The supplied
+configuration specifies a 2-layer encoder, a 27,874-gene axis, and 660 module
+definitions. Keep this configuration paired with the weights so that task
+runners can restore the corresponding architecture settings.
+
+## Tutorials
+
+| Task | Notebook | Main entry point | Default output style |
+| --- | --- | --- | --- |
+| Cell-type annotation | [Notebook](cell_annotation/tutorial.ipynb) | `cell_annotation/train.py` | metrics, classification report, confusion matrix |
+| Zero-shot batch integration | [Notebook](zero_shot_batch/tutorial.ipynb) | `zero_shot_batch/evaluate.py` | UMAP PDF/PNG, avgBatch, avgBio |
+| Gene perturbation | [Notebook](gene_perturbation/tutorial.ipynb) | `gene_perturbation/train.py` | selected model, configuration, parameters, and training log |
+| Drug response prediction | [Notebook](drug_response/tutorial.ipynb) | `drug_response/train.py` | scatter, residuals, ROC, per-drug and per-cancer tables |
+| Drug-combination synergy | [Notebook](drug_combination/tutorial.ipynb) | `drug_combination/train.py` | label balance, loss, validation metrics, and ROC curves |
+
+Each notebook derives paths from `PROJECT_ROOT = Path('../..').resolve()` when
+its kernel runs from the corresponding task directory, such as
+`BioMod/cell_annotation/`. With a different kernel working directory, set
+`PROJECT_ROOT` explicitly to the outer repository root. Training is disabled by
+default with `RUN_TRAINING = False`; zero-shot inference uses
+`RUN_EVALUATION = False`. Review the configuration and
+path checks before enabling execution. The shell variable `BIOMOD_ROOT` is for
+the command examples only and does not override notebook configuration cells.
+
+## Data Layout
+
+The Hugging Face repository includes datasets under the project-root `data/`
+directory. Individual tasks may require additional inputs or a different data
+layout. Check each task's required files, then configure its data paths for the
+local environment; the presence of `data/` does not imply that every tutorial
+has all of its inputs.
+
+### Single-Drug Response
+
+Expected clean data root:
+
+```text
+data/downstream/final_data/single_drug_response/data/
+|-- CCLE/
+|   |-- Cell_lines_annotations_20181226.txt
+|   |-- GDSC_IC50.csv
+|   `-- genomic_expression_561celllines_697genes_demap_features.csv
+`-- GDSC/
+    |-- 1.Drug_listMon Jun 24 09_00_55 2019.csv
+    `-- drug_graph_feat/
+```
+
+### Drug-Combination Synergy
+
+Expected clean data root:
+
+```text
+data/downstream/final_data/drug_combination_synergy/data_summary/
+|-- new_labels_0_10.csv
+|-- new_cell_features_954.csv
+|-- smiles_0_10.csv
+`-- test_data/
+    |-- independent_input.csv
+    `-- independent_cell_features_954.csv
+```
+
+The drug-combination tutorial reads the metric tables and ROC artifacts produced
+by its training script. The script does not export per-example prediction CSVs.
+
+## Example Commands
+
+Run these commands from the inner `BioMod/` source directory after setting
+`BIOMOD_ROOT` as shown above. Drug response prediction with the BioMod model weights:
+
+```bash
+python drug_response/train.py \
+  --gpu_id 0 \
+  --data_path "$BIOMOD_ROOT/data/downstream/final_data/single_drug_response/data" \
+  --checkpoint_path "$BIOMOD_ROOT/weight/BioMod.pth" \
+  --hyper_params_path "$BIOMOD_ROOT/weight/BioMod.pkl" \
+  --biomod_data_dir "$BIOMOD_ROOT/model_dependencies/biomod_data_all" \
+  --gene_list_path "$BIOMOD_ROOT/model_dependencies/selected_genes_27k.txt" \
+  --gene_mapping_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
+  --gene2vec_path "$BIOMOD_ROOT/model_dependencies/selected_gene2vec_27k.npy" \
+  --symbol2ens_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
+  --save_dir "$BIOMOD_ROOT/tutorial_runs/drug_response_prediction" \
+  --epochs 300 \
+  --batch_size 32 \
+  --pool_method max
+```
+
+Drug-combination synergy prediction:
+
+```bash
+python drug_combination/train.py \
+  --checkpoint_path "$BIOMOD_ROOT/weight/BioMod.pth" \
+  --hyper_params_path "$BIOMOD_ROOT/weight/BioMod.pkl" \
+  --biomod_data_dir "$BIOMOD_ROOT/model_dependencies/biomod_data_all" \
+  --gene_list_path "$BIOMOD_ROOT/model_dependencies/selected_genes_27k.txt" \
+  --gene_mapping_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
+  --gene2vec_path "$BIOMOD_ROOT/model_dependencies/selected_gene2vec_27k.npy" \
+  --symbol2ens_path "$BIOMOD_ROOT/model_dependencies/gene2ens.csv" \
+  --data_dir "$BIOMOD_ROOT/data/downstream/final_data/drug_combination_synergy/data_summary" \
+  --output_dir "$BIOMOD_ROOT/tutorial_runs/drug_combination/results" \
+  --epochs 500 \
+  --batch_size 256 \
+  --lr 1e-4 \
+  --n_folds 1 \
+  --pool_method max
+```
+
+## Display Outputs
+
+The notebooks regenerate figure files in task output directories. Typical files
+include:
+
+| Task | Generated display files |
+| --- | --- |
+| Zero-shot integration | A4 UMAP PDF/PNG panels, combined cell-type and batch views |
+| Drug response | `drug_response_overall_scatter.pdf`, `drug_response_median_threshold_roc.svg`, per-drug and per-cancer CSV summaries |
+| Drug combination | `drug_combination_metric_curves.pdf`, `drug_combination_roc_curves.pdf`, `drug_combination_label_balance.pdf` |
+| Cell annotation | confusion matrices and classification reports |
+| Gene perturbation | training logs containing perturbation evaluation metrics |
+
+For GitHub display, use PNG files. For manuscripts, use the PDF/SVG exports.
+The plotting cells use embedded, editable PDF fonts where supported.
+
+## Evaluation Results
+
+This README does not provide verified benchmark scores for the bundled BioMod
+model weights. Run the relevant task with a recorded data split and configuration
+before reporting performance. Notebook plotting cells and historical outputs
+alone do not establish performance for the supplied weights.
+
+## Reproducibility Conventions
+
+- Record the exact command line, checkpoint, hyper-parameter pkl, data version,
+  random seed, and output directory for every final run.
+- Keep raw data, cached embeddings, model checkpoints, metric tables, and paper
+  figures in separate directories.
+- Prefer shared storage for large `.h5ad`, `.npy`, and checkpoint outputs.
+- Do not commit private data, raw model weights, API keys, or personal absolute
+  paths.
+
+## Troubleshooting
+
+### Missing data
+
+Open the relevant notebook and run the path-check cell. It prints all required
+files and shows which entries are missing.
+
+### Missing model assets
+
+Check `BIOMOD_ROOT` for shell commands or `PROJECT_ROOT` in a notebook, and verify that the BioMod model weights, matching
+`.pkl` configuration, gene2vec file, gene list, gene mapping, and
+`pytorch_cont/` directory exist at the paths listed under Shared Assets.
+
+### No space left on device
+
+Large `.h5ad`, `.npy`, checkpoint, and figure outputs can fill local disks.
+Move result directories to shared storage and remove partial failed outputs only
+after confirming they are not needed.
+
+### Matplotlib font warnings
+
+The notebooks request Arial first and fall back to DejaVu Sans when Arial is not
+installed. PDF font embedding is enabled through `pdf.fonttype = 42`.
+
+## GitHub Release Checklist
+
+- Keep `README.md`, `requirements.txt`, `environment.yml`, `.gitignore`, and
+  `.gitattributes` at the repository root.
+- Confirm tutorial notebooks open without personal paths.
+- Keep large data and checkpoints outside Git, then document download or
+  placement instructions.
+- Add final figure examples under a small `docs/` or `assets/` directory if the
+  repository needs rendered images on the GitHub landing page.
+- Add license, citation, and data-availability text before public release.
+
+## Changelog
+
+| Date | Change |
+| --- | --- |
+| 2026-06-15 | Added paper-style plotting cells for single-drug and drug-combination tutorials. |
+| 2026-06-15 | Removed `stage2_alignment/` from the active repository scope. |
+| 2026-06-15 | Added root environment files and GitHub-ready repository documentation. |

BioMod/cell_annotation/README.md

@@ -0,0 +1,282 @@
+# Cell Annotation with BioMod
+
+Fine-tune BioMod to predict cell-type labels from single-cell gene expression.
+The training script supports frozen, partially unfrozen, and fully trainable
+backbones with several cell-embedding heads.
+
+## Notebook
+
+See the [task notebook](tutorial.ipynb) for a step-by-step workflow.
+
+## Files
+
+| File or directory | Purpose |
+| --- | --- |
+| [README.md](README.md) | File guide and command-line training example. |
+| [tutorial.ipynb](tutorial.ipynb) | Task configuration, execution, and result review in a Jupyter notebook. |
+| [train.py](train.py) | Train and evaluate the cell-type classifier with the BioMod backbone. |
+
+## Optional Module Sources
+
+The optional [Hallmark and Reactome source definitions](../../model_dependencies/biomod_data_sources/)
+are stored separately from this task. They include
+[Hallmark modules](../../model_dependencies/biomod_data_sources/biomod1_hallmarks.json),
+[filtered Reactome modules](../../model_dependencies/biomod_data_sources/biomod2_reactome_2022_filtered.json),
+and a [JSON loading helper](../../model_dependencies/biomod_data_sources/load_biomod.py).
+These source resources are not the default runtime module definitions for the
+supplied checkpoint. The training example below continues to use
+`model_dependencies/biomod_data_all/` with the matching BioMod weights and configuration.
+
+## Dataset Commands
+
+The manuscript's annotation task covers seven benchmarks. The commands below
+map those benchmarks to the files currently bundled in this repository; paired
+files are one benchmark, not two separate datasets. Kim Lung has an unresolved
+missing-label policy documented in its section.
+
+Run from the outer repository root after installing the environment. Explicit
+relative paths resolve from the working directory. Each command writes to its
+own `tutorial_runs/` directory. Ordinary `python` supports one GPU; CUDA DDP with
+multiple processes additionally requires a working NCCL environment.
+
+The command blocks use Bash line continuations (`\`). Run them in Bash, such as
+Linux, WSL or Git Bash. In PowerShell, join each command into a single line
+instead of copying the Bash continuation characters.
+
+### Training Protocol and Limits
+
+The settings shared by these examples follow the available options for the
+annotation method described in manuscript Section 4.5 (PDF pages 24–25): attention
+pooling, a three-layer classifier, dropout 0.2, full encoder-path fine-tuning,
+AdamW, backbone and head learning rates of `1e-4`, and weight decay `0.01`.
+The script applies cosine annealing internally. `--label_smoothing 0` selects
+the plain cross-entropy loss shown in the manuscript instead of the script's
+default label smoothing of 0.1.
+
+The manuscript does not specify the annotation batch size, epoch limit, patience,
+or complete seed list in this section. The examples explicitly use batch size
+32, 50 epochs, patience 10 and seed 0 as launch settings; these values are not
+claimed to reproduce the reported five-run results. Reduce the batch size to
+fit available GPU memory and retain the actual experiment settings for reporting.
+The best checkpoint is selected by validation macro-F1, then evaluated on the
+test set.
+
+The classification report and confusion matrix include every declared class,
+including rows with zero test support. The report's macro average covers this
+full class list. Training and evaluation `macro_f1` values use the classes
+present in the true or predicted labels, so the two averages can differ when
+some declared classes are absent from both. Checkpoint selection keeps the
+existing validation metric.
+
+For single files, the entry point creates a stratified 70% / 10% / 20% split.
+For paired files, it splits the supplied training file 90% / 10% and uses the
+entire supplied test file for final evaluation. Stored split columns in `obs`
+are not used. The manuscript does not list these exact filenames, so the
+reference/query mapping is based on the bundled file organization and metadata.
+
+Both expression filtering thresholds are explicitly zero to preserve the supplied
+cells and genes. The entry point reads `adata.X` as supplied: it does not select
+`layers["counts"]` or `.raw`, normalize counts, or apply `log1p`. Metadata checks
+do not establish that each expression matrix has the intended preprocessing.
+Model training and manuscript-result reproduction have not been validated by
+these command examples.
+
+### 1. Kim Lung
+
+**Conditional configuration: missing-label handling must be resolved before
+benchmarking.** The file contains 32,493 cells, including 2,021 cells with missing
+`cell_type`. The current classifier converts labels to strings, so the command
+below would treat missing values as an extra class instead of excluding them.
+The manuscript does not specify the annotation policy for these cells. This
+block records the intended file and fields; it is not a verified reproduction
+command. Do not silently relabel or remove cells from the original file.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --data_path data/downstream/final_data/Kim2020_Lung.h5ad \
+    --label_column cell_type --gene_name_column gene_name \
+    --val_ratio 0.1 --test_ratio 0.2 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_kim_lung --run_name seed0
+```
+
+### 2. Immune
+
+Use the 33,506-cell matrix and its 16 `final_annotation` classes. Gene symbols
+are stored in `var.index`. A new stratified 70% / 10% / 20% split is created.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --data_path data/downstream/final_data/Immune_ALL_human.h5ad \
+    --label_column final_annotation --gene_name_column index \
+    --val_ratio 0.1 --test_ratio 0.2 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_immune --run_name seed0
+```
+
+### 3. Liver
+
+Use the 8,444-cell matrix and its 20 `cell_type` classes. Gene symbols are stored
+in `var.index`. A new stratified 70% / 10% / 20% split is created.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --data_path data/downstream/final_data/liver.h5ad \
+    --label_column cell_type --gene_name_column index \
+    --val_ratio 0.1 --test_ratio 0.2 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_liver --run_name seed0
+```
+
+### 4. MS
+
+Keep the 13,468-cell external test file separate. The 7,844-cell training file
+is split 90% / 10% for training and validation. All 18 test labels occur in the
+training file. Some training classes contain only three or four cells, so a
+small validation split may not contain every rare class.
+
+Use `index_column`, which contains Ensembl IDs in both files. The training file
+has an index *named* `gene_name`, but no `var["gene_name"]` column. Consequently,
+the unmodified `--dataset_preset ms` gene-column default is incompatible with
+that file. The explicit configuration below avoids that default.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --train_data_path data/downstream/final_data/MS/c_data.h5ad \
+    --test_data_path data/downstream/final_data/MS/filtered_ms_adata.h5ad \
+    --label_column celltype --test_label_column celltype --gene_name_column index_column \
+    --val_ratio 0.1 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_ms --run_name seed0
+```
+
+### 5. hPancreas
+
+The 10,600-cell training file declares 14 categorical labels and contains cells
+from 13 of them. `MHC class II` is already declared in its `Celltype` categories
+with a training count of **0**; the 4,218-cell test file contains **5** such cells.
+The entry point preserves the predefined training categories when constructing
+the classifier and label encoder, including categories with zero training cells.
+It does not create cells, add test cells to training, or infer the class list from
+the test labels. Stratification still operates only on the observed training
+cells. Keeping the output category does not supply positive training examples
+for it.
+
+Both files use `Celltype` and gene symbols in `var.index`. This annotation
+benchmark is distinct from the 16,382-cell hPNC integration dataset.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --train_data_path data/downstream/final_data/hp14/demo_train_with_MHC_class_II.h5ad \
+    --test_data_path data/downstream/final_data/hp14/demo_test.h5ad \
+    --label_column Celltype --test_label_column Celltype --gene_name_column index \
+    --val_ratio 0.1 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_hpancreas --run_name seed0
+```
+
+### 6. Zheng68K
+
+Use the 68,450-cell matrix and its 11 `celltype` classes. Gene symbols are stored
+in `var.index`. The 20% test split contains 13,690 cells, matching the test-set
+size reported in the manuscript; exact cell identities still depend on the seed
+and input ordering.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --data_path data/downstream/final_data/Zheng68K.h5ad \
+    --label_column celltype --gene_name_column index \
+    --val_ratio 0.1 --test_ratio 0.2 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_zheng68k --run_name seed0
+```
+
+### 7. Myeloid
+
+Use the 9,748-cell reference file for training/validation and retain the
+3,430-cell query file as the external test set. The reference contains 21
+observed labels and the query contains 11; every query label occurs in the
+reference. Both files use `cell_type` and gene symbols in `var.index`.
+
+```bash
+python BioMod/cell_annotation/train.py \
+    --train_data_path data/downstream/final_data/Myeloid/reference_adata.h5ad \
+    --test_data_path data/downstream/final_data/Myeloid/query_adata.h5ad \
+    --label_column cell_type --test_label_column cell_type --gene_name_column index \
+    --val_ratio 0.1 \
+    --biomod_checkpoint weight/BioMod.pth --config_pkl weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene_list_path model_dependencies/selected_genes_27k.txt \
+    --gene2ens_file model_dependencies/gene2ens.csv --gene_mapping_path model_dependencies/gene2ens.csv \
+    --gene2vec_data_path model_dependencies/selected_gene2vec_27k.npy \
+    --head_type attn_pool --finetune_mode full --cls_decoder_layers 3 \
+    --dropout 0.2 --label_smoothing 0 \
+    --lr_head 1e-4 --lr_backbone 1e-4 --weight_decay 0.01 \
+    --min_genes 0 --min_cells 0 \
+    --batch_size 32 --epochs 50 --patience 10 --seed 0 \
+    --save_test_artifacts --output_dir tutorial_runs/cell_annotation_myeloid --run_name seed0
+```
+
+## Related Datasets
+
+`BMMC_processed.h5ad`, `human_pancreas_norm_complexBatch_with_gene_name.h5ad`,
+and the lung file without malignant cells belong to the documented
+[zero-shot integration conditions](../zero_shot_batch/README.md#dataset-commands).
+`norman/perturb_processed.h5ad` belongs to
+[gene perturbation prediction](../gene_perturbation/README.md).
+Their presence in the data directory does not add annotation benchmarks to the
+seven listed above.

BioMod/cell_annotation/cmd_finetune.md

@@ -1,21 +0,0 @@
-# Cell annotation command
-
-Run this command from the repository root in a CUDA/NCCL environment. Resource defaults resolve from the script location; explicit relative paths resolve from the working directory.
-
-```bash
-torchrun --nproc_per_node=4 BioMod/cell_annotation/biomod_finetune.py \
---config_pkl weight/BioMod.pkl \
---biomod_checkpoint weight/BioMod.pth \
---data_path data/downstream/final_data/Zheng68K.h5ad \
---label_column celltype \
---head_type attn_pool \
---finetune_mode last_n \
---unfreeze_last_n 1 \
---cls_decoder_layers 3 \
---lr_head 1e-4 \
---lr_backbone 1e-5 \
---batch_size 256 \
---epochs 300 \
---biomod_data_dir model_dependencies/biomod_data_all \
---output_dir tutorial_runs/cell_annotation_zheng68k
-```

BioMod/cell_annotation/{biomod_finetune.py → train.py}

@@ -244,9 +244,16 @@ def preprocess_adata(
         max_cells: Optional[int] = None,
         dataset_name: str = "dataset",
         gene_name_column: Optional[str] = None,
+        label_column: Optional[str] = None,
     ) -> Tuple[ad.AnnData, dict]:
         logger.info("Loading data from %s", adata_path)
         adata = ad.read_h5ad(adata_path)
+        label_dtype = None
+        if label_column is not None:
+            if label_column not in adata.obs:
+                raise ValueError(f"[{dataset_name}] Missing label column: {label_column}")
+            if isinstance(adata.obs[label_column].dtype, pd.CategoricalDtype):
+                label_dtype = adata.obs[label_column].dtype
         matrix = adata.X
 
         if sparse.issparse(matrix):
@@ -266,6 +273,10 @@ def preprocess_adata(
             chosen = np.sort(rng.choice(adata.n_obs, size=max_cells, replace=False))
             adata = adata[chosen].copy()
 
+        # AnnData slicing can drop declared classes with no remaining cells.
+        if label_dtype is not None:
+            adata.obs[label_column] = adata.obs[label_column].astype(label_dtype)
+
         reindex_state = self._build_reindex_state(
             adata,
             dataset_name=dataset_name,
@@ -869,6 +880,15 @@ def load_backbone(args, device: torch.device) -> Tuple["BioMod", List[List[int]]
     return backbone, biomod_data, model_config
 
 
+def fit_label_encoder(labels: pd.Series) -> LabelEncoder:
+    """Keep declared training classes even when their observed count is zero."""
+    values = labels.astype(str).to_numpy()
+    if isinstance(labels.dtype, pd.CategoricalDtype):
+        declared = labels.cat.categories.astype(str).to_numpy()
+        values = np.concatenate([values, declared])
+    return LabelEncoder().fit(values)
+
+
 def stratified_split(
     labels: np.ndarray,
     seed: int,
@@ -1247,6 +1267,7 @@ def main():
             max_cells=args.max_cells,
             dataset_name="train",
             gene_name_column=args.gene_name_column,
+            label_column=args.label_column,
         )
         test_adata, test_reindex_state = preprocessor.preprocess_adata(
             str(resolve_path(args.test_data_path)),
@@ -1255,20 +1276,23 @@ def main():
             max_cells=None,
             dataset_name="test",
             gene_name_column=args.gene_name_column,
+            label_column=args.test_label_column or args.label_column,
         )
 
         train_label_column = args.label_column
         test_label_column = args.test_label_column or args.label_column
-        label_encoder = LabelEncoder()
+        label_encoder = fit_label_encoder(train_adata.obs[train_label_column])
         train_labels_raw = train_adata.obs[train_label_column].astype(str).to_numpy()
-        label_encoder.fit(train_labels_raw)
         labels_train = label_encoder.transform(train_labels_raw)
+        zero_count_classes = sorted(set(label_encoder.classes_) - set(train_labels_raw))
+        if zero_count_classes:
+            logger.info("Declared classes with zero training-file cells: %s", zero_count_classes)
 
         test_labels_raw = test_adata.obs[test_label_column].astype(str).to_numpy()
         unseen_test_labels = sorted(set(test_labels_raw) - set(label_encoder.classes_))
         if unseen_test_labels:
             raise ValueError(
-                "External test set contains labels unseen in training data: "
+                "External test set contains labels not declared or observed in training data: "
                 + ", ".join(unseen_test_labels[:10])
             )
         labels_test = label_encoder.transform(test_labels_raw)
@@ -1304,10 +1328,11 @@ def main():
             max_cells=args.max_cells,
             dataset_name="data",
             gene_name_column=args.gene_name_column,
+            label_column=args.label_column,
         )
 
-        label_encoder = LabelEncoder()
-        labels = label_encoder.fit_transform(adata.obs[args.label_column].astype(str).to_numpy())
+        label_encoder = fit_label_encoder(adata.obs[args.label_column])
+        labels = label_encoder.transform(adata.obs[args.label_column].astype(str).to_numpy())
         num_classes = len(label_encoder.classes_)
         train_idx, val_idx, test_idx = stratified_split(
             labels=labels,
@@ -1547,8 +1572,10 @@ def main():
     report = classification_report(
         test_metrics["labels"],
         test_metrics["preds"],
+        labels=np.arange(num_classes),
         target_names=label_encoder.classes_.tolist(),
         digits=4,
+        zero_division=0,
     )
     conf, conf_norm, conf_df, conf_norm_df = build_confusion_tables(
         test_metrics["labels"],

BioMod/{toturials/cell_annotation.ipynb → cell_annotation/tutorial.ipynb}

@@ -15,7 +15,7 @@
    "source": [
     "## Project paths\n",
     "\n",
-    "Start Jupyter from `BioMod/toturials/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
+    "Start Jupyter from `BioMod/cell_annotation/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
    ]
   },
   {
@@ -24,7 +24,7 @@
    "source": [
     "## Inputs and workflow\n",
     "\n",
-    "- Entry point: `BioMod/cell_annotation/biomod_finetune.py`.\n",
+    "- Entry point: `BioMod/cell_annotation/train.py`.\n",
     "- Dataset: `data/downstream/final_data/Zheng68K.h5ad`.\n",
     "- Model weights and matching configuration: the `.pth` / `.pkl` pair under `weight/`.\n",
     "- Gene vocabulary, mapping, embeddings, and module definitions: `model_dependencies/`.\n",
@@ -56,7 +56,7 @@
     "PROJECT_ROOT = Path('../..').resolve()\n",
     "if not (PROJECT_ROOT / 'BioMod' / 'models_biomod.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
     "    raise FileNotFoundError(\n",
-    "        'Project root not found. Start Jupyter from BioMod/toturials, or set '\n",
+    "        'Project root not found. Start Jupyter from BioMod/cell_annotation, or set '\n",
     "        'PROJECT_ROOT to your repository root (use Path(\".\").resolve() if '\n",
     "        'Jupyter was started at the repository root).'\n",
     "    )\n",
@@ -64,7 +64,7 @@
     "\n",
     "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
     "ASSET_ROOT = PROJECT_ROOT / 'model_dependencies'\n",
-    "SCRIPT = SOURCE_DIR / 'cell_annotation' / 'biomod_finetune.py'\n",
+    "SCRIPT = SOURCE_DIR / 'cell_annotation' / 'train.py'\n",
     "DATA_PATH = PROJECT_ROOT / 'data' / 'downstream' / 'final_data' / 'Zheng68K.h5ad'\n",
     "\n",
     "# Keep the BioMod model weights and their matching configuration together.\n",

BioMod/drug_combination/README.md

@@ -8,32 +8,48 @@ It is adapted from scLong's drug-combination implementation
 (DeepDDs), replacing the scfm/geneformer cell-line representation with
 BioMod.
 
+## Notebook
+
+See the [task notebook](tutorial.ipynb) for a step-by-step workflow.
+
+## Files
+
+| File or directory | Purpose |
+| --- | --- |
+| [README.md](README.md) | File guide, model and data descriptions, training commands, and evaluation details. |
+| [tutorial.ipynb](tutorial.ipynb) | Task configuration, execution, and result review in a Jupyter notebook. |
+| [train.py](train.py) | Train and validate drug-combination synergy prediction with drug graph encoders and BioMod cell embeddings. |
+
 ## Model Architecture
 
 | Component | Description |
 | --- | --- |
 | Drug branch | Three PyG GCNConv layers (78 -> 78 -> 156 -> 312), `global_max_pool`, and two fully connected layers to 128 dimensions |
-| Cell branch | Pretrained BioMod encoder, module mean pooling, and an MLP to 128 dimensions |
+| Cell branch | Pretrained BioMod encoder, configurable pooling, and an MLP to 128 dimensions |
 | Fusion layer | `concat(drug1, drug2, cell) = 384 -> MLP -> 2-class prediction` |
 
 The drug1 and drug2 branches share weights, matching scLong.
 
-## Data
+## Bundled dataset
 
-The bundled data are under `data/downstream/final_data/drug_combination_synergy/data_summary/`:
+The current entry point uses one collection with separate training and validation
+inputs under `data/downstream/final_data/drug_combination_synergy/data_summary/`.
 
-```text
-data_summary/
-|-- new_labels_0_10.csv          # Training labels (drug1, drug2, cell, label)
-|-- new_cell_features_954.csv    # Training cell features (30 cell lines x 954 genes)
-|-- smiles_0_10.csv              # Drug SMILES
-`-- test_data/
-    |-- independent_input.csv              # Validation labels (668 samples)
-    `-- independent_cell_features_954.csv  # Validation cell features (1020 cell lines x 954 genes)
-```
+| Input below `data_summary/` | Purpose |
+| --- | --- |
+| `new_labels_0_10.csv` | 12,415 training records with `drug1`, `drug2`, `cell`, and `label` columns. |
+| `new_cell_features_954.csv` | Training expression; the loader removes the transcript-ID row and obtains 31 cell rows by 954 genes. |
+| `test_data/independent_input.csv` | 668 validation records used for checkpoint selection. |
+| `test_data/independent_cell_features_954.csv` | Validation expression; the loader removes the transcript-ID row and obtains 1,019 cell rows by 954 genes. |
 
-The directory name `test_data/` is retained for compatibility. These files are
-used as the validation set for checkpoint selection.
+The directory name `test_data/` is retained for compatibility. The current script
+uses these files as validation inputs. Drug molecular graphs are constructed from
+the SMILES strings already stored in the label tables' `drug1` and `drug2` columns.
+`smiles_0_10.csv` and the other source tables are retained for reference and are not
+additional required inputs to this entry point.
+
+The older `data/downstream/final_data/drug_response_pytorch/drug_combination/`
+package is a reference copy, not another dataset for this task.
 
 ## Dependencies
 
@@ -44,66 +60,75 @@ used as the validation set for checkpoint selection.
 - scikit-learn
 - pandas, numpy
 
-## Usage
-
-Run all commands below from the repository root. Built-in resource defaults resolve from the script location. Explicit relative paths resolve from the working directory.
-
-```bash
-pip install RDKit -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
-
-### Basic Run (Domain Generalization Mode, Default)
-
-```bash
-python BioMod/drug_combination/train_drug_combination_biomod.py \
-    --checkpoint_path weight/BioMod.pth \
-    --hyper_params_path weight/BioMod.pkl \
-    --biomod_data_dir model_dependencies/biomod_data_all \
-    --data_dir data/downstream/final_data/drug_combination_synergy/data_summary
-```
+## Run the bundled training and validation collection
 
-### Custom Paths
+Activate the environment described in the [project README](../../README.md), then
+run these commands in Bash from the outer repository root containing `BioMod/`,
+`data/`, `model_dependencies/`, and `weight/`. The backslashes continue commands
+across lines in Bash; they are not PowerShell continuation characters. Use a
+Linux or WSL Bash shell with the required environment. Built-in resource defaults
+resolve from the script location; explicit relative paths resolve from the
+working directory.
 
 ```bash
-python BioMod/drug_combination/train_drug_combination_biomod.py \
+python BioMod/drug_combination/train.py \
+    --data_dir data/downstream/final_data/drug_combination_synergy/data_summary \
+    --label_file new_labels_0_10.csv \
+    --cell_file new_cell_features_954.csv \
+    --validation_label_file test_data/independent_input.csv \
+    --validation_cell_file test_data/independent_cell_features_954.csv \
     --checkpoint_path weight/BioMod.pth \
     --hyper_params_path weight/BioMod.pkl \
     --biomod_data_dir model_dependencies/biomod_data_all \
-    --gene_list_path model_dependencies/selected_genes_27k.txt\
-    --gene_mapping_path model_dependencies/gene2ens.csv \
-    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
-    --symbol2ens_path model_dependencies/gene2ens.csv \
-    --data_dir data/downstream/final_data/drug_combination_synergy/data_summary \
-    --output_dir tutorial_runs/drug_combination \
+    --n_folds 1 \
     --epochs 500 \
     --batch_size 256 \
-    --lr 1e-4 \
-    --n_folds 1 \
-    --pool_method max
+    --pool_method max \
+    --output_dir tutorial_runs/drug_combination_synergy
 ```
 
-Reference note: `1e-4 500 0.6824301861702127`.
+These are example settings for the bundled inputs, not a verified reproduction
+of manuscript results. The default domain-generalization mode trains on the full
+training collection and selects a checkpoint using the separate validation
+collection. `--n_folds 1` requests one run; leaving its default of 5 in this mode
+repeats full-training/full-validation runs rather than performing five-fold
+cross-validation.
 
-### Five-Fold Cross-Validation (Disable Domain Generalization)
+The manuscript describes the 668-sample collection as an independent test set.
+The current code uses it for checkpoint selection, so its reported AUROC is a
+validation result and must not be presented as an independent final test estimate.
 
-```bash
-python BioMod/drug_combination/train_drug_combination_biomod.py \
-    --checkpoint_path weight/BioMod.pth \
-    --hyper_params_path weight/BioMod.pkl \
-    --biomod_data_dir model_dependencies/biomod_data_all \
-    --no_domain_gen
-```
+### Optional five-fold cross-validation on the same collection
 
-### Force Reprocessing After Changing the Model or Data
+This is an alternative validation protocol, not a second dataset. The current
+implementation still reads all four input tables before selecting the held-out
+folds from the training collection.
 
 ```bash
-python BioMod/drug_combination/train_drug_combination_biomod.py \
+python BioMod/drug_combination/train.py \
+    --data_dir data/downstream/final_data/drug_combination_synergy/data_summary \
     --checkpoint_path weight/BioMod.pth \
     --hyper_params_path weight/BioMod.pkl \
     --biomod_data_dir model_dependencies/biomod_data_all \
-    --force_preprocess
+    --no_domain_gen \
+    --n_folds 5 \
+    --epochs 500 \
+    --batch_size 256 \
+    --pool_method max \
+    --output_dir tutorial_runs/drug_combination_synergy_cv
 ```
 
+### Input and cache changes
+
+Use `--gene_list_path`, `--gene_mapping_path`, `--gene2vec_path`, and
+`--symbol2ens_path` only when replacing the bundled shared assets. After changing
+model weights, pooling, or input tables, append `--force_preprocess` to regenerate
+the embedding cache and the PyG datasets under `data_summary/processed/`; an
+existing cache is not automatically keyed by all model and input settings.
+
+The input files and command arguments have been checked. This documentation update
+did not execute training or verify checkpoint performance.
+
 ## Main Arguments
 
 | Argument | Default | Description |

BioMod/drug_combination/{train_drug_combination_biomod.py → train.py}

@@ -10,7 +10,7 @@
 
 Usage:
     # Run from the repository root.
-    python BioMod/drug_combination/train_drug_combination_biomod.py \
+    python BioMod/drug_combination/train.py \
         --checkpoint_path <ckpt> --hyper_params_path <pkl> \
         --biomod_data_dir model_dependencies/biomod_data_all \
         --data_dir data/downstream/final_data/drug_combination_synergy/data_summary

BioMod/{toturials/drug_combination.ipynb → drug_combination/tutorial.ipynb}

@@ -7,7 +7,7 @@
     "# Drug combination synergy prediction with BioMod\n",
     "\n",
     "This notebook checks the bundled drug-combination inputs, prepares a command for\n",
-    "`BioMod/drug_combination/train_drug_combination_biomod.py`, and plots\n",
+    "`BioMod/drug_combination/train.py`, and plots\n",
     "available training outputs. It uses the BioMod model weights and configuration.\n",
     "\n",
     "`RUN_TRAINING = False` by default. Set it to `True` only when you are ready to train\n",
@@ -21,7 +21,7 @@
    "source": [
     "## Project paths and dataset layout\n",
     "\n",
-    "Start Jupyter from `BioMod/toturials/`. The path cell uses `Path('../..')` for the\n",
+    "Start Jupyter from `BioMod/drug_combination/`. The path cell uses `Path('../..')` for the\n",
     "repository root and resolves project-relative paths for the training subprocess.\n",
     "If Jupyter starts at the repository root, use `Path('.')` instead. The dataset is\n",
     "already included in the project:\n",
@@ -65,13 +65,13 @@
     "    def display(obj):\n",
     "        print(obj)\n",
     "\n",
-    "# Start Jupyter with BioMod/toturials as the working directory.\n",
+    "# Start Jupyter with BioMod/drug_combination as the working directory.\n",
     "# If Jupyter starts at the repository root, change this line to Path('.').resolve().\n",
     "PROJECT_ROOT = Path('../..').resolve()\n",
     "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
     "if not (SOURCE_DIR / 'models_biomod.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
     "    raise FileNotFoundError(\n",
-    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        'Start Jupyter from BioMod/drug_combination, or set PROJECT_ROOT to the repository root '\n",
     "        \"(use Path('.').resolve() if Jupyter was launched at that root).\"\n",
     "    )\n",
     "REPO_ROOT = PROJECT_ROOT\n",
@@ -86,7 +86,7 @@
     ")\n",
     "\n",
     "TASK_ROOT = SOURCE_DIR / 'drug_combination'\n",
-    "TRAIN_SCRIPT = TASK_ROOT / 'train_drug_combination_biomod.py'\n",
+    "TRAIN_SCRIPT = TASK_ROOT / 'train.py'\n",
     "CHECKPOINT_PATH = WEIGHT_DIR / 'BioMod.pth'\n",
     "HYPER_PARAMS_PATH = WEIGHT_DIR / 'BioMod.pkl'\n",
     "GENE_LIST_PATH = DEPENDENCY_ROOT / 'selected_genes_27k.txt'\n",

BioMod/drug_response/README.md

@@ -0,0 +1,70 @@
+# Drug Response Prediction with BioMod
+
+Predict drug response from drug molecular graphs and BioMod cell embeddings.
+The training script combines a graph-based drug encoder with the pretrained
+BioMod backbone.
+
+## Notebook
+
+See the [task notebook](tutorial.ipynb) for a step-by-step workflow.
+
+## Files
+
+| File or directory | Purpose |
+| --- | --- |
+| [README.md](README.md) | File guide and command-line training example. |
+| [tutorial.ipynb](tutorial.ipynb) | Task configuration, execution, and result review in a Jupyter notebook. |
+| [train.py](train.py) | Load drug-response data, build the model, train, and evaluate validation metrics. |
+
+## Bundled dataset
+
+The current entry point uses one joint GDSC-CCLE collection. GDSC supplies drug
+information and responses; CCLE supplies cell-line annotations and expression.
+They are complementary inputs to one run, rather than separate training datasets.
+Use the cleaned collection under
+`data/downstream/final_data/single_drug_response/data/`.
+
+| Input below the data directory | Purpose |
+| --- | --- |
+| `GDSC/1.Drug_listMon Jun 24 09_00_55 2019.csv` | Map drug IDs to PubChem IDs. |
+| `GDSC/drug_graph_feat/*.hkl` | Precomputed molecular graphs; 223 files are bundled. |
+| `CCLE/Cell_lines_annotations_20181226.txt` | Map cell-line IDs to cancer types. |
+| `CCLE/genomic_expression_561celllines_697genes_demap_features.csv` | Expression for 561 cell lines and 697 genes. |
+| `CCLE/GDSC_IC50.csv` | Drug-response matrix used to construct matched response records. |
+
+The older `data/downstream/final_data/drug_response_pytorch/` package is retained
+for reference. It is not an additional dataset for this task. Files under
+`single_drug_response/optional_reference/` are not consumed by this entry point.
+
+## Run the bundled GDSC-CCLE collection
+
+Activate the environment described in the [project README](../../README.md), then
+run these commands in Bash from the outer repository root containing `BioMod/`,
+`data/`, `model_dependencies/`, and `weight/`. The backslashes continue commands
+across lines in Bash; they are not PowerShell continuation characters. Use a
+Linux or WSL Bash shell with the required environment. Built-in resource defaults
+resolve from the script location; explicit relative paths resolve from the
+working directory.
+
+```bash
+python BioMod/drug_response/train.py \
+    --gpu_id 0 \
+    --data_path data/downstream/final_data/single_drug_response/data \
+    --checkpoint_path weight/BioMod.pth \
+    --hyper_params_path weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --epochs 300 \
+    --batch_size 32 \
+    --pool_method max \
+    --save_dir tutorial_runs/drug_response_gdsc_ccle
+```
+
+This is a command for the bundled inputs, with example training settings. It has
+not been validated as a reproduction of the manuscript results. The current code
+uses a fixed seed of 0 and splits matched records by cancer type into approximately
+95% training and 5% validation. Validation Pearson correlation determines the best
+checkpoint. It does not create a separate final test set, and it does not implement
+the manuscript's 70%/10%/20% train/validation/test protocol.
+
+The data and asset paths have been checked for existence. A complete training run
+and checkpoint performance have not been verified by this documentation update.

BioMod/drug_response/cmd.txt

@@ -1,10 +0,0 @@
-# Run from the repository root.
-python BioMod/drug_response/train_drug_biomod.py \
-    --gpu_id 0 \
-    --data_path data/downstream/final_data/single_drug_response/data \
-    --checkpoint_path weight/BioMod.pth \
-    --hyper_params_path weight/BioMod.pkl \
-    --epochs 300 \
-    --biomod_data_dir model_dependencies/biomod_data_all \
-    --save_dir tutorial_runs/drug_response \
-    --pool_method max

BioMod/drug_response/{train_drug_biomod.py → train.py}

@@ -10,8 +10,8 @@
 - Optimization strategy: AdamW + cosine annealing + early stopping
 
 How to use:
-    # Run from the repository root; see drug_response/cmd.txt for a full example.
-    python BioMod/drug_response/train_drug_biomod.py \
+    # Run from the repository root; see drug_response/README.md for a full example.
+    python BioMod/drug_response/train.py \
         --gpu_id 0 --epochs 300 \
         --checkpoint_path weight/BioMod.pth \
         --hyper_params_path weight/BioMod.pkl

BioMod/{toturials/drug_response_prediction.ipynb → drug_response/tutorial.ipynb}

@@ -7,7 +7,7 @@
     "# Drug response prediction with BioMod\n",
     "\n",
     "This notebook uses the BioMod model weights, drug graph features, and the bundled\n",
-    "GDSC/CCLE response data with `train_drug_biomod.py`.\n",
+    "GDSC/CCLE response data with `train.py`.\n",
     "\n",
     "By default, `RUN_TRAINING = False`: the notebook checks inputs and prepares the\n",
     "training command. Result and plotting cells run only when `predictions.npz`\n",
@@ -23,7 +23,7 @@
    "source": [
     "## Project paths\n",
     "\n",
-    "Start Jupyter from `BioMod/toturials/`. The next cell uses `Path('../..')` for the\n",
+    "Start Jupyter from `BioMod/drug_response/`. The next cell uses `Path('../..')` for the\n",
     "repository root and resolves project-relative paths for the training subprocess.\n",
     "If Jupyter starts at the repository root, use `Path('.')` instead. All inputs come\n",
     "from this project; no private server alias or external reference run is required.\n"
@@ -35,7 +35,7 @@
    "source": [
     "## Inputs and outputs\n",
     "\n",
-    "- Training script: `BioMod/drug_response/train_drug_biomod.py`\n",
+    "- Training script: `BioMod/drug_response/train.py`\n",
     "- Dataset: `data/downstream/final_data/single_drug_response/data/`\n",
     "- BioMod model weights and configuration: `weight/`\n",
     "- Shared model assets: `model_dependencies/`\n",
@@ -72,13 +72,13 @@
     "\n",
     "from scipy.stats import pearsonr\n",
     "\n",
-    "# Start Jupyter with BioMod/toturials as the working directory.\n",
+    "# Start Jupyter with BioMod/drug_response as the working directory.\n",
     "# If Jupyter starts at the repository root, change this line to Path('.').resolve().\n",
     "PROJECT_ROOT = Path('../..').resolve()\n",
     "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
     "if not (SOURCE_DIR / 'models_biomod.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
     "    raise FileNotFoundError(\n",
-    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        'Start Jupyter from BioMod/drug_response, or set PROJECT_ROOT to the repository root '\n",
     "        \"(use Path('.').resolve() if Jupyter was launched at that root).\"\n",
     "    )\n",
     "REPO_ROOT = PROJECT_ROOT\n",
@@ -93,7 +93,7 @@
     ")\n",
     "\n",
     "TASK_ROOT = SOURCE_DIR / 'drug_response'\n",
-    "SCRIPT = TASK_ROOT / 'train_drug_biomod.py'\n",
+    "SCRIPT = TASK_ROOT / 'train.py'\n",
     "DATA_ROOT = REPO_ROOT / 'data/downstream/final_data/single_drug_response/data'\n",
     "CKPT_PATH = WEIGHT_DIR / 'BioMod.pth'\n",
     "CONFIG_PKL = WEIGHT_DIR / 'BioMod.pkl'\n",

BioMod/gene_perturbation/README.md

@@ -5,6 +5,27 @@ the `BioMod` encoder. The
 [BioMod model weights and matching configuration](../../README.md#biomod-model-weights)
 are shared with the other downstream tasks.
 
+## Notebook
+
+See the [task notebook](tutorial.ipynb) for a step-by-step workflow.
+
+## Files
+
+| File or directory | Purpose |
+| --- | --- |
+| [README.md](README.md) | File guide, model details, training commands, and troubleshooting notes. |
+| [tutorial.ipynb](tutorial.ipynb) | Task configuration, execution, and result review in a Jupyter notebook. |
+| [train.py](train.py) | Configure and launch distributed gene perturbation training. |
+| [components/](components/) | Model, data-processing, training, and evaluation helpers. |
+| [components/model.py](components/model.py) | Define the BioMod-backed GEARS model and recover gene-level features. |
+| [components/trainer.py](components/trainer.py) | Manage model training, checkpointing, and evaluation. |
+| [components/dataset.py](components/dataset.py) | Prepare perturbation datasets, splits, and graph data loaders. |
+| [components/data_utils.py](components/data_utils.py) | Compute differential-expression annotations and construct data splits. |
+| [components/inference.py](components/inference.py) | Run inference and compute perturbation prediction metrics. |
+| [components/training_utils.py](components/training_utils.py) | Provide distributed training, checkpoint, logging, and optimization helpers. |
+| [components/utils.py](components/utils.py) | Provide perturbation parsing, gene-network, loss, and prediction utilities. |
+| [components/__init__.py](components/__init__.py) | Mark the helper directory as a Python package. |
+
 ## Overview
 
 This adaptation replaces the original GEARS gene perturbation model based on
@@ -17,7 +38,7 @@ are:
 
 ## Main Adaptations
 
-### 1. `SCFM_BioMod` (`model_biomod.py`)
+### 1. `SCFM_BioMod` (`components/model.py`)
 
 `SCFM_BioMod` wraps `BioMod` and provides these functions:
 
@@ -42,7 +63,7 @@ for gene_idx in range(len(self.scfm_index_positions) - 1):
         gene_emb = module_encodings.mean(dim=1)  # B, D (global average)
 ```
 
-### 2. `GEARS_Model_BioMod` (`model_biomod.py`)
+### 2. `GEARS_Model_BioMod` (`components/model.py`)
 
 `GEARS_Model_BioMod` inherits the original GEARS architecture. Main changes:
 
@@ -50,7 +71,7 @@ for gene_idx in range(len(self.scfm_index_positions) - 1):
 - **Module data loading** supports loading module data from hyper-parameters or file paths.
 - **Other components remain unchanged**, including the GNN layers and decoder.
 
-### 3. Training Script (`train_biomod.py`)
+### 3. Training Script (`train.py`)
 
 The training script is based on `GEARS/train_dual_ddp_reduce_dec.py`. Main changes:
 
@@ -62,38 +83,80 @@ The training script is based on `GEARS/train_dual_ddp_reduce_dec.py`. Main chang
 
 ## Usage
 
-### 1. Prepare Data
+### 1. Bundled dataset: Norman
+
+Norman is the only perturbation dataset currently bundled with this repository.
+The following files are present under `data/downstream/final_data/norman/`:
+
+| Input | Purpose |
+| --- | --- |
+| `perturb_processed.h5ad` | Processed expression and perturbation annotations. |
+| `data_pyg/cell_graphs.pkl` | Prepared per-cell PyG graph cache. |
+| `splits/norman_simulation_1_0.75.pkl` | Existing simulation split for seed 1 and training-gene fraction 0.75. |
+| `splits/norman_simulation_1_0.75_subgroup.pkl` | Test-condition categories for that simulation split. |
+| `go.csv` | Gene Ontology similarity graph. |
+| `simulation_1_0.75_0.4_20_co_expression_network.csv` | Coexpression graph for the same split and default graph settings. |
 
-Make sure the following files exist:
+These files were checked for existence without deserializing the graph cache,
+split files, or model weights. The shared BioMod weights, configuration, module
+JSON, gene list, gene2vec array, and gene-mapping CSV are also present at the paths
+used below.
 
-- Perturbation data, such as Norman, Adamson, or Dixit.
-- BioMod model weights and their matching configuration file.
-- Module data in JSON format.
-- Gene mapping file, such as `gene2ens.csv`.
+The loader also contains support for other public perturbation datasets, including
+Adamson and Dixit, but their data are not bundled. They require separate data
+preparation or downloading and are not additional ready-to-run local collections.
 
-### 2. Run Training
+### 2. Run the bundled Norman simulation split
 
-Run from the repository root in a Linux CUDA/NCCL environment. Built-in asset defaults are derived from the script location; explicit relative paths resolve from the working directory.
+Activate the environment described in the [project README](../../README.md), then
+run this command in Bash from the outer repository root containing `BioMod/`,
+`data/`, `model_dependencies/`, and `weight/`. A Linux CUDA/NCCL environment is
+required, including when using Linux through WSL. The backslashes continue the
+command across lines in Bash; they are not PowerShell continuation characters.
+One GPU still requires the distributed launcher. Built-in asset defaults resolve
+from the script location; explicit relative paths resolve from the working
+directory.
 
 ```bash
-# DDP training on multiple GPUs.
-python -m torch.distributed.run --standalone --nproc_per_node=4 \
-    BioMod/gene_perturbation/train_biomod.py \
+python -m torch.distributed.run --standalone --nproc_per_node=1 \
+    BioMod/gene_perturbation/train.py \
     --data_dir data/downstream/final_data \
     --data_name norman \
+    --split simulation \
+    --seed 1 \
+    --train_gene_set_size 0.75 \
     --scfm_hyper_params_path weight/BioMod.pkl \
     --scfm_ckpt_path weight/BioMod.pth \
     --biomod_data_dir model_dependencies/biomod_data_all \
     --gene_mapping_path model_dependencies/gene2ens.csv \
     --scfm_genes_list_path model_dependencies/selected_genes_27k.txt \
     --scfm_gene2vec_file model_dependencies/selected_gene2vec_27k.npy \
-    --result_dir tutorial_runs/gene_perturbation \
-    --run_name norman_module_experiment \
+    --finetune_method frozen \
+    --result_dir tutorial_runs/gene_perturbation_norman \
+    --run_name norman_biomod \
     --epochs 20 \
     --batch_size 32 \
     --lr 1e-3
 ```
 
+The split arguments select the bundled split and graph caches. The explicit
+`--finetune_method frozen` setting selects the frozen pretrained branch described
+in the manuscript; omitting it uses the script's current full-fine-tuning default.
+To use several GPUs, change `--nproc_per_node` to the number of available GPUs.
+
+This is a single bundled-data run with example training settings, not a verified
+reproduction of the manuscript's repeated-seed results. The training code selects
+the checkpoint with the lowest validation MSE on the top 20 differentially
+expressed genes and then evaluates that checkpoint on the test split. The stored
+split contents, reported cell counts, and model performance were not revalidated
+by this documentation update.
+
+Norman already includes `go.csv`, so no external GO annotation file is required
+for this command. If regenerating a missing GO graph for prepared data, supply a
+compatible `gene2go_all.pkl` through `BIOMOD_GENE2GO_PATH`; that optional annotation
+resource is not bundled. Different seeds or split settings require new split and
+coexpression caches and do not reuse the files listed above.
+
 ### 3. Arguments
 
 **Model arguments** (defaults point to the bundled BioMod assets):
@@ -133,19 +196,11 @@ python -m torch.distributed.run --standalone --nproc_per_node=4 \
 
 To compare model performance:
 
-1. **Run original sclong**:
+1. **Run original scLong** using its own documentation and a separate checkout
+   of the upstream GEARS code. That external workflow is not bundled here.
 
-   ```bash
-   # Requires a separate checkout of the original scLong/GEARS code.
-   python /path/to/upstream/GEARS/train_dual_ddp_reduce_dec.py [arguments...]
-   ```
-
-2. **Run the module version**:
-
-   ```bash
-   # Run from the BioMod repository root.
-   python -m torch.distributed.run --standalone --nproc_per_node=1 BioMod/gene_perturbation/train_biomod.py [arguments...]
-   ```
+2. **Run BioMod** using the complete
+   [Norman simulation command](#2-run-the-bundled-norman-simulation-split) above.
 
 3. **Compare metrics**:
 
@@ -189,16 +244,19 @@ This strategy:
 ## File Structure
 
 ```text
-BioMod/gene_perturbation/                                # Perturbation response prediction: processing, training and inference
-├── data_utils_local.py
-├── gears_reduce_local.py
-├── inference_local.py
-├── model_biomod.py
-├── pertdata_local.py
+BioMod/gene_perturbation/                   # Perturbation response prediction
+├── components/                            # Model, data, training, and evaluation helpers
+│   ├── __init__.py
+│   ├── data_utils.py
+│   ├── dataset.py
+│   ├── inference.py
+│   ├── model.py
+│   ├── trainer.py
+│   ├── training_utils.py
+│   └── utils.py
 ├── README.md
-├── scfm_utils_local.py
-├── train_biomod.py
-└── utils.py
+├── train.py
+└── tutorial.ipynb
 ```
 
 ## Notes
@@ -232,10 +290,9 @@ This implementation supports two feature extraction modes controlled by
 
 ### Method 1: Gene-Based (Default, Recommended)
 
-```bash
-# Default mode; do not pass --use_biomod_level or set it to False.
-python -m torch.distributed.run --standalone --nproc_per_node=1 BioMod/gene_perturbation/train_biomod.py [other arguments...]
-```
+Use the complete [Norman simulation command](#2-run-the-bundled-norman-simulation-split)
+above. It uses gene-level recovery by omitting `--use_biomod_level`; this is a
+boolean switch, so do not append a value such as `False`.
 
 **Characteristics**:
 
@@ -246,10 +303,10 @@ python -m torch.distributed.run --standalone --nproc_per_node=1 BioMod/gene_pert
 
 ### Method 2: Module-Based (Experimental, Not Fully Implemented)
 
-```bash
-# Enable module-level mode.
-python -m torch.distributed.run --standalone --nproc_per_node=1 BioMod/gene_perturbation/train_biomod.py --use_biomod_level [other arguments...]
-```
+The `--use_biomod_level` switch selects an experimental mode whose downstream
+architecture is incomplete. It is not included in the runnable
+[Norman simulation command](#2-run-the-bundled-norman-simulation-split). Use the
+default gene-level workflow for the bundled dataset.
 
 **Characteristics**:
 

BioMod/gene_perturbation/components/__init__.py

@@ -0,0 +1 @@
+"""Implementation components for gene perturbation prediction."""

BioMod/gene_perturbation/{data_utils_local.py → components/data_utils.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Data processing utility functions (data_utils_local.py)
+Data processing utility functions (data_utils.py)
 ================================================================================
 
 [Module function]
@@ -28,7 +28,7 @@
 import warnings
 warnings.filterwarnings("ignore")
 
-from utils import parse_single_pert, parse_combo_pert, parse_any_pert, print_sys
+from .utils import parse_single_pert, parse_combo_pert, parse_any_pert, print_sys
 
 def rank_genes_groups_by_cov(
     adata,

BioMod/gene_perturbation/{pertdata_local.py → components/dataset.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Perturbation data processing module (pertdata_local.py)
+Perturbation data processing module (dataset.py)
 ================================================================================
 
 [Module function]
@@ -45,8 +45,8 @@
 # ============================================================
 # Use local modules and avoid external gears_reduce imports
 # ============================================================
-from utils import print_sys, zip_data_download_wrapper, filter_pert_cond_in_go
-from data_utils_local import get_DE_genes, get_dropout_non_zero_genes, DataSplitter
+from .utils import print_sys, zip_data_download_wrapper, filter_pert_cond_in_go
+from .data_utils import get_DE_genes, get_dropout_non_zero_genes, DataSplitter
 
 class PertData:
 

BioMod/gene_perturbation/{inference_local.py → components/inference.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Model evaluation module (inference_local.py)
+Model evaluation module (inference.py)
 ================================================================================
 
 [Module function]

BioMod/gene_perturbation/{model_biomod.py → components/model.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Genetic perturbation task model definition (model_biomod.py)
+Genetic perturbation task model definition (model.py)
 ================================================================================
 
 [Model Architecture Overview]
@@ -55,7 +55,7 @@
         sanitize_biomod_config,
     )
 except ModuleNotFoundError:
-    project_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+    project_root = str(Path(__file__).resolve().parents[3])
     if project_root not in sys.path:
         sys.path.insert(0, project_root)
     from BioMod.models_biomod import (
@@ -485,7 +485,7 @@ def __init__(self, args):
                 module_data_path=args.get('module_data_path'),
                 module_data_dir=args.get('module_data_dir'),
                 gene_list_path=args.get('scfm_genes_list_path'),
-                gene_mapping_path=args.get('gene_mapping_path', str(Path(__file__).resolve().parents[2] / 'model_dependencies' / 'gene2ens.csv')),
+                gene_mapping_path=args.get('gene_mapping_path', str(Path(__file__).resolve().parents[3] / 'model_dependencies' / 'gene2ens.csv')),
                 verbose=True
             )
         else:

BioMod/gene_perturbation/{gears_reduce_local.py → components/trainer.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-GEARS training and evaluation controller (gears_reduce_local.py)
+GEARS training and evaluation controller (trainer.py)
 ================================================================================
 
 [Module function]
@@ -46,14 +46,14 @@
 # ============================================================
 # Use native modules (optimized versions)
 # ============================================================
-from model_biomod import GEARS_Model_BioMod
-from inference_local import evaluate, compute_metrics, deeper_analysis, \
+from .model import GEARS_Model_BioMod
+from .inference import evaluate, compute_metrics, deeper_analysis, \
                   non_dropout_analysis, compute_synergy_loss
-from utils import loss_fct, uncertainty_loss_fct, parse_any_pert, \
+from .utils import loss_fct, uncertainty_loss_fct, parse_any_pert, \
                   get_similarity_network, print_sys, GeneSimNetwork, \
                   create_cell_graph_dataset_for_prediction, get_mean_control, \
                   get_GI_genes_idx, get_GI_params
-from scfm_utils_local import *
+from .training_utils import *
 
 torch.manual_seed(0)
 

BioMod/gene_perturbation/{scfm_utils_local.py → components/training_utils.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Single cell basic model tool module (scfm_utils_local.py)
+Single cell basic model tool module (training_utils.py)
 ================================================================================
 
 [Module function]

BioMod/gene_perturbation/{train_biomod.py → train.py}

@@ -1,6 +1,6 @@
 """
 ================================================================================
-Genetic perturbation task training script (train_biomod.py)
+Genetic perturbation task training script (train.py)
 ================================================================================
 
 [Task Overview]
@@ -49,23 +49,10 @@
 DOWNSTREAM_DATA_DIR = REPO_ROOT / "data" / "downstream" / "final_data"
 WEIGHTS_DIR = REPO_ROOT / "weight"
 
-# ============================================================
-# [Key] Correctly set the order of sys.path
-# Local modules must take precedence and avoid importing into external utils.py
-# This is a common Python module import trap: if a module with the same name exists in the project root directory,
-# can be imported incorrectly, causing problems that are difficult to debug
-# ============================================================
-current_dir = os.path.dirname(os.path.abspath(__file__))
-project_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-# First add the project root directory (for pytorch_cont)
+# Resolve package imports from the repository root, independently of the working directory.
+project_root = str(REPO_ROOT)
 if project_root not in sys.path:
-    sys.path.append(project_root)
-
-# Finally add the current directory (make sure local modules take precedence)
-# Use append instead of insert(0), but since the current directory is empty there will be no conflict
-if current_dir not in sys.path:
-    sys.path.insert(0, current_dir)
+    sys.path.insert(0, project_root)
 
 # ============================================================
 # [Key] Create an independent Numba cache directory for each process
@@ -82,12 +69,12 @@
 # [Module Import] Use completely independent local modules
 # These local modules are to avoid conflicts with modules of the same name in other parts of the project
 # ============================================================
-from gears_reduce_local import GEARS
-from utils import print_sys
-from pertdata_local import PertData  # Local version, does not rely on external gears_reduce
+from BioMod.gene_perturbation.components.trainer import GEARS
+from BioMod.gene_perturbation.components.utils import print_sys
+from BioMod.gene_perturbation.components.dataset import PertData  # Local version, does not rely on external gears_reduce
 
 # Import module model
-from model_biomod import GEARS_Model_BioMod
+from BioMod.gene_perturbation.components.model import GEARS_Model_BioMod
 
 def main(parser):
     """

BioMod/{toturials/gene_perturbation.ipynb → gene_perturbation/tutorial.ipynb}

@@ -15,7 +15,7 @@
    "source": [
     "## Project paths\n",
     "\n",
-    "Start Jupyter from `BioMod/toturials/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
+    "Start Jupyter from `BioMod/gene_perturbation/`. The setup cell uses `Path(\"../..\").resolve()` to select the repository root, following the project-relative path convention used in scGPT tutorials. All code, data, model weight, and output paths are derived from this root and resolved to absolute paths for the training command. If Jupyter starts at the repository root, change that line to `Path(\".\").resolve()`. For another launch location, set `PROJECT_ROOT` to the checkout root."
    ]
   },
   {
@@ -24,7 +24,7 @@
    "source": [
     "## Inputs and workflow\n",
     "\n",
-    "- Entry point: `BioMod/gene_perturbation/train_biomod.py`.\n",
+    "- Entry point: `BioMod/gene_perturbation/train.py`.\n",
     "- Dataset parent: `data/downstream/final_data/`; the loader appends `norman/`.\n",
     "- Processed expression: `norman/perturb_processed.h5ad` under that parent.\n",
     "- Split: `simulation`, seed `1`, training-gene fraction `0.75`.\n",
@@ -58,7 +58,7 @@
     "PROJECT_ROOT = Path('../..').resolve()\n",
     "if not (PROJECT_ROOT / 'BioMod' / 'models_biomod.py').is_file() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
     "    raise FileNotFoundError(\n",
-    "        'Project root not found. Start Jupyter from BioMod/toturials, or set '\n",
+    "        'Project root not found. Start Jupyter from BioMod/gene_perturbation, or set '\n",
     "        'PROJECT_ROOT to your repository root (use Path(\".\").resolve() if '\n",
     "        'Jupyter was started at the repository root).'\n",
     "    )\n",
@@ -67,7 +67,7 @@
     "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
     "ASSET_ROOT = PROJECT_ROOT / 'model_dependencies'\n",
     "TASK_ROOT = SOURCE_DIR / 'gene_perturbation'\n",
-    "TRAIN_SCRIPT = TASK_ROOT / 'train_biomod.py'\n",
+    "TRAIN_SCRIPT = TASK_ROOT / 'train.py'\n",
     "DATA_DIR = PROJECT_ROOT / 'data' / 'downstream' / 'final_data'\n",
     "DATA_NAME = 'norman'\n",
     "SPLIT = 'simulation'\n",
@@ -169,7 +169,7 @@
    "source": [
     "## Training command\n",
     "\n",
-    "`train_biomod.py` initializes distributed training even for one GPU. The command therefore uses `python -m torch.distributed.run --standalone --nproc_per_node=1`. Review the settings below, then set `RUN_TRAINING = True` in the setup cell to launch. The training script selects a checkpoint using validation MSE on the top 20 differentially expressed genes, then evaluates the selected checkpoint on the test split. Batch sizes, epoch count, and learning rate are configurable examples, not a verified benchmark recipe."
+    "`train.py` initializes distributed training even for one GPU. The command therefore uses `python -m torch.distributed.run --standalone --nproc_per_node=1`. Review the settings below, then set `RUN_TRAINING = True` in the setup cell to launch. The training script selects a checkpoint using validation MSE on the top 20 differentially expressed genes, then evaluates the selected checkpoint on the test split. Batch sizes, epoch count, and learning rate are configurable examples, not a verified benchmark recipe."
    ]
   },
   {

BioMod/zero_shot_batch/README.md

@@ -4,156 +4,184 @@ This directory contains code for extracting BioMod cell embeddings and
 evaluating batch effects without fine-tuning. Use the
 [BioMod model weights and matching configuration](../../README.md#biomod-model-weights).
 
-## Features
-
-- **Task objective**: evaluate batch effects in cell embeddings extracted with the BioMod model weights.
-- **Evaluation metrics**: compute ASW_label/batch metrics with the scib package.
-- **Data format**: supports single-cell data in h5ad format.
-
-## Usage
-
-Run these commands from the repository root. Built-in resource defaults resolve from the script location; explicit relative paths resolve from the working directory.
-
-### Basic Usage
+## Notebook
+
+See the [task notebook](tutorial.ipynb) for a step-by-step workflow.
+
+## Files
+
+| File or directory | Purpose |
+| --- | --- |
+| [README.md](README.md) | File guide, evaluation commands, arguments, and output descriptions. |
+| [tutorial.ipynb](tutorial.ipynb) | Task configuration, execution, and result review in a Jupyter notebook. |
+| [evaluate.py](evaluate.py) | Extract BioMod cell embeddings and evaluate cell-type and batch-effect metrics. |
+| [tools/](tools/) | Optional visualization and sequencing-depth analysis helpers. |
+| [tools/visualize_results.py](tools/visualize_results.py) | Plot saved embeddings and compare batch ASW metrics. |
+| [tools/sequencing_depth/](tools/sequencing_depth/) | Generate reduced-depth inputs and compare zero-shot evaluation results. |
+| [tools/sequencing_depth/make_low_depth_h5ad.py](tools/sequencing_depth/make_low_depth_h5ad.py) | Create reduced-depth h5ad datasets by thinning raw integer counts. |
+| [tools/sequencing_depth/run_zero_shot_low_depth.py](tools/sequencing_depth/run_zero_shot_low_depth.py) | Run evaluation over multiple input datasets and summarize the results. |
+| [tools/sequencing_depth/README.md](tools/sequencing_depth/README.md) | Commands for the sequencing-depth evaluation workflow. |
+
+## Dataset Commands
+
+The manuscript evaluates three atlases under four conditions: hPNC, BMMC, and
+Kim Lung with and without malignant cells. Each command below evaluates one
+complete bundled file with `--n_cells all`, max pooling and the scIB metric
+bundle selected by `--metrics_mode scgpt`. These choices follow manuscript
+Section 4.6 (PDF page 25). No additional gene or cell filtering is enabled.
+
+Run from the outer repository root after installing the environment. Each
+condition has a separate output directory. A batch size of 1 limits the per-step
+inference memory requirement; all cells are still processed. This is a launch
+setting, not a batch size reported in the manuscript. The complete expression
+matrix and metric calculations can still require substantial host memory.
+
+The command blocks use Bash line continuations (`\`). Run them in Bash, such as
+Linux, WSL or Git Bash. In PowerShell, join each command into a single line
+instead of copying the Bash continuation characters.
+
+The encoder receives only expression values, not batch or cell-type labels.
+The batch and label fields below are used for evaluation after embedding
+extraction. The entry point reads `adata.X` without additional normalization or
+`log1p`; it does not select counts layers or `.raw`. The commands establish
+file/field wiring, not preprocessing correctness or reproduced performance.
+
+| Condition | Cells embedded | Genes in file | Label field | Batch field | Observed batches |
+| --- | ---: | ---: | --- | --- | ---: |
+| hPNC | 16,382 | 19,093 | `celltype` | `tech` | 9 |
+| BMMC | 90,261 | 14,087 | `cell_type` | `batch` | 12 |
+| Kim Lung, malignant cells retained | 32,493 | 20,793 | `cell_type` | `sample` | 14 |
+| Kim Lung, malignant cells removed | 21,955 | 20,793 | `cell_type` | `sample` | 14 |
+
+### 1. hPNC
+
+Use the 16,382-cell pancreas atlas, not the paired `hp14/` files used for
+annotation. Its `tech` field contains nine observed values: `celseq`, `celseq2`,
+`fluidigmc1`, `inDrop1`–`inDrop4`, `smarter` and `smartseq2`. The manuscript states
+six batches. No six-batch harmonization is defined in the current entry point;
+this command evaluates the nine values stored in the file and must not be
+presented as a verified six-batch reproduction.
 
 ```bash
-python BioMod/zero_shot_batch/biomod_zero_shot.py \
-    --scfm_hyper_params_path weight/BioMod.pkl \
-    --scfm_ckpt_path weight/BioMod.pth \
-    --biomod_data_path model_dependencies/biomod_data_all/merged_biomods.json \
-    --data_path data/downstream/final_data/Kim2020_Lung.h5ad \
-    --dataset_preset lung
+python BioMod/zero_shot_batch/evaluate.py \
+    --data_path data/downstream/final_data/human_pancreas_norm_complexBatch_with_gene_name.h5ad \
+    --dataset_preset auto --gene_col gene_name --batch_key tech --label_key celltype \
+    --scfm_ckpt_path weight/BioMod.pth --scfm_hyper_params_path weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
+    --gene_mapping_path model_dependencies/gene2ens.csv \
+    --scfm_genes_file model_dependencies/selected_genes_27k.txt \
+    --n_cells all --batch_size 1 --pool_method max --metrics_mode scgpt \
+    --min_genes_per_cell 0 --min_cells_per_gene 0 \
+    --output_dir tutorial_runs/zero_shot_hPNC
 ```
 
-### Full Argument Example
+### 2. BMMC
+
+Use the stored `gene_id` column, which decodes to Ensembl identifiers, and the
+45 observed `cell_type` classes. The gene column has 98 missing entries and 36
+duplicate nonmissing identifiers. The current mapping leaves missing IDs
+unmapped and uses the last input column for duplicate IDs; it does not aggregate
+duplicates. These existing input semantics have not been changed.
 
 ```bash
-python BioMod/zero_shot_batch/biomod_zero_shot.py \
-    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
-    --scfm_hyper_params_path weight/BioMod.pkl \
-    --scfm_ckpt_path weight/BioMod.pth \
+python BioMod/zero_shot_batch/evaluate.py \
+    --data_path data/downstream/final_data/BMMC_processed.h5ad \
+    --dataset_preset auto --gene_col gene_id --batch_key batch --label_key cell_type \
+    --scfm_ckpt_path weight/BioMod.pth --scfm_hyper_params_path weight/BioMod.pkl \
     --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
     --gene_mapping_path model_dependencies/gene2ens.csv \
     --scfm_genes_file model_dependencies/selected_genes_27k.txt \
-    --data_path data/downstream/final_data/Kim2020_Lung.h5ad \
-    --dataset_preset lung \
-    --output_key X_sclong_module \
-    --n_cells all \
-    --batch_size 256 \
-    --output_dir tutorial_runs/zero_shot_first_module \
-    --metrics_mode simple \
-    --pool_method first_module
+    --n_cells all --batch_size 1 --pool_method max --metrics_mode scgpt \
+    --min_genes_per_cell 0 --min_cells_per_gene 0 \
+    --output_dir tutorial_runs/zero_shot_bmmc
 ```
 
+### 3. Kim Lung: Malignant Cells Retained
+
+All 32,493 cells receive embeddings. The file has 10 observed cell-type labels
+and 2,021 unlabeled cells. The evaluator excludes cells with missing labels from
+metrics, so 30,472 cells remain for labeled evaluation. This metadata exclusion
+does not remove rows from the input file or the saved embedding array.
+
 ```bash
-python BioMod/zero_shot_batch/biomod_zero_shot.py \
-    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
-    --scfm_hyper_params_path weight/BioMod.pkl \
-    --scfm_ckpt_path weight/BioMod.pth \
+python BioMod/zero_shot_batch/evaluate.py \
+    --data_path data/downstream/final_data/Kim2020_Lung.h5ad \
+    --dataset_preset auto --gene_col gene_name --batch_key sample --label_key cell_type \
+    --scfm_ckpt_path weight/BioMod.pth --scfm_hyper_params_path weight/BioMod.pkl \
     --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
     --gene_mapping_path model_dependencies/gene2ens.csv \
     --scfm_genes_file model_dependencies/selected_genes_27k.txt \
-    --dataset_preset lung \
-    --data_path data/downstream/final_data/Kim2020_Lung.h5ad \
-    --output_key X_sclong_module \
-    --batch_size 128 \
-    --metrics_mode scgpt \
-    --output_dir tutorial_runs/zero_shot_max \
-    --pool_method max
+    --n_cells all --batch_size 1 --pool_method max --metrics_mode scgpt \
+    --min_genes_per_cell 0 --min_cells_per_gene 0 \
+    --output_dir tutorial_runs/zero_shot_kim_lung
 ```
 
-### Arguments
-
-| Argument | Description | Default |
-| --- | --- | --- |
-| `--scfm_hyper_params_path` | Path to the matching BioMod model configuration; required | - |
-| `--scfm_ckpt_path` | Path to the BioMod model weights; required | - |
-| `--biomod_data_path` | Path to one module JSON file | None |
-| `--biomod_data_dir` | Directory containing multiple module JSON files | None |
-| `--gene2vec_path` | Path to the Gene2Vec file | model_dependencies/selected_gene2vec_27k.npy |
-| `--gene_mapping_path` | CSV mapping gene symbols to Ensembl IDs | model_dependencies/gene2ens.csv |
-| `--scfm_genes_file` | BioMod gene list file | model_dependencies/selected_genes_27k.txt |
-| `--data_path` | Input h5ad data file | Required |
-| `--output_key` | Key name for embeddings in `adata.obsm` | X_sclong_module |
-| `--n_cells` | Number of cells to process; `all` means all cells | all |
-| `--batch_size` | Inference batch size | 4 |
-| `--output_dir` | Output directory | `BioMod/zero_shot_batch/res/` under the repository |
-
-## Outputs
+### 4. Kim Lung: Malignant Cells Removed
 
-After the run finishes, the following files are generated under `--output_dir`:
+Use the existing 21,955-cell file with malignant cells already excluded. It
+retains nine observed cell-type labels and the same 2,021 unlabeled cells, giving
+19,934 cells for labeled evaluation. The command does not modify or regenerate
+either lung file.
 
-1. **`sclong_module_batch_cell_emb_mode.csv`**: evaluation metrics in CSV format.
-   - Contains three columns: metric, label, and value.
-   - Main metric: ASW_label/batch. Larger values are better and indicate stronger batch-correction performance.
-
-2. **`sclong_module_batch_cell_emb_mode_ce.npy`**: NumPy array of cell embeddings.
-   - Shape: `(n_cells, embed_dim)`.
-   - Can be used for downstream analysis or visualization.
-
-## Workflow
-
-1. **Data loading**: load single-cell data in h5ad format.
-2. **Gene mapping**: convert input gene symbols to Ensembl IDs and map them to the BioMod gene list.
-3. **Model inference**:
-   - Reindex gene expression values to the BioMod gene order.
-   - Obtain encodings with the BioMod model.
-   - Extract cell embeddings using the selected `--pool_method`.
-4. **Evaluation**: compute batch-correction metrics with scib.
-
-## Cell Embedding Extraction
-
-The model output must contain `module_encodings`. Pooling is selected explicitly:
-
-- `auto`: use `cls_encoding` when present, otherwise mean-pool module encodings.
-- `cls`: use `cls_encoding`; if absent, log a warning and use mean pooling.
-- `mean` or `max`: pool module encodings along the module dimension.
-- `first_module`: restore the original module order with `ids_restore` and select the first module.
-
-The current script does not fall back to decoder outputs when module encodings are absent.
-
-## Comparison with Original scLong
-
-| Feature | scLong | BioMod |
-| --- | --- | --- |
-| Model architecture | DualEncoder (mini + large) | Single Encoder |
-| Input format | Flat gene expression | Module-based |
-| Embedding extraction | `merged_decodings[:, 0, :]` | `module_encodings.mean(dim=1)` |
-| Gene handling | Directly uses 27k genes | Aggregates through modules |
-
-## Notes
-
-1. **Module data**: make sure the correct module data path is provided with `--biomod_data_path` or `--biomod_data_dir`.
-2. **Gene mapping**: make sure the mapping file (`gene2ens.csv`) contains the required gene-symbol to Ensembl-ID mappings.
-3. **Memory usage**: for large datasets, set `--n_cells` to limit the number of processed cells.
-4. **Batch size**: adjust `--batch_size` according to GPU memory.
-
-## Troubleshooting
-
-### Issue 1: Module Data Not Found
-
-**Error**: `KeyError: 'module_sets'`
+```bash
+python BioMod/zero_shot_batch/evaluate.py \
+    --data_path data/downstream/final_data/Kim2020_Lung_noMalignant.h5ad \
+    --dataset_preset auto --gene_col gene_name --batch_key sample --label_key cell_type \
+    --scfm_ckpt_path weight/BioMod.pth --scfm_hyper_params_path weight/BioMod.pkl \
+    --biomod_data_dir model_dependencies/biomod_data_all \
+    --gene2vec_path model_dependencies/selected_gene2vec_27k.npy \
+    --gene_mapping_path model_dependencies/gene2ens.csv \
+    --scfm_genes_file model_dependencies/selected_genes_27k.txt \
+    --n_cells all --batch_size 1 --pool_method max --metrics_mode scgpt \
+    --min_genes_per_cell 0 --min_cells_per_gene 0 \
+    --output_dir tutorial_runs/zero_shot_kim_lung_no_malignant
+```
 
-**Fix**: make sure `--biomod_data_path` or `--biomod_data_dir` is provided.
+## Outputs and Metric Definitions
 
-### Issue 2: Gene Mapping Failed
+Each output directory contains:
 
-**Error**: `KeyError: 'gene_symbol'`
+- `sclong_module_batch_cell_emb_mode.csv`: metrics with `metric`, `label` and
+  `value` columns.
+- `sclong_module_batch_cell_emb_mode_ce.npy`: one embedding row per processed
+  input cell, before the metric-only removal of missing batch/label values.
 
-**Fix**: check the format of `--gene_mapping_path` and make sure it contains the `Ensembl gene ID` column.
+With `--metrics_mode scgpt`, the script reports individual scIB metrics plus:
 
-### Issue 3: Model Output Key Missing
+- `avg_bio = (NMI_cluster/label + ARI_cluster/label + ASW_label) / 3`.
+- `avg_batch = (graph_conn + ASW_label/batch) / 2`.
 
-**Error**: `KeyError: 'module_encodings'`
+The manuscript defines `avg_score = 0.6 * avg_bio + 0.4 * avg_batch`, but the
+current entry point does not write an `avg_score` row. Compute that aggregate
+from the two reported values when needed; do not mistake `ASW_label/batch`
+alone for the overall score. `--metrics_mode simple` computes only the batch
+silhouette metric and is not the manuscript's complete metric bundle.
 
-**Fix**: use the BioMod model weights and their matching configuration, and
-check that the model provides the required output.
+The manuscript reports five runs with seeds 0–4. This entry point has no general
+`--seed` argument or five-run driver. The existing `--shuffle_seed` controls an
+optional gene-to-module reshuffling experiment; it is not an evaluation seed
+and must not be used as a substitute. Reproducing the reported run-to-run
+variation requires the original randomization protocol.
 
-### Issue 4: Out of Memory
+## Gene Alignment and Optional Workflows
 
-**Error**: `RuntimeError: CUDA out of memory`
+The gene column is mapped to the model vocabulary, with unavailable model genes
+filled with zero. `--gene_col` takes an actual `adata.var` column name; to use
+`var.index`, omit this argument with `--dataset_preset auto`. The optional
+`lung` preset supplies `gene_name`, `sample` and `cell_type`, while the examples
+above make every field explicit.
 
-**Fix**:
+Keep saved embedding rows aligned with the original input when plotting.
+The [task notebook](tutorial.ipynb) provides result-review cells, and
+[tools/](tools/) contains standalone plotting and sequencing-depth workflows.
+Reduced-depth evaluation is an additional analysis, not a fifth integration
+condition in the manuscript. The supplied counts layers and any source file
+must remain unchanged unless a separate preprocessing task is explicitly chosen.
 
-- Decrease `--batch_size`.
-- Use `--n_cells` to limit the number of processed cells.
+The remaining bundled H5AD files serve the documented
+[annotation benchmarks](../cell_annotation/README.md#dataset-commands) and
+[gene perturbation task](../gene_perturbation/README.md); their presence does not
+establish additional manuscript integration benchmarks.

BioMod/zero_shot_batch/{sequencing_depth/cmd.txt → tools/sequencing_depth/README.md}

@@ -1,42 +1,74 @@
-# Run from the repository root. The selected input/layer must contain raw integer counts.
-python BioMod/zero_shot_batch/sequencing_depth/make_low_depth_h5ad.py \
---input_path data/downstream/final_data/Kim2020_Lung.h5ad \
---output_dir tutorial_runs/sequencing_depth/inputs \
---keep_ratios 0.5,0.25,0.1,0.05 \
---seeds 0,1,2  
-
-python BioMod/zero_shot_batch/sequencing_depth/run_zero_shot_low_depth.py \
---data_paths \
-data/downstream/final_data/Kim2020_Lung.h5ad \
-tutorial_runs/sequencing_depth/inputs/*.h5ad \
---scfm_ckpt_path weight/BioMod.pth \
---scfm_hyper_params_path weight/BioMod.pkl \
---output_root tutorial_runs/sequencing_depth/results \
---gene_mapping_path model_dependencies/gene2ens.csv \
---scfm_genes_file model_dependencies/selected_genes_27k.txt \
---biomod_data_dir model_dependencies/biomod_data_all \
---dataset_preset lung \
---output_key X_sclong_module \
---batch_size 256 \
---metrics_mode scgpt \
---pool_method max
-
-
-python BioMod/zero_shot_batch/sequencing_depth/run_zero_shot_low_depth.py \
---data_paths \
-data/downstream/final_data/Kim2020_Lung.h5ad \
-tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p5_seed0.h5ad \
-tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p25_seed0.h5ad \
-tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p1_seed0.h5ad \
-tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p05_seed0.h5ad \
---scfm_ckpt_path weight/BioMod.pth \
---scfm_hyper_params_path weight/BioMod.pkl \
---output_root tutorial_runs/sequencing_depth/results \
---gene_mapping_path model_dependencies/gene2ens.csv \
---scfm_genes_file model_dependencies/selected_genes_27k.txt \
---biomod_data_dir model_dependencies/biomod_data_all \
---dataset_preset lung \
---output_key X_sclong_module \
---batch_size 256 \
---metrics_mode scgpt \
---pool_method max
+# Sequencing-Depth Evaluation with BioMod
+
+Create reduced-depth single-cell datasets by thinning raw integer counts, then
+compare BioMod zero-shot embeddings and evaluation metrics across input depths.
+See the [zero-shot task guide](../../README.md) and
+[task notebook](../../tutorial.ipynb) for the main workflow.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| [make_low_depth_h5ad.py](make_low_depth_h5ad.py) | Generate reduced-depth h5ad datasets at specified retention ratios and random seeds. |
+| [run_zero_shot_low_depth.py](run_zero_shot_low_depth.py) | Run the zero-shot evaluation entrypoint on each input and summarize the resulting metrics. |
+| [README.md](README.md) | Commands for generating inputs and comparing evaluation results. |
+
+## Usage
+
+Run from the repository root. The selected input matrix or layer must contain
+raw integer counts. Built-in resource defaults resolve from the script location;
+explicit relative paths resolve from the working directory.
+
+### Generate Reduced-Depth Inputs
+
+```bash
+# Run from the repository root. The selected input/layer must contain raw integer counts.
+python BioMod/zero_shot_batch/tools/sequencing_depth/make_low_depth_h5ad.py \
+--input_path data/downstream/final_data/Kim2020_Lung.h5ad \
+--output_dir tutorial_runs/sequencing_depth/inputs \
+--keep_ratios 0.5,0.25,0.1,0.05 \
+--seeds 0,1,2  
+```
+
+### Evaluate All Generated Inputs
+
+```bash
+python BioMod/zero_shot_batch/tools/sequencing_depth/run_zero_shot_low_depth.py \
+--data_paths \
+data/downstream/final_data/Kim2020_Lung.h5ad \
+tutorial_runs/sequencing_depth/inputs/*.h5ad \
+--scfm_ckpt_path weight/BioMod.pth \
+--scfm_hyper_params_path weight/BioMod.pkl \
+--output_root tutorial_runs/sequencing_depth/results \
+--gene_mapping_path model_dependencies/gene2ens.csv \
+--scfm_genes_file model_dependencies/selected_genes_27k.txt \
+--biomod_data_dir model_dependencies/biomod_data_all \
+--dataset_preset lung \
+--output_key X_sclong_module \
+--batch_size 256 \
+--metrics_mode scgpt \
+--pool_method max
+```
+
+### Evaluate Selected Inputs
+
+```bash
+python BioMod/zero_shot_batch/tools/sequencing_depth/run_zero_shot_low_depth.py \
+--data_paths \
+data/downstream/final_data/Kim2020_Lung.h5ad \
+tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p5_seed0.h5ad \
+tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p25_seed0.h5ad \
+tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p1_seed0.h5ad \
+tutorial_runs/sequencing_depth/inputs/Kim2020_Lung_ld0p05_seed0.h5ad \
+--scfm_ckpt_path weight/BioMod.pth \
+--scfm_hyper_params_path weight/BioMod.pkl \
+--output_root tutorial_runs/sequencing_depth/results \
+--gene_mapping_path model_dependencies/gene2ens.csv \
+--scfm_genes_file model_dependencies/selected_genes_27k.txt \
+--biomod_data_dir model_dependencies/biomod_data_all \
+--dataset_preset lung \
+--output_key X_sclong_module \
+--batch_size 256 \
+--metrics_mode scgpt \
+--pool_method max
+```

BioMod/zero_shot_batch/{sequencing_depth → tools/sequencing_depth}/run_zero_shot_low_depth.py

@@ -17,7 +17,7 @@ def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(
         description="Run biomod zero-shot on multiple full/low-depth h5ad files and aggregate metrics."
     )
-    parser.add_argument("--script_path", type=str, default=None, help="Path to biomod_zero_shot.py")
+    parser.add_argument("--script_path", type=str, default=None, help="Path to evaluate.py")
     parser.add_argument("--data_paths", nargs="+", required=True, help="One or more full/low-depth h5ad files")
     parser.add_argument("--output_root", type=str, required=True, help="Root directory for per-run outputs")
     parser.add_argument("--scfm_hyper_params_path", type=str, required=True)
@@ -26,7 +26,7 @@ def parse_args() -> argparse.Namespace:
     parser.add_argument("--biomod_data_dir", "--module_data_dir", type=str, default=None, dest='module_data_dir')
     parser.add_argument("--gene_mapping_path", type=str, required=True)
     parser.add_argument("--scfm_genes_file", type=str, required=True)
-    parser.add_argument("--gene2vec_path", type=str, default=str(Path(__file__).resolve().parents[3] / "model_dependencies" / "selected_gene2vec_27k.npy"))
+    parser.add_argument("--gene2vec_path", type=str, default=str(Path(__file__).resolve().parents[4] / "model_dependencies" / "selected_gene2vec_27k.npy"))
     parser.add_argument("--dataset_preset", type=str, default="lung")
     parser.add_argument("--gene_col", type=str, default=None)
     parser.add_argument("--batch_key", type=str, default=None)
@@ -49,7 +49,7 @@ def parse_args() -> argparse.Namespace:
 def resolve_script_path(script_path: Optional[str]) -> Path:
     if script_path is not None:
         return Path(script_path).resolve()
-    return (Path(__file__).resolve().parent.parent / "biomod_zero_shot.py").resolve()
+    return (Path(__file__).resolve().parents[2] / "evaluate.py").resolve()
 
 
 def slugify_path(path: Path) -> str:

BioMod/{toturials/zero_shot_batch_integration.ipynb → zero_shot_batch/tutorial.ipynb}

@@ -15,12 +15,12 @@
    "source": [
     "## Project paths and environment\n",
     "\n",
-    "Start Jupyter with `BioMod/toturials` as its working directory and select the environment described in [the project README](../../README.md). Paths below are relative to this notebook directory, following the project-relative layout used in scGPT tutorials. `Path.resolve()` turns them into absolute paths for subprocesses. If Jupyter was started at the repository root, set `PROJECT_ROOT = Path('.').resolve()` instead.\n",
+    "Start Jupyter with `BioMod/zero_shot_batch` as its working directory and select the environment described in [the project README](../../README.md). Paths below are relative to this notebook directory, following the project-relative layout used in scGPT tutorials. `Path.resolve()` turns them into absolute paths for subprocesses. If Jupyter was started at the repository root, set `PROJECT_ROOT = Path('.').resolve()` instead.\n",
     "\n",
     "- Data: `../../data/downstream/final_data/Kim2020_Lung_noMalignant.h5ad`\n",
     "- BioMod model weights and configuration: `../../weight/`\n",
     "- Shared assets: `../../model_dependencies/`\n",
-    "- Runner: `../zero_shot_batch/biomod_zero_shot.py`\n",
+    "- Runner: `evaluate.py`\n",
     "- Results: `../../tutorial_runs/zero_shot_kim_lung_no_malignant/`\n",
     "\n",
     "Review the configuration, then set `RUN_EVALUATION = True` to run inference. The runner consumes `adata.X` without normalization or log transformation; check that its expression preprocessing is appropriate for the checkpoint and your experiment. It loads the input dataset into memory before batching inference.\n"
@@ -41,12 +41,12 @@
     "PROJECT_ROOT = Path('../..').resolve()\n",
     "if not (PROJECT_ROOT / 'BioMod').is_dir() or not (PROJECT_ROOT / 'model_dependencies').is_dir():\n",
     "    raise FileNotFoundError(\n",
-    "        'Start Jupyter from BioMod/toturials, or set PROJECT_ROOT to the repository root '\n",
+    "        'Start Jupyter from BioMod/zero_shot_batch, or set PROJECT_ROOT to the repository root '\n",
     "        \"(use Path('.').resolve() when launched at the repository root).\"\n",
     "    )\n",
     "SOURCE_DIR = PROJECT_ROOT / 'BioMod'\n",
     "DEPENDENCIES_DIR = PROJECT_ROOT / 'model_dependencies'\n",
-    "SCRIPT = SOURCE_DIR / 'zero_shot_batch' / 'biomod_zero_shot.py'\n",
+    "SCRIPT = SOURCE_DIR / 'zero_shot_batch' / 'evaluate.py'\n",
     "DATA_PATH = PROJECT_ROOT / 'data' / 'downstream' / 'final_data' / 'Kim2020_Lung_noMalignant.h5ad'\n",
     "WEIGHT_DIR = PROJECT_ROOT / 'weight'\n",
     "CONFIG_PKL = WEIGHT_DIR / 'BioMod.pkl'\n",

README.md

@@ -53,6 +53,17 @@ Pretraining uses CUDA/NCCL. The zero-shot
 runner selects CUDA when available and otherwise uses CPU. Native Windows
 execution has not been validated for the supplied environment.
 
+To open a tutorial, choose its task directory. For cell annotation, run the
+following from the outer repository root:
+
+```bash
+cd BioMod/cell_annotation
+jupyter lab tutorial.ipynb
+```
+
+Use `drug_response`, `drug_combination`, `gene_perturbation`, or `zero_shot_batch`
+in place of `cell_annotation` to open another task.
+
 ## BioMod model weights
 
 This release includes the **BioMod model weights** and their matching
@@ -70,6 +81,9 @@ architecture and gene-module definitions.
 
 The model components are bundled under
 [`model_dependencies/pytorch_cont/`](model_dependencies/pytorch_cont/).
+Optional [Hallmark and Reactome source definitions](model_dependencies/biomod_data_sources/)
+are kept in `biomod_data_sources/`. They are separate from the default checkpoint
+runtime definitions in `biomod_data_all/`.
 Task-specific checkpoint arguments are listed under
 [Usage notes](#checkpoint-arguments).
 
@@ -81,11 +95,11 @@ to end with the bundled checkpoint.
 
 | Task | Implementation | Notebook |
 | --- | --- | --- |
-| Cell-type annotation | [Fine-tuning script](BioMod/cell_annotation/biomod_finetune.py) | [Notebook](BioMod/toturials/cell_annotation.ipynb) |
-| Zero-shot cell embeddings and batch-effect evaluation | [Embedding and evaluation script](BioMod/zero_shot_batch/biomod_zero_shot.py) | [Notebook](BioMod/toturials/zero_shot_batch_integration.ipynb) |
-| Single-drug response regression (IC50) | [Training script](BioMod/drug_response/train_drug_biomod.py) | [Notebook](BioMod/toturials/drug_response_prediction.ipynb) |
-| Drug-pair synergy classification | [Training script](BioMod/drug_combination/train_drug_combination_biomod.py) | [Notebook](BioMod/toturials/drug_combination.ipynb) |
-| Gene perturbation response prediction | [Training script](BioMod/gene_perturbation/train_biomod.py) | [Notebook](BioMod/toturials/gene_perturbation.ipynb) |
+| Cell-type annotation | [Fine-tuning script](BioMod/cell_annotation/train.py) | [Notebook](BioMod/cell_annotation/tutorial.ipynb) |
+| Zero-shot cell embeddings and batch-effect evaluation | [Embedding and evaluation script](BioMod/zero_shot_batch/evaluate.py) | [Notebook](BioMod/zero_shot_batch/tutorial.ipynb) |
+| Single-drug response regression (IC50) | [Training script](BioMod/drug_response/train.py) | [Notebook](BioMod/drug_response/tutorial.ipynb) |
+| Drug-pair synergy classification | [Training script](BioMod/drug_combination/train.py) | [Notebook](BioMod/drug_combination/tutorial.ipynb) |
+| Gene perturbation response prediction | [Training script](BioMod/gene_perturbation/train.py) | [Notebook](BioMod/gene_perturbation/tutorial.ipynb) |
 
 ### Pretraining
 
@@ -109,10 +123,12 @@ The project root contains `BioMod/`, `model_dependencies/`, `weight/`, and
 may assume different working directories; adjust relative paths when following
 a guide.
 
-Start Jupyter from `BioMod/toturials/`. Each notebook sets
-`PROJECT_ROOT = Path('../..').resolve()` and derives absolute runtime paths from
-the outer repository root. If Jupyter starts at the repository root, use
-`Path('.').resolve()` instead. Review the configuration cells before running:
+Each task keeps its notebook in `BioMod/<task>/tutorial.ipynb`. Start Jupyter
+from the selected task directory, as shown in [Installation](#installation).
+Each notebook sets `PROJECT_ROOT = Path('../..').resolve()` and derives absolute
+runtime paths from the outer repository root. If the notebook kernel uses a
+different working directory, set `PROJECT_ROOT` to the outer repository root
+explicitly before running the path checks. Review the configuration cells before running:
 
 | Setting | Location |
 | --- | --- |
@@ -170,64 +186,65 @@ readability. Folder descriptions explain where to find each component.
 <summary>Expand the full repository structure</summary>
 
 ```text
-BioMod/                                                   # Project root: source code, model dependencies and weights
-├── BioMod/                                               # Task code: training, evaluation and tutorials
-│   ├── cell_annotation/                                  # Cell-type annotation and model fine-tuning
-│   │   ├── biomod_data/                                  # Hallmark and Reactome modules with loading utilities
-│   │   │   ├── load_biomod.py
-│   │   │   ├── biomod1_hallmarks.json
-│   │   │   └── biomod2_reactome_2022_filtered.json
-│   │   ├── cmd_finetune.md
-│   │   └── biomod_finetune.py
-│   ├── drug_combination/                                 # Drug-combination synergy prediction
+BioMod/                                                        # Project root: source code, model dependencies and weights
+├── BioMod/                                                    # Task code: training, evaluation and tutorials
+│   ├── cell_annotation/                                       # Cell-type annotation and model fine-tuning
+│   │   ├── README.md
+│   │   ├── train.py
+│   │   └── tutorial.ipynb
+│   ├── drug_combination/                                      # Drug-combination synergy prediction
 │   │   ├── README.md
-│   │   └── train_drug_combination_biomod.py
-│   ├── drug_response/                                    # Single-drug response prediction and training
-│   │   ├── cmd.txt
-│   │   └── train_drug_biomod.py
-│   ├── gene_perturbation/                                # Perturbation response prediction: processing, training and inference
-│   │   ├── data_utils_local.py
-│   │   ├── gears_reduce_local.py
-│   │   ├── inference_local.py
-│   │   ├── model_biomod.py
-│   │   ├── pertdata_local.py
+│   │   ├── train.py
+│   │   └── tutorial.ipynb
+│   ├── drug_response/                                         # Single-drug response prediction
 │   │   ├── README.md
-│   │   ├── scfm_utils_local.py
-│   │   ├── train_biomod.py
-│   │   └── utils.py
-│   ├── tests/                                            # CPU tests for shared training utilities
+│   │   ├── train.py
+│   │   └── tutorial.ipynb
+│   ├── gene_perturbation/                                     # Gene perturbation response prediction
+│   │   ├── components/                                        # Task models, datasets, training and inference helpers
+│   │   │   ├── __init__.py
+│   │   │   ├── data_utils.py
+│   │   │   ├── dataset.py
+│   │   │   ├── inference.py
+│   │   │   ├── model.py
+│   │   │   ├── trainer.py
+│   │   │   ├── training_utils.py
+│   │   │   └── utils.py
+│   │   ├── README.md
+│   │   ├── train.py
+│   │   └── tutorial.ipynb
+│   ├── tests/                                                 # CPU tests for shared training utilities
 │   │   └── test_training_utils.py
-│   ├── toturials/                                        # Jupyter tutorials for five downstream tasks
-│   │   ├── cell_annotation.ipynb
-│   │   ├── drug_combination.ipynb
-│   │   ├── drug_response_prediction.ipynb
-│   │   ├── gene_perturbation.ipynb
-│   │   └── zero_shot_batch_integration.ipynb
-│   ├── zero_shot_batch/                                  # Zero-shot batch integration, evaluation and visualization
-│   │   ├── sequencing_depth/                             # Low-depth data generation and zero-shot evaluation
-│   │   │   ├── cmd.txt
-│   │   │   ├── make_low_depth_h5ad.py
-│   │   │   └── run_zero_shot_low_depth.py
-│   │   ├── biomod_zero_shot.py
+│   ├── zero_shot_batch/                                       # Zero-shot integration and evaluation
+│   │   ├── tools/                                             # Optional visualization and evaluation tools
+│   │   │   ├── sequencing_depth/                              # Low-depth expression generation and evaluation
+│   │   │   │   ├── make_low_depth_h5ad.py
+│   │   │   │   ├── README.md
+│   │   │   │   └── run_zero_shot_low_depth.py
+│   │   │   └── visualize_results.py
+│   │   ├── evaluate.py
 │   │   ├── README.md
-│   │   └── visualize_results.py
-│   ├── .env.example
+│   │   └── tutorial.ipynb
 │   ├── .gitattributes
 │   ├── .gitignore
+│   ├── biomod.py
+│   ├── biomod_dataset.py
 │   ├── command.txt
 │   ├── environment.yml
 │   ├── models_biomod.py
-│   ├── biomod_dataset.py
-│   ├── biomod.py
 │   ├── README.md
 │   ├── requirements.txt
 │   └── training_utils.py
-├── model_dependencies/                                   # Shared gene mappings, gene lists, embeddings and model components
-│   ├── biomod_data_all/                                  # Shared module definitions and construction tools
+├── model_dependencies/                                        # Shared gene mappings, embeddings and model components
+│   ├── biomod_data_all/                                       # Default merged module definitions and construction tools
 │   │   ├── create_uncovered_biomods.py
 │   │   ├── merge_biomods.py
 │   │   └── merged_biomods.json
-│   ├── pytorch_cont/                           # Attention and model components
+│   ├── biomod_data_sources/                                   # Optional source gene sets and their loading helper
+│   │   ├── biomod1_hallmarks.json
+│   │   ├── biomod2_reactome_2022_filtered.json
+│   │   └── load_biomod.py
+│   ├── pytorch_cont/                                          # Attention and model components
 │   │   ├── __init__.py
 │   │   ├── ding_models.py
 │   │   ├── pytorch.py
@@ -236,7 +253,7 @@ BioMod/                                                   # Project root: source
 │   ├── gene2ens.csv
 │   ├── selected_gene2vec_27k.npy
 │   └── selected_genes_27k.txt
-├── weight/                                               # Pretrained weights and matching model configurations
+├── weight/                                                    # Pretrained weights and matching model configuration
 │   ├── BioMod.pkl
 │   └── BioMod.pth
 ├── .gitattributes

data/downstream/final_data/drug_combination_synergy/README.md

@@ -9,7 +9,7 @@ data/downstream/final_data/drug_response_pytorch/drug_combination
 
 ## Runnable data path
 
-From the repository root, use this directory as `--data_dir` for `BioMod/drug_combination/train_drug_combination_biomod.py`:
+From the repository root, use this directory as `--data_dir` for `BioMod/drug_combination/train.py`:
 
 ```text
 data/downstream/final_data/drug_combination_synergy/data_summary

data/downstream/final_data/single_drug_response/README.md

@@ -9,7 +9,7 @@ data/downstream/final_data/drug_response_pytorch/drug_response_pytorch
 
 ## Runnable data path
 
-From the repository root, use this directory as `--data_path` for `BioMod/drug_response/train_drug_biomod.py`:
+From the repository root, use this directory as `--data_path` for `BioMod/drug_response/train.py`:
 
 ```text
 data/downstream/final_data/single_drug_response/data