CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this repository is

This is the **final deliverable for CS3319 Project 2** — a research/experiment codebase for an
academic **author↔paper link-prediction** (reading-recommendation) task on a heterogeneous graph.
It is *not* a software product: there is **no build system and no test suite**. The "commands" are
individual Python experiment scripts; "running an experiment" = invoking one script.

- **Task:** predict 0/1 for ~2.05M author-paper test pairs (`data_and_docs/bipartite_test_ann.txt`).
  Metric is **F1**. Submission CSV has columns `Index,Predicted` (0/1).
- **Graph:** 6,611 authors, 79,937 papers. Edges: author→paper cites (`bipartite_train_ann.txt`),
  author↔author co-authorship (`author_file_ann.txt`), paper→paper citation (`paper_file_ann.txt`).
  `feature.pkl` holds a 512-d USE embedding per paper (a torch tensor; load with `pickle.load`,
  then call `.numpy()`).
- The `README.md` has the results table and exact reproduction commands — read it first for that.
  This file covers the *architecture and developer workflow* that the README assumes.

## Environment

```bash
# Conda (full original env)
conda env create -f env/environment-cs3319.yml
# or minimal pinned deps (Python 3.10)
pip install -r env/requirements-minimal.txt
```

Core stack: numpy, pandas, scipy, scikit-learn, lightgbm, torch, torch-geometric, gensim, networkx,
node2vec. LightGCN/BPR/HGT/SAGE training needs a GPU (`--device cuda:0`); the LightGBM second-stage
stackers are CPU-only.

## How scripts are run (CLI conventions)

Every non-legacy script uses `argparse` and takes `--package-root` as its first argument, defaulting
to the repo root:

```python
parser.add_argument("--package-root", type=Path, default=Path(__file__).resolve().parents[1])
```

All data is resolved relative to `package-root` — there is no path-config file. Inputs live under
`data_and_docs/`; outputs under `validation_runs/`. Recurring flags:

| Flag | Meaning |
|---|---|
| `--package-root` | Repo root (default = parent of `code/`). |
| `--split-seed` | Selects the validation split subtree `validation_runs/dynamic_seed{N}/`. **Almost everything is pinned to `202`.** |
| `--seed` | RNG seed for LightGBM / numpy / torch within a stage. |
| `--n-splits` | StratifiedKFold folds for OOF evaluation (default 5). |
| `--device` | `cuda:0` / `cpu` for torch models. |
| `--main-val-score-file` | Path to the primary LightGCN ensemble score `.npy` a stacker builds on. |
| `--versions` | Which DeepWalk/Node2Vec config blocks to include (space-separated). |
| `--ratios` / `--thresholds` | Positive-ratio (rank-cutoff) or absolute-threshold values for submission generation. |
| `--make-submission` | Gate: also score test pairs and write submission CSVs (otherwise validation-only). |

### Reproduce the final result

Fastest: the submission is already generated — see `README.md` → "Quick Verification".
To regenerate from the included cached features / RW weights (the whole pipeline's intermediate
outputs are cached in the package):

```bash
python code/high_order_graph_stack.py --package-root . --split-seed 202 --seed 202 --n-splits 5 --make-submission
```

To regenerate the earlier 6-model LightGCN ensemble submission from checkpoints:

```bash
python code/generate_ens6_submission.py --package-root . --device cuda:0   # or cpu
```

### Running a single experiment

There is no "single test" — the unit of work is an **ablation script**. Ablations evaluate one
feature source added to the stacker via 5-fold OOF and write `validation_summary.csv`:

```bash
python code/content_rich_ablation.py \
  --package-root . --split-seed 202 \
  --main-val-score-file validation_runs/dynamic_seed202/dyn202_l2d512_bpr_bigbatch_more/scores/val_vanilla_ensemble_mean.npy
```

### End-to-end stage order (if regenerating from scratch, not from cache)

1. `train_val_lgcn_ensemble.py` → primary LightGCN ensemble scores (`.../scores/val_vanilla_ensemble_mean.npy`).
2. `generate_post95_submission.py` → selects top-N GNN score variants + their test counterparts.
3. `extra_score_sources_ablation.py` → BPR-MF + content-mean-cos scores.
4. `randomwalk_systematic_ablation.py` (`--mode small`, then `--mode graph`) → 7 DeepWalk/Node2Vec Word2Vec models.
5. `high_order_graph_stack.py` → final stacker + submission.

Almost every stage's output is already cached in the package, so in practice you only re-run the
stage you are changing.

## Architecture: the big picture

### Two-stage stacking

The final model (`code/high_order_graph_stack.py`) is a **LightGBM second-stage meta-learner** over
~259 features. Stage 1 produces raw scores/embeddings from several independent models; stage 2
combines them. The feature groups, each with a producer script:

| Feature group | Producer script |
|---|---|
| LightGCN score + rank features | `train_val_lgcn_ensemble.py` (the primary score) |
| Explicit graph / meta-path features | `stack_rank_calibration.py` (`ExplicitGraphFeatures`, `add_rank_features`) |
| Content mean-cos, top-k content similarity | `extra_score_sources_ablation.py`, `generate_post95_submission.py` |
| BPR-MF scores | `extra_score_sources_ablation.py` |
| Rich author-content profile (18 feats from `feature.pkl`) | `content_rich_ablation.py` |
| 7 DeepWalk/Node2Vec random-walk blocks (11 feats each) | `randomwalk_systematic_ablation.py` |
| Random-walk agreement aggregate | `generate_randomwalk_ensemble_submission.py` (`aggregate`) |
| High-order citation propagation (`A-P-P^k`, `A-A-P-P^k`, fwd/bwd/undir) | `high_order_graph_stack.py` (`build_high_order*`) |

The "ablation" scripts are how each group was validated in isolation (and combinations thereof).

### No shared utils module — scripts load each other at runtime

There is **no `utils.py` or common package**. Code reuse is done by loading sibling scripts as
modules with a near-identical `load_module()` helper (`importlib.util`):

```python
def load_module(name, path):
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module
```

Two scripts are the **de-facto shared libraries** and are loaded by ~14 others each:

- **`train_val_lgcn_ensemble.py`** — data loading (`build_parts`, `build_data`), the notebook-style
  split (`make_notebook_style_split`), and `best_f1()` (optimal-F1 threshold via the PR curve).
- **`stack_rank_calibration.py`** — `ExplicitGraphFeatures` and `add_rank_features()` used by every
  stacker.

Other frequently-loaded peers: `generate_post95_submission.py` (variant selection, `topk_content_similarity_fast`),
`post95_ablation.py` (`negative_evidence_features`), `extra_score_sources_ablation.py`
(`content_mean_score`, `score_to_features`), `randomwalk_systematic_ablation.py` (`build_base_features`,
`pair_feature_block`), `content_rich_ablation.py` (`content_rich_features`).

**Implication:** when editing a function in one of these shared scripts, you are changing behavior
for every downstream script that loads it. Each script also redefines its own local `read_txt`.

### Validation framework: `dynamic_seed202`

"dynamic_seed{N}" = the train/val split materialized at runtime with `split_seed=N`
(`make_notebook_style_split`): hold out 10% (`--train-frac 0.9`) of known author-paper edges as
validation **positives**, then sample an **equal number of random non-edges as negatives** — an
artificial **1:1** positive/negative validation set. All stacking experiments write into
`validation_runs/dynamic_seed202/<stage>/`.

Because the split seed is baked into every score/feature file, **changing `--split-seed` invalidates
the entire pipeline** — scores are split-specific and must be regenerated end-to-end.

### Score & feature file conventions

- **Scores are `.npy`**, one float per pair, named `val_*` (validation/OOF) or `test_*` (test pairs).
  e.g. `.../scores/val_vanilla_ensemble_mean.npy`. Test counterparts mirror val paths under
  `validation_runs/dynamic_seed202/post95_test_scores/` with `val_`→`test_`.
- **OOF evaluation:** stackers don't train on the raw GNN score; they run 5-fold `StratifiedKFold`
  LightGBM (`fit_lgb_oof`), then call `best_f1(y, oof)` to get a leak-free validation F1. The OOF
  array is saved as `*_oof.npy`.
- **Feature cache** (`validation_runs/feature_cache/`): expensive content/high-order features use a
  **cache-or-compute** pattern keyed by an identity filename
  `{tag}_{npairs}_{sum(author_ids)}_{sum(paper_ids)}[_k{topk}].npy`. If the file exists it is loaded
  verbatim; delete it to force recomputation.
- Random-walk pair features are cached as `.npz` under
  `validation_runs/dynamic_seed202/randomwalk_systematic/pair_features/`; the 7 Word2Vec models live
  in `.../randomwalk_systematic/models/`.

### Submission decision rule (important)

The final test decision is **rank-cutoff, not a probability threshold**:

```
sort test pairs by final LightGBM score → predict the top ratio (0.500) as 1
→ force any test pair that also appears in the training edges to 1 (test_known_mask.npy)
```

This is deliberately *not* the validation-optimal probability threshold, because the 1:1 validation
split is artificial and LightGBM probabilities don't calibrate across the val→test distribution
shift. Submission generators sweep a small set of ratios (e.g. 0.498–0.502) and pick the best public
file. The mask of train/test-overlap positives (`cached_scores/test_known_mask.npy`) is loaded by
every submission generator.

## Critical conventions & gotchas

- **Hardcoded graph dimensions** `6611` (authors) and `79937` (papers) appear literally in
  `high_order_graph_stack.py` (sparse-matrix shapes) and `train_val_lgcn_ensemble.py` (sampling
  loops). They match the provided dataset; don't "refactor" them into variables casually.
- **Legacy scripts are not runnable as-is.** `run_baseline.py`, `run_improved.py`, `run_v2.py`,
  `run_final.py`, `run_ultimate.py`, `run_lgcn_final.py`, `run_lgcn_v2.py`, `run_graph_features.py`,
  and `compare_gnn.py` are early prototypes with **hardcoded `/home/lzc/cs3319-project` paths and no
  argparse**. They are kept for provenance only — edit paths before running, or use the modern
  `train_val_*` / `generate_*` / `*_ablation` scripts instead.
- **`--split-seed` is load-bearing.** Default 202 everywhere; the cached `.npy`/`.npz`/`.model`
  artifacts are only valid for seed 202. A different seed requires regenerating the whole chain.
- **Course rules forbid pre-trained models and external datasets** (see
  `data_and_docs/project_description.md`). Everything is built from scratch on the provided files.
- Some inherited metadata files contain absolute paths from the original workspace; for curated
  artifacts use the files included here or paths relative to `--package-root`.

## Key entry points

- **Final method:** `code/high_order_graph_stack.py` (validation-only by default; `--make-submission`
  for the test CSV). Its output dir is `validation_runs/dynamic_seed202/high_order_graph_stack/`.
- **Primary score producer:** `code/train_val_lgcn_ensemble.py`.
- **Script categories:** training = `train_val_*` / `run_*`(legacy); feature/score ablation =
  `*_ablation.py`; submission generation = `generate_*submission.py`; stacking/calibration/search =
  `stack_*`, `score_level_*`, `error_group_*`, `search_*`, `rich_randomwalk_stack.py`. See the
  README's "Core Scripts" table for per-script purposes.

## Reports & docs

Read in order for the full experimental narrative: `reports/preliminary_report.md` →
`reports/exploration_summary.md` → `reports/final_report.md` → `notes/experiment_history.md`.
Task/eval specs: `data_and_docs/project_description.md`, `dataset.md`, `project_evaluation.md`.