CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repository is

This is the final deliverable for CS3319 Project 2 — a research/experiment codebase for an academic author↔paper link-prediction (reading-recommendation) task on a heterogeneous graph. It is not a software product: there is no build system and no test suite. The "commands" are individual Python experiment scripts; "running an experiment" = invoking one script.

• Task: predict 0/1 for ~2.05M author-paper test pairs (data_and_docs/bipartite_test_ann.txt). Metric is F1. Submission CSV has columns Index,Predicted (0/1).
• Graph: 6,611 authors, 79,937 papers. Edges: author→paper cites (bipartite_train_ann.txt), author↔author co-authorship (author_file_ann.txt), paper→paper citation (paper_file_ann.txt). feature.pkl holds a 512-d USE embedding per paper (a torch tensor; load with pickle.load, then call .numpy()).
• The README.md has the results table and exact reproduction commands — read it first for that. This file covers the architecture and developer workflow that the README assumes.

Environment

# Conda (full original env)
conda env create -f env/environment-cs3319.yml
# or minimal pinned deps (Python 3.10)
pip install -r env/requirements-minimal.txt

Core stack: numpy, pandas, scipy, scikit-learn, lightgbm, torch, torch-geometric, gensim, networkx, node2vec. LightGCN/BPR/HGT/SAGE training needs a GPU (--device cuda:0); the LightGBM second-stage stackers are CPU-only.

How scripts are run (CLI conventions)

Every non-legacy script uses argparse and takes --package-root as its first argument, defaulting to the repo root:

parser.add_argument("--package-root", type=Path, default=Path(__file__).resolve().parents[1])

All data is resolved relative to package-root — there is no path-config file. Inputs live under data_and_docs/; outputs under validation_runs/. Recurring flags:

Flag	Meaning
--package-root	Repo root (default = parent of code/).
--split-seed	Selects the validation split subtree validation_runs/dynamic_seed{N}/. Almost everything is pinned to 202.
--seed	RNG seed for LightGBM / numpy / torch within a stage.
--n-splits	StratifiedKFold folds for OOF evaluation (default 5).
--device	cuda:0 / cpu for torch models.
--main-val-score-file	Path to the primary LightGCN ensemble score .npy a stacker builds on.
--versions	Which DeepWalk/Node2Vec config blocks to include (space-separated).
--ratios / --thresholds	Positive-ratio (rank-cutoff) or absolute-threshold values for submission generation.
--make-submission	Gate: also score test pairs and write submission CSVs (otherwise validation-only).

Reproduce the final result

Fastest: the submission is already generated — see README.md → "Quick Verification". To regenerate from the included cached features / RW weights (the whole pipeline's intermediate outputs are cached in the package):

python code/high_order_graph_stack.py --package-root . --split-seed 202 --seed 202 --n-splits 5 --make-submission

To regenerate the earlier 6-model LightGCN ensemble submission from checkpoints:

python code/generate_ens6_submission.py --package-root . --device cuda:0   # or cpu

Running a single experiment

There is no "single test" — the unit of work is an ablation script. Ablations evaluate one feature source added to the stacker via 5-fold OOF and write validation_summary.csv:

python code/content_rich_ablation.py \
  --package-root . --split-seed 202 \
  --main-val-score-file validation_runs/dynamic_seed202/dyn202_l2d512_bpr_bigbatch_more/scores/val_vanilla_ensemble_mean.npy

End-to-end stage order (if regenerating from scratch, not from cache)

1. train_val_lgcn_ensemble.py → primary LightGCN ensemble scores (.../scores/val_vanilla_ensemble_mean.npy).
2. generate_post95_submission.py → selects top-N GNN score variants + their test counterparts.
3. extra_score_sources_ablation.py → BPR-MF + content-mean-cos scores.
4. randomwalk_systematic_ablation.py (--mode small, then --mode graph) → 7 DeepWalk/Node2Vec Word2Vec models.
5. high_order_graph_stack.py → final stacker + submission.

Almost every stage's output is already cached in the package, so in practice you only re-run the stage you are changing.

Architecture: the big picture

Two-stage stacking

The final model (code/high_order_graph_stack.py) is a LightGBM second-stage meta-learner over ~259 features. Stage 1 produces raw scores/embeddings from several independent models; stage 2 combines them. The feature groups, each with a producer script:

Feature group	Producer script
LightGCN score + rank features	train_val_lgcn_ensemble.py (the primary score)
Explicit graph / meta-path features	stack_rank_calibration.py (ExplicitGraphFeatures, add_rank_features)
Content mean-cos, top-k content similarity	extra_score_sources_ablation.py, generate_post95_submission.py
BPR-MF scores	extra_score_sources_ablation.py
Rich author-content profile (18 feats from feature.pkl)	content_rich_ablation.py
7 DeepWalk/Node2Vec random-walk blocks (11 feats each)	randomwalk_systematic_ablation.py
Random-walk agreement aggregate	generate_randomwalk_ensemble_submission.py (aggregate)
High-order citation propagation (A-P-P^k, A-A-P-P^k, fwd/bwd/undir)	high_order_graph_stack.py (build_high_order*)

The "ablation" scripts are how each group was validated in isolation (and combinations thereof).

No shared utils module — scripts load each other at runtime

There is no utils.py or common package. Code reuse is done by loading sibling scripts as modules with a near-identical load_module() helper (importlib.util):

def load_module(name, path):
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module

Two scripts are the de-facto shared libraries and are loaded by ~14 others each:

• train_val_lgcn_ensemble.py — data loading (build_parts, build_data), the notebook-style split (make_notebook_style_split), and best_f1() (optimal-F1 threshold via the PR curve).
• stack_rank_calibration.py — ExplicitGraphFeatures and add_rank_features() used by every stacker.

Other frequently-loaded peers: generate_post95_submission.py (variant selection, topk_content_similarity_fast), post95_ablation.py (negative_evidence_features), extra_score_sources_ablation.py (content_mean_score, score_to_features), randomwalk_systematic_ablation.py (build_base_features, pair_feature_block), content_rich_ablation.py (content_rich_features).

Implication: when editing a function in one of these shared scripts, you are changing behavior for every downstream script that loads it. Each script also redefines its own local read_txt.

Validation framework: dynamic_seed202

"dynamic_seed{N}" = the train/val split materialized at runtime with split_seed=N (make_notebook_style_split): hold out 10% (--train-frac 0.9) of known author-paper edges as validation positives, then sample an equal number of random non-edges as negatives — an artificial 1:1 positive/negative validation set. All stacking experiments write into validation_runs/dynamic_seed202/<stage>/.

Because the split seed is baked into every score/feature file, changing --split-seed invalidates the entire pipeline — scores are split-specific and must be regenerated end-to-end.

Score & feature file conventions

• Scores are .npy, one float per pair, named val_* (validation/OOF) or test_* (test pairs). e.g. .../scores/val_vanilla_ensemble_mean.npy. Test counterparts mirror val paths under validation_runs/dynamic_seed202/post95_test_scores/ with val_→test_.
• OOF evaluation: stackers don't train on the raw GNN score; they run 5-fold StratifiedKFold LightGBM (fit_lgb_oof), then call best_f1(y, oof) to get a leak-free validation F1. The OOF array is saved as *_oof.npy.
• Feature cache (validation_runs/feature_cache/): expensive content/high-order features use a cache-or-compute pattern keyed by an identity filename {tag}_{npairs}_{sum(author_ids)}_{sum(paper_ids)}[_k{topk}].npy. If the file exists it is loaded verbatim; delete it to force recomputation.
• Random-walk pair features are cached as .npz under validation_runs/dynamic_seed202/randomwalk_systematic/pair_features/; the 7 Word2Vec models live in .../randomwalk_systematic/models/.

Submission decision rule (important)

The final test decision is rank-cutoff, not a probability threshold:

sort test pairs by final LightGBM score → predict the top ratio (0.500) as 1
→ force any test pair that also appears in the training edges to 1 (test_known_mask.npy)

This is deliberately not the validation-optimal probability threshold, because the 1:1 validation split is artificial and LightGBM probabilities don't calibrate across the val→test distribution shift. Submission generators sweep a small set of ratios (e.g. 0.498–0.502) and pick the best public file. The mask of train/test-overlap positives (cached_scores/test_known_mask.npy) is loaded by every submission generator.

Critical conventions & gotchas

• Hardcoded graph dimensions 6611 (authors) and 79937 (papers) appear literally in high_order_graph_stack.py (sparse-matrix shapes) and train_val_lgcn_ensemble.py (sampling loops). They match the provided dataset; don't "refactor" them into variables casually.
• Legacy scripts are not runnable as-is. run_baseline.py, run_improved.py, run_v2.py, run_final.py, run_ultimate.py, run_lgcn_final.py, run_lgcn_v2.py, run_graph_features.py, and compare_gnn.py are early prototypes with hardcoded /home/lzc/cs3319-project paths and no argparse. They are kept for provenance only — edit paths before running, or use the modern train_val_* / generate_* / *_ablation scripts instead.
• --split-seed is load-bearing. Default 202 everywhere; the cached .npy/.npz/.model artifacts are only valid for seed 202. A different seed requires regenerating the whole chain.
• Course rules forbid pre-trained models and external datasets (see data_and_docs/project_description.md). Everything is built from scratch on the provided files.
• Some inherited metadata files contain absolute paths from the original workspace; for curated artifacts use the files included here or paths relative to --package-root.

Key entry points

• Final method: code/high_order_graph_stack.py (validation-only by default; --make-submission for the test CSV). Its output dir is validation_runs/dynamic_seed202/high_order_graph_stack/.
• Primary score producer: code/train_val_lgcn_ensemble.py.
• Script categories: training = train_val_* / run_*(legacy); feature/score ablation = *_ablation.py; submission generation = generate_*submission.py; stacking/calibration/search = stack_*, score_level_*, error_group_*, search_*, rich_randomwalk_stack.py. See the README's "Core Scripts" table for per-script purposes.

Reports & docs

Read in order for the full experimental narrative: reports/preliminary_report.md → reports/exploration_summary.md → reports/final_report.md → notes/experiment_history.md. Task/eval specs: data_and_docs/project_description.md, dataset.md, project_evaluation.md.