notes/experiment_history.md

Experiment History

This note records the main experiments and what they were intended to test.

1. Official Notebook Baseline

Script: code/run_baseline.py

Purpose:

• Convert the official notebook into a local Python script.
• Build a heterogeneous graph with authors and papers.
• Use a simple custom HeteroMeanConv layer.
• Train with a hinge/ranking-style objective.
• Predict with cosine similarity and validation threshold search.

Outcome:

• Validation F1 was around 0.885.
• Generated /home/lzc/submission.csv.
• This was the earliest upload candidate, but the exact leaderboard score was not preserved.

Main limitations:

• Custom mean aggregation was weak.
• Negative sampling was simple.
• No strong hard-negative strategy.
• Only a single model.

2. Improved Heterogeneous GNN

Script: code/run_improved.py

Purpose:

• Replace the custom baseline with PyG HeteroConv + SAGEConv.
• Add residual connections, layer normalization, and dropout.
• Use an MLP decoder.
• Add graph structural features.
• Use hard negatives:
  • random negatives.
  • popular papers.
  • papers read by coauthors.
• Train two seeds and ensemble.

Outcome:

• Better than the notebook-style baseline, but still below the later LightGCN models.
• Generated:
  • submission_improved.csv
  • submission_model1.csv
  • submission_model2.csv

Conclusion:

• More expressive SAGE/MLP models did not generalize as well as the simpler LightGCN-style model.

3. SAGEConv + BPR Ranking

Script: code/run_v2.py

Purpose:

• Keep a SAGEConv heterogeneous encoder.
• Switch to BPR ranking loss.
• Use cosine similarity for prediction.
• Force train/test overlap pairs to positive.
• Ensemble two seeds.

Outcome:

• Improved over the BCE/MLP version in recommendation-style validation.
• Generated:
  • submission_v2.csv
  • submission_v2_m1.csv
  • submission_v2_m2.csv

Conclusion:

• BPR was clearly better aligned with the recommendation task than BCE.
• However, SAGEConv transformations still underperformed vanilla LightGCN.

4. Vanilla LightGCN Main Line

Scripts:

• code/run_final.py
• code/run_lgcn_final.py

Purpose:

• Use a LightGCN-style architecture:
  • trainable author embeddings.
  • linear projection for paper features.
  • mean aggregation over heterogeneous relations.
  • no nonlinear activation inside propagation.
  • no learned transform per propagation layer.
  • average layer embeddings.
• Use BPR loss.
• Use hard negative sampling.
• Train multiple seeds on full training data.
• Sweep decision thresholds.

Important checkpoints:

• model_lgcn_s0.pt
• model_lgcn_s42.pt
• model_lgcn_s2024.pt
• model_lgcn_s10.pt
• model_lgcn_s100.pt

Generated submissions:

• sub_lgcn_t0.30.csv through sub_lgcn_t0.50.csv

Outcome:

• Validation F1 around 0.934 to 0.935.
• This became the strongest stable model family.

Conclusion:

• LightGCN's simplicity was a strength for this recommendation task.
• Nonlinear GNNs and extra feature transformations tended to overfit or degrade ranking quality.

5. Confirmed 6-Model LightGCN Ensemble

Script source:

• Five 256d checkpoints came from code/run_lgcn_final.py.
• The sixth 384d checkpoint and the 6-model CSV ensemble were produced in a previous inline Python run.
• This package includes code/generate_ens6_submission.py as the portable regeneration script.

Method:

• Ensemble members:
  • 5 x LightGCN, 256 dimensions, 4 layers.
  • 1 x LightGCN, 384 dimensions, 4 layers.
• Score:
  • cosine similarity between author and paper embeddings.
• Ensemble:
  • arithmetic mean of six score vectors.
• Post-processing:
  • train/test overlap pairs forced to label 1.
• Threshold:
  • 0.36 for confirmed submission.

Confirmed CSV:

• submissions/sub_ens6_t0.36.csv

Confirmed public leaderboard F1:

• 0.93044

Nearby threshold files included:

• sub_ens6_t0.35.csv
• sub_ens6_t0.37.csv
• sub_ens6_t0.38.csv

Conclusion:

• This is the best confirmed result in the preserved records.

6. L2-Normalized LightGCN Variant

Script: code/run_lgcn_v2.py

Purpose:

• Normalize embeddings during training and evaluation.
• Align training scores with cosine similarity.

Outcome:

• Underperformed vanilla LightGCN.

Conclusion:

• L2 normalization during training reduced useful ranking signal.
• The better setup was to train with BPR on dot products and evaluate with cosine similarity.

7. LightGBM Structural Feature Model

Script: code/run_graph_features.py

Purpose:

• Build a non-GNN baseline and potential ensemble component.
• Compute hand-engineered graph features for author-paper pairs:
  • author degree.
  • paper degree.
  • coauthor degree.
  • citation in/out degree.
  • preferential attachment.
  • popularity percentiles.
  • whether a coauthor read the paper.
  • author average paper embedding cosine similarity.
  • cold-start indicators.
• Train LightGBM.

Cached outputs:

• cached_scores/lgb_model.pkl
• cached_scores/lgb_v2_model.pkl
• cached_scores/test_lgb_scores.npy
• cached_scores/test_lgb_v2_scores.npy

Outcome:

• Validation F1 around 0.897 in the preserved notes.
• Useful as a baseline and ensemble attempt, but not stronger than LightGCN.

Conclusion:

• Structural features are good for reporting and comparison, but the GNN captured the main signal better.

8. BPR/MF-Style Baseline

Cached outputs:

• cached_scores/test_bpr_cos.npy
• cached_scores/test_bpr_dot.npy

Purpose:

• Test a more traditional recommendation baseline.
• Provide a diverse signal for ensembles.

Outcome:

• Validation was around 0.90 in previous notes.
• Predictions were highly correlated with GNN scores, so ensemble gains were limited.

Conclusion:

• Useful baseline, but not a replacement for LightGCN.

9. GNN Architecture Search

Script: code/compare_gnn.py

Architectures compared:

• Vanilla LightGCN, 256d, 4 layers.
• Learnable layer weights.
• GAT aggregation.
• SAGE aggregation.
• Deep LightGCN.
• Wide LightGCN, 384d.
• Vanilla LightGCN, 5 layers.
• GAT with more heads/layers.

Preserved results from previous session:

Architecture	Validation F1
Vanilla LightGCN, 256d, 4L	about 0.9350
Learnable layer weights, 256d, 4L	about 0.9360
Wide LightGCN, 384d, 3L	about 0.9365
Wide LightGCN, 512d, 3L	about 0.9363
Deep LightGCN, 256d, 6L	about 0.9319
GAT aggregation	about 0.9331
SAGE aggregation	about 0.9185

Conclusion:

• Wider models gave tiny validation gains, but did not clearly improve public leaderboard behavior.
• Deep/nonlinear/attention/SAGE variants were not better.
• The final practical choice remained vanilla LightGCN.

10. Later Threshold and Force/No-Force Experiments

Generated after the confirmed 0.93044 score:

• sub_final3_t*.csv
• sub_noforce_t*.csv
• sub_forced_t*.csv

Purpose:

• Try more seeds and threshold sweeps.
• Test whether forcing known positives was helping.
• Search better thresholds based on public leaderboard feedback.

Included in this package:

• sub_final3_t0.35.csv
• sub_final3_t0.36.csv
• sub_final3_t0.37.csv
• sub_noforce_t0.34.csv
• sub_noforce_t0.36.csv
• sub_noforce_t0.38.csv
• sub_forced_t0.36.csv

Outcome:

• No preserved Kaggle scores are available for these later files.
• They should be treated as candidates, not confirmed improvements.

Practical Recommendation

For transferring and resuming work:

1. Start from submissions/sub_ens6_t0.36.csv as the confirmed baseline.
2. Recreate environment with env/environment-cs3319.yml.
3. Verify code/generate_ens6_submission.py runs on the new machine.
4. If more Kaggle submissions are available, tune thresholds around the final ensemble:
   • 0.35
   • 0.36
   • 0.37
   • 0.38
5. For the course report, present:
   • official baseline.
   • SAGE/BPR model.
   • LightGBM structural baseline.
   • LightGCN final model.
   • architecture ablation from compare_gnn.py.