reports/final_report.md

# Final Report: Citation-Aware High-Order Graph Recommendation

## Abstract

This project solves an academic paper recommendation task as author-paper link prediction on a
heterogeneous academic graph. The final system combines LightGCN collaborative filtering, explicit
graph/meta-path features, content features from `feature.pkl`, BPR-MF scores, DeepWalk / Node2Vec
random-walk embeddings, and a new citation-aware high-order propagation feature family. The best
confirmed public leaderboard score is **0.96626 F1**, achieved by
`submission_rich_rw7_highorder_directed_r0.500000.csv`.

## Data

The official data includes:

- `bipartite_train_ann.txt`: author-paper training positives.
- `bipartite_test_ann.txt`: author-paper pairs to predict.
- `author_file_ann.txt`: author-author collaboration edges.
- `paper_file_ann.txt`: paper-paper citation edges.
- `feature.pkl`: 512-dimensional paper content features.

The graph has 6,611 authors and about 79,937 papers. The test set contains 2,047,262 author-paper
pairs.

## Baseline And Early Models

The initial notebook-style heterogeneous GNN baseline reached validation F1 around 0.885. Several
variants were tried:

- SAGEConv heterogeneous GNN with MLP decoder.
- BPR ranking loss.
- LightGBM structural feature baselines.
- BPR-MF recommender baselines.
- Multiple LightGCN variants.

The first stable confirmed public result was a 6-model LightGCN ensemble:

```text
submissions/sub_ens6_t0.36.csv
public F1 = 0.93044
```

This ensemble averaged cosine scores from six LightGCN checkpoints and forced known train/test
overlap positives to 1.

## First Major Breakthrough: Feature Stacking

The first large improvement came from moving beyond pure LightGCN scores. The model stacked:

- LightGCN score and rank features.
- Author degree and paper degree.
- Coauthor evidence.
- Citation in/out degree.
- Author-history and candidate-paper citation overlaps.
- Meta-path counts such as A-A-P, A-P-P, and A-P-A-P.
- Content similarity features.
- BPR-MF score features.

The second-stage model was LightGBM with OOF validation. This pushed public performance to about
0.95996 with the content + BPR-MF stacker.

## Second Major Breakthrough: DeepWalk / Node2Vec

The next improvement came from random-walk graph embedding score sources. DeepWalk and Node2Vec
were trained on mixed academic graphs using author-paper, paper-paper citation, and author-author
coauthor edges. For each author-paper pair, the model constructed:

- dot product.
- cosine similarity.
- global rank.
- author-wise rank / percentile.

Adding DeepWalk and Node2Vec to the content + BPR-MF stacker improved public F1 to about 0.96252.
Further systematic random-walk experiments showed that higher-dimensional DeepWalk and longer walks
improved validation, but larger random-walk ensembles began to overfit the seed202 validation split.

## Third Major Breakthrough: High-Order Citation Propagation

The final and most important innovation was explicit high-order citation propagation. Instead of
training more random-walk embeddings, we computed deterministic propagation features over typed
meta-paths.

Let:

- `R` be the row-normalized author-paper interaction matrix.
- `C` be the row-normalized paper-paper citation matrix.
- `S` be the row-normalized author-author coauthor matrix.

Author-history citation propagation is:

```text
H_k = R C^k
```

For a candidate pair `(a, p)`, `H_k[a, p]` measures whether candidate paper `p` is reachable from
author `a`'s historical papers through k citation steps.

Coauthor-based propagation is:

```text
G_k = S R C^k
```

This captures whether candidate paper `p` is reachable from the historical papers of author `a`'s
collaborators.

The final version uses three citation directions:

- forward citation.
- backward citation.
- undirected citation.

It also includes popularity-normalized scores:

```text
propagation_score / log(1 + paper_degree + citation_degree)
```

This reduces the tendency to over-score globally popular papers.

### Validation Results

| Stage | Validation F1 | AUC |
|---|---:|---:|
| rich content + 7 random-walk blocks | 0.964947 | 0.994555 |
| + undirected high-order propagation | 0.966556 | 0.994890 |
| + directed high-order propagation | **0.966874** | **0.994918** |

The final public submission is:

```text
validation_runs/dynamic_seed202/high_order_graph_stack/submissions/submission_rich_rw7_highorder_directed_r0.500000.csv
public F1 = 0.96626
```

## Threshold Calibration Observation

An important finding is that validation-optimal probability thresholds do not transfer reliably to
test. For the final model, the validation-optimal threshold was:

```text
0.461730808
```

Applying this threshold directly to test produced a positive ratio of:

```text
0.524195
```

The public-best final submission instead used rank cutoff:

```text
rank top 50.0% -> positive
force known positives -> positive
```

This gives a stable test positive ratio of 0.500000.

The reason is that the validation set is an artificial 1:1 positive/negative split, while the test
candidate distribution is different. LightGBM scores are strong ranking scores but are not calibrated
probabilities under this distribution shift. Therefore, rank cutoff is more robust than transferring
the raw validation probability threshold.

## Final Files

Best final submission:

```text
validation_runs/dynamic_seed202/high_order_graph_stack/submissions/submission_rich_rw7_highorder_directed_r0.500000.csv
```

Final validation summary:

```text
validation_runs/dynamic_seed202/high_order_graph_stack/validation_summary.csv
```

Final test scores:

```text
validation_runs/dynamic_seed202/high_order_graph_stack/rich_rw7_highorder_directed_test_pred.npy
```

High-order feature code:

```text
code/high_order_graph_stack.py
```

## Conclusion

The final system improves by combining three complementary signals:

1. LightGCN-style collaborative filtering.
2. Random-walk graph embedding proximity.
3. Explicit citation-aware high-order meta-path propagation.

The high-order propagation features are the most distinctive final contribution. They preserve
interpretable path semantics such as `A-P-P^k` and `A-A-P-P^k`, separate citation directionality,
and reduce popularity bias through normalization. This turned out to be a real public leaderboard
improvement rather than only a validation gain.