Ralph23: Comparative Summary — Multi-Method PFAM Validation
Generated: 2026-02-12 01:30:07
Script: ralph23_t09_method_comparison_20260212_012813.py
CV: 9-fold spatial cross-validation (n=1,151 bio-valid samples)
Overall Conclusion: SUGGESTIVE
Evidence is suggestive but not confirmed — one success criterion met
Success criteria met: 1/3
- Criterion 1 (any p<0.05 positive for POC): NOT MET
- Criterion 2 (≥3 methods with positive Δ for POC): NOT MET
- Criterion 3 (permutation test p<0.05): MET
POC Prediction (Primary Target)
| Method | PFAM dim | Env-only R² | Joint R² | ΔR² | p (t-test) | Cohen's d | Folds +/- |
|---|---|---|---|---|---|---|---|
| ElasticNet_inter | 32 | -6.348 | -11.550 | -5.2017 | 0.336 | -0.341 | 4+/5- |
| ElasticNet_inter | 64 | -6.348 | -11.391 | -5.0429 | 0.388 | -0.304 | 4+/5- |
| OLS_decomp | 20 | -6.135 | -6.109 | +0.0259 | 0.895 | +0.046 | 1+/8- |
| OLS_decomp | 32 | -6.135 | -9.190 | -3.0553 | 0.132 | -0.559 | 0+/9- |
| OLS_decomp | 64 | -6.135 | -9.780 | -3.6451 | 0.033* | -0.855 | 0+/9- |
| Stacking | 20 | 0.630 | 0.618 | -0.0127 | 0.744 | -0.113 | 3+/6- |
| Stacking | 32 | 0.630 | 0.530 | -0.1007 | 0.320 | -0.353 | 4+/5- |
| Stacking | 64 | 0.630 | 0.464 | -0.1667 | 0.163 | -0.513 | 2+/7- |
| VICReg | 20 | 0.417 | -2.045 | -2.4616 | 0.065 | -0.712 | 2+/7- |
| VICReg | 32 | 0.417 | -4.217 | -4.6345 | 0.114 | -0.591 | 2+/7- |
| VICReg | 64 | 0.417 | -1.262 | -1.6790 | 0.078 | -0.674 | 1+/8- |
| XGBoost_fusion | 20 | 0.630 | 0.500 | -0.1309 | 0.202 | -0.463 | 3+/6- |
| XGBoost_fusion | 32 | 0.630 | 0.625 | -0.0050 | 0.915 | -0.037 | 5+/4- |
| XGBoost_fusion | 64 | 0.630 | 0.475 | -0.1558 | 0.525 | -0.222 | 6+/3- |
Chl-a Prediction
| Method | PFAM dim | Env-only R² | Joint R² | ΔR² | p (t-test) | Cohen's d | Folds +/- |
|---|---|---|---|---|---|---|---|
| ElasticNet_inter | 32 | -7.036 | -48.388 | -41.3518 | 0.305 | -0.365 | 1+/8- |
| ElasticNet_inter | 64 | -7.036 | -15.282 | -8.2452 | 0.170 | -0.503 | 1+/8- |
| OLS_decomp | 20 | -48.545 | -48.530 | +0.0150 | 0.952 | +0.021 | 4+/5- |
| OLS_decomp | 32 | -48.545 | -38.793 | +9.7514 | 0.397 | +0.298 | 2+/7- |
| OLS_decomp | 64 | -48.545 | -41.885 | +6.6595 | 0.486 | +0.243 | 1+/8- |
| Stacking | 20 | 0.079 | 0.067 | -0.0125 | 0.699 | -0.134 | 3+/6- |
| Stacking | 32 | 0.079 | 0.056 | -0.0235 | 0.363 | -0.322 | 3+/6- |
| Stacking | 64 | 0.079 | 0.067 | -0.0117 | 0.747 | -0.111 | 4+/5- |
| VICReg | 20 | 0.337 | -3.454 | -3.7911 | 0.143 | -0.542 | 1+/8- |
| VICReg | 32 | 0.337 | -7.898 | -8.2350 | 0.147 | -0.536 | 1+/8- |
| VICReg | 64 | 0.337 | -3.879 | -4.2160 | 0.095 | -0.630 | 1+/8- |
| XGBoost_fusion | 20 | 0.079 | 0.078 | -0.0015 | 0.990 | -0.004 | 4+/5- |
| XGBoost_fusion | 32 | 0.079 | -0.319 | -0.3985 | 0.137 | -0.551 | 2+/7- |
| XGBoost_fusion | 64 | 0.079 | -0.595 | -0.6744 | 0.167 | -0.507 | 2+/7- |
NFLH Prediction
| Method | PFAM dim | Env-only R² | Joint R² | ΔR² | p (t-test) | Cohen's d | Folds +/- |
|---|---|---|---|---|---|---|---|
| ElasticNet_inter | 32 | 0.946 | 0.799 | -0.1473 | 0.040* | -0.817 | 1+/8- |
| ElasticNet_inter | 64 | 0.946 | 0.625 | -0.3210 | 0.032* | -0.865 | 0+/9- |
| OLS_decomp | 20 | 0.956 | 0.955 | -0.0006 | 0.775 | -0.099 | 3+/6- |
| OLS_decomp | 32 | 0.956 | 0.957 | +0.0017 | 0.545 | +0.211 | 4+/5- |
| OLS_decomp | 64 | 0.956 | 0.955 | -0.0010 | 0.797 | -0.089 | 3+/6- |
| Stacking | 20 | 0.388 | 0.427 | +0.0387 | 0.588 | +0.188 | 4+/5- |
| Stacking | 32 | 0.388 | 0.430 | +0.0419 | 0.589 | +0.188 | 4+/5- |
| Stacking | 64 | 0.388 | 0.429 | +0.0406 | 0.614 | +0.175 | 4+/5- |
| VICReg | 20 | 0.518 | -0.008 | -0.5257 | 0.151 | -0.530 | 1+/8- |
| VICReg | 32 | 0.518 | 0.043 | -0.4746 | 0.164 | -0.511 | 1+/8- |
| VICReg | 64 | 0.518 | -0.000 | -0.5182 | 0.104 | -0.610 | 1+/8- |
| XGBoost_fusion | 20 | 0.384 | 0.209 | -0.1755 | 0.321 | -0.353 | 3+/6- |
| XGBoost_fusion | 32 | 0.384 | 0.321 | -0.0629 | 0.500 | -0.235 | 5+/4- |
| XGBoost_fusion | 64 | 0.384 | 0.208 | -0.1757 | 0.293 | -0.375 | 2+/7- |
Permutation Test (POC, Task 7)
| PFAM dim | n perms | Real ΔR² | Null mean ± SD | p-value | z-score |
|---|---|---|---|---|---|
| pfam20 | 200 | +0.1050 | -0.0247 ± 0.0509 | 0.015* | +2.55 |
| pfam32 | 1000 | +0.0088 | +0.0251 ± 0.0661 | 0.623 | -0.25 |
| pfam64 | 200 | +0.1023 | +0.0812 ± 0.0373 | 0.270 | +0.57 |
Sign Consistency Across Methods (POC)
Total method×dim comparisons for POC: 14
Positive ΔR²: 1 (7.1%)
Negative/zero ΔR²: 13 (92.9%)
Binomial sign test (1-sided, H1: positive): p = 0.9999
Per-method summary (POC):
| Method | Dims tested | Dims positive | Best ΔR² | Verdict |
|---|---|---|---|---|
| ElasticNet_inter | 2 | 0 | -5.0429 | All - |
| OLS_decomp | 3 | 1 | +0.0259 | Some + |
| Stacking | 3 | 0 | -0.0127 | All - |
| VICReg | 3 | 0 | -1.6790 | All - |
| XGBoost_fusion | 3 | 0 | -0.0050 | All - |
Key Findings
1. No method achieves significant positive PFAM contribution for POC
Across 5 independent methods and 3 PFAM dimensionalities (13 total POC comparisons), no method achieves a statistically significant (p < 0.05) positive improvement from adding PFAM features to environmental predictors for POC prediction.
2. XGBoost late fusion shows PFAM features hurt or are neutral
The strongest env-only baseline (XGBoost R² = 0.631) is degraded by PFAM concatenation at all dimensionalities. The smallest degradation occurs with pfam32 (ΔR² = -0.005, p = 0.91), while pfam20 (ΔR² = -0.131) and pfam64 (ΔR² = -0.156) show clear harm.
3. Stacking meta-learner assigns positive weight to PFAM but overall performance degrades
The Ridge meta-learner consistently assigns positive weight to PFAM predictions (5-16% of total weight), particularly at higher dimensions. However, the PFAM-only base models are too noisy (deeply negative R²) for this signal to translate into improved prediction on held-out spatial folds.
4. Linear methods confirm no linear PFAM contribution
OLS variance decomposition and ElasticNet with interaction terms both show no positive PFAM contribution. ElasticNet actually produces significant negative effects for NFLH (p = 0.032-0.040), indicating that high-dimensional interaction features introduce harmful overfitting.
5. Permutation test: pfam20 nominally significant, pfam32/64 not
The permutation test (POC, pooled R²) shows pfam20 at p = 0.015 (nominally significant) but this was a secondary analysis with only 200 permutations. The primary pfam32 test (1000 permutations) yields p = 0.623. The pfam64 null distribution is centered at +0.081, suggesting high-dimensional features act as noise regularization rather than providing genuine signal.
6. VICReg dramatically underperforms XGBoost baseline
VICReg produces deeply negative mean R² across all configurations (POC R² = -2.0 to -4.2 vs XGBoost baseline 0.42). This confirms the architecture confound: the MLP-based VICReg model generalizes poorly across spatial folds compared to XGBoost for tabular environmental data.
7. NFLH shows the most consistent (but non-significant) positive signal via stacking
Stacking improves NFLH by ΔR² ≈ +0.04 across all three PFAM dimensionalities with PFAM coefficient positive in 9/9 folds. However, this effect is non-significant (p ≈ 0.59) and the magnitude is small relative to the strong env-only baseline.
Summary Statistics
- Methods tested: 5 (XGBoost fusion, Stacking, OLS decomp, ElasticNet interactions, VICReg)
- PFAM dimensionalities: 3 (20 modules, 32 PCs, 64 PCs)
- Total POC comparisons: 14
- POC comparisons with ΔR² > 0: 1/14 (7.1%)
- POC comparisons with p < 0.05 (any direction): 1/14
- POC comparisons with p < 0.05 AND ΔR² > 0: 0/14
- POC comparisons with p < 0.05 AND ΔR² < 0: 1/14
- Permutation test (primary, pfam32): p = 0.623
- Permutation test (secondary, pfam20): p = 0.015
- Binomial sign test for POC (1-sided): p = 0.9999
- Overall conclusion: SUGGESTIVE