lookback-lens/extensions/phi-4/predefined.log

======== Loading data from /workspace/generalizable-interpretability/artifacts/lookback-lens/inputs/anno-nq-7b.jsonl and /workspace/generalizable-interpretability/artifacts/lookback-lens/extensions/phi-4/lookback-ratio-nq.pt...
A trivial baseline: if higher lookback ratio means less hallucination.
Baseline AUROC: 0.376940530
Train AUROC of the classifier: 0.984272984
Test AUROC of the classifier: 0.901579162
Top-10 important features:
lookback-mean-L33-H2: -1.446306408
lookback-mean-L16-H23: -1.348257371
lookback-mean-L22-H29: 1.223903212
lookback-mean-L25-H27: 1.215140050
lookback-mean-L33-H30: -1.161465338
lookback-mean-L25-H15: -1.145237004
lookback-mean-L31-H12: 1.125003287
lookback-mean-L29-H35: -1.115291894
lookback-mean-L25-H5: 1.106424866
lookback-mean-L24-H20: -1.082534317
Train AUROC of the classifier: 0.983784088
Test AUROC of the classifier: 0.906376732
Top-10 important features:
lookback-mean-L19-H5: 1.623896451
lookback-mean-L27-H17: 1.392790224
lookback-mean-L27-H11: 1.323177604
lookback-mean-L25-H15: -1.280765250
lookback-mean-L6-H39: 1.272454927
lookback-mean-L33-H2: -1.187557396
lookback-mean-L24-H4: 1.154348591
lookback-mean-L22-H21: -1.146721635
lookback-mean-L21-H16: 1.128777123
lookback-mean-L34-H27: 1.123702872
======== Transfer to data from /workspace/generalizable-interpretability/artifacts/lookback-lens/inputs/anno-cnndm-7b.jsonl and /workspace/generalizable-interpretability/artifacts/lookback-lens/extensions/phi-4/lookback-ratio-cnndm.pt...
A trivial baseline: if higher lookback ratio means less hallucination.
Transfer Baseline AUROC: 0.440770517
Transfer AUROC of the classifier: 0.871386665
======== Loading data from /workspace/generalizable-interpretability/artifacts/lookback-lens/inputs/anno-cnndm-7b.jsonl and /workspace/generalizable-interpretability/artifacts/lookback-lens/extensions/phi-4/lookback-ratio-cnndm.pt...
A trivial baseline: if higher lookback ratio means less hallucination.
Baseline AUROC: 0.440770517
Train AUROC of the classifier: 0.990813460
Test AUROC of the classifier: 0.884228612
Top-10 important features:
lookback-mean-L33-H21: -1.445000751
lookback-mean-L29-H30: -1.249933483
lookback-mean-L30-H11: 1.125202441
lookback-mean-L33-H30: -1.051773286
lookback-mean-L28-H30: 1.040183283
lookback-mean-L32-H9: -0.980378540
lookback-mean-L26-H35: -0.972076249
lookback-mean-L3-H18: 0.957095169
lookback-mean-L25-H39: 0.912895748
lookback-mean-L30-H38: -0.848151462
Train AUROC of the classifier: 0.982959107
Test AUROC of the classifier: 0.904578328
Top-10 important features:
lookback-mean-L30-H38: -1.249216261
lookback-mean-L5-H12: 1.133405217
lookback-mean-L29-H35: -1.096084248
lookback-mean-L28-H39: -1.068296549
lookback-mean-L18-H31: -1.019666741
lookback-mean-L16-H23: -0.988021026
lookback-mean-L30-H5: 0.959250148
lookback-mean-L17-H33: 0.946749663
lookback-mean-L29-H30: -0.933067216
lookback-mean-L10-H17: 0.932735972
======== Transfer to data from /workspace/generalizable-interpretability/artifacts/lookback-lens/inputs/anno-nq-7b.jsonl and /workspace/generalizable-interpretability/artifacts/lookback-lens/extensions/phi-4/lookback-ratio-nq.pt...
A trivial baseline: if higher lookback ratio means less hallucination.
Transfer Baseline AUROC: 0.376940530
Transfer AUROC of the classifier: 0.830823083
======== Results:
                  , Train AUROC (on A), Test AUROC (on A), Transfer AUROC (on B)
A=nq-7b;B=cnndm-7b, 0.9840285357240282, 0.9039779471120246, 0.8713866645065729
A=cnndm-7b;B=nq-7b, 0.9868862837584778, 0.8944034697551608, 0.8308230833532361