=== evaluating v2-sft-baseline ===
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/sft/final

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6777.58it/s]
[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/v2-sft-baseline.json
[eval] OVERALL: n=1743  EM=0.305  F1=0.366
[eval] by complexity:
  1hop: n=800  EM=0.328  F1=0.411
  2hop: n=800  EM=0.246  F1=0.283
  3plus: n=143  EM=0.510  F1=0.579
[eval] by operator:
  before_after: n=517  EM=0.257  F1=0.305
  interval: n=163  EM=0.994  F1=0.994
  point_in_time: n=535  EM=0.200  F1=0.284
  sequence: n=528  EM=0.246  F1=0.313
=== evaluating v2-grpo-9000 ===
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/grpo/checkpoint-9000

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6308.96it/s]
[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/v2-grpo-9000.json
[eval] OVERALL: n=1743  EM=0.364  F1=0.418
[eval] by complexity:
  1hop: n=800  EM=0.383  F1=0.450
  2hop: n=800  EM=0.304  F1=0.340
  3plus: n=143  EM=0.594  F1=0.670
[eval] by operator:
  before_after: n=517  EM=0.335  F1=0.381
  interval: n=163  EM=1.000  F1=1.000
  point_in_time: n=535  EM=0.250  F1=0.318
  sequence: n=528  EM=0.311  F1=0.376
=== evaluating v2-grpo-9500 ===
[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/grpo/checkpoint-9500

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6631.69it/s]
[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/v2-grpo-9500.json
[eval] OVERALL: n=1743  EM=0.366  F1=0.420
[eval] by complexity:
  1hop: n=800  EM=0.384  F1=0.455
  2hop: n=800  EM=0.306  F1=0.341
  3plus: n=143  EM=0.601  F1=0.670
[eval] by operator:
  before_after: n=517  EM=0.340  F1=0.386
  interval: n=163  EM=1.000  F1=1.000
  point_in_time: n=535  EM=0.254  F1=0.319
  sequence: n=528  EM=0.309  F1=0.377
=== evaluating v2-grpo-10000 ===
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/grpo/checkpoint-10000

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6324.96it/s]
[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/v2-grpo-10000.json
[eval] OVERALL: n=1743  EM=0.364  F1=0.420
[eval] by complexity:
  1hop: n=800  EM=0.381  F1=0.453
  2hop: n=800  EM=0.305  F1=0.341
  3plus: n=143  EM=0.601  F1=0.670
[eval] by operator:
  before_after: n=517  EM=0.335  F1=0.382
  interval: n=163  EM=1.000  F1=1.000
  point_in_time: n=535  EM=0.252  F1=0.320
  sequence: n=528  EM=0.311  F1=0.378
=== all evals done ===
-rw-rw-r-- 1 ubuntu ubuntu 562510 Apr 25 09:28 ../outputs/eval/v2-grpo-10000.json
-rw-rw-r-- 1 ubuntu ubuntu 561712 Apr 25 09:20 ../outputs/eval/v2-grpo-9000.json
-rw-rw-r-- 1 ubuntu ubuntu 562262 Apr 25 09:24 ../outputs/eval/v2-grpo-9500.json
-rw-rw-r-- 1 ubuntu ubuntu 566615 Apr 25 09:16 ../outputs/eval/v2-sft-baseline.json