| === evaluating v2-sft-baseline === |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/sft/final |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6777.58it/s] |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/v2-sft-baseline.json |
| [eval] OVERALL: n=1743 EM=0.305 F1=0.366 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.328 F1=0.411 |
| 2hop: n=800 EM=0.246 F1=0.283 |
| 3plus: n=143 EM=0.510 F1=0.579 |
| [eval] by operator: |
| before_after: n=517 EM=0.257 F1=0.305 |
| interval: n=163 EM=0.994 F1=0.994 |
| point_in_time: n=535 EM=0.200 F1=0.284 |
| sequence: n=528 EM=0.246 F1=0.313 |
| === evaluating v2-grpo-9000 === |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/grpo/checkpoint-9000 |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6308.96it/s] |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/v2-grpo-9000.json |
| [eval] OVERALL: n=1743 EM=0.364 F1=0.418 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.383 F1=0.450 |
| 2hop: n=800 EM=0.304 F1=0.340 |
| 3plus: n=143 EM=0.594 F1=0.670 |
| [eval] by operator: |
| before_after: n=517 EM=0.335 F1=0.381 |
| interval: n=163 EM=1.000 F1=1.000 |
| point_in_time: n=535 EM=0.250 F1=0.318 |
| sequence: n=528 EM=0.311 F1=0.376 |
| === evaluating v2-grpo-9500 === |
| [eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/grpo/checkpoint-9500 |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6631.69it/s] |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/v2-grpo-9500.json |
| [eval] OVERALL: n=1743 EM=0.366 F1=0.420 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.384 F1=0.455 |
| 2hop: n=800 EM=0.306 F1=0.341 |
| 3plus: n=143 EM=0.601 F1=0.670 |
| [eval] by operator: |
| before_after: n=517 EM=0.340 F1=0.386 |
| interval: n=163 EM=1.000 F1=1.000 |
| point_in_time: n=535 EM=0.254 F1=0.319 |
| sequence: n=528 EM=0.309 F1=0.377 |
| === evaluating v2-grpo-10000 === |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/grpo/checkpoint-10000 |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6324.96it/s] |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/v2-grpo-10000.json |
| [eval] OVERALL: n=1743 EM=0.364 F1=0.420 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.381 F1=0.453 |
| 2hop: n=800 EM=0.305 F1=0.341 |
| 3plus: n=143 EM=0.601 F1=0.670 |
| [eval] by operator: |
| before_after: n=517 EM=0.335 F1=0.382 |
| interval: n=163 EM=1.000 F1=1.000 |
| point_in_time: n=535 EM=0.252 F1=0.320 |
| sequence: n=528 EM=0.311 F1=0.378 |
| === all evals done === |
| -rw-rw-r-- 1 ubuntu ubuntu 562510 Apr 25 09:28 ../outputs/eval/v2-grpo-10000.json |
| -rw-rw-r-- 1 ubuntu ubuntu 561712 Apr 25 09:20 ../outputs/eval/v2-grpo-9000.json |
| -rw-rw-r-- 1 ubuntu ubuntu 562262 Apr 25 09:24 ../outputs/eval/v2-grpo-9500.json |
| -rw-rw-r-- 1 ubuntu ubuntu 566615 Apr 25 09:16 ../outputs/eval/v2-sft-baseline.json |
|
|
