| [eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/sft-v3/final |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6192.84it/s] |
| [eval] retrieval mode ON — building retriever (k=15) |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv |
| [eval] retrieval done: 1743/1743 questions got at least one triple |
| [eval] generating predictions (bs=16) |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [eval] progress: 16/1743 |
| [eval] progress: 416/1743 |
| [eval] progress: 816/1743 |
| [eval] progress: 1216/1743 |
| [eval] progress: 1616/1743 |
|
|
| [eval] wrote ../outputs/eval/v3-sft-baseline.json |
| [eval] OVERALL: n=1743 EM=0.903 F1=0.265 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.998 F1=0.343 |
| 2hop: n=800 EM=0.871 F1=0.200 |
| 3plus: n=143 EM=0.552 F1=0.194 |
| [eval] by operator: |
| before_after: n=517 EM=0.901 F1=0.283 |
| interval: n=163 EM=1.000 F1=0.138 |
| point_in_time: n=535 EM=0.908 F1=0.291 |
| sequence: n=528 EM=0.869 F1=0.261 |
|
|
