| [shuf] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl | |
| [shuf] 1743 questions | |
| [shuf] loading policy from ../checkpoints/sft-v3/final | |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. | |
| Loading weights: 0%| | 0/399 [00:00<?, ?it/s] Loading weights: 100%|██████████| 399/399 [00:00<00:00, 7072.28it/s] | |
| [shuf] building retriever (k=15, hops=2) | |
| [Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv | |
| [shuf] retrieving true evidence per question | |
| [shuf] generating predictions (bs=16) | |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. | |
| [shuf] progress: 16/1743 | |
| [shuf] progress: 416/1743 | |
| [shuf] progress: 816/1743 | |
| [shuf] progress: 1216/1743 | |
| [shuf] progress: 1616/1743 | |
| [shuf] wrote ../outputs/eval/v3-sft-shuffled.json | |
| [shuf] OVERALL: n=1743 EM=0.196 F1=0.079 | |