| [mtq-eval] loading KG from ~/temporal-aware-graphrag/data/MultiTQ/MultiTQ/kg |
| [mtq-eval] 461,329 triples |
| [mtq-eval] building retriever (k=15, hops=2) |
| [mtq-eval] loading questions from ~/temporal-aware-graphrag/data/MultiTQ/MultiTQ/questions/test.json |
| [mtq-eval] stratified subset: 1,496 of 54,584 |
| [mtq-eval] pre-retrieving evidence per question |
| [mtq-eval] 1,496/1,496 got >=1 triple |
| [mtq-eval] loading policy from ../checkpoints/sft-multitq/final |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
|
Loading weights: 0%| | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 5259.26it/s] |
| [mtq-eval] generating predictions (bs=8) |
| [transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details. |
| [mtq-eval] progress: 8/1496 |
| [mtq-eval] progress: 208/1496 |
| [mtq-eval] progress: 408/1496 |
| [mtq-eval] progress: 608/1496 |
| [mtq-eval] progress: 808/1496 |
| [mtq-eval] progress: 1008/1496 |
| [mtq-eval] progress: 1208/1496 |
| [mtq-eval] progress: 1408/1496 |
|
|
| [mtq-eval] wrote ../outputs/eval/multitq-v3-sft.json |
| [mtq-eval] OVERALL: n=1496 EM=0.280 F1=0.315 |
| [mtq-eval] by qtype: |
| after_first: n=187 EM=0.091 |
| before_after: n=187 EM=0.481 |
| before_last: n=187 EM=0.118 |
| equal: n=374 EM=0.447 |
| equal_multi: n=187 EM=0.246 |
| first_last: n=374 EM=0.206 |
| [mtq-eval] by answer_type: |
| entity: n=1122 EM=0.289 |
| time: n=374 EM=0.254 |
|
|
