| [mtq-eval] loading KG from ../data/MultiTQ/MultiTQ/kg |
| [mtq-eval] 461,329 triples |
| [mtq-eval] building retriever (k=15, hops=2) |
| [mtq-eval] loading questions from ../data/MultiTQ/MultiTQ/questions/test.json |
| [mtq-eval] stratified subset: 1,496 of 54,584 |
| [mtq-eval] pre-retrieving evidence per question |
| [mtq-eval] 1,496/1,496 got >=1 triple |
| [mtq-eval] loading policy from ../checkpoints/sft-multitq-mistral/final |
|
Loading weights: 0%| | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 5607.05it/s] |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [mtq-eval] generating predictions (bs=16) |
| [mtq-eval] progress: 16/1496 |
| [mtq-eval] progress: 416/1496 |
| [mtq-eval] progress: 816/1496 |
| [mtq-eval] progress: 1216/1496 |
|
|
| [mtq-eval] wrote ../outputs/eval/multitq-mistral-sft.json |
| [mtq-eval] OVERALL: n=1496 EM=0.337 F1=0.368 |
| [mtq-eval] by qtype: |
| after_first: n=187 EM=0.160 |
| before_after: n=187 EM=0.545 |
| before_last: n=187 EM=0.182 |
| equal: n=374 EM=0.524 |
| equal_multi: n=187 EM=0.262 |
| first_last: n=374 EM=0.249 |
| [mtq-eval] by answer_type: |
| entity: n=1122 EM=0.362 |
| time: n=374 EM=0.262 |
|
|
