| [mtq-eval] loading KG from ../data/MultiTQ/MultiTQ/kg |
| [mtq-eval] 461,329 triples |
| [mtq-eval] building retriever (k=15, hops=2) |
| [mtq-eval] loading questions from ../data/MultiTQ/MultiTQ/questions/test.json |
| [mtq-eval] stratified subset: 1,496 of 54,584 |
| [mtq-eval] pre-retrieving evidence per question |
| [mtq-eval] 1,496/1,496 got >=1 triple |
| [mtq-eval] loading policy from ../checkpoints/sft-multitq-llama/final |
|
Loading weights: 0%| | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 5934.87it/s] |
| [transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. |
| [mtq-eval] generating predictions (bs=16) |
| [mtq-eval] progress: 16/1496 |
| [mtq-eval] progress: 416/1496 |
| [mtq-eval] progress: 816/1496 |
| [mtq-eval] progress: 1216/1496 |
|
|
| [mtq-eval] wrote ../outputs/eval/multitq-llama-sft.json |
| [mtq-eval] OVERALL: n=1496 EM=0.308 F1=0.346 |
| [mtq-eval] by qtype: |
| after_first: n=187 EM=0.134 |
| before_after: n=187 EM=0.497 |
| before_last: n=187 EM=0.150 |
| equal: n=374 EM=0.505 |
| equal_multi: n=187 EM=0.267 |
| first_last: n=374 EM=0.203 |
| [mtq-eval] by answer_type: |
| entity: n=1122 EM=0.327 |
| time: n=374 EM=0.251 |
|
|
