| [eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/sft-llama31/final |
|
Loading weights: 0%| | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 5407.50it/s] |
| [transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. |
| [eval] retrieval mode ON — building retriever (k=15) |
| [Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv |
| [eval] retrieval done: 1743/1743 questions got at least one triple |
| [eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/llama31-sft.json |
| [eval] OVERALL: n=1743 EM=0.944 F1=0.273 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.990 F1=0.340 |
| 2hop: n=800 EM=0.932 F1=0.217 |
| 3plus: n=143 EM=0.748 F1=0.217 |
| [eval] by operator: |
| before_after: n=517 EM=0.946 F1=0.294 |
| interval: n=163 EM=1.000 F1=0.138 |
| point_in_time: n=535 EM=0.940 F1=0.299 |
| sequence: n=528 EM=0.928 F1=0.269 |
| 9 |
| sequence: n=528 EM=0.928 F1=0.269 |
|
|
| [2/2] Post-extracting Answer: spans... |
| [extract] wrote ../outputs/eval/llama31-sft.extracted.json |
| [extract] 1707/1743 predictions had an 'Answer:' marker (97.9%) |
| [extract] OVERALL: n=1743 EM=0.9225 F1=0.9224 |
| [extract] by complexity: |
| 1hop: n=800 EM=0.9725 F1=0.9769 |
| 2hop: n=800 EM=0.9175 F1=0.9081 |
| 3plus: n=143 EM=0.6713 F1=0.6975 |
| [extract] by operator: |
| before_after: n=517 EM=0.9304 F1=0.9343 |
| interval: n=163 EM=0.9755 F1=0.9419 |
| point_in_time: n=535 EM=0.9346 F1=0.9449 |
| sequence: n=528 EM=0.8864 F1=0.8819 |
|
|
| Done. Headline numbers in: |
| ../outputs/eval/llama31-sft.extracted.json (post-extracted EM/F1) |
| ../outputs/eval/llama31-sft.json (raw predictions) |
|
|
