| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test) |
| [eval] 1743 questions |
| [eval] loading policy from ../checkpoints/sft-mistral/final |
|
Loading weights: 0%| | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 4556.05it/s] |
| [eval] retrieval mode ON — building retriever (k=15) |
| [Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv |
| [eval] retrieval done: 1743/1743 questions got at least one triple |
| [eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv |
| [Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv |
| [eval] generating predictions (bs=4) |
| [eval] progress: 4/1743 |
| [eval] progress: 104/1743 |
| [eval] progress: 204/1743 |
| [eval] progress: 304/1743 |
| [eval] progress: 404/1743 |
| [eval] progress: 504/1743 |
| [eval] progress: 604/1743 |
| [eval] progress: 704/1743 |
| [eval] progress: 804/1743 |
| [eval] progress: 904/1743 |
| [eval] progress: 1004/1743 |
| [eval] progress: 1104/1743 |
| [eval] progress: 1204/1743 |
| [eval] progress: 1304/1743 |
| [eval] progress: 1404/1743 |
| [eval] progress: 1504/1743 |
| [eval] progress: 1604/1743 |
| [eval] progress: 1704/1743 |
|
|
| [eval] wrote ../outputs/eval/mistral-sft.json |
| [eval] OVERALL: n=1743 EM=0.943 F1=0.276 |
| [eval] by complexity: |
| 1hop: n=800 EM=0.998 F1=0.343 |
| 2hop: n=800 EM=0.945 F1=0.219 |
| 3plus: n=143 EM=0.622 F1=0.216 |
| [eval] by operator: |
| before_after: n=517 EM=0.954 F1=0.297 |
| interval: n=163 EM=1.000 F1=0.138 |
| point_in_time: n=535 EM=0.929 F1=0.299 |
| sequence: n=528 EM=0.928 F1=0.275 |
| 9 |
| sequence: n=528 EM=0.928 F1=0.275 |
|
|
| [2/2] Post-extracting Answer: spans... |
| [extract] wrote ../outputs/eval/mistral-sft.extracted.json |
| [extract] 1583/1743 predictions had an 'Answer:' marker (90.8%) |
| [extract] OVERALL: n=1743 EM=0.8910 F1=0.8677 |
| [extract] by complexity: |
| 1hop: n=800 EM=0.9938 F1=0.9940 |
| 2hop: n=800 EM=0.8387 F1=0.8530 |
| 3plus: n=143 EM=0.6084 F1=0.2426 |
| [extract] by operator: |
| before_after: n=517 EM=0.9110 F1=0.8984 |
| interval: n=163 EM=0.8344 F1=0.7649 |
| point_in_time: n=535 EM=0.8916 F1=0.8795 |
| sequence: n=528 EM=0.8883 F1=0.8573 |
|
|
| Done. Headline: ../outputs/eval/mistral-sft.extracted.json |
|
|
