[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/sft-llama31/final

Loading weights:   0%|          | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 5407.50it/s]
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
[eval] retrieval mode ON — building retriever (k=15)
[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
[eval]   retrieval done: 1743/1743 questions got at least one triple
[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/llama31-sft.json
[eval] OVERALL: n=1743  EM=0.944  F1=0.273
[eval] by complexity:
  1hop: n=800  EM=0.990  F1=0.340
  2hop: n=800  EM=0.932  F1=0.217
  3plus: n=143  EM=0.748  F1=0.217
[eval] by operator:
  before_after: n=517  EM=0.946  F1=0.294
  interval: n=163  EM=1.000  F1=0.138
  point_in_time: n=535  EM=0.940  F1=0.299
  sequence: n=528  EM=0.928  F1=0.269
9
  sequence: n=528  EM=0.928  F1=0.269

[2/2] Post-extracting Answer: spans...
[extract] wrote ../outputs/eval/llama31-sft.extracted.json
[extract] 1707/1743 predictions had an 'Answer:' marker (97.9%)
[extract] OVERALL: n=1743  EM=0.9225  F1=0.9224
[extract] by complexity:
  1hop: n=800  EM=0.9725  F1=0.9769
  2hop: n=800  EM=0.9175  F1=0.9081
  3plus: n=143  EM=0.6713  F1=0.6975
[extract] by operator:
  before_after: n=517  EM=0.9304  F1=0.9343
  interval: n=163  EM=0.9755  F1=0.9419
  point_in_time: n=535  EM=0.9346  F1=0.9449
  sequence: n=528  EM=0.8864  F1=0.8819

Done. Headline numbers in:
  ../outputs/eval/llama31-sft.extracted.json  (post-extracted EM/F1)
  ../outputs/eval/llama31-sft.json            (raw predictions)