Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/sft-mistral/final

Loading weights:   0%|          | 0/291 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 291/291 [00:00<00:00, 4556.05it/s]
[eval] retrieval mode ON — building retriever (k=15)
[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
[eval]   retrieval done: 1743/1743 questions got at least one triple
[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
[eval] generating predictions (bs=4)
[eval]   progress: 4/1743
[eval]   progress: 104/1743
[eval]   progress: 204/1743
[eval]   progress: 304/1743
[eval]   progress: 404/1743
[eval]   progress: 504/1743
[eval]   progress: 604/1743
[eval]   progress: 704/1743
[eval]   progress: 804/1743
[eval]   progress: 904/1743
[eval]   progress: 1004/1743
[eval]   progress: 1104/1743
[eval]   progress: 1204/1743
[eval]   progress: 1304/1743
[eval]   progress: 1404/1743
[eval]   progress: 1504/1743
[eval]   progress: 1604/1743
[eval]   progress: 1704/1743

[eval] wrote ../outputs/eval/mistral-sft.json
[eval] OVERALL: n=1743  EM=0.943  F1=0.276
[eval] by complexity:
  1hop: n=800  EM=0.998  F1=0.343
  2hop: n=800  EM=0.945  F1=0.219
  3plus: n=143  EM=0.622  F1=0.216
[eval] by operator:
  before_after: n=517  EM=0.954  F1=0.297
  interval: n=163  EM=1.000  F1=0.138
  point_in_time: n=535  EM=0.929  F1=0.299
  sequence: n=528  EM=0.928  F1=0.275
9
  sequence: n=528  EM=0.928  F1=0.275

[2/2] Post-extracting Answer: spans...
[extract] wrote ../outputs/eval/mistral-sft.extracted.json
[extract] 1583/1743 predictions had an 'Answer:' marker (90.8%)
[extract] OVERALL: n=1743  EM=0.8910  F1=0.8677
[extract] by complexity:
  1hop: n=800  EM=0.9938  F1=0.9940
  2hop: n=800  EM=0.8387  F1=0.8530
  3plus: n=143  EM=0.6084  F1=0.2426
[extract] by operator:
  before_after: n=517  EM=0.9110  F1=0.8984
  interval: n=163  EM=0.8344  F1=0.7649
  point_in_time: n=535  EM=0.8916  F1=0.8795
  sequence: n=528  EM=0.8883  F1=0.8573

Done. Headline: ../outputs/eval/mistral-sft.extracted.json