[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
[eval]   1743 questions
[eval] loading policy from ../checkpoints/sft-v3/final
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Loading weights:   0%|          | 0/399 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 399/399 [00:00<00:00, 6192.84it/s]
[eval] retrieval mode ON — building retriever (k=15)
[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
[eval]   retrieval done: 1743/1743 questions got at least one triple
[eval] generating predictions (bs=16)
[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
[eval]   progress: 16/1743
[eval]   progress: 416/1743
[eval]   progress: 816/1743
[eval]   progress: 1216/1743
[eval]   progress: 1616/1743

[eval] wrote ../outputs/eval/v3-sft-baseline.json
[eval] OVERALL: n=1743  EM=0.903  F1=0.265
[eval] by complexity:
  1hop: n=800  EM=0.998  F1=0.343
  2hop: n=800  EM=0.871  F1=0.200
  3plus: n=143  EM=0.552  F1=0.194
[eval] by operator:
  before_after: n=517  EM=0.901  F1=0.283
  interval: n=163  EM=1.000  F1=0.138
  point_in_time: n=535  EM=0.908  F1=0.291
  sequence: n=528  EM=0.869  F1=0.261