Add files using upload-large-folder tool

adapters/sft-llama31/README.md

@@ -0,0 +1,207 @@
+---
+base_model: meta-llama/Llama-3.1-8B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:meta-llama/Llama-3.1-8B-Instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

adapters/sft-llama31/adapter_config.json

@@ -0,0 +1,45 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "o_proj",
+    "v_proj",
+    "k_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapters/sft-llama31/chat_template.jinja

@@ -0,0 +1,109 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- set date_string = "26 Jul 2024" %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message + builtin tools #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if builtin_tools is defined or tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{%- if builtin_tools is defined %}
+    {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- "<|python_tag|>" + tool_call.name + ".call(" }}
+            {%- for arg_name, arg_val in tool_call.arguments | items %}
+                {{- arg_name + '="' + arg_val + '"' }}
+                {%- if not loop.last %}
+                    {{- ", " }}
+                {%- endif %}
+                {%- endfor %}
+            {{- ")" }}
+        {%- else  %}
+            {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+            {{- '{"name": "' + tool_call.name + '", ' }}
+            {{- '"parameters": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- "}" }}
+        {%- endif %}
+        {%- if builtin_tools is defined %}
+            {#- This means we're in ipython mode #}
+            {{- "<|eom_id|>" }}
+        {%- else %}
+            {{- "<|eot_id|>" }}
+        {%- endif %}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}

adapters/sft-llama31/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|eot_id|>",
+  "tokenizer_class": "TokenizersBackend"
+}

eval/v3-sft-baseline.judged.json

@@ -0,0 +1,184 @@
+{
+  "name": "v3-sft-baseline",
+  "policy_path": "../checkpoints/sft-v3/final",
+  "split": "test",
+  "n_samples": 1743,
+  "temperature": 0.0,
+  "max_new_tokens": 96,
+  "answer_extracted": true,
+  "n_with_answer_marker": 1738,
+  "judge_provider": "anthropic",
+  "judge_model": "claude-haiku-4-5-20251001",
+  "overall": {
+    "n": 10,
+    "em": 0.4,
+    "f1": 0.5404761904761906,
+    "judge_em": 0.6
+  },
+  "by_complexity": {
+    "3plus": {
+      "n": 10,
+      "em": 0.4,
+      "f1": 0.5404761904761906,
+      "judge_em": 0.6
+    }
+  },
+  "by_operator": {
+    "before_after": {
+      "n": 2,
+      "em": 0.5,
+      "f1": 0.5,
+      "judge_em": 0.5
+    },
+    "interval": {
+      "n": 1,
+      "em": 1.0,
+      "f1": 1.0,
+      "judge_em": 1.0
+    },
+    "point_in_time": {
+      "n": 1,
+      "em": 1.0,
+      "f1": 1.0,
+      "judge_em": 1.0
+    },
+    "sequence": {
+      "n": 6,
+      "em": 0.16666666666666666,
+      "f1": 0.40079365079365076,
+      "judge_em": 0.5
+    }
+  },
+  "per_example": [
+    {
+      "id": "q_006291",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 2017.0,
+      "gold": "chairperson",
+      "pred": "President of the Portuguese Football Federation",
+      "em": 0.0,
+      "f1": 0.0,
+      "pred_raw": "Hop 1: Jonas -> member of sports team -> S.L. Benfica (2017)\n Hop 2: S.L. Benfica -> chairperson -> Lu\u00eds Filipe Vieira (2017)\n Hop 3: Lu\u00eds Filipe Vieira -> position held -> President of the Portuguese Football Federation (2017)\n Answer: President of the Portuguese Football Federation",
+      "judge_em": 0,
+      "judge_reply": "NO"
+    },
+    {
+      "id": "q_007337",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 1907.0,
+      "gold": "Nicholas II of Russia",
+      "pred": "Nikolay II",
+      "em": 0.0,
+      "f1": 0.3333333333333333,
+      "pred_raw": "Hop 1: Ararat -> located in -> Erivan Governorate (1907)\n Hop 2: Erivan Governorate -> country -> Russian Empire (1907)\n Hop 3: Russian Empire -> head of state -> Nikolay II (1907)\n Answer: Nikolay II",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_000021",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 1959.0,
+      "gold": "President of Italy",
+      "pred": "President of the Italian Republic",
+      "em": 0.0,
+      "f1": 0.5,
+      "pred_raw": "Hop 1: Galvano Della Volpe -> country of citizenship -> Italy (1959)\n Hop 2: Italy -> head of state -> Antonio Segni (1959)\n Hop 3: Antonio Segni -> position held -> President of the Italian Republic (1959)\n Answer: President of the Italian Republic",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_007809",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 2011.0,
+      "gold": "Irina Bokova",
+      "pred": "David Elkind",
+      "em": 0.0,
+      "f1": 0.0,
+      "pred_raw": "Hop 1: Kach -> designated as terrorist by -> United States (2011)\n Hop 2: United States -> member of -> UNESCO (2011)\n Hop 3: UNESCO -> chairperson -> David Elkind (2011)\n Answer: David Elkind",
+      "judge_em": 0,
+      "judge_reply": "NO"
+    },
+    {
+      "id": "q_001979",
+      "complexity": "3plus",
+      "operator_type": "interval",
+      "t_query": 1998.0,
+      "gold": "1998",
+      "pred": "1998",
+      "em": 1.0,
+      "f1": 1.0,
+      "pred_raw": "Hop 1: Arnold J. Levine -> award received -> Paul Ehrlich and Ludwig Darmstaedter Prize (1998)\n Answer: 1998",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_005455",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 1982.0,
+      "gold": "Golden Globe Award for Best Actress \u2013 Motion Picture Drama",
+      "pred": "Academy Award for Best Actress",
+      "em": 0.0,
+      "f1": 0.5714285714285714,
+      "pred_raw": "Hop 1: On Golden Pond -> nominated for -> Academy Award for Best Actress (1982)\n Hop 2: Academy Award for Best Actress -> winner -> Meryl Streep (1982)\n Hop 3: Meryl Streep -> award received -> Academy Award for Best Actress (1982)\n Answer: Academy Award for Best Actress",
+      "judge_em": 0,
+      "judge_reply": "NO"
+    },
+    {
+      "id": "q_008005",
+      "complexity": "3plus",
+      "operator_type": "before_after",
+      "t_query": 1961.0,
+      "gold": "President of Italy",
+      "pred": "President of Italy",
+      "em": 1.0,
+      "f1": 1.0,
+      "pred_raw": "Hop 1: Urbano Lazzaro -> country of citizenship -> Italy (1961)\n Hop 2: Italy -> head of state -> Giuseppe Saragat (1961)\n Hop 3: Giuseppe Saragat -> position held -> President of Italy (1961)\n Answer: President of Italy",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_008690",
+      "complexity": "3plus",
+      "operator_type": "sequence",
+      "t_query": 1968.0,
+      "gold": "President of Italy",
+      "pred": "President of Italy",
+      "em": 1.0,
+      "f1": 1.0,
+      "pred_raw": "Hop 1: Bruno Pesaola -> country of citizenship -> Italy (1968)\n Hop 2: Italy -> head of state -> Giovanni Gronchi (1968)\n Hop 3: Giovanni Gronchi -> position held -> President of Italy (1968)\n Answer: President of Italy",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_008321",
+      "complexity": "3plus",
+      "operator_type": "point_in_time",
+      "t_query": 1976.0,
+      "gold": "Prime Minister of Italy",
+      "pred": "Prime Minister of Italy",
+      "em": 1.0,
+      "f1": 1.0,
+      "pred_raw": "Hop 1: Vito Lattanzio -> country of citizenship -> Italy (1976)\n Hop 2: Italy -> head of government -> Giovanni Spadolini (1976)\n Hop 3: Giovanni Spadolini -> position held -> Prime Minister of Italy (1976)\n Answer: Prime Minister of Italy",
+      "judge_em": 1,
+      "judge_reply": "YES"
+    },
+    {
+      "id": "q_007761",
+      "complexity": "3plus",
+      "operator_type": "before_after",
+      "t_query": 2002.0,
+      "gold": "Arista Records",
+      "pred": "Sony Music",
+      "em": 0.0,
+      "f1": 0.0,
+      "pred_raw": "Hop 1: Jennifer Love Hewitt -> unmarried partner -> John Mayer (2002)\n Hop 2: John Mayer -> nominated for -> Grammy Award for Best Male Pop Vocal Performance (2002)\n Hop 3: Grammy Award for Best Male Pop Vocal Performance -> winner -> Epic Records (2002)\n Hop 4: Epic Records -> record label -> Sony Music (2002)\n Answer: Sony Music",
+      "judge_em": 0,
+      "judge_reply": "NO"
+    }
+  ]
+}

logs/eval-llama31-seed1337.log

@@ -0,0 +1,41 @@
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-llama31-seed1337/final
+
+[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/llama31-sft-seed1337.json
+[eval] OVERALL: n=1743  EM=0.942  F1=0.273
+[eval] by complexity:
+  1hop: n=800  EM=0.993  F1=0.341
+  2hop: n=800  EM=0.929  F1=0.216
+  3plus: n=143  EM=0.734  F1=0.215
+[eval] by operator:
+  before_after: n=517  EM=0.942  F1=0.292
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.935  F1=0.298
+  sequence: n=528  EM=0.932  F1=0.272

logs/eval-llama31-seed7.log

@@ -0,0 +1,41 @@
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-llama31-seed7/final
+
+[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/llama31-sft-seed7.json
+[eval] OVERALL: n=1743  EM=0.941  F1=0.273
+[eval] by complexity:
+  1hop: n=800  EM=0.991  F1=0.341
+  2hop: n=800  EM=0.932  F1=0.217
+  3plus: n=143  EM=0.713  F1=0.213
+[eval] by operator:
+  before_after: n=517  EM=0.940  F1=0.292
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.946  F1=0.301
+  sequence: n=528  EM=0.920  F1=0.269

logs/eval-llama31.log

@@ -0,0 +1,61 @@
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-llama31/final
+
+[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/llama31-sft.json
+[eval] OVERALL: n=1743  EM=0.944  F1=0.273
+[eval] by complexity:
+  1hop: n=800  EM=0.990  F1=0.340
+  2hop: n=800  EM=0.932  F1=0.217
+  3plus: n=143  EM=0.748  F1=0.217
+[eval] by operator:
+  before_after: n=517  EM=0.946  F1=0.294
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.940  F1=0.299
+  sequence: n=528  EM=0.928  F1=0.269
+9
+  sequence: n=528  EM=0.928  F1=0.269
+
+[2/2] Post-extracting Answer: spans...
+[extract] wrote ../outputs/eval/llama31-sft.extracted.json
+[extract] 1707/1743 predictions had an 'Answer:' marker (97.9%)
+[extract] OVERALL: n=1743  EM=0.9225  F1=0.9224
+[extract] by complexity:
+  1hop: n=800  EM=0.9725  F1=0.9769
+  2hop: n=800  EM=0.9175  F1=0.9081
+  3plus: n=143  EM=0.6713  F1=0.6975
+[extract] by operator:
+  before_after: n=517  EM=0.9304  F1=0.9343
+  interval: n=163  EM=0.9755  F1=0.9419
+  point_in_time: n=535  EM=0.9346  F1=0.9449
+  sequence: n=528  EM=0.8864  F1=0.8819
+
+Done. Headline numbers in:
+  ../outputs/eval/llama31-sft.extracted.json  (post-extracted EM/F1)
+  ../outputs/eval/llama31-sft.json            (raw predictions)

logs/eval-mistral-seed1337.log

@@ -0,0 +1,40 @@
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-mistral-seed1337/final
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/mistral-sft-seed1337.json
+[eval] OVERALL: n=1743  EM=0.944  F1=0.276
+[eval] by complexity:
+  1hop: n=800  EM=0.996  F1=0.343
+  2hop: n=800  EM=0.946  F1=0.220
+  3plus: n=143  EM=0.636  F1=0.221
+[eval] by operator:
+  before_after: n=517  EM=0.954  F1=0.297
+  interval: n=163  EM=1.000  F1=0.139
+  point_in_time: n=535  EM=0.929  F1=0.300
+  sequence: n=528  EM=0.932  F1=0.275

logs/eval-mistral-seed7.log

@@ -0,0 +1,40 @@
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-mistral-seed7/final
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/mistral-sft-seed7.json
+[eval] OVERALL: n=1743  EM=0.943  F1=0.276
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.342
+  2hop: n=800  EM=0.945  F1=0.219
+  3plus: n=143  EM=0.629  F1=0.219
+[eval] by operator:
+  before_after: n=517  EM=0.956  F1=0.298
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.927  F1=0.299
+  sequence: n=528  EM=0.930  F1=0.273

logs/eval-mistral.log

@@ -0,0 +1,59 @@
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[eval] loading benchmark from ../outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-mistral/final
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] loading KG indexer from data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/mistral-sft.json
+[eval] OVERALL: n=1743  EM=0.943  F1=0.276
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.343
+  2hop: n=800  EM=0.945  F1=0.219
+  3plus: n=143  EM=0.622  F1=0.216
+[eval] by operator:
+  before_after: n=517  EM=0.954  F1=0.297
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.929  F1=0.299
+  sequence: n=528  EM=0.928  F1=0.275
+9
+  sequence: n=528  EM=0.928  F1=0.275
+
+[2/2] Post-extracting Answer: spans...
+[extract] wrote ../outputs/eval/mistral-sft.extracted.json
+[extract] 1583/1743 predictions had an 'Answer:' marker (90.8%)
+[extract] OVERALL: n=1743  EM=0.8910  F1=0.8677
+[extract] by complexity:
+  1hop: n=800  EM=0.9938  F1=0.9940
+  2hop: n=800  EM=0.8387  F1=0.8530
+  3plus: n=143  EM=0.6084  F1=0.2426
+[extract] by operator:
+  before_after: n=517  EM=0.9110  F1=0.8984
+  interval: n=163  EM=0.8344  F1=0.7649
+  point_in_time: n=535  EM=0.8916  F1=0.8795
+  sequence: n=528  EM=0.8883  F1=0.8573
+
+Done. Headline: ../outputs/eval/mistral-sft.extracted.json

logs/eval-multitq-llama.log

@@ -0,0 +1,28 @@
+[mtq-eval] loading KG from ../data/MultiTQ/MultiTQ/kg
+[mtq-eval]   461,329 triples
+[mtq-eval] building retriever (k=15, hops=2)
+[mtq-eval] loading questions from ../data/MultiTQ/MultiTQ/questions/test.json
+[mtq-eval]   stratified subset: 1,496 of 54,584
+[mtq-eval] pre-retrieving evidence per question
+[mtq-eval]   1,496/1,496 got >=1 triple
+[mtq-eval] loading policy from ../checkpoints/sft-multitq-llama/final
+
+[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
+[mtq-eval] generating predictions (bs=16)
+[mtq-eval]   progress: 16/1496
+[mtq-eval]   progress: 416/1496
+[mtq-eval]   progress: 816/1496
+[mtq-eval]   progress: 1216/1496
+
+[mtq-eval] wrote ../outputs/eval/multitq-llama-sft.json
+[mtq-eval] OVERALL: n=1496  EM=0.308  F1=0.346
+[mtq-eval] by qtype:
+  after_first: n=187 EM=0.134
+  before_after: n=187 EM=0.497
+  before_last: n=187 EM=0.150
+  equal: n=374 EM=0.505
+  equal_multi: n=187 EM=0.267
+  first_last: n=374 EM=0.203
+[mtq-eval] by answer_type:
+  entity: n=1122 EM=0.327
+  time: n=374 EM=0.251

logs/eval-multitq-mistral.log

@@ -0,0 +1,28 @@
+[mtq-eval] loading KG from ../data/MultiTQ/MultiTQ/kg
+[mtq-eval]   461,329 triples
+[mtq-eval] building retriever (k=15, hops=2)
+[mtq-eval] loading questions from ../data/MultiTQ/MultiTQ/questions/test.json
+[mtq-eval]   stratified subset: 1,496 of 54,584
+[mtq-eval] pre-retrieving evidence per question
+[mtq-eval]   1,496/1,496 got >=1 triple
+[mtq-eval] loading policy from ../checkpoints/sft-multitq-mistral/final
+
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[mtq-eval] generating predictions (bs=16)
+[mtq-eval]   progress: 16/1496
+[mtq-eval]   progress: 416/1496
+[mtq-eval]   progress: 816/1496
+[mtq-eval]   progress: 1216/1496
+
+[mtq-eval] wrote ../outputs/eval/multitq-mistral-sft.json
+[mtq-eval] OVERALL: n=1496  EM=0.337  F1=0.368
+[mtq-eval] by qtype:
+  after_first: n=187 EM=0.160
+  before_after: n=187 EM=0.545
+  before_last: n=187 EM=0.182
+  equal: n=374 EM=0.524
+  equal_multi: n=187 EM=0.262
+  first_last: n=374 EM=0.249
+[mtq-eval] by answer_type:
+  entity: n=1122 EM=0.362
+  time: n=374 EM=0.262

logs/eval-multitq.log

@@ -0,0 +1,33 @@
+[mtq-eval] loading KG from ~/temporal-aware-graphrag/data/MultiTQ/MultiTQ/kg
+[mtq-eval]   461,329 triples
+[mtq-eval] building retriever (k=15, hops=2)
+[mtq-eval] loading questions from ~/temporal-aware-graphrag/data/MultiTQ/MultiTQ/questions/test.json
+[mtq-eval]   stratified subset: 1,496 of 54,584
+[mtq-eval] pre-retrieving evidence per question
+[mtq-eval]   1,496/1,496 got >=1 triple
+[mtq-eval] loading policy from ../checkpoints/sft-multitq/final
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+
+[mtq-eval] generating predictions (bs=8)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[mtq-eval]   progress: 8/1496
+[mtq-eval]   progress: 208/1496
+[mtq-eval]   progress: 408/1496
+[mtq-eval]   progress: 608/1496
+[mtq-eval]   progress: 808/1496
+[mtq-eval]   progress: 1008/1496
+[mtq-eval]   progress: 1208/1496
+[mtq-eval]   progress: 1408/1496
+
+[mtq-eval] wrote ../outputs/eval/multitq-v3-sft.json
+[mtq-eval] OVERALL: n=1496  EM=0.280  F1=0.315
+[mtq-eval] by qtype:
+  after_first: n=187 EM=0.091
+  before_after: n=187 EM=0.481
+  before_last: n=187 EM=0.118
+  equal: n=374 EM=0.447
+  equal_multi: n=187 EM=0.246
+  first_last: n=374 EM=0.206
+[mtq-eval] by answer_type:
+  entity: n=1122 EM=0.289
+  time: n=374 EM=0.254

logs/eval-v2.log

@@ -0,0 +1,164 @@
+=== evaluating v2-sft-baseline ===
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft/final
+
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/v2-sft-baseline.json
+[eval] OVERALL: n=1743  EM=0.305  F1=0.366
+[eval] by complexity:
+  1hop: n=800  EM=0.328  F1=0.411
+  2hop: n=800  EM=0.246  F1=0.283
+  3plus: n=143  EM=0.510  F1=0.579
+[eval] by operator:
+  before_after: n=517  EM=0.257  F1=0.305
+  interval: n=163  EM=0.994  F1=0.994
+  point_in_time: n=535  EM=0.200  F1=0.284
+  sequence: n=528  EM=0.246  F1=0.313
+=== evaluating v2-grpo-9000 ===
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/grpo/checkpoint-9000
+
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/v2-grpo-9000.json
+[eval] OVERALL: n=1743  EM=0.364  F1=0.418
+[eval] by complexity:
+  1hop: n=800  EM=0.383  F1=0.450
+  2hop: n=800  EM=0.304  F1=0.340
+  3plus: n=143  EM=0.594  F1=0.670
+[eval] by operator:
+  before_after: n=517  EM=0.335  F1=0.381
+  interval: n=163  EM=1.000  F1=1.000
+  point_in_time: n=535  EM=0.250  F1=0.318
+  sequence: n=528  EM=0.311  F1=0.376
+=== evaluating v2-grpo-9500 ===
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/grpo/checkpoint-9500
+
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/v2-grpo-9500.json
+[eval] OVERALL: n=1743  EM=0.366  F1=0.420
+[eval] by complexity:
+  1hop: n=800  EM=0.384  F1=0.455
+  2hop: n=800  EM=0.306  F1=0.341
+  3plus: n=143  EM=0.601  F1=0.670
+[eval] by operator:
+  before_after: n=517  EM=0.340  F1=0.386
+  interval: n=163  EM=1.000  F1=1.000
+  point_in_time: n=535  EM=0.254  F1=0.319
+  sequence: n=528  EM=0.309  F1=0.377
+=== evaluating v2-grpo-10000 ===
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/grpo/checkpoint-10000
+
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval] loading KG indexer from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval] generating predictions (bs=4)
+[eval]   progress: 4/1743
+[eval]   progress: 104/1743
+[eval]   progress: 204/1743
+[eval]   progress: 304/1743
+[eval]   progress: 404/1743
+[eval]   progress: 504/1743
+[eval]   progress: 604/1743
+[eval]   progress: 704/1743
+[eval]   progress: 804/1743
+[eval]   progress: 904/1743
+[eval]   progress: 1004/1743
+[eval]   progress: 1104/1743
+[eval]   progress: 1204/1743
+[eval]   progress: 1304/1743
+[eval]   progress: 1404/1743
+[eval]   progress: 1504/1743
+[eval]   progress: 1604/1743
+[eval]   progress: 1704/1743
+
+[eval] wrote ../outputs/eval/v2-grpo-10000.json
+[eval] OVERALL: n=1743  EM=0.364  F1=0.420
+[eval] by complexity:
+  1hop: n=800  EM=0.381  F1=0.453
+  2hop: n=800  EM=0.305  F1=0.341
+  3plus: n=143  EM=0.601  F1=0.670
+[eval] by operator:
+  before_after: n=517  EM=0.335  F1=0.382
+  interval: n=163  EM=1.000  F1=1.000
+  point_in_time: n=535  EM=0.252  F1=0.320
+  sequence: n=528  EM=0.311  F1=0.378
+=== all evals done ===
+-rw-rw-r-- 1 ubuntu ubuntu 562510 Apr 25 09:28 ../outputs/eval/v2-grpo-10000.json
+-rw-rw-r-- 1 ubuntu ubuntu 561712 Apr 25 09:20 ../outputs/eval/v2-grpo-9000.json
+-rw-rw-r-- 1 ubuntu ubuntu 562262 Apr 25 09:24 ../outputs/eval/v2-grpo-9500.json
+-rw-rw-r-- 1 ubuntu ubuntu 566615 Apr 25 09:16 ../outputs/eval/v2-sft-baseline.json

logs/eval-v3-1k.log

@@ -0,0 +1,26 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3-1k/final
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-1k.json
+[eval] OVERALL: n=1743  EM=0.861  F1=0.252
+[eval] by complexity:
+  1hop: n=800  EM=0.985  F1=0.320
+  2hop: n=800  EM=0.864  F1=0.200
+  3plus: n=143  EM=0.154  F1=0.167
+[eval] by operator:
+  before_after: n=517  EM=0.847  F1=0.262
+  interval: n=163  EM=0.969  F1=0.130
+  point_in_time: n=535  EM=0.865  F1=0.278
+  sequence: n=528  EM=0.837  F1=0.255

logs/eval-v3-2k.log

@@ -0,0 +1,27 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3-2k/final
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-2k.json
+[eval] OVERALL: n=1743  EM=0.880  F1=0.261
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.338
+  2hop: n=800  EM=0.854  F1=0.198
+  3plus: n=143  EM=0.364  F1=0.180
+[eval] by operator:
+  before_after: n=517  EM=0.867  F1=0.273
+  interval: n=163  EM=1.000  F1=0.137
+  point_in_time: n=535  EM=0.879  F1=0.287
+  sequence: n=528  EM=0.856  F1=0.260

logs/eval-v3-noevidence.log

@@ -0,0 +1,27 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3/final
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+
+[eval] retrieval mode ON — building retriever (k=0)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 0/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-noevidence.json
+[eval] OVERALL: n=1743  EM=0.233  F1=0.088
+[eval] by complexity:
+  1hop: n=800  EM=0.247  F1=0.108
+  2hop: n=800  EM=0.181  F1=0.054
+  3plus: n=143  EM=0.441  F1=0.168
+[eval] by operator:
+  before_after: n=517  EM=0.170  F1=0.088
+  interval: n=163  EM=0.693  F1=0.102
+  point_in_time: n=535  EM=0.181  F1=0.090
+  sequence: n=528  EM=0.205  F1=0.082

logs/eval-v3-seed1337.log

@@ -0,0 +1,27 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3-seed1337/final
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-seed1337.json
+[eval] OVERALL: n=1743  EM=0.898  F1=0.266
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.343
+  2hop: n=800  EM=0.871  F1=0.202
+  3plus: n=143  EM=0.490  F1=0.189
+[eval] by operator:
+  before_after: n=517  EM=0.897  F1=0.284
+  interval: n=163  EM=1.000  F1=0.139
+  point_in_time: n=535  EM=0.897  F1=0.290
+  sequence: n=528  EM=0.867  F1=0.263

logs/eval-v3-seed7.log

@@ -0,0 +1,26 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3-seed7/final
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-seed7.json
+[eval] OVERALL: n=1743  EM=0.904  F1=0.266
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.343
+  2hop: n=800  EM=0.875  F1=0.202
+  3plus: n=143  EM=0.545  F1=0.194
+[eval] by operator:
+  before_after: n=517  EM=0.907  F1=0.284
+  interval: n=163  EM=1.000  F1=0.139
+  point_in_time: n=535  EM=0.905  F1=0.291
+  sequence: n=528  EM=0.871  F1=0.261

logs/eval-v3-sft-3hop.log

@@ -0,0 +1,31 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3/final
+
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+[eval] retrieval mode ON — building retriever (k=25)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=8)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 8/1743
+[eval]   progress: 208/1743
+[eval]   progress: 408/1743
+[eval]   progress: 608/1743
+[eval]   progress: 808/1743
+[eval]   progress: 1008/1743
+[eval]   progress: 1208/1743
+[eval]   progress: 1408/1743
+[eval]   progress: 1608/1743
+
+[eval] wrote ../outputs/eval/v3-sft-3hop.json
+[eval] OVERALL: n=1743  EM=0.922  F1=0.267
+[eval] by complexity:
+  1hop: n=800  EM=0.995  F1=0.341
+  2hop: n=800  EM=0.882  F1=0.202
+  3plus: n=143  EM=0.734  F1=0.225
+[eval] by operator:
+  before_after: n=517  EM=0.901  F1=0.282
+  interval: n=163  EM=1.000  F1=0.137
+  point_in_time: n=535  EM=0.925  F1=0.294
+  sequence: n=528  EM=0.915  F1=0.267

logs/eval-v3-sft.log

@@ -0,0 +1,27 @@
+[eval] loading benchmark from ~/temporal-aware-graphrag/outputs/benchmark/benchmark_labelled.jsonl (split=test)
+[eval]   1743 questions
+[eval] loading policy from ../checkpoints/sft-v3/final
+Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+
+[eval] retrieval mode ON — building retriever (k=15)
+[Indexer] Loaded 550,376 triples from ~/temporal-aware-graphrag/code/data/tkgl-smallpedia_edgelist.csv
+[eval]   retrieval done: 1743/1743 questions got at least one triple
+[eval] generating predictions (bs=16)
+[transformers] The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
+[eval]   progress: 16/1743
+[eval]   progress: 416/1743
+[eval]   progress: 816/1743
+[eval]   progress: 1216/1743
+[eval]   progress: 1616/1743
+
+[eval] wrote ../outputs/eval/v3-sft-baseline.json
+[eval] OVERALL: n=1743  EM=0.903  F1=0.265
+[eval] by complexity:
+  1hop: n=800  EM=0.998  F1=0.343
+  2hop: n=800  EM=0.871  F1=0.200
+  3plus: n=143  EM=0.552  F1=0.194
+[eval] by operator:
+  before_after: n=517  EM=0.901  F1=0.283
+  interval: n=163  EM=1.000  F1=0.138
+  point_in_time: n=535  EM=0.908  F1=0.291
+  sequence: n=528  EM=0.869  F1=0.261