Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+cls_gemma_orderkd_mid_l02/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+cls_gemma_orderkd_mid_l005/checkpoint-200/tokenizer.json filter=lfs diff=lfs merge=lfs -text

cls_gemma_baseline/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l005/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l005/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l005/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l005/checkpoint-200/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l005/checkpoint-200/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l005/checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccd01c86f18a73793308f6a7ef344301848044ff6c0511d87f3579e3d675c343
+size 52231312

cls_gemma_orderkd_first_l005/checkpoint-200/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l005/checkpoint-400/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l005/checkpoint-400/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l005/checkpoint-400/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l005/train_metrics.json

@@ -0,0 +1,1874 @@
+{
+  "loss_log": [
+    {
+      "global_step": 20,
+      "optimizer_step": 2,
+      "lr": 4.347826086956522e-06,
+      "lm_loss": 3.2664432525634766,
+      "order_loss": 63.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 6.438318252563477
+    },
+    {
+      "global_step": 40,
+      "optimizer_step": 5,
+      "lr": 1.0869565217391305e-05,
+      "lm_loss": 1.2319077253341675,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.060032844543457
+    },
+    {
+      "global_step": 60,
+      "optimizer_step": 7,
+      "lr": 1.5217391304347828e-05,
+      "lm_loss": 0.5355294346809387,
+      "order_loss": 42.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.644904375076294
+    },
+    {
+      "global_step": 80,
+      "optimizer_step": 10,
+      "lr": 2.173913043478261e-05,
+      "lm_loss": 1.8286396265029907,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.594264507293701
+    },
+    {
+      "global_step": 100,
+      "optimizer_step": 12,
+      "lr": 2.608695652173913e-05,
+      "lm_loss": 0.5796006917953491,
+      "order_loss": 43.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7514758110046387
+    },
+    {
+      "global_step": 120,
+      "optimizer_step": 15,
+      "lr": 3.260869565217392e-05,
+      "lm_loss": 2.7209701538085938,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 5.439720153808594
+    },
+    {
+      "global_step": 140,
+      "optimizer_step": 17,
+      "lr": 3.695652173913043e-05,
+      "lm_loss": 0.6651992797851562,
+      "order_loss": 54.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3683242797851562
+    },
+    {
+      "global_step": 160,
+      "optimizer_step": 20,
+      "lr": 4.347826086956522e-05,
+      "lm_loss": 0.4559280574321747,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.784053087234497
+    },
+    {
+      "global_step": 180,
+      "optimizer_step": 22,
+      "lr": 4.782608695652174e-05,
+      "lm_loss": 0.05015219375491142,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.76890230178833
+    },
+    {
+      "global_step": 200,
+      "optimizer_step": 25,
+      "lr": 4.9997541293989384e-05,
+      "lm_loss": 0.025382647290825844,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7441327571868896
+    },
+    {
+      "global_step": 220,
+      "optimizer_step": 27,
+      "lr": 4.999016565957633e-05,
+      "lm_loss": 0.019361622631549835,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.081861734390259
+    },
+    {
+      "global_step": 240,
+      "optimizer_step": 30,
+      "lr": 4.996988640512931e-05,
+      "lm_loss": 0.34661865234375,
+      "order_loss": 39.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.33099365234375
+    },
+    {
+      "global_step": 260,
+      "optimizer_step": 32,
+      "lr": 4.995022691166621e-05,
+      "lm_loss": 0.028115127235651016,
+      "order_loss": 62.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1374900341033936
+    },
+    {
+      "global_step": 280,
+      "optimizer_step": 35,
+      "lr": 4.991153735294049e-05,
+      "lm_loss": 0.4418671131134033,
+      "order_loss": 60.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.4731171131134033
+    },
+    {
+      "global_step": 300,
+      "optimizer_step": 37,
+      "lr": 4.987961816680492e-05,
+      "lm_loss": 0.07714522629976273,
+      "order_loss": 54.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8115203380584717
+    },
+    {
+      "global_step": 320,
+      "optimizer_step": 40,
+      "lr": 4.9822565862832806e-05,
+      "lm_loss": 0.12331299483776093,
+      "order_loss": 40.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1545629501342773
+    },
+    {
+      "global_step": 340,
+      "optimizer_step": 42,
+      "lr": 4.9778426220595456e-05,
+      "lm_loss": 0.3274025619029999,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8274025917053223
+    },
+    {
+      "global_step": 360,
+      "optimizer_step": 45,
+      "lr": 4.970308130276272e-05,
+      "lm_loss": 0.7888674139976501,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.616992473602295
+    },
+    {
+      "global_step": 380,
+      "optimizer_step": 47,
+      "lr": 4.96467754629559e-05,
+      "lm_loss": 0.03701641783118248,
+      "order_loss": 41.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1307663917541504
+    },
+    {
+      "global_step": 400,
+      "optimizer_step": 50,
+      "lr": 4.955323054879298e-05,
+      "lm_loss": 0.12001635134220123,
+      "order_loss": 47.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.47939133644104
+    },
+    {
+      "global_step": 420,
+      "optimizer_step": 52,
+      "lr": 4.948482772521345e-05,
+      "lm_loss": 0.1201663538813591,
+      "order_loss": 66.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.417041301727295
+    },
+    {
+      "global_step": 440,
+      "optimizer_step": 55,
+      "lr": 4.937319780454559e-05,
+      "lm_loss": 0.16920027136802673,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8879501819610596
+    },
+    {
+      "global_step": 460,
+      "optimizer_step": 57,
+      "lr": 4.929278208117378e-05,
+      "lm_loss": 0.6665996313095093,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.650974750518799
+    },
+    {
+      "global_step": 480,
+      "optimizer_step": 60,
+      "lr": 4.916320437477008e-05,
+      "lm_loss": 0.3026118874549866,
+      "order_loss": 57.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.161986827850342
+    },
+    {
+      "global_step": 500,
+      "optimizer_step": 62,
+      "lr": 4.907087460240999e-05,
+      "lm_loss": 0.13276802003383636,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.195267915725708
+    },
+    {
+      "global_step": 520,
+      "optimizer_step": 65,
+      "lr": 4.892350839330522e-05,
+      "lm_loss": 0.08697174489498138,
+      "order_loss": 50.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6182217597961426
+    },
+    {
+      "global_step": 540,
+      "optimizer_step": 67,
+      "lr": 4.881937806807241e-05,
+      "lm_loss": 0.13747315108776093,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.9499731063842773
+    },
+    {
+      "global_step": 560,
+      "optimizer_step": 70,
+      "lr": 4.865440450576874e-05,
+      "lm_loss": 0.023917367681860924,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8676674365997314
+    },
+    {
+      "global_step": 580,
+      "optimizer_step": 72,
+      "lr": 4.853860162957552e-05,
+      "lm_loss": 0.09218831360340118,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.9359383583068848
+    },
+    {
+      "global_step": 600,
+      "optimizer_step": 75,
+      "lr": 4.8356223507364996e-05,
+      "lm_loss": 0.016055572777986526,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6566805839538574
+    },
+    {
+      "global_step": 620,
+      "optimizer_step": 77,
+      "lr": 4.822889043057446e-05,
+      "lm_loss": 0.2428482174873352,
+      "order_loss": 49.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7115981578826904
+    },
+    {
+      "global_step": 640,
+      "optimizer_step": 80,
+      "lr": 4.802933193625593e-05,
+      "lm_loss": 0.007987729273736477,
+      "order_loss": 67.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3517377376556396
+    },
+    {
+      "global_step": 660,
+      "optimizer_step": 82,
+      "lr": 4.789062518269812e-05,
+      "lm_loss": 0.03567792847752571,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.676302909851074
+    },
+    {
+      "global_step": 680,
+      "optimizer_step": 85,
+      "lr": 4.767413162299501e-05,
+      "lm_loss": 0.05696840584278107,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0413434505462646
+    },
+    {
+      "global_step": 700,
+      "optimizer_step": 87,
+      "lr": 4.752422169756048e-05,
+      "lm_loss": 0.24159936606884003,
+      "order_loss": 67.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.6165993213653564
+    },
+    {
+      "global_step": 720,
+      "optimizer_step": 90,
+      "lr": 4.729105919657818e-05,
+      "lm_loss": 0.3990735113620758,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.274073600769043
+    },
+    {
+      "global_step": 740,
+      "optimizer_step": 92,
+      "lr": 4.7130130375625305e-05,
+      "lm_loss": 0.17991729080677032,
+      "order_loss": 55.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.929917335510254
+    },
+    {
+      "global_step": 760,
+      "optimizer_step": 95,
+      "lr": 4.6880585547718845e-05,
+      "lm_loss": 0.28898534178733826,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.163985252380371
+    },
+    {
+      "global_step": 780,
+      "optimizer_step": 97,
+      "lr": 4.670883565255264e-05,
+      "lm_loss": 0.01540749054402113,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1404075622558594
+    },
+    {
+      "global_step": 800,
+      "optimizer_step": 100,
+      "lr": 4.6443215250006806e-05,
+      "lm_loss": 0.3447962701320648,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1416711807250977
+    },
+    {
+      "global_step": 820,
+      "optimizer_step": 102,
+      "lr": 4.626085540370762e-05,
+      "lm_loss": 0.01032446138560772,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.900949478149414
+    },
+    {
+      "global_step": 840,
+      "optimizer_step": 105,
+      "lr": 4.597948593966256e-05,
+      "lm_loss": 0.06883961707353592,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0532145500183105
+    },
+    {
+      "global_step": 860,
+      "optimizer_step": 107,
+      "lr": 4.5786740307563636e-05,
+      "lm_loss": 0.06082785874605179,
+      "order_loss": 63.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2483279705047607
+    },
+    {
+      "global_step": 880,
+      "optimizer_step": 110,
+      "lr": 4.5489967654649474e-05,
+      "lm_loss": 0.09451040625572205,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.250760316848755
+    },
+    {
+      "global_step": 900,
+      "optimizer_step": 112,
+      "lr": 4.528707316878232e-05,
+      "lm_loss": 0.12780378758907318,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5965538024902344
+    },
+    {
+      "global_step": 920,
+      "optimizer_step": 115,
+      "lr": 4.497526213395623e-05,
+      "lm_loss": 0.14049623906612396,
+      "order_loss": 55.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.921746253967285
+    },
+    {
+      "global_step": 940,
+      "optimizer_step": 117,
+      "lr": 4.476246820180259e-05,
+      "lm_loss": 0.6046736240386963,
+      "order_loss": 52.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2140486240386963
+    },
+    {
+      "global_step": 960,
+      "optimizer_step": 120,
+      "lr": 4.443600207791081e-05,
+      "lm_loss": 0.29507577419281006,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3732008934020996
+    },
+    {
+      "global_step": 980,
+      "optimizer_step": 122,
+      "lr": 4.421357027581921e-05,
+      "lm_loss": 0.8163134455680847,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.5819385051727295
+    },
+    {
+      "global_step": 1000,
+      "optimizer_step": 125,
+      "lr": 4.38728503704354e-05,
+      "lm_loss": 0.06714347004890442,
+      "order_loss": 55.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.848393440246582
+    },
+    {
+      "global_step": 1020,
+      "optimizer_step": 127,
+      "lr": 4.364105412207914e-05,
+      "lm_loss": 0.26185736060142517,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.511857271194458
+    },
+    {
+      "global_step": 1040,
+      "optimizer_step": 130,
+      "lr": 4.328649926419819e-05,
+      "lm_loss": 0.05918440222740173,
+      "order_loss": 52.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6685593128204346
+    },
+    {
+      "global_step": 1060,
+      "optimizer_step": 132,
+      "lr": 4.304562350447008e-05,
+      "lm_loss": 0.18948712944984436,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.9863622188568115
+    },
+    {
+      "global_step": 1080,
+      "optimizer_step": 135,
+      "lr": 4.267766952966369e-05,
+      "lm_loss": 0.02881915122270584,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1538190841674805
+    },
+    {
+      "global_step": 1100,
+      "optimizer_step": 137,
+      "lr": 4.242801035442058e-05,
+      "lm_loss": 0.44544386863708496,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.539193868637085
+    },
+    {
+      "global_step": 1120,
+      "optimizer_step": 140,
+      "lr": 4.204710956908763e-05,
+      "lm_loss": 0.7407297492027283,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.834479808807373
+    },
+    {
+      "global_step": 1140,
+      "optimizer_step": 142,
+      "lr": 4.178897387117546e-05,
+      "lm_loss": 0.07381007820367813,
+      "order_loss": 61.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.12068510055542
+    },
+    {
+      "global_step": 1160,
+      "optimizer_step": 145,
+      "lr": 4.139559449654561e-05,
+      "lm_loss": 0.04309108480811119,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.714966058731079
+    },
+    {
+      "global_step": 1180,
+      "optimizer_step": 147,
+      "lr": 4.1129299588552193e-05,
+      "lm_loss": 0.16437874734401703,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2268786430358887
+    },
+    {
+      "global_step": 1200,
+      "optimizer_step": 150,
+      "lr": 4.072392518512623e-05,
+      "lm_loss": 0.5126318335533142,
+      "order_loss": 63.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.684506893157959
+    },
+    {
+      "global_step": 1220,
+      "optimizer_step": 152,
+      "lr": 4.0449798409325665e-05,
+      "lm_loss": 0.042548954486846924,
+      "order_loss": 52.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.667548894882202
+    },
+    {
+      "global_step": 1240,
+      "optimizer_step": 155,
+      "lr": 4.0032927282460146e-05,
+      "lm_loss": 0.368450790643692,
+      "order_loss": 60.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.384075880050659
+    },
+    {
+      "global_step": 1260,
+      "optimizer_step": 157,
+      "lr": 3.9751305608428205e-05,
+      "lm_loss": 0.20985375344753265,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0536036491394043
+    },
+    {
+      "global_step": 1280,
+      "optimizer_step": 160,
+      "lr": 3.932345019579495e-05,
+      "lm_loss": 0.05628350004553795,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.150033473968506
+    },
+    {
+      "global_step": 1300,
+      "optimizer_step": 162,
+      "lr": 3.90346798061902e-05,
+      "lm_loss": 0.06509767472743988,
+      "order_loss": 50.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5807225704193115
+    },
+    {
+      "global_step": 1320,
+      "optimizer_step": 165,
+      "lr": 3.859636604786372e-05,
+      "lm_loss": 0.32651910185813904,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.732769012451172
+    },
+    {
+      "global_step": 1340,
+      "optimizer_step": 167,
+      "lr": 3.830080191288342e-05,
+      "lm_loss": 0.35849064588546753,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0772407054901123
+    },
+    {
+      "global_step": 1360,
+      "optimizer_step": 170,
+      "lr": 3.785256860483054e-05,
+      "lm_loss": 0.06221141666173935,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.734086513519287
+    },
+    {
+      "global_step": 1380,
+      "optimizer_step": 172,
+      "lr": 3.755057404586455e-05,
+      "lm_loss": 0.15530270338058472,
+      "order_loss": 39.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1162402629852295
+    },
+    {
+      "global_step": 1400,
+      "optimizer_step": 175,
+      "lr": 3.7092972177631e-05,
+      "lm_loss": 0.13956251740455627,
+      "order_loss": 65.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3895626068115234
+    },
+    {
+      "global_step": 1420,
+      "optimizer_step": 177,
+      "lr": 3.678491842064995e-05,
+      "lm_loss": 0.18673621118068695,
+      "order_loss": 51.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7492361068725586
+    },
+    {
+      "global_step": 1440,
+      "optimizer_step": 180,
+      "lr": 3.631851049805793e-05,
+      "lm_loss": 0.28595873713493347,
+      "order_loss": 45.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.567208766937256
+    },
+    {
+      "global_step": 1460,
+      "optimizer_step": 182,
+      "lr": 3.600477621728473e-05,
+      "lm_loss": 0.01681588962674141,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8293159008026123
+    },
+    {
+      "global_step": 1480,
+      "optimizer_step": 185,
+      "lr": 3.5530135570974274e-05,
+      "lm_loss": 0.25463736057281494,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7858872413635254
+    },
+    {
+      "global_step": 1500,
+      "optimizer_step": 187,
+      "lr": 3.521110642339991e-05,
+      "lm_loss": 0.4409516453742981,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0815765857696533
+    },
+    {
+      "global_step": 1520,
+      "optimizer_step": 190,
+      "lr": 3.4728816504063726e-05,
+      "lm_loss": 0.2549166679382324,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3486666679382324
+    },
+    {
+      "global_step": 1540,
+      "optimizer_step": 192,
+      "lr": 3.440488465537943e-05,
+      "lm_loss": 0.006924055051058531,
+      "order_loss": 65.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2569241523742676
+    },
+    {
+      "global_step": 1560,
+      "optimizer_step": 195,
+      "lr": 3.391553831655782e-05,
+      "lm_loss": 0.018992431461811066,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4877424240112305
+    },
+    {
+      "global_step": 1580,
+      "optimizer_step": 197,
+      "lr": 3.358710195908653e-05,
+      "lm_loss": 0.2616627514362335,
+      "order_loss": 70.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.761662721633911
+    },
+    {
+      "global_step": 1600,
+      "optimizer_step": 200,
+      "lr": 3.309130072840379e-05,
+      "lm_loss": 0.22181908786296844,
+      "order_loss": 58.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1280691623687744
+    },
+    {
+      "global_step": 1620,
+      "optimizer_step": 202,
+      "lr": 3.2758763591623295e-05,
+      "lm_loss": 0.1705959141254425,
+      "order_loss": 50.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.68622088432312
+    },
+    {
+      "global_step": 1640,
+      "optimizer_step": 205,
+      "lr": 3.225711693136156e-05,
+      "lm_loss": 0.08368847519159317,
+      "order_loss": 60.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.114938497543335
+    },
+    {
+      "global_step": 1660,
+      "optimizer_step": 207,
+      "lr": 3.1920887785621235e-05,
+      "lm_loss": 0.046722203493118286,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.140472173690796
+    },
+    {
+      "global_step": 1680,
+      "optimizer_step": 210,
+      "lr": 3.141401234354067e-05,
+      "lm_loss": 0.12700414657592773,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6113791465759277
+    },
+    {
+      "global_step": 1700,
+      "optimizer_step": 212,
+      "lr": 3.10745044975816e-05,
+      "lm_loss": 0.16074301302433014,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.691992998123169
+    },
+    {
+      "global_step": 1720,
+      "optimizer_step": 215,
+      "lr": 3.056302334890786e-05,
+      "lm_loss": 0.01594642922282219,
+      "order_loss": 74.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.719071388244629
+    },
+    {
+      "global_step": 1740,
+      "optimizer_step": 217,
+      "lr": 3.022065414180425e-05,
+      "lm_loss": 0.006087217014282942,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.131087303161621
+    },
+    {
+      "global_step": 1760,
+      "optimizer_step": 220,
+      "lr": 2.9705196023315006e-05,
+      "lm_loss": 0.010915451683104038,
+      "order_loss": 59.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.979665517807007
+    },
+    {
+      "global_step": 1780,
+      "optimizer_step": 222,
+      "lr": 2.9360386311461275e-05,
+      "lm_loss": 0.002609483664855361,
+      "order_loss": 47.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.3619844913482666
+    },
+    {
+      "global_step": 1800,
+      "optimizer_step": 225,
+      "lr": 2.884158484861325e-05,
+      "lm_loss": 0.0126071497797966,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8876070976257324
+    },
+    {
+      "global_step": 1820,
+      "optimizer_step": 227,
+      "lr": 2.849475848838749e-05,
+      "lm_loss": 0.07058226317167282,
+      "order_loss": 39.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.047144651412964
+    },
+    {
+      "global_step": 1840,
+      "optimizer_step": 230,
+      "lr": 2.7973251416434175e-05,
+      "lm_loss": 0.05446998029947281,
+      "order_loss": 29.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.5310324430465698
+    },
+    {
+      "global_step": 1860,
+      "optimizer_step": 232,
+      "lr": 2.7624834743173867e-05,
+      "lm_loss": 0.537248432636261,
+      "order_loss": 75.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.287248611450195
+    },
+    {
+      "global_step": 1880,
+      "optimizer_step": 235,
+      "lr": 2.710126312323119e-05,
+      "lm_loss": 0.12687210738658905,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2518720626831055
+    },
+    {
+      "global_step": 1900,
+      "optimizer_step": 237,
+      "lr": 2.6751684427161683e-05,
+      "lm_loss": 0.14577153325080872,
+      "order_loss": 40.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1613965034484863
+    },
+    {
+      "global_step": 1920,
+      "optimizer_step": 240,
+      "lr": 2.6226691858185454e-05,
+      "lm_loss": 0.0037754380609840155,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.2537753582000732
+    },
+    {
+      "global_step": 1940,
+      "optimizer_step": 242,
+      "lr": 2.587638085794541e-05,
+      "lm_loss": 0.04501320794224739,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.873138189315796
+    },
+    {
+      "global_step": 1960,
+      "optimizer_step": 245,
+      "lr": 2.535061268558906e-05,
+      "lm_loss": 0.11622508615255356,
+      "order_loss": 71.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.694350004196167
+    },
+    {
+      "global_step": 1980,
+      "optimizer_step": 247,
+      "lr": 2.5e-05,
+      "lm_loss": 0.252244770526886,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.595994710922241
+    },
+    {
+      "global_step": 2000,
+      "optimizer_step": 250,
+      "lr": 2.447410252332528e-05,
+      "lm_loss": 0.010590744204819202,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7293407917022705
+    },
+    {
+      "global_step": 2020,
+      "optimizer_step": 252,
+      "lr": 2.4123619142054597e-05,
+      "lm_loss": 0.039978720247745514,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5399787425994873
+    },
+    {
+      "global_step": 2040,
+      "optimizer_step": 255,
+      "lr": 2.3598238819070202e-05,
+      "lm_loss": 0.14826305210590363,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2263879776000977
+    },
+    {
+      "global_step": 2060,
+      "optimizer_step": 257,
+      "lr": 2.3248315572838316e-05,
+      "lm_loss": 0.0014903320698067546,
+      "order_loss": 44.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.235865354537964
+    },
+    {
+      "global_step": 2080,
+      "optimizer_step": 260,
+      "lr": 2.272409822584325e-05,
+      "lm_loss": 0.0028078618925064802,
+      "order_loss": 43.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.190307855606079
+    },
+    {
+      "global_step": 2100,
+      "optimizer_step": 262,
+      "lr": 2.237516525682614e-05,
+      "lm_loss": 0.016735367476940155,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.360485315322876
+    },
+    {
+      "global_step": 2120,
+      "optimizer_step": 265,
+      "lr": 2.185275527853437e-05,
+      "lm_loss": 0.011359093710780144,
+      "order_loss": 43.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1676089763641357
+    },
+    {
+      "global_step": 2140,
+      "optimizer_step": 267,
+      "lr": 2.1505241511612522e-05,
+      "lm_loss": 0.008438815362751484,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4771888256073
+    },
+    {
+      "global_step": 2160,
+      "optimizer_step": 270,
+      "lr": 2.0985281073035076e-05,
+      "lm_loss": 0.11193981766700745,
+      "order_loss": 44.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.3306899070739746
+    },
+    {
+      "global_step": 2180,
+      "optimizer_step": 272,
+      "lr": 2.0639613688538734e-05,
+      "lm_loss": 0.0006088608643040061,
+      "order_loss": 51.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5474839210510254
+    },
+    {
+      "global_step": 2200,
+      "optimizer_step": 275,
+      "lr": 2.0122741949596797e-05,
+      "lm_loss": 0.010579620487987995,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8855795860290527
+    },
+    {
+      "global_step": 2220,
+      "optimizer_step": 277,
+      "lr": 1.977934585819576e-05,
+      "lm_loss": 0.31700122356414795,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7232513427734375
+    },
+    {
+      "global_step": 2240,
+      "optimizer_step": 280,
+      "lr": 1.9266198182035058e-05,
+      "lm_loss": 0.007454684004187584,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0855796337127686
+    },
+    {
+      "global_step": 2260,
+      "optimizer_step": 282,
+      "lr": 1.8925495502418406e-05,
+      "lm_loss": 0.06350623816251755,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8760061264038086
+    },
+    {
+      "global_step": 2280,
+      "optimizer_step": 285,
+      "lr": 1.841670267439088e-05,
+      "lm_loss": 0.06775510311126709,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6615052223205566
+    },
+    {
+      "global_step": 2300,
+      "optimizer_step": 287,
+      "lr": 1.8079112214378768e-05,
+      "lm_loss": 0.06462709605693817,
+      "order_loss": 58.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.002127170562744
+    },
+    {
+      "global_step": 2320,
+      "optimizer_step": 290,
+      "lr": 1.757529966665141e-05,
+      "lm_loss": 0.08016367256641388,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.4082887172698975
+    },
+    {
+      "global_step": 2340,
+      "optimizer_step": 292,
+      "lr": 1.72412364083767e-05,
+      "lm_loss": 0.03637836128473282,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1926283836364746
+    },
+    {
+      "global_step": 2360,
+      "optimizer_step": 295,
+      "lr": 1.6743023451120832e-05,
+      "lm_loss": 0.18999110162258148,
+      "order_loss": 58.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.096240997314453
+    },
+    {
+      "global_step": 2380,
+      "optimizer_step": 297,
+      "lr": 1.641289804091347e-05,
+      "lm_loss": 0.059739310294389725,
+      "order_loss": 48.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.465989351272583
+    },
+    {
+      "global_step": 2400,
+      "optimizer_step": 300,
+      "lr": 1.5920897101019383e-05,
+      "lm_loss": 0.21012945473194122,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.616379499435425
+    },
+    {
+      "global_step": 2420,
+      "optimizer_step": 302,
+      "lr": 1.5595115344620576e-05,
+      "lm_loss": 0.012707713060081005,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8877077102661133
+    },
+    {
+      "global_step": 2440,
+      "optimizer_step": 305,
+      "lr": 1.5109931212873534e-05,
+      "lm_loss": 0.023759279400110245,
+      "order_loss": 61.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.070634365081787
+    },
+    {
+      "global_step": 2460,
+      "optimizer_step": 307,
+      "lr": 1.4788893576600099e-05,
+      "lm_loss": 0.24138568341732025,
+      "order_loss": 64.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.4601356983184814
+    },
+    {
+      "global_step": 2480,
+      "optimizer_step": 310,
+      "lr": 1.4311122664242954e-05,
+      "lm_loss": 0.006877885665744543,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.585002899169922
+    },
+    {
+      "global_step": 2500,
+      "optimizer_step": 312,
+      "lr": 1.399522378271527e-05,
+      "lm_loss": 0.8298333287239075,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.6735832691192627
+    },
+    {
+      "global_step": 2520,
+      "optimizer_step": 315,
+      "lr": 1.3525453388311554e-05,
+      "lm_loss": 0.12899236381053925,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5977423191070557
+    },
+    {
+      "global_step": 2540,
+      "optimizer_step": 317,
+      "lr": 1.3215081579350058e-05,
+      "lm_loss": 0.7274826765060425,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.071232795715332
+    },
+    {
+      "global_step": 2560,
+      "optimizer_step": 320,
+      "lr": 1.2753889166848909e-05,
+      "lm_loss": 0.007098065689206123,
+      "order_loss": 63.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1633479595184326
+    },
+    {
+      "global_step": 2580,
+      "optimizer_step": 322,
+      "lr": 1.2449425954135452e-05,
+      "lm_loss": 0.011103707365691662,
+      "order_loss": 46.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.307978630065918
+    },
+    {
+      "global_step": 2600,
+      "optimizer_step": 325,
+      "lr": 1.1997378443025634e-05,
+      "lm_loss": 0.005687281023710966,
+      "order_loss": 50.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5369372367858887
+    },
+    {
+      "global_step": 2620,
+      "optimizer_step": 327,
+      "lr": 1.1699198087116589e-05,
+      "lm_loss": 0.001761247985996306,
+      "order_loss": 47.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.3767611980438232
+    },
+    {
+      "global_step": 2640,
+      "optimizer_step": 330,
+      "lr": 1.1256851155542298e-05,
+      "lm_loss": 0.001718214014545083,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.845468282699585
+    },
+    {
+      "global_step": 2660,
+      "optimizer_step": 332,
+      "lr": 1.0965320193809808e-05,
+      "lm_loss": 0.013945301994681358,
+      "order_loss": 52.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6389453411102295
+    },
+    {
+      "global_step": 2680,
+      "optimizer_step": 335,
+      "lr": 1.0533217595504858e-05,
+      "lm_loss": 0.0011550714261829853,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.673030138015747
+    },
+    {
+      "global_step": 2700,
+      "optimizer_step": 337,
+      "lr": 1.0248694391571801e-05,
+      "lm_loss": 0.06350190937519073,
+      "order_loss": 51.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6260018348693848
+    },
+    {
+      "global_step": 2720,
+      "optimizer_step": 340,
+      "lr": 9.82736728745186e-06,
+      "lm_loss": 0.06553089618682861,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.940530776977539
+    },
+    {
+      "global_step": 2740,
+      "optimizer_step": 342,
+      "lr": 9.550201590674343e-06,
+      "lm_loss": 0.016463955864310265,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5945889949798584
+    },
+    {
+      "global_step": 2760,
+      "optimizer_step": 345,
+      "lr": 9.140167895908867e-06,
+      "lm_loss": 0.021131625398993492,
+      "order_loss": 45.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.2867565155029297
+    },
+    {
+      "global_step": 2780,
+      "optimizer_step": 347,
+      "lr": 8.870700411447816e-06,
+      "lm_loss": 0.07255371659994125,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6663036346435547
+    },
+    {
+      "global_step": 2800,
+      "optimizer_step": 350,
+      "lr": 8.472464158814255e-06,
+      "lm_loss": 0.0543871708214283,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.304387092590332
+    },
+    {
+      "global_step": 2820,
+      "optimizer_step": 352,
+      "lr": 8.211026128824539e-06,
+      "lm_loss": 0.003727344097569585,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.800602436065674
+    },
+    {
+      "global_step": 2840,
+      "optimizer_step": 355,
+      "lr": 7.825076849127458e-06,
+      "lm_loss": 0.013382216915488243,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3415071964263916
+    },
+    {
+      "global_step": 2860,
+      "optimizer_step": 357,
+      "lr": 7.571989645579419e-06,
+      "lm_loss": 0.10955724865198135,
+      "order_loss": 60.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1095571517944336
+    },
+    {
+      "global_step": 2880,
+      "optimizer_step": 360,
+      "lr": 7.198801765896115e-06,
+      "lm_loss": 0.21780653297901154,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.420931577682495
+    },
+    {
+      "global_step": 2900,
+      "optimizer_step": 362,
+      "lr": 6.9543764955299265e-06,
+      "lm_loss": 0.02760743536055088,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4963574409484863
+    },
+    {
+      "global_step": 2920,
+      "optimizer_step": 365,
+      "lr": 6.594408756022272e-06,
+      "lm_loss": 0.18195956945419312,
+      "order_loss": 58.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.088209629058838
+    },
+    {
+      "global_step": 2940,
+      "optimizer_step": 367,
+      "lr": 6.358945877920861e-06,
+      "lm_loss": 0.002865887712687254,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.596615791320801
+    },
+    {
+      "global_step": 2960,
+      "optimizer_step": 370,
+      "lr": 6.012640767930217e-06,
+      "lm_loss": 0.014526529237627983,
+      "order_loss": 54.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.717651605606079
+    },
+    {
+      "global_step": 2980,
+      "optimizer_step": 372,
+      "lr": 5.7864297241808e-06,
+      "lm_loss": 0.0003817374526988715,
+      "order_loss": 55.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7816317081451416
+    },
+    {
+      "global_step": 3000,
+      "optimizer_step": 375,
+      "lr": 5.454212938299255e-06,
+      "lm_loss": 0.0004669543413911015,
+      "order_loss": 41.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.04734206199646
+    },
+    {
+      "global_step": 3020,
+      "optimizer_step": 377,
+      "lr": 5.237531798197415e-06,
+      "lm_loss": 0.004930449184030294,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4893054962158203
+    },
+    {
+      "global_step": 3040,
+      "optimizer_step": 380,
+      "lr": 4.91981171298388e-06,
+      "lm_loss": 0.0053327856585383415,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5834577083587646
+    },
+    {
+      "global_step": 3060,
+      "optimizer_step": 382,
+      "lr": 4.712926831217685e-06,
+      "lm_loss": 0.07085570693016052,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5552306175231934
+    },
+    {
+      "global_step": 3080,
+      "optimizer_step": 385,
+      "lr": 4.410094003202134e-06,
+      "lm_loss": 0.008409897796809673,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.2584099769592285
+    },
+    {
+      "global_step": 3100,
+      "optimizer_step": 387,
+      "lr": 4.213259692436367e-06,
+      "lm_loss": 0.0022729835473001003,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4866480827331543
+    },
+    {
+      "global_step": 3120,
+      "optimizer_step": 390,
+      "lr": 3.925686378029331e-06,
+      "lm_loss": 0.008358649909496307,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8521087169647217
+    },
+    {
+      "global_step": 3140,
+      "optimizer_step": 392,
+      "lr": 3.739144596292382e-06,
+      "lm_loss": 0.10174979269504547,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.992374897003174
+    },
+    {
+      "global_step": 3160,
+      "optimizer_step": 395,
+      "lr": 3.4671842941897765e-06,
+      "lm_loss": 0.38187363743782043,
+      "order_loss": 71.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.959998607635498
+    },
+    {
+      "global_step": 3180,
+      "optimizer_step": 397,
+      "lr": 3.2911643474473646e-06,
+      "lm_loss": 0.0064330208115279675,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8189330101013184
+    },
+    {
+      "global_step": 3200,
+      "optimizer_step": 400,
+      "lr": 3.035151364093214e-06,
+      "lm_loss": 0.00030503649031743407,
+      "order_loss": 60.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.000304937362671
+    },
+    {
+      "global_step": 3220,
+      "optimizer_step": 402,
+      "lr": 2.869869624374699e-06,
+      "lm_loss": 0.16525502502918243,
+      "order_loss": 44.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.384005069732666
+    },
+    {
+      "global_step": 3240,
+      "optimizer_step": 405,
+      "lr": 2.6301186630158485e-06,
+      "lm_loss": 0.005688054021447897,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.099437952041626
+    },
+    {
+      "global_step": 3260,
+      "optimizer_step": 407,
+      "lr": 2.475778302439524e-06,
+      "lm_loss": 0.044212035834789276,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.9348371028900146
+    },
+    {
+      "global_step": 3280,
+      "optimizer_step": 410,
+      "lr": 2.2525840762775863e-06,
+      "lm_loss": 0.18914005160331726,
+      "order_loss": 59.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1578900814056396
+    },
+    {
+      "global_step": 3300,
+      "optimizer_step": 412,
+      "lr": 2.1093748173018848e-06,
+      "lm_loss": 0.008175029419362545,
+      "order_loss": 58.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.9300501346588135
+    },
+    {
+      "global_step": 3320,
+      "optimizer_step": 415,
+      "lr": 1.9030116872178316e-06,
+      "lm_loss": 0.016908491030335426,
+      "order_loss": 68.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.4387834072113037
+    },
+    {
+      "global_step": 3340,
+      "optimizer_step": 417,
+      "lr": 1.771109569425547e-06,
+      "lm_loss": 0.0193562563508749,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1756062507629395
+    },
+    {
+      "global_step": 3360,
+      "optimizer_step": 420,
+      "lr": 1.5818312067223672e-06,
+      "lm_loss": 0.13524892926216125,
+      "order_loss": 44.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.353998899459839
+    },
+    {
+      "global_step": 3380,
+      "optimizer_step": 422,
+      "lr": 1.4613983704244826e-06,
+      "lm_loss": 0.03589756041765213,
+      "order_loss": 67.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3796474933624268
+    },
+    {
+      "global_step": 3400,
+      "optimizer_step": 425,
+      "lr": 1.2894374450024338e-06,
+      "lm_loss": 0.008794480003416538,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.8994195461273193
+    },
+    {
+      "global_step": 3420,
+      "optimizer_step": 427,
+      "lr": 1.180621931927592e-06,
+      "lm_loss": 0.011885435320436954,
+      "order_loss": 45.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.293135404586792
+    },
+    {
+      "global_step": 3440,
+      "optimizer_step": 430,
+      "lr": 1.0261898262753812e-06,
+      "lm_loss": 0.002837997628375888,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.205962896347046
+    },
+    {
+      "global_step": 3460,
+      "optimizer_step": 432,
+      "lr": 9.291253975900139e-07,
+      "lm_loss": 0.008270316757261753,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.992645263671875
+    },
+    {
+      "global_step": 3480,
+      "optimizer_step": 435,
+      "lr": 7.924119469434665e-07,
+      "lm_loss": 0.010878940112888813,
+      "order_loss": 43.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1671290397644043
+    },
+    {
+      "global_step": 3500,
+      "optimizer_step": 437,
+      "lr": 7.072179188262251e-07,
+      "lm_loss": 0.13964684307575226,
+      "order_loss": 59.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.1083967685699463
+    },
+    {
+      "global_step": 3520,
+      "optimizer_step": 440,
+      "lr": 5.883911778138324e-07,
+      "lm_loss": 0.1216261014342308,
+      "order_loss": 51.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.668501138687134
+    },
+    {
+      "global_step": 3540,
+      "optimizer_step": 442,
+      "lr": 5.151722747865434e-07,
+      "lm_loss": 0.5792491436004639,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.391749143600464
+    },
+    {
+      "global_step": 3560,
+      "optimizer_step": 445,
+      "lr": 4.1437831084878974e-07,
+      "lm_loss": 0.20101477205753326,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.669764757156372
+    },
+    {
+      "global_step": 3580,
+      "optimizer_step": 447,
+      "lr": 3.5322453704410286e-07,
+      "lm_loss": 0.0024946837220340967,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.5337445735931396
+    },
+    {
+      "global_step": 3600,
+      "optimizer_step": 450,
+      "lr": 2.705872508804747e-07,
+      "lm_loss": 0.004365373402833939,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2074904441833496
+    },
+    {
+      "global_step": 3620,
+      "optimizer_step": 452,
+      "lr": 2.215737794045436e-07,
+      "lm_loss": 0.07040327042341232,
+      "order_loss": 62.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.2110283374786377
+    },
+    {
+      "global_step": 3640,
+      "optimizer_step": 455,
+      "lr": 1.571947526689349e-07,
+      "lm_loss": 0.009811030700802803,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.3379359245300293
+    },
+    {
+      "global_step": 3660,
+      "optimizer_step": 457,
+      "lr": 1.2038183319507955e-07,
+      "lm_loss": 0.005419779568910599,
+      "order_loss": 33.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.6929197311401367
+    },
+    {
+      "global_step": 3680,
+      "optimizer_step": 460,
+      "lr": 7.434020362682704e-08,
+      "lm_loss": 0.0229925699532032,
+      "order_loss": 55.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7729926109313965
+    },
+    {
+      "global_step": 3700,
+      "optimizer_step": 462,
+      "lr": 4.977308833379324e-08,
+      "lm_loss": 0.00041012815199792385,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.4691600799560547
+    },
+    {
+      "global_step": 3720,
+      "optimizer_step": 465,
+      "lr": 2.2125452477828047e-08,
+      "lm_loss": 0.016065869480371475,
+      "order_loss": 38.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.9379408359527588
+    },
+    {
+      "global_step": 3740,
+      "optimizer_step": 467,
+      "lr": 9.834340423678368e-09,
+      "lm_loss": 0.03799429163336754,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.537994384765625
+    }
+  ]
+}

cls_gemma_orderkd_first_l02/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l02/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l02/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l02/checkpoint-200/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l02/checkpoint-200/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l02/checkpoint-200/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l02/checkpoint-400/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_first_l02/checkpoint-400/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_first_l02/checkpoint-400/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_first_l02/train_metrics.json

@@ -0,0 +1,1874 @@
+{
+  "loss_log": [
+    {
+      "global_step": 20,
+      "optimizer_step": 2,
+      "lr": 4.347826086956522e-06,
+      "lm_loss": 3.1184232234954834,
+      "order_loss": 63.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 15.805923461914062
+    },
+    {
+      "global_step": 40,
+      "optimizer_step": 5,
+      "lr": 1.0869565217391305e-05,
+      "lm_loss": 1.0916705131530762,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.404170989990234
+    },
+    {
+      "global_step": 60,
+      "optimizer_step": 7,
+      "lr": 1.5217391304347828e-05,
+      "lm_loss": 0.6041339039802551,
+      "order_loss": 42.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.041633605957031
+    },
+    {
+      "global_step": 80,
+      "optimizer_step": 10,
+      "lr": 2.173913043478261e-05,
+      "lm_loss": 1.8147910833358765,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.877290725708008
+    },
+    {
+      "global_step": 100,
+      "optimizer_step": 12,
+      "lr": 2.608695652173913e-05,
+      "lm_loss": 0.5634512305259705,
+      "order_loss": 43.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.250950813293457
+    },
+    {
+      "global_step": 120,
+      "optimizer_step": 15,
+      "lr": 3.260869565217392e-05,
+      "lm_loss": 2.721200942993164,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.596200942993164
+    },
+    {
+      "global_step": 140,
+      "optimizer_step": 17,
+      "lr": 3.695652173913043e-05,
+      "lm_loss": 0.6781831979751587,
+      "order_loss": 54.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.490683555603027
+    },
+    {
+      "global_step": 160,
+      "optimizer_step": 20,
+      "lr": 4.347826086956522e-05,
+      "lm_loss": 0.41744333505630493,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.72994327545166
+    },
+    {
+      "global_step": 180,
+      "optimizer_step": 22,
+      "lr": 4.782608695652174e-05,
+      "lm_loss": 0.06321373581886292,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.938213348388672
+    },
+    {
+      "global_step": 200,
+      "optimizer_step": 25,
+      "lr": 4.9997541293989384e-05,
+      "lm_loss": 0.02435319498181343,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.89935302734375
+    },
+    {
+      "global_step": 220,
+      "optimizer_step": 27,
+      "lr": 4.999016565957633e-05,
+      "lm_loss": 0.019743939861655235,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.269743919372559
+    },
+    {
+      "global_step": 240,
+      "optimizer_step": 30,
+      "lr": 4.996988640512931e-05,
+      "lm_loss": 0.3459993302822113,
+      "order_loss": 39.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.283499717712402
+    },
+    {
+      "global_step": 260,
+      "optimizer_step": 32,
+      "lr": 4.995022691166621e-05,
+      "lm_loss": 0.0298320259898901,
+      "order_loss": 62.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.467331886291504
+    },
+    {
+      "global_step": 280,
+      "optimizer_step": 35,
+      "lr": 4.991153735294049e-05,
+      "lm_loss": 0.46130573749542236,
+      "order_loss": 60.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.586305618286133
+    },
+    {
+      "global_step": 300,
+      "optimizer_step": 37,
+      "lr": 4.987961816680492e-05,
+      "lm_loss": 0.1003972664475441,
+      "order_loss": 54.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.037897109985352
+    },
+    {
+      "global_step": 320,
+      "optimizer_step": 40,
+      "lr": 4.9822565862832806e-05,
+      "lm_loss": 0.13187472522258759,
+      "order_loss": 40.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.256875038146973
+    },
+    {
+      "global_step": 340,
+      "optimizer_step": 42,
+      "lr": 4.9778426220595456e-05,
+      "lm_loss": 0.32335221767425537,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.323351860046387
+    },
+    {
+      "global_step": 360,
+      "optimizer_step": 45,
+      "lr": 4.970308130276272e-05,
+      "lm_loss": 0.7211238741874695,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.033623695373535
+    },
+    {
+      "global_step": 380,
+      "optimizer_step": 47,
+      "lr": 4.96467754629559e-05,
+      "lm_loss": 0.03598100692033768,
+      "order_loss": 41.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.410981178283691
+    },
+    {
+      "global_step": 400,
+      "optimizer_step": 50,
+      "lr": 4.955323054879298e-05,
+      "lm_loss": 0.10630018264055252,
+      "order_loss": 47.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.543800354003906
+    },
+    {
+      "global_step": 420,
+      "optimizer_step": 52,
+      "lr": 4.948482772521345e-05,
+      "lm_loss": 0.11145680397748947,
+      "order_loss": 66.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.298956871032715
+    },
+    {
+      "global_step": 440,
+      "optimizer_step": 55,
+      "lr": 4.937319780454559e-05,
+      "lm_loss": 0.18087011575698853,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.055870056152344
+    },
+    {
+      "global_step": 460,
+      "optimizer_step": 57,
+      "lr": 4.929278208117378e-05,
+      "lm_loss": 0.6563955545425415,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.59389591217041
+    },
+    {
+      "global_step": 480,
+      "optimizer_step": 60,
+      "lr": 4.916320437477008e-05,
+      "lm_loss": 0.32431942224502563,
+      "order_loss": 57.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.761819839477539
+    },
+    {
+      "global_step": 500,
+      "optimizer_step": 62,
+      "lr": 4.907087460240999e-05,
+      "lm_loss": 0.14991332590579987,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.399913787841797
+    },
+    {
+      "global_step": 520,
+      "optimizer_step": 65,
+      "lr": 4.892350839330522e-05,
+      "lm_loss": 0.1010974645614624,
+      "order_loss": 50.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.226097106933594
+    },
+    {
+      "global_step": 540,
+      "optimizer_step": 67,
+      "lr": 4.881937806807241e-05,
+      "lm_loss": 0.15496166050434113,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.404961585998535
+    },
+    {
+      "global_step": 560,
+      "optimizer_step": 70,
+      "lr": 4.865440450576874e-05,
+      "lm_loss": 0.026232725009322166,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.401232719421387
+    },
+    {
+      "global_step": 580,
+      "optimizer_step": 72,
+      "lr": 4.853860162957552e-05,
+      "lm_loss": 0.09959332644939423,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.474593162536621
+    },
+    {
+      "global_step": 600,
+      "optimizer_step": 75,
+      "lr": 4.8356223507364996e-05,
+      "lm_loss": 0.01794126257300377,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.58044147491455
+    },
+    {
+      "global_step": 620,
+      "optimizer_step": 77,
+      "lr": 4.822889043057446e-05,
+      "lm_loss": 0.26736268401145935,
+      "order_loss": 49.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.142362594604492
+    },
+    {
+      "global_step": 640,
+      "optimizer_step": 80,
+      "lr": 4.802933193625593e-05,
+      "lm_loss": 0.007746524177491665,
+      "order_loss": 67.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.382746696472168
+    },
+    {
+      "global_step": 660,
+      "optimizer_step": 82,
+      "lr": 4.789062518269812e-05,
+      "lm_loss": 0.03736429661512375,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.59986400604248
+    },
+    {
+      "global_step": 680,
+      "optimizer_step": 85,
+      "lr": 4.767413162299501e-05,
+      "lm_loss": 0.06276468187570572,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.000265121459961
+    },
+    {
+      "global_step": 700,
+      "optimizer_step": 87,
+      "lr": 4.752422169756048e-05,
+      "lm_loss": 0.22046461701393127,
+      "order_loss": 67.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.720464706420898
+    },
+    {
+      "global_step": 720,
+      "optimizer_step": 90,
+      "lr": 4.729105919657818e-05,
+      "lm_loss": 0.4013015925884247,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.901301383972168
+    },
+    {
+      "global_step": 740,
+      "optimizer_step": 92,
+      "lr": 4.7130130375625305e-05,
+      "lm_loss": 0.15300190448760986,
+      "order_loss": 55.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.15300178527832
+    },
+    {
+      "global_step": 760,
+      "optimizer_step": 95,
+      "lr": 4.6880585547718845e-05,
+      "lm_loss": 0.3297501802444458,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.829750061035156
+    },
+    {
+      "global_step": 780,
+      "optimizer_step": 97,
+      "lr": 4.670883565255264e-05,
+      "lm_loss": 0.017043394967913628,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.517043113708496
+    },
+    {
+      "global_step": 800,
+      "optimizer_step": 100,
+      "lr": 4.6443215250006806e-05,
+      "lm_loss": 0.3523687720298767,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.539868354797363
+    },
+    {
+      "global_step": 820,
+      "optimizer_step": 102,
+      "lr": 4.626085540370762e-05,
+      "lm_loss": 0.012201391160488129,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.574701309204102
+    },
+    {
+      "global_step": 840,
+      "optimizer_step": 105,
+      "lr": 4.597948593966256e-05,
+      "lm_loss": 0.07073015719652176,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.008230209350586
+    },
+    {
+      "global_step": 860,
+      "optimizer_step": 107,
+      "lr": 4.5786740307563636e-05,
+      "lm_loss": 0.0668209120631218,
+      "order_loss": 63.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.816821098327637
+    },
+    {
+      "global_step": 880,
+      "optimizer_step": 110,
+      "lr": 4.5489967654649474e-05,
+      "lm_loss": 0.1039772778749466,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.72897720336914
+    },
+    {
+      "global_step": 900,
+      "optimizer_step": 112,
+      "lr": 4.528707316878232e-05,
+      "lm_loss": 0.1377195566892624,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.01271915435791
+    },
+    {
+      "global_step": 920,
+      "optimizer_step": 115,
+      "lr": 4.497526213395623e-05,
+      "lm_loss": 0.1522826999425888,
+      "order_loss": 55.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.27728271484375
+    },
+    {
+      "global_step": 940,
+      "optimizer_step": 117,
+      "lr": 4.476246820180259e-05,
+      "lm_loss": 0.6368707418441772,
+      "order_loss": 52.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.074370384216309
+    },
+    {
+      "global_step": 960,
+      "optimizer_step": 120,
+      "lr": 4.443600207791081e-05,
+      "lm_loss": 0.2829626798629761,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.595462799072266
+    },
+    {
+      "global_step": 980,
+      "optimizer_step": 122,
+      "lr": 4.421357027581921e-05,
+      "lm_loss": 0.8739344477653503,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.936434745788574
+    },
+    {
+      "global_step": 1000,
+      "optimizer_step": 125,
+      "lr": 4.38728503704354e-05,
+      "lm_loss": 0.07859532535076141,
+      "order_loss": 55.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.203595161437988
+    },
+    {
+      "global_step": 1020,
+      "optimizer_step": 127,
+      "lr": 4.364105412207914e-05,
+      "lm_loss": 0.25118908286094666,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.251189231872559
+    },
+    {
+      "global_step": 1040,
+      "optimizer_step": 130,
+      "lr": 4.328649926419819e-05,
+      "lm_loss": 0.06467773020267487,
+      "order_loss": 52.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.502178192138672
+    },
+    {
+      "global_step": 1060,
+      "optimizer_step": 132,
+      "lr": 4.304562350447008e-05,
+      "lm_loss": 0.18095380067825317,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.368453979492188
+    },
+    {
+      "global_step": 1080,
+      "optimizer_step": 135,
+      "lr": 4.267766952966369e-05,
+      "lm_loss": 0.02313944138586521,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.523138999938965
+    },
+    {
+      "global_step": 1100,
+      "optimizer_step": 137,
+      "lr": 4.242801035442058e-05,
+      "lm_loss": 0.5191475749015808,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.894147872924805
+    },
+    {
+      "global_step": 1120,
+      "optimizer_step": 140,
+      "lr": 4.204710956908763e-05,
+      "lm_loss": 0.81064772605896,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.185647964477539
+    },
+    {
+      "global_step": 1140,
+      "optimizer_step": 142,
+      "lr": 4.178897387117546e-05,
+      "lm_loss": 0.0647754892706871,
+      "order_loss": 61.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.252275466918945
+    },
+    {
+      "global_step": 1160,
+      "optimizer_step": 145,
+      "lr": 4.139559449654561e-05,
+      "lm_loss": 0.048019688576459885,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.735519409179688
+    },
+    {
+      "global_step": 1180,
+      "optimizer_step": 147,
+      "lr": 4.1129299588552193e-05,
+      "lm_loss": 0.160772904753685,
+      "order_loss": 61.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.410773277282715
+    },
+    {
+      "global_step": 1200,
+      "optimizer_step": 150,
+      "lr": 4.072392518512623e-05,
+      "lm_loss": 0.4824674129486084,
+      "order_loss": 63.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.169967651367188
+    },
+    {
+      "global_step": 1220,
+      "optimizer_step": 152,
+      "lr": 4.0449798409325665e-05,
+      "lm_loss": 0.05526652932167053,
+      "order_loss": 52.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.555266380310059
+    },
+    {
+      "global_step": 1240,
+      "optimizer_step": 155,
+      "lr": 4.0032927282460146e-05,
+      "lm_loss": 0.33561477065086365,
+      "order_loss": 60.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.398115158081055
+    },
+    {
+      "global_step": 1260,
+      "optimizer_step": 157,
+      "lr": 3.9751305608428205e-05,
+      "lm_loss": 0.17131206393241882,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.54631233215332
+    },
+    {
+      "global_step": 1280,
+      "optimizer_step": 160,
+      "lr": 3.932345019579495e-05,
+      "lm_loss": 0.04662935063242912,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.421628952026367
+    },
+    {
+      "global_step": 1300,
+      "optimizer_step": 162,
+      "lr": 3.90346798061902e-05,
+      "lm_loss": 0.050219446420669556,
+      "order_loss": 50.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.112719535827637
+    },
+    {
+      "global_step": 1320,
+      "optimizer_step": 165,
+      "lr": 3.859636604786372e-05,
+      "lm_loss": 0.34347474575042725,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.968474388122559
+    },
+    {
+      "global_step": 1340,
+      "optimizer_step": 167,
+      "lr": 3.830080191288342e-05,
+      "lm_loss": 0.4303498864173889,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.305350303649902
+    },
+    {
+      "global_step": 1360,
+      "optimizer_step": 170,
+      "lr": 3.785256860483054e-05,
+      "lm_loss": 0.07431875914335251,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.761818885803223
+    },
+    {
+      "global_step": 1380,
+      "optimizer_step": 172,
+      "lr": 3.755057404586455e-05,
+      "lm_loss": 0.14596708118915558,
+      "order_loss": 39.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 7.98971700668335
+    },
+    {
+      "global_step": 1400,
+      "optimizer_step": 175,
+      "lr": 3.7092972177631e-05,
+      "lm_loss": 0.13190682232379913,
+      "order_loss": 65.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.131906509399414
+    },
+    {
+      "global_step": 1420,
+      "optimizer_step": 177,
+      "lr": 3.678491842064995e-05,
+      "lm_loss": 0.15802578628063202,
+      "order_loss": 51.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.408025741577148
+    },
+    {
+      "global_step": 1440,
+      "optimizer_step": 180,
+      "lr": 3.631851049805793e-05,
+      "lm_loss": 0.30685389041900635,
+      "order_loss": 45.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.431854248046875
+    },
+    {
+      "global_step": 1460,
+      "optimizer_step": 182,
+      "lr": 3.600477621728473e-05,
+      "lm_loss": 0.01724999025464058,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.267250061035156
+    },
+    {
+      "global_step": 1480,
+      "optimizer_step": 185,
+      "lr": 3.5530135570974274e-05,
+      "lm_loss": 0.25045639276504517,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.375456809997559
+    },
+    {
+      "global_step": 1500,
+      "optimizer_step": 187,
+      "lr": 3.521110642339991e-05,
+      "lm_loss": 0.4265330731868744,
+      "order_loss": 52.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.989032745361328
+    },
+    {
+      "global_step": 1520,
+      "optimizer_step": 190,
+      "lr": 3.4728816504063726e-05,
+      "lm_loss": 0.23969930410385132,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.614699363708496
+    },
+    {
+      "global_step": 1540,
+      "optimizer_step": 192,
+      "lr": 3.440488465537943e-05,
+      "lm_loss": 0.006110867951065302,
+      "order_loss": 65.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.006111145019531
+    },
+    {
+      "global_step": 1560,
+      "optimizer_step": 195,
+      "lr": 3.391553831655782e-05,
+      "lm_loss": 0.01415237132459879,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.889152526855469
+    },
+    {
+      "global_step": 1580,
+      "optimizer_step": 197,
+      "lr": 3.358710195908653e-05,
+      "lm_loss": 0.2871032953262329,
+      "order_loss": 70.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 14.287103652954102
+    },
+    {
+      "global_step": 1600,
+      "optimizer_step": 200,
+      "lr": 3.309130072840379e-05,
+      "lm_loss": 0.23844397068023682,
+      "order_loss": 58.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.863444328308105
+    },
+    {
+      "global_step": 1620,
+      "optimizer_step": 202,
+      "lr": 3.2758763591623295e-05,
+      "lm_loss": 0.1546662300825119,
+      "order_loss": 50.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.21716594696045
+    },
+    {
+      "global_step": 1640,
+      "optimizer_step": 205,
+      "lr": 3.225711693136156e-05,
+      "lm_loss": 0.08392491936683655,
+      "order_loss": 60.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.208925247192383
+    },
+    {
+      "global_step": 1660,
+      "optimizer_step": 207,
+      "lr": 3.1920887785621235e-05,
+      "lm_loss": 0.043332748115062714,
+      "order_loss": 61.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.418333053588867
+    },
+    {
+      "global_step": 1680,
+      "optimizer_step": 210,
+      "lr": 3.141401234354067e-05,
+      "lm_loss": 0.13834291696548462,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.07584285736084
+    },
+    {
+      "global_step": 1700,
+      "optimizer_step": 212,
+      "lr": 3.10745044975816e-05,
+      "lm_loss": 0.2134607881307602,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.338460922241211
+    },
+    {
+      "global_step": 1720,
+      "optimizer_step": 215,
+      "lr": 3.056302334890786e-05,
+      "lm_loss": 0.01857478730380535,
+      "order_loss": 74.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 14.831074714660645
+    },
+    {
+      "global_step": 1740,
+      "optimizer_step": 217,
+      "lr": 3.022065414180425e-05,
+      "lm_loss": 0.006793432869017124,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.506793022155762
+    },
+    {
+      "global_step": 1760,
+      "optimizer_step": 220,
+      "lr": 2.9705196023315006e-05,
+      "lm_loss": 0.013562669977545738,
+      "order_loss": 59.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.888562202453613
+    },
+    {
+      "global_step": 1780,
+      "optimizer_step": 222,
+      "lr": 2.9360386311461275e-05,
+      "lm_loss": 0.0029967674054205418,
+      "order_loss": 47.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.440496444702148
+    },
+    {
+      "global_step": 1800,
+      "optimizer_step": 225,
+      "lr": 2.884158484861325e-05,
+      "lm_loss": 0.012671629898250103,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.51267147064209
+    },
+    {
+      "global_step": 1820,
+      "optimizer_step": 227,
+      "lr": 2.849475848838749e-05,
+      "lm_loss": 0.0835554227232933,
+      "order_loss": 39.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 7.989805221557617
+    },
+    {
+      "global_step": 1840,
+      "optimizer_step": 230,
+      "lr": 2.7973251416434175e-05,
+      "lm_loss": 0.05105651170015335,
+      "order_loss": 29.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 5.957306385040283
+    },
+    {
+      "global_step": 1860,
+      "optimizer_step": 232,
+      "lr": 2.7624834743173867e-05,
+      "lm_loss": 0.5029534101486206,
+      "order_loss": 75.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 15.50295352935791
+    },
+    {
+      "global_step": 1880,
+      "optimizer_step": 235,
+      "lr": 2.710126312323119e-05,
+      "lm_loss": 0.12120761722326279,
+      "order_loss": 62.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.621207237243652
+    },
+    {
+      "global_step": 1900,
+      "optimizer_step": 237,
+      "lr": 2.6751684427161683e-05,
+      "lm_loss": 0.17923295497894287,
+      "order_loss": 40.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.241732597351074
+    },
+    {
+      "global_step": 1920,
+      "optimizer_step": 240,
+      "lr": 2.6226691858185454e-05,
+      "lm_loss": 0.004263256676495075,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.004262924194336
+    },
+    {
+      "global_step": 1940,
+      "optimizer_step": 242,
+      "lr": 2.587638085794541e-05,
+      "lm_loss": 0.04619055613875389,
+      "order_loss": 56.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.35869026184082
+    },
+    {
+      "global_step": 1960,
+      "optimizer_step": 245,
+      "lr": 2.535061268558906e-05,
+      "lm_loss": 0.11890175938606262,
+      "order_loss": 71.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 14.431402206420898
+    },
+    {
+      "global_step": 1980,
+      "optimizer_step": 247,
+      "lr": 2.5e-05,
+      "lm_loss": 0.25617969036102295,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.631179809570312
+    },
+    {
+      "global_step": 2000,
+      "optimizer_step": 250,
+      "lr": 2.447410252332528e-05,
+      "lm_loss": 0.013576528988778591,
+      "order_loss": 54.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.88857650756836
+    },
+    {
+      "global_step": 2020,
+      "optimizer_step": 252,
+      "lr": 2.4123619142054597e-05,
+      "lm_loss": 0.04630662128329277,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.046306610107422
+    },
+    {
+      "global_step": 2040,
+      "optimizer_step": 255,
+      "lr": 2.3598238819070202e-05,
+      "lm_loss": 0.14818225800991058,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.460681915283203
+    },
+    {
+      "global_step": 2060,
+      "optimizer_step": 257,
+      "lr": 2.3248315572838316e-05,
+      "lm_loss": 0.0021958579309284687,
+      "order_loss": 44.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.939696311950684
+    },
+    {
+      "global_step": 2080,
+      "optimizer_step": 260,
+      "lr": 2.272409822584325e-05,
+      "lm_loss": 0.002902108011767268,
+      "order_loss": 43.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.752902030944824
+    },
+    {
+      "global_step": 2100,
+      "optimizer_step": 262,
+      "lr": 2.237516525682614e-05,
+      "lm_loss": 0.016654551029205322,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.391654968261719
+    },
+    {
+      "global_step": 2120,
+      "optimizer_step": 265,
+      "lr": 2.185275527853437e-05,
+      "lm_loss": 0.012752072885632515,
+      "order_loss": 43.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.637752532958984
+    },
+    {
+      "global_step": 2140,
+      "optimizer_step": 267,
+      "lr": 2.1505241511612522e-05,
+      "lm_loss": 0.009639913216233253,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.884639739990234
+    },
+    {
+      "global_step": 2160,
+      "optimizer_step": 270,
+      "lr": 2.0985281073035076e-05,
+      "lm_loss": 0.11398650705814362,
+      "order_loss": 44.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.98898696899414
+    },
+    {
+      "global_step": 2180,
+      "optimizer_step": 272,
+      "lr": 2.0639613688538734e-05,
+      "lm_loss": 0.0007875743904151022,
+      "order_loss": 51.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.188287734985352
+    },
+    {
+      "global_step": 2200,
+      "optimizer_step": 275,
+      "lr": 2.0122741949596797e-05,
+      "lm_loss": 0.009382687509059906,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.509382247924805
+    },
+    {
+      "global_step": 2220,
+      "optimizer_step": 277,
+      "lr": 1.977934585819576e-05,
+      "lm_loss": 0.2916325628757477,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.916632652282715
+    },
+    {
+      "global_step": 2240,
+      "optimizer_step": 280,
+      "lr": 1.9266198182035058e-05,
+      "lm_loss": 0.006090455688536167,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.31859016418457
+    },
+    {
+      "global_step": 2260,
+      "optimizer_step": 282,
+      "lr": 1.8925495502418406e-05,
+      "lm_loss": 0.06058869510889053,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.310588836669922
+    },
+    {
+      "global_step": 2280,
+      "optimizer_step": 285,
+      "lr": 1.841670267439088e-05,
+      "lm_loss": 0.0836532935500145,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.458653450012207
+    },
+    {
+      "global_step": 2300,
+      "optimizer_step": 287,
+      "lr": 1.8079112214378768e-05,
+      "lm_loss": 0.05897977575659752,
+      "order_loss": 58.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.808979988098145
+    },
+    {
+      "global_step": 2320,
+      "optimizer_step": 290,
+      "lr": 1.757529966665141e-05,
+      "lm_loss": 0.1328563094139099,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.445356369018555
+    },
+    {
+      "global_step": 2340,
+      "optimizer_step": 292,
+      "lr": 1.72412364083767e-05,
+      "lm_loss": 0.044375427067279816,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.6693754196167
+    },
+    {
+      "global_step": 2360,
+      "optimizer_step": 295,
+      "lr": 1.6743023451120832e-05,
+      "lm_loss": 0.18005213141441345,
+      "order_loss": 58.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.805051803588867
+    },
+    {
+      "global_step": 2380,
+      "optimizer_step": 297,
+      "lr": 1.641289804091347e-05,
+      "lm_loss": 0.07155724614858627,
+      "order_loss": 48.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.69655704498291
+    },
+    {
+      "global_step": 2400,
+      "optimizer_step": 300,
+      "lr": 1.5920897101019383e-05,
+      "lm_loss": 0.21017329394817352,
+      "order_loss": 48.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.835173606872559
+    },
+    {
+      "global_step": 2420,
+      "optimizer_step": 302,
+      "lr": 1.5595115344620576e-05,
+      "lm_loss": 0.013050842098891735,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.51305103302002
+    },
+    {
+      "global_step": 2440,
+      "optimizer_step": 305,
+      "lr": 1.5109931212873534e-05,
+      "lm_loss": 0.022372275590896606,
+      "order_loss": 61.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.209872245788574
+    },
+    {
+      "global_step": 2460,
+      "optimizer_step": 307,
+      "lr": 1.4788893576600099e-05,
+      "lm_loss": 0.2720758318901062,
+      "order_loss": 64.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.147075653076172
+    },
+    {
+      "global_step": 2480,
+      "optimizer_step": 310,
+      "lr": 1.4311122664242954e-05,
+      "lm_loss": 0.0060526239685714245,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.31855297088623
+    },
+    {
+      "global_step": 2500,
+      "optimizer_step": 312,
+      "lr": 1.399522378271527e-05,
+      "lm_loss": 0.8424654603004456,
+      "order_loss": 57.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.2174654006958
+    },
+    {
+      "global_step": 2520,
+      "optimizer_step": 315,
+      "lr": 1.3525453388311554e-05,
+      "lm_loss": 0.12010786682367325,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.995107650756836
+    },
+    {
+      "global_step": 2540,
+      "optimizer_step": 317,
+      "lr": 1.3215081579350058e-05,
+      "lm_loss": 0.6840029954910278,
+      "order_loss": 47.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.059002876281738
+    },
+    {
+      "global_step": 2560,
+      "optimizer_step": 320,
+      "lr": 1.2753889166848909e-05,
+      "lm_loss": 0.005617386195808649,
+      "order_loss": 63.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.630617141723633
+    },
+    {
+      "global_step": 2580,
+      "optimizer_step": 322,
+      "lr": 1.2449425954135452e-05,
+      "lm_loss": 0.009596875868737698,
+      "order_loss": 46.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.197096824645996
+    },
+    {
+      "global_step": 2600,
+      "optimizer_step": 325,
+      "lr": 1.1997378443025634e-05,
+      "lm_loss": 0.006365991663187742,
+      "order_loss": 50.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.131365776062012
+    },
+    {
+      "global_step": 2620,
+      "optimizer_step": 327,
+      "lr": 1.1699198087116589e-05,
+      "lm_loss": 0.0015480580041185021,
+      "order_loss": 47.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.501547813415527
+    },
+    {
+      "global_step": 2640,
+      "optimizer_step": 330,
+      "lr": 1.1256851155542298e-05,
+      "lm_loss": 0.0021843877620995045,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.37718391418457
+    },
+    {
+      "global_step": 2660,
+      "optimizer_step": 332,
+      "lr": 1.0965320193809808e-05,
+      "lm_loss": 0.01388256810605526,
+      "order_loss": 52.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.513882637023926
+    },
+    {
+      "global_step": 2680,
+      "optimizer_step": 335,
+      "lr": 1.0533217595504858e-05,
+      "lm_loss": 0.0009903698228299618,
+      "order_loss": 53.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.68848991394043
+    },
+    {
+      "global_step": 2700,
+      "optimizer_step": 337,
+      "lr": 1.0248694391571801e-05,
+      "lm_loss": 0.0581960529088974,
+      "order_loss": 51.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.308196067810059
+    },
+    {
+      "global_step": 2720,
+      "optimizer_step": 340,
+      "lr": 9.82736728745186e-06,
+      "lm_loss": 0.05928004905581474,
+      "order_loss": 57.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.559280395507812
+    },
+    {
+      "global_step": 2740,
+      "optimizer_step": 342,
+      "lr": 9.550201590674343e-06,
+      "lm_loss": 0.00916436966508627,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.321664810180664
+    },
+    {
+      "global_step": 2760,
+      "optimizer_step": 345,
+      "lr": 9.140167895908867e-06,
+      "lm_loss": 0.02078745700418949,
+      "order_loss": 45.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.083287239074707
+    },
+    {
+      "global_step": 2780,
+      "optimizer_step": 347,
+      "lr": 8.870700411447816e-06,
+      "lm_loss": 0.0645354613661766,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.439535140991211
+    },
+    {
+      "global_step": 2800,
+      "optimizer_step": 350,
+      "lr": 8.472464158814255e-06,
+      "lm_loss": 0.03742121905088425,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.037421226501465
+    },
+    {
+      "global_step": 2820,
+      "optimizer_step": 352,
+      "lr": 8.211026128824539e-06,
+      "lm_loss": 0.004177003167569637,
+      "order_loss": 56.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.19167709350586
+    },
+    {
+      "global_step": 2840,
+      "optimizer_step": 355,
+      "lr": 7.825076849127458e-06,
+      "lm_loss": 0.017688076943159103,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.330187797546387
+    },
+    {
+      "global_step": 2860,
+      "optimizer_step": 357,
+      "lr": 7.571989645579419e-06,
+      "lm_loss": 0.0998288244009018,
+      "order_loss": 60.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.099828720092773
+    },
+    {
+      "global_step": 2880,
+      "optimizer_step": 360,
+      "lr": 7.198801765896115e-06,
+      "lm_loss": 0.24300594627857208,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.055505752563477
+    },
+    {
+      "global_step": 2900,
+      "optimizer_step": 362,
+      "lr": 6.9543764955299265e-06,
+      "lm_loss": 0.03150952234864235,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.906509399414062
+    },
+    {
+      "global_step": 2920,
+      "optimizer_step": 365,
+      "lr": 6.594408756022272e-06,
+      "lm_loss": 0.1402754932641983,
+      "order_loss": 58.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.765275955200195
+    },
+    {
+      "global_step": 2940,
+      "optimizer_step": 367,
+      "lr": 6.358945877920861e-06,
+      "lm_loss": 0.002310896525159478,
+      "order_loss": 51.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.377310752868652
+    },
+    {
+      "global_step": 2960,
+      "optimizer_step": 370,
+      "lr": 6.012640767930217e-06,
+      "lm_loss": 0.018192529678344727,
+      "order_loss": 54.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.830692291259766
+    },
+    {
+      "global_step": 2980,
+      "optimizer_step": 372,
+      "lr": 5.7864297241808e-06,
+      "lm_loss": 0.00038043089443817735,
+      "order_loss": 55.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.125380516052246
+    },
+    {
+      "global_step": 3000,
+      "optimizer_step": 375,
+      "lr": 5.454212938299255e-06,
+      "lm_loss": 0.0005834042676724494,
+      "order_loss": 41.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.18808364868164
+    },
+    {
+      "global_step": 3020,
+      "optimizer_step": 377,
+      "lr": 5.237531798197415e-06,
+      "lm_loss": 0.00628126272931695,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.943780899047852
+    },
+    {
+      "global_step": 3040,
+      "optimizer_step": 380,
+      "lr": 4.91981171298388e-06,
+      "lm_loss": 0.006694844923913479,
+      "order_loss": 51.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.319194793701172
+    },
+    {
+      "global_step": 3060,
+      "optimizer_step": 382,
+      "lr": 4.712926831217685e-06,
+      "lm_loss": 0.12685316801071167,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.064352989196777
+    },
+    {
+      "global_step": 3080,
+      "optimizer_step": 385,
+      "lr": 4.410094003202134e-06,
+      "lm_loss": 0.009770600125193596,
+      "order_loss": 45.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.009770393371582
+    },
+    {
+      "global_step": 3100,
+      "optimizer_step": 387,
+      "lr": 4.213259692436367e-06,
+      "lm_loss": 0.002140387427061796,
+      "order_loss": 49.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.939640045166016
+    },
+    {
+      "global_step": 3120,
+      "optimizer_step": 390,
+      "lr": 3.925686378029331e-06,
+      "lm_loss": 0.00836945790797472,
+      "order_loss": 56.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.383369445800781
+    },
+    {
+      "global_step": 3140,
+      "optimizer_step": 392,
+      "lr": 3.739144596292382e-06,
+      "lm_loss": 0.12146015465259552,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.6839599609375
+    },
+    {
+      "global_step": 3160,
+      "optimizer_step": 395,
+      "lr": 3.4671842941897765e-06,
+      "lm_loss": 0.37217986583709717,
+      "order_loss": 71.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 14.684679985046387
+    },
+    {
+      "global_step": 3180,
+      "optimizer_step": 397,
+      "lr": 3.2911643474473646e-06,
+      "lm_loss": 0.00539372256025672,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.255393981933594
+    },
+    {
+      "global_step": 3200,
+      "optimizer_step": 400,
+      "lr": 3.035151364093214e-06,
+      "lm_loss": 0.0002799053327180445,
+      "order_loss": 60.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.000280380249023
+    },
+    {
+      "global_step": 3220,
+      "optimizer_step": 402,
+      "lr": 2.869869624374699e-06,
+      "lm_loss": 0.13275915384292603,
+      "order_loss": 44.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.007759094238281
+    },
+    {
+      "global_step": 3240,
+      "optimizer_step": 405,
+      "lr": 2.6301186630158485e-06,
+      "lm_loss": 0.006312482059001923,
+      "order_loss": 62.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.381312370300293
+    },
+    {
+      "global_step": 3260,
+      "optimizer_step": 407,
+      "lr": 2.475778302439524e-06,
+      "lm_loss": 0.048964012414216995,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.61146354675293
+    },
+    {
+      "global_step": 3280,
+      "optimizer_step": 410,
+      "lr": 2.2525840762775863e-06,
+      "lm_loss": 0.2303650975227356,
+      "order_loss": 59.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.105364799499512
+    },
+    {
+      "global_step": 3300,
+      "optimizer_step": 412,
+      "lr": 2.1093748173018848e-06,
+      "lm_loss": 0.007891875691711903,
+      "order_loss": 58.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.695391654968262
+    },
+    {
+      "global_step": 3320,
+      "optimizer_step": 415,
+      "lr": 1.9030116872178316e-06,
+      "lm_loss": 0.013733092695474625,
+      "order_loss": 68.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.70123291015625
+    },
+    {
+      "global_step": 3340,
+      "optimizer_step": 417,
+      "lr": 1.771109569425547e-06,
+      "lm_loss": 0.021751023828983307,
+      "order_loss": 63.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.646751403808594
+    },
+    {
+      "global_step": 3360,
+      "optimizer_step": 420,
+      "lr": 1.5818312067223672e-06,
+      "lm_loss": 0.12683044373989105,
+      "order_loss": 44.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.001830101013184
+    },
+    {
+      "global_step": 3380,
+      "optimizer_step": 422,
+      "lr": 1.4613983704244826e-06,
+      "lm_loss": 0.03174825385212898,
+      "order_loss": 67.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.406747817993164
+    },
+    {
+      "global_step": 3400,
+      "optimizer_step": 425,
+      "lr": 1.2894374450024338e-06,
+      "lm_loss": 0.009823920205235481,
+      "order_loss": 57.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.5723237991333
+    },
+    {
+      "global_step": 3420,
+      "optimizer_step": 427,
+      "lr": 1.180621931927592e-06,
+      "lm_loss": 0.010670330375432968,
+      "order_loss": 45.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.13567066192627
+    },
+    {
+      "global_step": 3440,
+      "optimizer_step": 430,
+      "lr": 1.0261898262753812e-06,
+      "lm_loss": 0.003456930862739682,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.815957069396973
+    },
+    {
+      "global_step": 3460,
+      "optimizer_step": 432,
+      "lr": 9.291253975900139e-07,
+      "lm_loss": 0.007835650816559792,
+      "order_loss": 59.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.945335388183594
+    },
+    {
+      "global_step": 3480,
+      "optimizer_step": 435,
+      "lr": 7.924119469434665e-07,
+      "lm_loss": 0.00920460931956768,
+      "order_loss": 43.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 8.634204864501953
+    },
+    {
+      "global_step": 3500,
+      "optimizer_step": 437,
+      "lr": 7.072179188262251e-07,
+      "lm_loss": 0.15896598994731903,
+      "order_loss": 59.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.033966064453125
+    },
+    {
+      "global_step": 3520,
+      "optimizer_step": 440,
+      "lr": 5.883911778138324e-07,
+      "lm_loss": 0.11559732258319855,
+      "order_loss": 51.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.30309772491455
+    },
+    {
+      "global_step": 3540,
+      "optimizer_step": 442,
+      "lr": 5.151722747865434e-07,
+      "lm_loss": 0.548110842704773,
+      "order_loss": 56.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.798110961914062
+    },
+    {
+      "global_step": 3560,
+      "optimizer_step": 445,
+      "lr": 4.1437831084878974e-07,
+      "lm_loss": 0.2126510888338089,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.087651252746582
+    },
+    {
+      "global_step": 3580,
+      "optimizer_step": 447,
+      "lr": 3.5322453704410286e-07,
+      "lm_loss": 0.0028603943064808846,
+      "order_loss": 50.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.127860069274902
+    },
+    {
+      "global_step": 3600,
+      "optimizer_step": 450,
+      "lr": 2.705872508804747e-07,
+      "lm_loss": 0.0031662171240895987,
+      "order_loss": 64.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.815666198730469
+    },
+    {
+      "global_step": 3620,
+      "optimizer_step": 452,
+      "lr": 2.215737794045436e-07,
+      "lm_loss": 0.052710987627506256,
+      "order_loss": 62.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 12.61521053314209
+    },
+    {
+      "global_step": 3640,
+      "optimizer_step": 455,
+      "lr": 1.571947526689349e-07,
+      "lm_loss": 0.008811240084469318,
+      "order_loss": 66.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 13.321310997009277
+    },
+    {
+      "global_step": 3660,
+      "optimizer_step": 457,
+      "lr": 1.2038183319507955e-07,
+      "lm_loss": 0.005917538423091173,
+      "order_loss": 33.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 6.755917549133301
+    },
+    {
+      "global_step": 3680,
+      "optimizer_step": 460,
+      "lr": 7.434020362682704e-08,
+      "lm_loss": 0.032587382942438126,
+      "order_loss": 55.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 11.032587051391602
+    },
+    {
+      "global_step": 3700,
+      "optimizer_step": 462,
+      "lr": 4.977308833379324e-08,
+      "lm_loss": 0.0004790124949067831,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.875478744506836
+    },
+    {
+      "global_step": 3720,
+      "optimizer_step": 465,
+      "lr": 2.2125452477828047e-08,
+      "lm_loss": 0.01606699638068676,
+      "order_loss": 38.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 7.703567028045654
+    },
+    {
+      "global_step": 3740,
+      "optimizer_step": 467,
+      "lr": 9.834340423678368e-09,
+      "lm_loss": 0.03486189246177673,
+      "order_loss": 50.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 10.03486156463623
+    }
+  ]
+}

cls_gemma_orderkd_last_l005/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_last_l005/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_last_l005/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_last_l005/checkpoint-200/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_last_l005/checkpoint-200/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_last_l005/checkpoint-200/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_last_l005/checkpoint-200/order_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b21bd38abed33cc6980122ed7572b780acde66075fce2e5ab6c11fde094bbdc0
+size 4723259

cls_gemma_orderkd_last_l005/checkpoint-200/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

cls_gemma_orderkd_last_l005/order_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49880ebe931bec67ca63dfd24e0139c046d1f17fe6f9de0ffaac5c3f9f97ea54
+size 4723259

cls_gemma_orderkd_last_l005/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

cls_gemma_orderkd_last_l005/train_metrics.json

@@ -0,0 +1,1874 @@
+{
+  "loss_log": [
+    {
+      "global_step": 20,
+      "optimizer_step": 2,
+      "lr": 4.347826086956522e-06,
+      "lm_loss": 3.154064655303955,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 6.232189655303955
+    },
+    {
+      "global_step": 40,
+      "optimizer_step": 5,
+      "lr": 1.0869565217391305e-05,
+      "lm_loss": 1.2224328517913818,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.941182851791382
+    },
+    {
+      "global_step": 60,
+      "optimizer_step": 7,
+      "lr": 1.5217391304347828e-05,
+      "lm_loss": 0.6663519740104675,
+      "order_loss": 42.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.7757270336151123
+    },
+    {
+      "global_step": 80,
+      "optimizer_step": 10,
+      "lr": 2.173913043478261e-05,
+      "lm_loss": 1.8447270393371582,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.610352039337158
+    },
+    {
+      "global_step": 100,
+      "optimizer_step": 12,
+      "lr": 2.608695652173913e-05,
+      "lm_loss": 0.5551432371139526,
+      "order_loss": 43.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.727018356323242
+    },
+    {
+      "global_step": 120,
+      "optimizer_step": 15,
+      "lr": 3.260869565217392e-05,
+      "lm_loss": 2.73148775100708,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 5.20023775100708
+    },
+    {
+      "global_step": 140,
+      "optimizer_step": 17,
+      "lr": 3.695652173913043e-05,
+      "lm_loss": 0.702565610408783,
+      "order_loss": 46.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0306906700134277
+    },
+    {
+      "global_step": 160,
+      "optimizer_step": 20,
+      "lr": 4.347826086956522e-05,
+      "lm_loss": 0.417985200881958,
+      "order_loss": 42.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.558610200881958
+    },
+    {
+      "global_step": 180,
+      "optimizer_step": 22,
+      "lr": 4.782608695652174e-05,
+      "lm_loss": 0.03622516244649887,
+      "order_loss": 18.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.9737251400947571
+    },
+    {
+      "global_step": 200,
+      "optimizer_step": 25,
+      "lr": 4.9997541293989384e-05,
+      "lm_loss": 0.024076402187347412,
+      "order_loss": 21.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.1100139617919922
+    },
+    {
+      "global_step": 220,
+      "optimizer_step": 27,
+      "lr": 4.999016565957633e-05,
+      "lm_loss": 0.026584738865494728,
+      "order_loss": 13.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.6906472444534302
+    },
+    {
+      "global_step": 240,
+      "optimizer_step": 30,
+      "lr": 4.996988640512931e-05,
+      "lm_loss": 0.32719624042510986,
+      "order_loss": 26.75,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.6631337404251099
+    },
+    {
+      "global_step": 260,
+      "optimizer_step": 32,
+      "lr": 4.995022691166621e-05,
+      "lm_loss": 0.03321271017193794,
+      "order_loss": 6.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.36524397134780884
+    },
+    {
+      "global_step": 280,
+      "optimizer_step": 35,
+      "lr": 4.991153735294049e-05,
+      "lm_loss": 0.583421528339386,
+      "order_loss": 20.125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.5912339687347412
+    },
+    {
+      "global_step": 300,
+      "optimizer_step": 37,
+      "lr": 4.987961816680492e-05,
+      "lm_loss": 0.13797391951084137,
+      "order_loss": 21.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.1848489046096802
+    },
+    {
+      "global_step": 320,
+      "optimizer_step": 40,
+      "lr": 4.9822565862832806e-05,
+      "lm_loss": 0.07645569741725922,
+      "order_loss": 16.875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.9202057123184204
+    },
+    {
+      "global_step": 340,
+      "optimizer_step": 42,
+      "lr": 4.9778426220595456e-05,
+      "lm_loss": 0.25512945652008057,
+      "order_loss": 15.3125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.0207544565200806
+    },
+    {
+      "global_step": 360,
+      "optimizer_step": 45,
+      "lr": 4.970308130276272e-05,
+      "lm_loss": 0.5102730989456177,
+      "order_loss": 12.9375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.1587105989456177
+    },
+    {
+      "global_step": 380,
+      "optimizer_step": 47,
+      "lr": 4.96467754629559e-05,
+      "lm_loss": 0.021939177066087723,
+      "order_loss": 3.890625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21627511084079742
+    },
+    {
+      "global_step": 400,
+      "optimizer_step": 50,
+      "lr": 4.955323054879298e-05,
+      "lm_loss": 0.10563353449106216,
+      "order_loss": 15.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.882977306842804
+    },
+    {
+      "global_step": 420,
+      "optimizer_step": 52,
+      "lr": 4.948482772521345e-05,
+      "lm_loss": 0.10611257702112198,
+      "order_loss": 4.3125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3219328820705414
+    },
+    {
+      "global_step": 440,
+      "optimizer_step": 55,
+      "lr": 4.937319780454559e-05,
+      "lm_loss": 0.2084236443042755,
+      "order_loss": 0.94140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2555427849292755
+    },
+    {
+      "global_step": 460,
+      "optimizer_step": 57,
+      "lr": 4.929278208117378e-05,
+      "lm_loss": 0.6529907584190369,
+      "order_loss": 13.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.3326783180236816
+    },
+    {
+      "global_step": 480,
+      "optimizer_step": 60,
+      "lr": 4.916320437477008e-05,
+      "lm_loss": 0.35254889726638794,
+      "order_loss": 1.4140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.42334967851638794
+    },
+    {
+      "global_step": 500,
+      "optimizer_step": 62,
+      "lr": 4.907087460240999e-05,
+      "lm_loss": 0.13709476590156555,
+      "order_loss": 2.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.26795414090156555
+    },
+    {
+      "global_step": 520,
+      "optimizer_step": 65,
+      "lr": 4.892350839330522e-05,
+      "lm_loss": 0.09153623133897781,
+      "order_loss": 1.3515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15891903638839722
+    },
+    {
+      "global_step": 540,
+      "optimizer_step": 67,
+      "lr": 4.881937806807241e-05,
+      "lm_loss": 0.18383583426475525,
+      "order_loss": 3.734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.37035927176475525
+    },
+    {
+      "global_step": 560,
+      "optimizer_step": 70,
+      "lr": 4.865440450576874e-05,
+      "lm_loss": 0.03243543952703476,
+      "order_loss": 1.7578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12032606452703476
+    },
+    {
+      "global_step": 580,
+      "optimizer_step": 72,
+      "lr": 4.853860162957552e-05,
+      "lm_loss": 0.08569248020648956,
+      "order_loss": 0.78515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12499912083148956
+    },
+    {
+      "global_step": 600,
+      "optimizer_step": 75,
+      "lr": 4.8356223507364996e-05,
+      "lm_loss": 0.015551693737506866,
+      "order_loss": 0.85546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.058276303112506866
+    },
+    {
+      "global_step": 620,
+      "optimizer_step": 77,
+      "lr": 4.822889043057446e-05,
+      "lm_loss": 0.2337014377117157,
+      "order_loss": 0.80078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2737405002117157
+    },
+    {
+      "global_step": 640,
+      "optimizer_step": 80,
+      "lr": 4.802933193625593e-05,
+      "lm_loss": 0.008940918371081352,
+      "order_loss": 0.76953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0475151389837265
+    },
+    {
+      "global_step": 660,
+      "optimizer_step": 82,
+      "lr": 4.789062518269812e-05,
+      "lm_loss": 0.04632673040032387,
+      "order_loss": 0.62890625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07782086730003357
+    },
+    {
+      "global_step": 680,
+      "optimizer_step": 85,
+      "lr": 4.767413162299501e-05,
+      "lm_loss": 0.09334154427051544,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12068529427051544
+    },
+    {
+      "global_step": 700,
+      "optimizer_step": 87,
+      "lr": 4.752422169756048e-05,
+      "lm_loss": 0.21684736013412476,
+      "order_loss": 0.68359375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.25102704763412476
+    },
+    {
+      "global_step": 720,
+      "optimizer_step": 90,
+      "lr": 4.729105919657818e-05,
+      "lm_loss": 0.4077761769294739,
+      "order_loss": 0.7578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4456179738044739
+    },
+    {
+      "global_step": 740,
+      "optimizer_step": 92,
+      "lr": 4.7130130375625305e-05,
+      "lm_loss": 0.1819460242986679,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2106325477361679
+    },
+    {
+      "global_step": 760,
+      "optimizer_step": 95,
+      "lr": 4.6880585547718845e-05,
+      "lm_loss": 0.3036164343357086,
+      "order_loss": 0.5546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3313263952732086
+    },
+    {
+      "global_step": 780,
+      "optimizer_step": 97,
+      "lr": 4.670883565255264e-05,
+      "lm_loss": 0.01904880627989769,
+      "order_loss": 0.6953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05371677502989769
+    },
+    {
+      "global_step": 800,
+      "optimizer_step": 100,
+      "lr": 4.6443215250006806e-05,
+      "lm_loss": 0.33073675632476807,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.36064398288726807
+    },
+    {
+      "global_step": 820,
+      "optimizer_step": 102,
+      "lr": 4.626085540370762e-05,
+      "lm_loss": 0.011265389621257782,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03982984274625778
+    },
+    {
+      "global_step": 840,
+      "optimizer_step": 105,
+      "lr": 4.597948593966256e-05,
+      "lm_loss": 0.03181800618767738,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05989417806267738
+    },
+    {
+      "global_step": 860,
+      "optimizer_step": 107,
+      "lr": 4.5786740307563636e-05,
+      "lm_loss": 0.04456106200814247,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07251515984535217
+    },
+    {
+      "global_step": 880,
+      "optimizer_step": 110,
+      "lr": 4.5489967654649474e-05,
+      "lm_loss": 0.08861854672431946,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11620643734931946
+    },
+    {
+      "global_step": 900,
+      "optimizer_step": 112,
+      "lr": 4.528707316878232e-05,
+      "lm_loss": 0.1683403104543686,
+      "order_loss": 2.0625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2713676691055298
+    },
+    {
+      "global_step": 920,
+      "optimizer_step": 115,
+      "lr": 4.497526213395623e-05,
+      "lm_loss": 0.1865161657333374,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2144702672958374
+    },
+    {
+      "global_step": 940,
+      "optimizer_step": 117,
+      "lr": 4.476246820180259e-05,
+      "lm_loss": 0.5643274188041687,
+      "order_loss": 0.671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5980188250541687
+    },
+    {
+      "global_step": 960,
+      "optimizer_step": 120,
+      "lr": 4.443600207791081e-05,
+      "lm_loss": 0.25739675760269165,
+      "order_loss": 0.5859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.28669363260269165
+    },
+    {
+      "global_step": 980,
+      "optimizer_step": 122,
+      "lr": 4.421357027581921e-05,
+      "lm_loss": 0.807464599609375,
+      "order_loss": 0.62109375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.838470458984375
+    },
+    {
+      "global_step": 1000,
+      "optimizer_step": 125,
+      "lr": 4.38728503704354e-05,
+      "lm_loss": 0.07562808692455292,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10565738379955292
+    },
+    {
+      "global_step": 1020,
+      "optimizer_step": 127,
+      "lr": 4.364105412207914e-05,
+      "lm_loss": 0.22450821101665497,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.25356096029281616
+    },
+    {
+      "global_step": 1040,
+      "optimizer_step": 130,
+      "lr": 4.328649926419819e-05,
+      "lm_loss": 0.10565733164548874,
+      "order_loss": 0.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13532042503356934
+    },
+    {
+      "global_step": 1060,
+      "optimizer_step": 132,
+      "lr": 4.304562350447008e-05,
+      "lm_loss": 0.21472251415252686,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.24133384227752686
+    },
+    {
+      "global_step": 1080,
+      "optimizer_step": 135,
+      "lr": 4.267766952966369e-05,
+      "lm_loss": 0.032229773700237274,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.058963172137737274
+    },
+    {
+      "global_step": 1100,
+      "optimizer_step": 137,
+      "lr": 4.242801035442058e-05,
+      "lm_loss": 0.571881115436554,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.598858654499054
+    },
+    {
+      "global_step": 1120,
+      "optimizer_step": 140,
+      "lr": 4.204710956908763e-05,
+      "lm_loss": 0.8310734629631042,
+      "order_loss": 0.74609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.8684269785881042
+    },
+    {
+      "global_step": 1140,
+      "optimizer_step": 142,
+      "lr": 4.178897387117546e-05,
+      "lm_loss": 0.06434237211942673,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09485995024442673
+    },
+    {
+      "global_step": 1160,
+      "optimizer_step": 145,
+      "lr": 4.139559449654561e-05,
+      "lm_loss": 0.04827670380473137,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07586459815502167
+    },
+    {
+      "global_step": 1180,
+      "optimizer_step": 147,
+      "lr": 4.1129299588552193e-05,
+      "lm_loss": 0.1848830282688141,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2139357626438141
+    },
+    {
+      "global_step": 1200,
+      "optimizer_step": 150,
+      "lr": 4.072392518512623e-05,
+      "lm_loss": 0.5498246550559998,
+      "order_loss": 0.58984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5793656706809998
+    },
+    {
+      "global_step": 1220,
+      "optimizer_step": 152,
+      "lr": 4.0449798409325665e-05,
+      "lm_loss": 0.04877238720655441,
+      "order_loss": 0.6171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07965617626905441
+    },
+    {
+      "global_step": 1240,
+      "optimizer_step": 155,
+      "lr": 4.0032927282460146e-05,
+      "lm_loss": 0.3415184020996094,
+      "order_loss": 0.6171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3724021911621094
+    },
+    {
+      "global_step": 1260,
+      "optimizer_step": 157,
+      "lr": 3.9751305608428205e-05,
+      "lm_loss": 0.18798238039016724,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21862202882766724
+    },
+    {
+      "global_step": 1280,
+      "optimizer_step": 160,
+      "lr": 3.932345019579495e-05,
+      "lm_loss": 0.05325399711728096,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08230672776699066
+    },
+    {
+      "global_step": 1300,
+      "optimizer_step": 162,
+      "lr": 3.90346798061902e-05,
+      "lm_loss": 0.06249101832509041,
+      "order_loss": 0.62109375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09349687397480011
+    },
+    {
+      "global_step": 1320,
+      "optimizer_step": 165,
+      "lr": 3.859636604786372e-05,
+      "lm_loss": 0.36135539412498474,
+      "order_loss": 0.65625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.39407023787498474
+    },
+    {
+      "global_step": 1340,
+      "optimizer_step": 167,
+      "lr": 3.830080191288342e-05,
+      "lm_loss": 0.4695039987564087,
+      "order_loss": 0.65625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5022188425064087
+    },
+    {
+      "global_step": 1360,
+      "optimizer_step": 170,
+      "lr": 3.785256860483054e-05,
+      "lm_loss": 0.09971931576728821,
+      "order_loss": 0.90234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1448853313922882
+    },
+    {
+      "global_step": 1380,
+      "optimizer_step": 172,
+      "lr": 3.755057404586455e-05,
+      "lm_loss": 0.1477055549621582,
+      "order_loss": 0.6484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1801762580871582
+    },
+    {
+      "global_step": 1400,
+      "optimizer_step": 175,
+      "lr": 3.7092972177631e-05,
+      "lm_loss": 0.13573057949543,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.16429503262043
+    },
+    {
+      "global_step": 1420,
+      "optimizer_step": 177,
+      "lr": 3.678491842064995e-05,
+      "lm_loss": 0.14675013720989227,
+      "order_loss": 0.640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.17873255908489227
+    },
+    {
+      "global_step": 1440,
+      "optimizer_step": 180,
+      "lr": 3.631851049805793e-05,
+      "lm_loss": 0.3483402132987976,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3789798617362976
+    },
+    {
+      "global_step": 1460,
+      "optimizer_step": 182,
+      "lr": 3.600477621728473e-05,
+      "lm_loss": 0.018764883279800415,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.046108633279800415
+    },
+    {
+      "global_step": 1480,
+      "optimizer_step": 185,
+      "lr": 3.5530135570974274e-05,
+      "lm_loss": 0.2707812190055847,
+      "order_loss": 0.640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3027636408805847
+    },
+    {
+      "global_step": 1500,
+      "optimizer_step": 187,
+      "lr": 3.521110642339991e-05,
+      "lm_loss": 0.44444000720977783,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.47349274158477783
+    },
+    {
+      "global_step": 1520,
+      "optimizer_step": 190,
+      "lr": 3.4728816504063726e-05,
+      "lm_loss": 0.19416525959968567,
+      "order_loss": 0.5859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.22346213459968567
+    },
+    {
+      "global_step": 1540,
+      "optimizer_step": 192,
+      "lr": 3.440488465537943e-05,
+      "lm_loss": 0.0048671611584723,
+      "order_loss": 0.6484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03733786568045616
+    },
+    {
+      "global_step": 1560,
+      "optimizer_step": 195,
+      "lr": 3.391553831655782e-05,
+      "lm_loss": 0.015687981620430946,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0446186438202858
+    },
+    {
+      "global_step": 1580,
+      "optimizer_step": 197,
+      "lr": 3.358710195908653e-05,
+      "lm_loss": 0.3525557219982147,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3824629485607147
+    },
+    {
+      "global_step": 1600,
+      "optimizer_step": 200,
+      "lr": 3.309130072840379e-05,
+      "lm_loss": 0.2939654588699341,
+      "order_loss": 0.71484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3296099901199341
+    },
+    {
+      "global_step": 1620,
+      "optimizer_step": 202,
+      "lr": 3.2758763591623295e-05,
+      "lm_loss": 0.22797633707523346,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.25800561904907227
+    },
+    {
+      "global_step": 1640,
+      "optimizer_step": 205,
+      "lr": 3.225711693136156e-05,
+      "lm_loss": 0.0836845114827156,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1139579489827156
+    },
+    {
+      "global_step": 1660,
+      "optimizer_step": 207,
+      "lr": 3.1920887785621235e-05,
+      "lm_loss": 0.0530904196202755,
+      "order_loss": 0.494140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0777486264705658
+    },
+    {
+      "global_step": 1680,
+      "optimizer_step": 210,
+      "lr": 3.141401234354067e-05,
+      "lm_loss": 0.12712986767292023,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15606053173542023
+    },
+    {
+      "global_step": 1700,
+      "optimizer_step": 212,
+      "lr": 3.10745044975816e-05,
+      "lm_loss": 0.21335665881633759,
+      "order_loss": 0.58984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.24289767444133759
+    },
+    {
+      "global_step": 1720,
+      "optimizer_step": 215,
+      "lr": 3.056302334890786e-05,
+      "lm_loss": 0.016052905470132828,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04303044453263283
+    },
+    {
+      "global_step": 1740,
+      "optimizer_step": 217,
+      "lr": 3.022065414180425e-05,
+      "lm_loss": 0.007782397326081991,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03646891936659813
+    },
+    {
+      "global_step": 1760,
+      "optimizer_step": 220,
+      "lr": 2.9705196023315006e-05,
+      "lm_loss": 0.010225682519376278,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03659287095069885
+    },
+    {
+      "global_step": 1780,
+      "optimizer_step": 222,
+      "lr": 2.9360386311461275e-05,
+      "lm_loss": 0.0031987405382096767,
+      "order_loss": 0.72265625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.039331551641225815
+    },
+    {
+      "global_step": 1800,
+      "optimizer_step": 225,
+      "lr": 2.884158484861325e-05,
+      "lm_loss": 0.015869250521063805,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.042602650821208954
+    },
+    {
+      "global_step": 1820,
+      "optimizer_step": 227,
+      "lr": 2.849475848838749e-05,
+      "lm_loss": 0.07672224193811417,
+      "order_loss": 0.5546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10443220287561417
+    },
+    {
+      "global_step": 1840,
+      "optimizer_step": 230,
+      "lr": 2.7973251416434175e-05,
+      "lm_loss": 0.06295819580554962,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09298749268054962
+    },
+    {
+      "global_step": 1860,
+      "optimizer_step": 232,
+      "lr": 2.7624834743173867e-05,
+      "lm_loss": 0.498603492975235,
+      "order_loss": 0.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5298534631729126
+    },
+    {
+      "global_step": 1880,
+      "optimizer_step": 235,
+      "lr": 2.710126312323119e-05,
+      "lm_loss": 0.12692563235759735,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15329281985759735
+    },
+    {
+      "global_step": 1900,
+      "optimizer_step": 237,
+      "lr": 2.6751684427161683e-05,
+      "lm_loss": 0.15096162259578705,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.17769502103328705
+    },
+    {
+      "global_step": 1920,
+      "optimizer_step": 240,
+      "lr": 2.6226691858185454e-05,
+      "lm_loss": 0.004965375643223524,
+      "order_loss": 0.48828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.02937943860888481
+    },
+    {
+      "global_step": 1940,
+      "optimizer_step": 242,
+      "lr": 2.587638085794541e-05,
+      "lm_loss": 0.05458242818713188,
+      "order_loss": 0.63671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08632071316242218
+    },
+    {
+      "global_step": 1960,
+      "optimizer_step": 245,
+      "lr": 2.535061268558906e-05,
+      "lm_loss": 0.11468195915222168,
+      "order_loss": 0.388671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13409113883972168
+    },
+    {
+      "global_step": 1980,
+      "optimizer_step": 247,
+      "lr": 2.5e-05,
+      "lm_loss": 0.29130151867866516,
+      "order_loss": 0.486328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.31559351086616516
+    },
+    {
+      "global_step": 2000,
+      "optimizer_step": 250,
+      "lr": 2.447410252332528e-05,
+      "lm_loss": 0.008092333562672138,
+      "order_loss": 0.458984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.031041551381349564
+    },
+    {
+      "global_step": 2020,
+      "optimizer_step": 252,
+      "lr": 2.4123619142054597e-05,
+      "lm_loss": 0.06767340004444122,
+      "order_loss": 0.50390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09281988441944122
+    },
+    {
+      "global_step": 2040,
+      "optimizer_step": 255,
+      "lr": 2.3598238819070202e-05,
+      "lm_loss": 0.13988685607910156,
+      "order_loss": 0.470703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.16344642639160156
+    },
+    {
+      "global_step": 2060,
+      "optimizer_step": 257,
+      "lr": 2.3248315572838316e-05,
+      "lm_loss": 0.002519852016121149,
+      "order_loss": 0.451171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.025102859362959862
+    },
+    {
+      "global_step": 2080,
+      "optimizer_step": 260,
+      "lr": 2.272409822584325e-05,
+      "lm_loss": 0.0040147737599909306,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03160266578197479
+    },
+    {
+      "global_step": 2100,
+      "optimizer_step": 262,
+      "lr": 2.237516525682614e-05,
+      "lm_loss": 0.01599397324025631,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04309358447790146
+    },
+    {
+      "global_step": 2120,
+      "optimizer_step": 265,
+      "lr": 2.185275527853437e-05,
+      "lm_loss": 0.014318106696009636,
+      "order_loss": 0.6328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.046056389808654785
+    },
+    {
+      "global_step": 2140,
+      "optimizer_step": 267,
+      "lr": 2.1505241511612522e-05,
+      "lm_loss": 0.009834249503910542,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03876491263508797
+    },
+    {
+      "global_step": 2160,
+      "optimizer_step": 270,
+      "lr": 2.0985281073035076e-05,
+      "lm_loss": 0.06978010386228561,
+      "order_loss": 0.484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09395002573728561
+    },
+    {
+      "global_step": 2180,
+      "optimizer_step": 272,
+      "lr": 2.0639613688538734e-05,
+      "lm_loss": 0.0009673925815150142,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.028555283322930336
+    },
+    {
+      "global_step": 2200,
+      "optimizer_step": 275,
+      "lr": 2.0122741949596797e-05,
+      "lm_loss": 0.010685226880013943,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04095866531133652
+    },
+    {
+      "global_step": 2220,
+      "optimizer_step": 277,
+      "lr": 1.977934585819576e-05,
+      "lm_loss": 0.36696794629096985,
+      "order_loss": 0.69140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.40163591504096985
+    },
+    {
+      "global_step": 2240,
+      "optimizer_step": 280,
+      "lr": 1.9266198182035058e-05,
+      "lm_loss": 0.006959175691008568,
+      "order_loss": 0.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03662226349115372
+    },
+    {
+      "global_step": 2260,
+      "optimizer_step": 282,
+      "lr": 1.8925495502418406e-05,
+      "lm_loss": 0.05435534194111824,
+      "order_loss": 0.486328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07864733040332794
+    },
+    {
+      "global_step": 2280,
+      "optimizer_step": 285,
+      "lr": 1.841670267439088e-05,
+      "lm_loss": 0.0729665756225586,
+      "order_loss": 0.451171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0955495834350586
+    },
+    {
+      "global_step": 2300,
+      "optimizer_step": 287,
+      "lr": 1.8079112214378768e-05,
+      "lm_loss": 0.06447143107652664,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09242553263902664
+    },
+    {
+      "global_step": 2320,
+      "optimizer_step": 290,
+      "lr": 1.757529966665141e-05,
+      "lm_loss": 0.08052024245262146,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11079367995262146
+    },
+    {
+      "global_step": 2340,
+      "optimizer_step": 292,
+      "lr": 1.72412364083767e-05,
+      "lm_loss": 0.04206604138016701,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06818908452987671
+    },
+    {
+      "global_step": 2360,
+      "optimizer_step": 295,
+      "lr": 1.6743023451120832e-05,
+      "lm_loss": 0.16625602543354034,
+      "order_loss": 0.67578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.19994743168354034
+    },
+    {
+      "global_step": 2380,
+      "optimizer_step": 297,
+      "lr": 1.641289804091347e-05,
+      "lm_loss": 0.06510496139526367,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09415769577026367
+    },
+    {
+      "global_step": 2400,
+      "optimizer_step": 300,
+      "lr": 1.5920897101019383e-05,
+      "lm_loss": 0.27654147148132324,
+      "order_loss": 0.423828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.29778170585632324
+    },
+    {
+      "global_step": 2420,
+      "optimizer_step": 302,
+      "lr": 1.5595115344620576e-05,
+      "lm_loss": 0.011664744466543198,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0386422835290432
+    },
+    {
+      "global_step": 2440,
+      "optimizer_step": 305,
+      "lr": 1.5109931212873534e-05,
+      "lm_loss": 0.022359207272529602,
+      "order_loss": 0.40625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0426228791475296
+    },
+    {
+      "global_step": 2460,
+      "optimizer_step": 307,
+      "lr": 1.4788893576600099e-05,
+      "lm_loss": 0.27348706126213074,
+      "order_loss": 0.58984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.30302807688713074
+    },
+    {
+      "global_step": 2480,
+      "optimizer_step": 310,
+      "lr": 1.4311122664242954e-05,
+      "lm_loss": 0.009061649441719055,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.039701297879219055
+    },
+    {
+      "global_step": 2500,
+      "optimizer_step": 312,
+      "lr": 1.399522378271527e-05,
+      "lm_loss": 0.8569443225860596,
+      "order_loss": 0.46875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.8803818225860596
+    },
+    {
+      "global_step": 2520,
+      "optimizer_step": 315,
+      "lr": 1.3525453388311554e-05,
+      "lm_loss": 0.07292108982801437,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10087519139051437
+    },
+    {
+      "global_step": 2540,
+      "optimizer_step": 317,
+      "lr": 1.3215081579350058e-05,
+      "lm_loss": 0.7619665861129761,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.7919958829879761
+    },
+    {
+      "global_step": 2560,
+      "optimizer_step": 320,
+      "lr": 1.2753889166848909e-05,
+      "lm_loss": 0.008005357347428799,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03534910827875137
+    },
+    {
+      "global_step": 2580,
+      "optimizer_step": 322,
+      "lr": 1.2449425954135452e-05,
+      "lm_loss": 0.01037638634443283,
+      "order_loss": 0.671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04406779259443283
+    },
+    {
+      "global_step": 2600,
+      "optimizer_step": 325,
+      "lr": 1.1997378443025634e-05,
+      "lm_loss": 0.005304152145981789,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03435688465833664
+    },
+    {
+      "global_step": 2620,
+      "optimizer_step": 327,
+      "lr": 1.1699198087116589e-05,
+      "lm_loss": 0.001738856895826757,
+      "order_loss": 4.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.23123104870319366
+    },
+    {
+      "global_step": 2640,
+      "optimizer_step": 330,
+      "lr": 1.1256851155542298e-05,
+      "lm_loss": 0.0020115836523473263,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.030331896618008614
+    },
+    {
+      "global_step": 2660,
+      "optimizer_step": 332,
+      "lr": 1.0965320193809808e-05,
+      "lm_loss": 0.01876344159245491,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04940309002995491
+    },
+    {
+      "global_step": 2680,
+      "optimizer_step": 335,
+      "lr": 1.0533217595504858e-05,
+      "lm_loss": 0.0010787693317979574,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0317184180021286
+    },
+    {
+      "global_step": 2700,
+      "optimizer_step": 337,
+      "lr": 1.0248694391571801e-05,
+      "lm_loss": 0.04407581314444542,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07300648093223572
+    },
+    {
+      "global_step": 2720,
+      "optimizer_step": 340,
+      "lr": 9.82736728745186e-06,
+      "lm_loss": 0.059230297803878784,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08950373530387878
+    },
+    {
+      "global_step": 2740,
+      "optimizer_step": 342,
+      "lr": 9.550201590674343e-06,
+      "lm_loss": 0.014554117806255817,
+      "order_loss": 0.6171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04543790593743324
+    },
+    {
+      "global_step": 2760,
+      "optimizer_step": 345,
+      "lr": 9.140167895908867e-06,
+      "lm_loss": 0.014679068699479103,
+      "order_loss": 0.6484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.047149769961833954
+    },
+    {
+      "global_step": 2780,
+      "optimizer_step": 347,
+      "lr": 8.870700411447816e-06,
+      "lm_loss": 0.07543246448040009,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10375277698040009
+    },
+    {
+      "global_step": 2800,
+      "optimizer_step": 350,
+      "lr": 8.472464158814255e-06,
+      "lm_loss": 0.034201085567474365,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.062155187129974365
+    },
+    {
+      "global_step": 2820,
+      "optimizer_step": 352,
+      "lr": 8.211026128824539e-06,
+      "lm_loss": 0.0049630808643996716,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03548065945506096
+    },
+    {
+      "global_step": 2840,
+      "optimizer_step": 355,
+      "lr": 7.825076849127458e-06,
+      "lm_loss": 0.016284892335534096,
+      "order_loss": 0.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04204172641038895
+    },
+    {
+      "global_step": 2860,
+      "optimizer_step": 357,
+      "lr": 7.571989645579419e-06,
+      "lm_loss": 0.10921858251094818,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13680647313594818
+    },
+    {
+      "global_step": 2880,
+      "optimizer_step": 360,
+      "lr": 7.198801765896115e-06,
+      "lm_loss": 0.25717654824256897,
+      "order_loss": 0.494140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.28183475136756897
+    },
+    {
+      "global_step": 2900,
+      "optimizer_step": 362,
+      "lr": 6.9543764955299265e-06,
+      "lm_loss": 0.026610566303133965,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.053343966603279114
+    },
+    {
+      "global_step": 2920,
+      "optimizer_step": 365,
+      "lr": 6.594408756022272e-06,
+      "lm_loss": 0.1350451558828354,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1649523824453354
+    },
+    {
+      "global_step": 2940,
+      "optimizer_step": 367,
+      "lr": 6.358945877920861e-06,
+      "lm_loss": 0.0024681584909558296,
+      "order_loss": 0.458984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.025417376309633255
+    },
+    {
+      "global_step": 2960,
+      "optimizer_step": 370,
+      "lr": 6.012640767930217e-06,
+      "lm_loss": 0.01191006787121296,
+      "order_loss": 0.4140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03266201913356781
+    },
+    {
+      "global_step": 2980,
+      "optimizer_step": 372,
+      "lr": 5.7864297241808e-06,
+      "lm_loss": 0.00042700982885435224,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.028503181412816048
+    },
+    {
+      "global_step": 3000,
+      "optimizer_step": 375,
+      "lr": 5.454212938299255e-06,
+      "lm_loss": 0.00046226487029343843,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.02780601568520069
+    },
+    {
+      "global_step": 3020,
+      "optimizer_step": 377,
+      "lr": 5.237531798197415e-06,
+      "lm_loss": 0.0036918141413480043,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.030425213277339935
+    },
+    {
+      "global_step": 3040,
+      "optimizer_step": 380,
+      "lr": 4.91981171298388e-06,
+      "lm_loss": 0.007289454340934753,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03438906371593475
+    },
+    {
+      "global_step": 3060,
+      "optimizer_step": 382,
+      "lr": 4.712926831217685e-06,
+      "lm_loss": 0.12698543071746826,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15750300884246826
+    },
+    {
+      "global_step": 3080,
+      "optimizer_step": 385,
+      "lr": 4.410094003202134e-06,
+      "lm_loss": 0.012188535183668137,
+      "order_loss": 0.71875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04807720705866814
+    },
+    {
+      "global_step": 3100,
+      "optimizer_step": 387,
+      "lr": 4.213259692436367e-06,
+      "lm_loss": 0.002271081320941448,
+      "order_loss": 0.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03193416818976402
+    },
+    {
+      "global_step": 3120,
+      "optimizer_step": 390,
+      "lr": 3.925686378029331e-06,
+      "lm_loss": 0.008490247651934624,
+      "order_loss": 0.470703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03204981982707977
+    },
+    {
+      "global_step": 3140,
+      "optimizer_step": 392,
+      "lr": 3.739144596292382e-06,
+      "lm_loss": 0.1310400366783142,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1580175757408142
+    },
+    {
+      "global_step": 3160,
+      "optimizer_step": 395,
+      "lr": 3.4671842941897765e-06,
+      "lm_loss": 0.32363763451576233,
+      "order_loss": 0.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.34939447045326233
+    },
+    {
+      "global_step": 3180,
+      "optimizer_step": 397,
+      "lr": 3.2911643474473646e-06,
+      "lm_loss": 0.005729279480874538,
+      "order_loss": 0.49609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.030509553849697113
+    },
+    {
+      "global_step": 3200,
+      "optimizer_step": 400,
+      "lr": 3.035151364093214e-06,
+      "lm_loss": 0.00038222724106162786,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.030655665323138237
+    },
+    {
+      "global_step": 3220,
+      "optimizer_step": 402,
+      "lr": 2.869869624374699e-06,
+      "lm_loss": 0.06640136986970901,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09313476830720901
+    },
+    {
+      "global_step": 3240,
+      "optimizer_step": 405,
+      "lr": 2.6301186630158485e-06,
+      "lm_loss": 0.006062754429876804,
+      "order_loss": 0.458984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.02901197224855423
+    },
+    {
+      "global_step": 3260,
+      "optimizer_step": 407,
+      "lr": 2.475778302439524e-06,
+      "lm_loss": 0.05398666486144066,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08096420764923096
+    },
+    {
+      "global_step": 3280,
+      "optimizer_step": 410,
+      "lr": 2.2525840762775863e-06,
+      "lm_loss": 0.1962323933839798,
+      "order_loss": 0.62109375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2272382527589798
+    },
+    {
+      "global_step": 3300,
+      "optimizer_step": 412,
+      "lr": 2.1093748173018848e-06,
+      "lm_loss": 0.006871367804706097,
+      "order_loss": 0.51171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03250613436102867
+    },
+    {
+      "global_step": 3320,
+      "optimizer_step": 415,
+      "lr": 1.9030116872178316e-06,
+      "lm_loss": 0.016944333910942078,
+      "order_loss": 0.50390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04209081828594208
+    },
+    {
+      "global_step": 3340,
+      "optimizer_step": 417,
+      "lr": 1.771109569425547e-06,
+      "lm_loss": 0.02157888002693653,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05148610472679138
+    },
+    {
+      "global_step": 3360,
+      "optimizer_step": 420,
+      "lr": 1.5818312067223672e-06,
+      "lm_loss": 0.1343194842338562,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1616632342338562
+    },
+    {
+      "global_step": 3380,
+      "optimizer_step": 422,
+      "lr": 1.4613983704244826e-06,
+      "lm_loss": 0.04511866718530655,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07221827656030655
+    },
+    {
+      "global_step": 3400,
+      "optimizer_step": 425,
+      "lr": 1.2894374450024338e-06,
+      "lm_loss": 0.008890017867088318,
+      "order_loss": 0.6953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04355798661708832
+    },
+    {
+      "global_step": 3420,
+      "optimizer_step": 427,
+      "lr": 1.180621931927592e-06,
+      "lm_loss": 0.007558132987469435,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03563430532813072
+    },
+    {
+      "global_step": 3440,
+      "optimizer_step": 430,
+      "lr": 1.0261898262753812e-06,
+      "lm_loss": 0.003311870386824012,
+      "order_loss": 0.46875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.026749370619654655
+    },
+    {
+      "global_step": 3460,
+      "optimizer_step": 432,
+      "lr": 9.291253975900139e-07,
+      "lm_loss": 0.007772153243422508,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03389520198106766
+    },
+    {
+      "global_step": 3480,
+      "optimizer_step": 435,
+      "lr": 7.924119469434665e-07,
+      "lm_loss": 0.010266868397593498,
+      "order_loss": 0.51953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03626784682273865
+    },
+    {
+      "global_step": 3500,
+      "optimizer_step": 437,
+      "lr": 7.072179188262251e-07,
+      "lm_loss": 0.1630493700504303,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1910034716129303
+    },
+    {
+      "global_step": 3520,
+      "optimizer_step": 440,
+      "lr": 5.883911778138324e-07,
+      "lm_loss": 0.09392284601926804,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12102245539426804
+    },
+    {
+      "global_step": 3540,
+      "optimizer_step": 442,
+      "lr": 5.151722747865434e-07,
+      "lm_loss": 0.5235670208930969,
+      "order_loss": 0.482421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5477369427680969
+    },
+    {
+      "global_step": 3560,
+      "optimizer_step": 445,
+      "lr": 4.1437831084878974e-07,
+      "lm_loss": 0.26141470670700073,
+      "order_loss": 0.69921875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.29632681608200073
+    },
+    {
+      "global_step": 3580,
+      "optimizer_step": 447,
+      "lr": 3.5322453704410286e-07,
+      "lm_loss": 0.0019069636473432183,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.02900657244026661
+    },
+    {
+      "global_step": 3600,
+      "optimizer_step": 450,
+      "lr": 2.705872508804747e-07,
+      "lm_loss": 0.0021807048469781876,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.029524454846978188
+    },
+    {
+      "global_step": 3620,
+      "optimizer_step": 452,
+      "lr": 2.215737794045436e-07,
+      "lm_loss": 0.0944581925868988,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1208253800868988
+    },
+    {
+      "global_step": 3640,
+      "optimizer_step": 455,
+      "lr": 1.571947526689349e-07,
+      "lm_loss": 0.008120126090943813,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03644043952226639
+    },
+    {
+      "global_step": 3660,
+      "optimizer_step": 457,
+      "lr": 1.2038183319507955e-07,
+      "lm_loss": 0.004870518576353788,
+      "order_loss": 0.51171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.030505284667015076
+    },
+    {
+      "global_step": 3680,
+      "optimizer_step": 460,
+      "lr": 7.434020362682704e-08,
+      "lm_loss": 0.042267005890607834,
+      "order_loss": 0.62890625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07376115024089813
+    },
+    {
+      "global_step": 3700,
+      "optimizer_step": 462,
+      "lr": 4.977308833379324e-08,
+      "lm_loss": 0.0006822891882620752,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03071158565580845
+    },
+    {
+      "global_step": 3720,
+      "optimizer_step": 465,
+      "lr": 2.2125452477828047e-08,
+      "lm_loss": 0.012706868350505829,
+      "order_loss": 0.4609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.03577815741300583
+    },
+    {
+      "global_step": 3740,
+      "optimizer_step": 467,
+      "lr": 9.834340423678368e-09,
+      "lm_loss": 0.04615507274866104,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07227811962366104
+    }
+  ]
+}

cls_gemma_orderkd_last_l01/checkpoint-200/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_last_l01/checkpoint-400/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1

cls_gemma_orderkd_last_l01/checkpoint-400/adapter_config.json

@@ -0,0 +1,44 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Gemma3ForCausalLM",
+    "parent_library": "transformers.models.gemma3.modeling_gemma3",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/gemma-3-1b-it",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "(?:.*?(?:language|text).*?(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer).*?(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj))|(?:\\bmodel\\.layers\\.[\\d]{1,}\\.(?:self_attn|attention|attn|mixer|mlp|feed_forward|ffn|dense|mixer)\\.(?:(?:q_proj|k_proj|v_proj|o_proj|up_proj|down_proj|gate_proj)))",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

cls_gemma_orderkd_last_l01/checkpoint-400/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

cls_gemma_orderkd_last_l01/train_metrics.json

@@ -0,0 +1,1874 @@
+{
+  "loss_log": [
+    {
+      "global_step": 20,
+      "optimizer_step": 2,
+      "lr": 4.347826086956522e-06,
+      "lm_loss": 3.085415840148926,
+      "order_loss": 61.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 9.241665840148926
+    },
+    {
+      "global_step": 40,
+      "optimizer_step": 5,
+      "lr": 1.0869565217391305e-05,
+      "lm_loss": 1.1733170747756958,
+      "order_loss": 54.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 6.610816955566406
+    },
+    {
+      "global_step": 60,
+      "optimizer_step": 7,
+      "lr": 1.5217391304347828e-05,
+      "lm_loss": 0.6860678195953369,
+      "order_loss": 42.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.904817581176758
+    },
+    {
+      "global_step": 80,
+      "optimizer_step": 10,
+      "lr": 2.173913043478261e-05,
+      "lm_loss": 1.8527085781097412,
+      "order_loss": 55.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 7.38395881652832
+    },
+    {
+      "global_step": 100,
+      "optimizer_step": 12,
+      "lr": 2.608695652173913e-05,
+      "lm_loss": 0.5747762322425842,
+      "order_loss": 43.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.9185261726379395
+    },
+    {
+      "global_step": 120,
+      "optimizer_step": 15,
+      "lr": 3.260869565217392e-05,
+      "lm_loss": 2.70076060295105,
+      "order_loss": 49.25,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 7.638260841369629
+    },
+    {
+      "global_step": 140,
+      "optimizer_step": 17,
+      "lr": 3.695652173913043e-05,
+      "lm_loss": 0.6722566485404968,
+      "order_loss": 44.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 5.0785064697265625
+    },
+    {
+      "global_step": 160,
+      "optimizer_step": 20,
+      "lr": 4.347826086956522e-05,
+      "lm_loss": 0.45723122358322144,
+      "order_loss": 42.0,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 4.644731044769287
+    },
+    {
+      "global_step": 180,
+      "optimizer_step": 22,
+      "lr": 4.782608695652174e-05,
+      "lm_loss": 0.04450434073805809,
+      "order_loss": 18.875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.9351292848587036
+    },
+    {
+      "global_step": 200,
+      "optimizer_step": 25,
+      "lr": 4.9997541293989384e-05,
+      "lm_loss": 0.02420174516737461,
+      "order_loss": 21.375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.1648266315460205
+    },
+    {
+      "global_step": 220,
+      "optimizer_step": 27,
+      "lr": 4.999016565957633e-05,
+      "lm_loss": 0.05216272175312042,
+      "order_loss": 17.125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.763100266456604
+    },
+    {
+      "global_step": 240,
+      "optimizer_step": 30,
+      "lr": 4.996988640512931e-05,
+      "lm_loss": 0.3272031247615814,
+      "order_loss": 26.875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 3.0147030353546143
+    },
+    {
+      "global_step": 260,
+      "optimizer_step": 32,
+      "lr": 4.995022691166621e-05,
+      "lm_loss": 0.03404615446925163,
+      "order_loss": 1.390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.17271803319454193
+    },
+    {
+      "global_step": 280,
+      "optimizer_step": 35,
+      "lr": 4.991153735294049e-05,
+      "lm_loss": 0.6312856078147888,
+      "order_loss": 20.125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.6469106674194336
+    },
+    {
+      "global_step": 300,
+      "optimizer_step": 37,
+      "lr": 4.987961816680492e-05,
+      "lm_loss": 0.18162043392658234,
+      "order_loss": 20.875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 2.2753703594207764
+    },
+    {
+      "global_step": 320,
+      "optimizer_step": 40,
+      "lr": 4.9822565862832806e-05,
+      "lm_loss": 0.07100138813257217,
+      "order_loss": 16.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.735063910484314
+    },
+    {
+      "global_step": 340,
+      "optimizer_step": 42,
+      "lr": 4.9778426220595456e-05,
+      "lm_loss": 0.23070411384105682,
+      "order_loss": 15.3125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.7619540691375732
+    },
+    {
+      "global_step": 360,
+      "optimizer_step": 45,
+      "lr": 4.970308130276272e-05,
+      "lm_loss": 0.6419835686683655,
+      "order_loss": 9.375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.5794835090637207
+    },
+    {
+      "global_step": 380,
+      "optimizer_step": 47,
+      "lr": 4.96467754629559e-05,
+      "lm_loss": 0.017068510875105858,
+      "order_loss": 1.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1684356927871704
+    },
+    {
+      "global_step": 400,
+      "optimizer_step": 50,
+      "lr": 4.955323054879298e-05,
+      "lm_loss": 0.10028436779975891,
+      "order_loss": 13.125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 1.4127843379974365
+    },
+    {
+      "global_step": 420,
+      "optimizer_step": 52,
+      "lr": 4.948482772521345e-05,
+      "lm_loss": 0.10965537279844284,
+      "order_loss": 4.0625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5159053802490234
+    },
+    {
+      "global_step": 440,
+      "optimizer_step": 55,
+      "lr": 4.937319780454559e-05,
+      "lm_loss": 0.24064283072948456,
+      "order_loss": 0.8515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.32560378313064575
+    },
+    {
+      "global_step": 460,
+      "optimizer_step": 57,
+      "lr": 4.929278208117378e-05,
+      "lm_loss": 0.5075720548629761,
+      "order_loss": 4.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.9665564298629761
+    },
+    {
+      "global_step": 480,
+      "optimizer_step": 60,
+      "lr": 4.916320437477008e-05,
+      "lm_loss": 0.3363575339317322,
+      "order_loss": 1.1796875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4545215964317322
+    },
+    {
+      "global_step": 500,
+      "optimizer_step": 62,
+      "lr": 4.907087460240999e-05,
+      "lm_loss": 0.14325940608978271,
+      "order_loss": 0.7109375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21454846858978271
+    },
+    {
+      "global_step": 520,
+      "optimizer_step": 65,
+      "lr": 4.892350839330522e-05,
+      "lm_loss": 0.12182406336069107,
+      "order_loss": 0.890625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21069124341011047
+    },
+    {
+      "global_step": 540,
+      "optimizer_step": 67,
+      "lr": 4.881937806807241e-05,
+      "lm_loss": 0.17294147610664368,
+      "order_loss": 0.7890625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2520430386066437
+    },
+    {
+      "global_step": 560,
+      "optimizer_step": 70,
+      "lr": 4.865440450576874e-05,
+      "lm_loss": 0.04529055207967758,
+      "order_loss": 0.703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11560305207967758
+    },
+    {
+      "global_step": 580,
+      "optimizer_step": 72,
+      "lr": 4.853860162957552e-05,
+      "lm_loss": 0.08526869863271713,
+      "order_loss": 0.64453125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.14972183108329773
+    },
+    {
+      "global_step": 600,
+      "optimizer_step": 75,
+      "lr": 4.8356223507364996e-05,
+      "lm_loss": 0.011337379924952984,
+      "order_loss": 0.73828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08506784588098526
+    },
+    {
+      "global_step": 620,
+      "optimizer_step": 77,
+      "lr": 4.822889043057446e-05,
+      "lm_loss": 0.2071656882762909,
+      "order_loss": 0.71875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2789430320262909
+    },
+    {
+      "global_step": 640,
+      "optimizer_step": 80,
+      "lr": 4.802933193625593e-05,
+      "lm_loss": 0.01071592140942812,
+      "order_loss": 0.65625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07614561170339584
+    },
+    {
+      "global_step": 660,
+      "optimizer_step": 82,
+      "lr": 4.789062518269812e-05,
+      "lm_loss": 0.048627037554979324,
+      "order_loss": 0.65625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11405672132968903
+    },
+    {
+      "global_step": 680,
+      "optimizer_step": 85,
+      "lr": 4.767413162299501e-05,
+      "lm_loss": 0.10063856840133667,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15386122465133667
+    },
+    {
+      "global_step": 700,
+      "optimizer_step": 87,
+      "lr": 4.752422169756048e-05,
+      "lm_loss": 0.2241210788488388,
+      "order_loss": 0.67578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.29150390625
+    },
+    {
+      "global_step": 720,
+      "optimizer_step": 90,
+      "lr": 4.729105919657818e-05,
+      "lm_loss": 0.4192332923412323,
+      "order_loss": 0.71875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4910106360912323
+    },
+    {
+      "global_step": 740,
+      "optimizer_step": 92,
+      "lr": 4.7130130375625305e-05,
+      "lm_loss": 0.22080935537815094,
+      "order_loss": 0.482421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.26914918422698975
+    },
+    {
+      "global_step": 760,
+      "optimizer_step": 95,
+      "lr": 4.6880585547718845e-05,
+      "lm_loss": 0.2958816885948181,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3500809073448181
+    },
+    {
+      "global_step": 780,
+      "optimizer_step": 97,
+      "lr": 4.670883565255264e-05,
+      "lm_loss": 0.02738013304769993,
+      "order_loss": 0.6328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09085669368505478
+    },
+    {
+      "global_step": 800,
+      "optimizer_step": 100,
+      "lr": 4.6443215250006806e-05,
+      "lm_loss": 0.35482561588287354,
+      "order_loss": 0.58984375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.41390764713287354
+    },
+    {
+      "global_step": 820,
+      "optimizer_step": 102,
+      "lr": 4.626085540370762e-05,
+      "lm_loss": 0.013227682560682297,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0698683112859726
+    },
+    {
+      "global_step": 840,
+      "optimizer_step": 105,
+      "lr": 4.597948593966256e-05,
+      "lm_loss": 0.04100118577480316,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09910665452480316
+    },
+    {
+      "global_step": 860,
+      "optimizer_step": 107,
+      "lr": 4.5786740307563636e-05,
+      "lm_loss": 0.042670246213674545,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09784603118896484
+    },
+    {
+      "global_step": 880,
+      "optimizer_step": 110,
+      "lr": 4.5489967654649474e-05,
+      "lm_loss": 0.08819571137428284,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.14605703949928284
+    },
+    {
+      "global_step": 900,
+      "optimizer_step": 112,
+      "lr": 4.528707316878232e-05,
+      "lm_loss": 0.13730522990226746,
+      "order_loss": 2.015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.33847710490226746
+    },
+    {
+      "global_step": 920,
+      "optimizer_step": 115,
+      "lr": 4.497526213395623e-05,
+      "lm_loss": 0.1977616548538208,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2512284517288208
+    },
+    {
+      "global_step": 940,
+      "optimizer_step": 117,
+      "lr": 4.476246820180259e-05,
+      "lm_loss": 0.5777482986450195,
+      "order_loss": 0.65234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.6431779861450195
+    },
+    {
+      "global_step": 960,
+      "optimizer_step": 120,
+      "lr": 4.443600207791081e-05,
+      "lm_loss": 0.29217880964279175,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.35199326276779175
+    },
+    {
+      "global_step": 980,
+      "optimizer_step": 122,
+      "lr": 4.421357027581921e-05,
+      "lm_loss": 0.7417739033699036,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.8028090596199036
+    },
+    {
+      "global_step": 1000,
+      "optimizer_step": 125,
+      "lr": 4.38728503704354e-05,
+      "lm_loss": 0.0897814929485321,
+      "order_loss": 0.5859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1483752429485321
+    },
+    {
+      "global_step": 1020,
+      "optimizer_step": 127,
+      "lr": 4.364105412207914e-05,
+      "lm_loss": 0.2607654333114624,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3188709020614624
+    },
+    {
+      "global_step": 1040,
+      "optimizer_step": 130,
+      "lr": 4.328649926419819e-05,
+      "lm_loss": 0.08412925153970718,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1402815878391266
+    },
+    {
+      "global_step": 1060,
+      "optimizer_step": 132,
+      "lr": 4.304562350447008e-05,
+      "lm_loss": 0.2278326153755188,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2810552716255188
+    },
+    {
+      "global_step": 1080,
+      "optimizer_step": 135,
+      "lr": 4.267766952966369e-05,
+      "lm_loss": 0.035670991986989975,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09084677696228027
+    },
+    {
+      "global_step": 1100,
+      "optimizer_step": 137,
+      "lr": 4.242801035442058e-05,
+      "lm_loss": 0.5935589671134949,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.6467816233634949
+    },
+    {
+      "global_step": 1120,
+      "optimizer_step": 140,
+      "lr": 4.204710956908763e-05,
+      "lm_loss": 0.7752993106842041,
+      "order_loss": 0.75390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.8504946231842041
+    },
+    {
+      "global_step": 1140,
+      "optimizer_step": 142,
+      "lr": 4.178897387117546e-05,
+      "lm_loss": 0.07537111639976501,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13640627264976501
+    },
+    {
+      "global_step": 1160,
+      "optimizer_step": 145,
+      "lr": 4.139559449654561e-05,
+      "lm_loss": 0.03182560205459595,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08602482080459595
+    },
+    {
+      "global_step": 1180,
+      "optimizer_step": 147,
+      "lr": 4.1129299588552193e-05,
+      "lm_loss": 0.19479840993881226,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.25290387868881226
+    },
+    {
+      "global_step": 1200,
+      "optimizer_step": 150,
+      "lr": 4.072392518512623e-05,
+      "lm_loss": 0.5517619848251343,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.6098674535751343
+    },
+    {
+      "global_step": 1220,
+      "optimizer_step": 152,
+      "lr": 4.0449798409325665e-05,
+      "lm_loss": 0.03966657817363739,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09948103129863739
+    },
+    {
+      "global_step": 1240,
+      "optimizer_step": 155,
+      "lr": 4.0032927282460146e-05,
+      "lm_loss": 0.343088299036026,
+      "order_loss": 0.64453125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.407541424036026
+    },
+    {
+      "global_step": 1260,
+      "optimizer_step": 157,
+      "lr": 3.9751305608428205e-05,
+      "lm_loss": 0.16970181465148926,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.23024868965148926
+    },
+    {
+      "global_step": 1280,
+      "optimizer_step": 160,
+      "lr": 3.932345019579495e-05,
+      "lm_loss": 0.05827689170837402,
+      "order_loss": 0.58203125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11638236045837402
+    },
+    {
+      "global_step": 1300,
+      "optimizer_step": 162,
+      "lr": 3.90346798061902e-05,
+      "lm_loss": 0.06468262523412704,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12522950768470764
+    },
+    {
+      "global_step": 1320,
+      "optimizer_step": 165,
+      "lr": 3.859636604786372e-05,
+      "lm_loss": 0.23940643668174744,
+      "order_loss": 0.66015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.30532440543174744
+    },
+    {
+      "global_step": 1340,
+      "optimizer_step": 167,
+      "lr": 3.830080191288342e-05,
+      "lm_loss": 0.4246941804885864,
+      "order_loss": 0.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4871941804885864
+    },
+    {
+      "global_step": 1360,
+      "optimizer_step": 170,
+      "lr": 3.785256860483054e-05,
+      "lm_loss": 0.08750709891319275,
+      "order_loss": 0.79296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.16660866141319275
+    },
+    {
+      "global_step": 1380,
+      "optimizer_step": 172,
+      "lr": 3.755057404586455e-05,
+      "lm_loss": 0.13699057698249817,
+      "order_loss": 0.640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.20095542073249817
+    },
+    {
+      "global_step": 1400,
+      "optimizer_step": 175,
+      "lr": 3.7092972177631e-05,
+      "lm_loss": 0.11647278815507889,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1711602807044983
+    },
+    {
+      "global_step": 1420,
+      "optimizer_step": 177,
+      "lr": 3.678491842064995e-05,
+      "lm_loss": 0.15342631936073303,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21397319436073303
+    },
+    {
+      "global_step": 1440,
+      "optimizer_step": 180,
+      "lr": 3.631851049805793e-05,
+      "lm_loss": 0.3482745587825775,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4080890119075775
+    },
+    {
+      "global_step": 1460,
+      "optimizer_step": 182,
+      "lr": 3.600477621728473e-05,
+      "lm_loss": 0.02179204486310482,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07525884360074997
+    },
+    {
+      "global_step": 1480,
+      "optimizer_step": 185,
+      "lr": 3.5530135570974274e-05,
+      "lm_loss": 0.27009710669517517,
+      "order_loss": 0.65234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.33552679419517517
+    },
+    {
+      "global_step": 1500,
+      "optimizer_step": 187,
+      "lr": 3.521110642339991e-05,
+      "lm_loss": 0.5374942421913147,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5948672890663147
+    },
+    {
+      "global_step": 1520,
+      "optimizer_step": 190,
+      "lr": 3.4728816504063726e-05,
+      "lm_loss": 0.2017577439546585,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2588866353034973
+    },
+    {
+      "global_step": 1540,
+      "optimizer_step": 192,
+      "lr": 3.440488465537943e-05,
+      "lm_loss": 0.004454552661627531,
+      "order_loss": 0.625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06695455312728882
+    },
+    {
+      "global_step": 1560,
+      "optimizer_step": 195,
+      "lr": 3.391553831655782e-05,
+      "lm_loss": 0.01754581928253174,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07540714740753174
+    },
+    {
+      "global_step": 1580,
+      "optimizer_step": 197,
+      "lr": 3.358710195908653e-05,
+      "lm_loss": 0.36603447794914246,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.42340752482414246
+    },
+    {
+      "global_step": 1600,
+      "optimizer_step": 200,
+      "lr": 3.309130072840379e-05,
+      "lm_loss": 0.23801112174987793,
+      "order_loss": 0.6796875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.30588221549987793
+    },
+    {
+      "global_step": 1620,
+      "optimizer_step": 202,
+      "lr": 3.2758763591623295e-05,
+      "lm_loss": 0.20070801675319672,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2573486566543579
+    },
+    {
+      "global_step": 1640,
+      "optimizer_step": 205,
+      "lr": 3.225711693136156e-05,
+      "lm_loss": 0.07082575559616089,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1279546618461609
+    },
+    {
+      "global_step": 1660,
+      "optimizer_step": 207,
+      "lr": 3.1920887785621235e-05,
+      "lm_loss": 0.05186420679092407,
+      "order_loss": 0.470703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09898334741592407
+    },
+    {
+      "global_step": 1680,
+      "optimizer_step": 210,
+      "lr": 3.141401234354067e-05,
+      "lm_loss": 0.12349201738834381,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1794002205133438
+    },
+    {
+      "global_step": 1700,
+      "optimizer_step": 212,
+      "lr": 3.10745044975816e-05,
+      "lm_loss": 0.19202591478824615,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.24866653978824615
+    },
+    {
+      "global_step": 1720,
+      "optimizer_step": 215,
+      "lr": 3.056302334890786e-05,
+      "lm_loss": 0.014021377079188824,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0667557492852211
+    },
+    {
+      "global_step": 1740,
+      "optimizer_step": 217,
+      "lr": 3.022065414180425e-05,
+      "lm_loss": 0.006658232305198908,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06085745245218277
+    },
+    {
+      "global_step": 1760,
+      "optimizer_step": 220,
+      "lr": 2.9705196023315006e-05,
+      "lm_loss": 0.00958278402686119,
+      "order_loss": 0.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06109645590186119
+    },
+    {
+      "global_step": 1780,
+      "optimizer_step": 222,
+      "lr": 2.9360386311461275e-05,
+      "lm_loss": 0.0031068390235304832,
+      "order_loss": 0.69140625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07244277745485306
+    },
+    {
+      "global_step": 1800,
+      "optimizer_step": 225,
+      "lr": 2.884158484861325e-05,
+      "lm_loss": 0.01922924444079399,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07147534191608429
+    },
+    {
+      "global_step": 1820,
+      "optimizer_step": 227,
+      "lr": 2.849475848838749e-05,
+      "lm_loss": 0.07677049189805984,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13414353132247925
+    },
+    {
+      "global_step": 1840,
+      "optimizer_step": 230,
+      "lr": 2.7973251416434175e-05,
+      "lm_loss": 0.051096681505441666,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10822558403015137
+    },
+    {
+      "global_step": 1860,
+      "optimizer_step": 232,
+      "lr": 2.7624834743173867e-05,
+      "lm_loss": 0.47954103350639343,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.5408203601837158
+    },
+    {
+      "global_step": 1880,
+      "optimizer_step": 235,
+      "lr": 2.710126312323119e-05,
+      "lm_loss": 0.13911223411560059,
+      "order_loss": 0.5,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.18916106224060059
+    },
+    {
+      "global_step": 1900,
+      "optimizer_step": 237,
+      "lr": 2.6751684427161683e-05,
+      "lm_loss": 0.16649648547172546,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.21971914172172546
+    },
+    {
+      "global_step": 1920,
+      "optimizer_step": 240,
+      "lr": 2.6226691858185454e-05,
+      "lm_loss": 0.004812562372535467,
+      "order_loss": 0.474609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.052175842225551605
+    },
+    {
+      "global_step": 1940,
+      "optimizer_step": 242,
+      "lr": 2.587638085794541e-05,
+      "lm_loss": 0.05702533200383186,
+      "order_loss": 0.63671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12050189077854156
+    },
+    {
+      "global_step": 1960,
+      "optimizer_step": 245,
+      "lr": 2.535061268558906e-05,
+      "lm_loss": 0.10227977484464645,
+      "order_loss": 0.3828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.14060986042022705
+    },
+    {
+      "global_step": 1980,
+      "optimizer_step": 247,
+      "lr": 2.5e-05,
+      "lm_loss": 0.10830750316381454,
+      "order_loss": 0.470703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15542665123939514
+    },
+    {
+      "global_step": 2000,
+      "optimizer_step": 250,
+      "lr": 2.447410252332528e-05,
+      "lm_loss": 0.010663356631994247,
+      "order_loss": 0.4375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05436452850699425
+    },
+    {
+      "global_step": 2020,
+      "optimizer_step": 252,
+      "lr": 2.4123619142054597e-05,
+      "lm_loss": 0.07362121343612671,
+      "order_loss": 0.4921875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12293761968612671
+    },
+    {
+      "global_step": 2040,
+      "optimizer_step": 255,
+      "lr": 2.3598238819070202e-05,
+      "lm_loss": 0.14120566844940186,
+      "order_loss": 0.4375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.18490684032440186
+    },
+    {
+      "global_step": 2060,
+      "optimizer_step": 257,
+      "lr": 2.3248315572838316e-05,
+      "lm_loss": 0.0024538934230804443,
+      "order_loss": 0.4296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.045422643423080444
+    },
+    {
+      "global_step": 2080,
+      "optimizer_step": 260,
+      "lr": 2.272409822584325e-05,
+      "lm_loss": 0.00425880728289485,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05748146399855614
+    },
+    {
+      "global_step": 2100,
+      "optimizer_step": 262,
+      "lr": 2.237516525682614e-05,
+      "lm_loss": 0.019106872379779816,
+      "order_loss": 0.51953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07110882550477982
+    },
+    {
+      "global_step": 2120,
+      "optimizer_step": 265,
+      "lr": 2.185275527853437e-05,
+      "lm_loss": 0.012871557846665382,
+      "order_loss": 0.61328125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07415085285902023
+    },
+    {
+      "global_step": 2140,
+      "optimizer_step": 267,
+      "lr": 2.1505241511612522e-05,
+      "lm_loss": 0.007741750217974186,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06364995241165161
+    },
+    {
+      "global_step": 2160,
+      "optimizer_step": 270,
+      "lr": 2.0985281073035076e-05,
+      "lm_loss": 0.05910928547382355,
+      "order_loss": 0.46484375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10549600422382355
+    },
+    {
+      "global_step": 2180,
+      "optimizer_step": 272,
+      "lr": 2.0639613688538734e-05,
+      "lm_loss": 0.0008628124487586319,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.055550310760736465
+    },
+    {
+      "global_step": 2200,
+      "optimizer_step": 275,
+      "lr": 2.0122741949596797e-05,
+      "lm_loss": 0.0134747214615345,
+      "order_loss": 0.56640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0701153427362442
+    },
+    {
+      "global_step": 2220,
+      "optimizer_step": 277,
+      "lr": 1.977934585819576e-05,
+      "lm_loss": 0.35245659947395325,
+      "order_loss": 0.63671875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.41593316197395325
+    },
+    {
+      "global_step": 2240,
+      "optimizer_step": 280,
+      "lr": 1.9266198182035058e-05,
+      "lm_loss": 0.008748134598135948,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0649004802107811
+    },
+    {
+      "global_step": 2260,
+      "optimizer_step": 282,
+      "lr": 1.8925495502418406e-05,
+      "lm_loss": 0.04881581291556358,
+      "order_loss": 0.478515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09666737914085388
+    },
+    {
+      "global_step": 2280,
+      "optimizer_step": 285,
+      "lr": 1.841670267439088e-05,
+      "lm_loss": 0.06688515096902847,
+      "order_loss": 0.40234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10716835409402847
+    },
+    {
+      "global_step": 2300,
+      "optimizer_step": 287,
+      "lr": 1.8079112214378768e-05,
+      "lm_loss": 0.053591735661029816,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10876751691102982
+    },
+    {
+      "global_step": 2320,
+      "optimizer_step": 290,
+      "lr": 1.757529966665141e-05,
+      "lm_loss": 0.05451134219765663,
+      "order_loss": 0.57421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11188438534736633
+    },
+    {
+      "global_step": 2340,
+      "optimizer_step": 292,
+      "lr": 1.72412364083767e-05,
+      "lm_loss": 0.03790012001991272,
+      "order_loss": 0.498046875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08770480751991272
+    },
+    {
+      "global_step": 2360,
+      "optimizer_step": 295,
+      "lr": 1.6743023451120832e-05,
+      "lm_loss": 0.19022615253925323,
+      "order_loss": 0.6640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.25663238763809204
+    },
+    {
+      "global_step": 2380,
+      "optimizer_step": 297,
+      "lr": 1.641289804091347e-05,
+      "lm_loss": 0.05954565480351448,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11423315107822418
+    },
+    {
+      "global_step": 2400,
+      "optimizer_step": 300,
+      "lr": 1.5920897101019383e-05,
+      "lm_loss": 0.27603620290756226,
+      "order_loss": 0.390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.31509870290756226
+    },
+    {
+      "global_step": 2420,
+      "optimizer_step": 302,
+      "lr": 1.5595115344620576e-05,
+      "lm_loss": 0.011492199264466763,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06373829394578934
+    },
+    {
+      "global_step": 2440,
+      "optimizer_step": 305,
+      "lr": 1.5109931212873534e-05,
+      "lm_loss": 0.03387191519141197,
+      "order_loss": 0.40234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07415512204170227
+    },
+    {
+      "global_step": 2460,
+      "optimizer_step": 307,
+      "lr": 1.4788893576600099e-05,
+      "lm_loss": 0.24835948646068573,
+      "order_loss": 0.5859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3069532513618469
+    },
+    {
+      "global_step": 2480,
+      "optimizer_step": 310,
+      "lr": 1.4311122664242954e-05,
+      "lm_loss": 0.008892997168004513,
+      "order_loss": 0.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06821916997432709
+    },
+    {
+      "global_step": 2500,
+      "optimizer_step": 312,
+      "lr": 1.399522378271527e-05,
+      "lm_loss": 0.8724842667579651,
+      "order_loss": 0.455078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.9178944230079651
+    },
+    {
+      "global_step": 2520,
+      "optimizer_step": 315,
+      "lr": 1.3525453388311554e-05,
+      "lm_loss": 0.06692298501729965,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12161048501729965
+    },
+    {
+      "global_step": 2540,
+      "optimizer_step": 317,
+      "lr": 1.3215081579350058e-05,
+      "lm_loss": 0.8130843639373779,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.8702132701873779
+    },
+    {
+      "global_step": 2560,
+      "optimizer_step": 320,
+      "lr": 1.2753889166848909e-05,
+      "lm_loss": 0.008002248592674732,
+      "order_loss": 0.51953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06000420078635216
+    },
+    {
+      "global_step": 2580,
+      "optimizer_step": 322,
+      "lr": 1.2449425954135452e-05,
+      "lm_loss": 0.01037616841495037,
+      "order_loss": 0.64453125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07482929527759552
+    },
+    {
+      "global_step": 2600,
+      "optimizer_step": 325,
+      "lr": 1.1997378443025634e-05,
+      "lm_loss": 0.004813759587705135,
+      "order_loss": 0.5625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06096610426902771
+    },
+    {
+      "global_step": 2620,
+      "optimizer_step": 327,
+      "lr": 1.1699198087116589e-05,
+      "lm_loss": 0.0018768347799777985,
+      "order_loss": 4.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.4608612060546875
+    },
+    {
+      "global_step": 2640,
+      "optimizer_step": 330,
+      "lr": 1.1256851155542298e-05,
+      "lm_loss": 0.0017797905020415783,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.057687994092702866
+    },
+    {
+      "global_step": 2660,
+      "optimizer_step": 332,
+      "lr": 1.0965320193809808e-05,
+      "lm_loss": 0.030651530250906944,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0885128602385521
+    },
+    {
+      "global_step": 2680,
+      "optimizer_step": 335,
+      "lr": 1.0533217595504858e-05,
+      "lm_loss": 0.0006764253485016525,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05853775516152382
+    },
+    {
+      "global_step": 2700,
+      "optimizer_step": 337,
+      "lr": 1.0248694391571801e-05,
+      "lm_loss": 0.03630198538303375,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09147776663303375
+    },
+    {
+      "global_step": 2720,
+      "optimizer_step": 340,
+      "lr": 9.82736728745186e-06,
+      "lm_loss": 0.04801255464553833,
+      "order_loss": 0.5859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10660630464553833
+    },
+    {
+      "global_step": 2740,
+      "optimizer_step": 342,
+      "lr": 9.550201590674343e-06,
+      "lm_loss": 0.014596512541174889,
+      "order_loss": 0.59765625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07441096752882004
+    },
+    {
+      "global_step": 2760,
+      "optimizer_step": 345,
+      "lr": 9.140167895908867e-06,
+      "lm_loss": 0.03127565234899521,
+      "order_loss": 0.640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09524049609899521
+    },
+    {
+      "global_step": 2780,
+      "optimizer_step": 347,
+      "lr": 8.870700411447816e-06,
+      "lm_loss": 0.07532689720392227,
+      "order_loss": 0.55078125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13050267100334167
+    },
+    {
+      "global_step": 2800,
+      "optimizer_step": 350,
+      "lr": 8.472464158814255e-06,
+      "lm_loss": 0.07896865904331207,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.13219131529331207
+    },
+    {
+      "global_step": 2820,
+      "optimizer_step": 352,
+      "lr": 8.211026128824539e-06,
+      "lm_loss": 0.00585238728672266,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06371371448040009
+    },
+    {
+      "global_step": 2840,
+      "optimizer_step": 355,
+      "lr": 7.825076849127458e-06,
+      "lm_loss": 0.012425755150616169,
+      "order_loss": 0.51953125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06442771106958389
+    },
+    {
+      "global_step": 2860,
+      "optimizer_step": 357,
+      "lr": 7.571989645579419e-06,
+      "lm_loss": 0.1399037390947342,
+      "order_loss": 0.5390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1938588172197342
+    },
+    {
+      "global_step": 2880,
+      "optimizer_step": 360,
+      "lr": 7.198801765896115e-06,
+      "lm_loss": 0.260348379611969,
+      "order_loss": 0.474609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.307711660861969
+    },
+    {
+      "global_step": 2900,
+      "optimizer_step": 362,
+      "lr": 6.9543764955299265e-06,
+      "lm_loss": 0.0293747428804636,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08210911601781845
+    },
+    {
+      "global_step": 2920,
+      "optimizer_step": 365,
+      "lr": 6.594408756022272e-06,
+      "lm_loss": 0.13667038083076477,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.19453170895576477
+    },
+    {
+      "global_step": 2940,
+      "optimizer_step": 367,
+      "lr": 6.358945877920861e-06,
+      "lm_loss": 0.0033789623994380236,
+      "order_loss": 0.466796875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.050009820610284805
+    },
+    {
+      "global_step": 2960,
+      "optimizer_step": 370,
+      "lr": 6.012640767930217e-06,
+      "lm_loss": 0.010715465061366558,
+      "order_loss": 0.41015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05173109099268913
+    },
+    {
+      "global_step": 2980,
+      "optimizer_step": 372,
+      "lr": 5.7864297241808e-06,
+      "lm_loss": 0.0005227762740105391,
+      "order_loss": 0.55859375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05643098056316376
+    },
+    {
+      "global_step": 3000,
+      "optimizer_step": 375,
+      "lr": 5.454212938299255e-06,
+      "lm_loss": 0.000452561245765537,
+      "order_loss": 0.50390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05074552819132805
+    },
+    {
+      "global_step": 3020,
+      "optimizer_step": 377,
+      "lr": 5.237531798197415e-06,
+      "lm_loss": 0.0023832994047552347,
+      "order_loss": 0.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05389697104692459
+    },
+    {
+      "global_step": 3040,
+      "optimizer_step": 380,
+      "lr": 4.91981171298388e-06,
+      "lm_loss": 0.005122009664773941,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05834466591477394
+    },
+    {
+      "global_step": 3060,
+      "optimizer_step": 382,
+      "lr": 4.712926831217685e-06,
+      "lm_loss": 0.09252365678548813,
+      "order_loss": 0.5703125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.14965257048606873
+    },
+    {
+      "global_step": 3080,
+      "optimizer_step": 385,
+      "lr": 4.410094003202134e-06,
+      "lm_loss": 0.01632733643054962,
+      "order_loss": 0.74609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.09103436768054962
+    },
+    {
+      "global_step": 3100,
+      "optimizer_step": 387,
+      "lr": 4.213259692436367e-06,
+      "lm_loss": 0.0018161230254918337,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.061874717473983765
+    },
+    {
+      "global_step": 3120,
+      "optimizer_step": 390,
+      "lr": 3.925686378029331e-06,
+      "lm_loss": 0.008418104611337185,
+      "order_loss": 0.44921875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05333998054265976
+    },
+    {
+      "global_step": 3140,
+      "optimizer_step": 392,
+      "lr": 3.739144596292382e-06,
+      "lm_loss": 0.12710820138454437,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.17935429513454437
+    },
+    {
+      "global_step": 3160,
+      "optimizer_step": 395,
+      "lr": 3.4671842941897765e-06,
+      "lm_loss": 0.2975465655326843,
+      "order_loss": 0.515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.3490602374076843
+    },
+    {
+      "global_step": 3180,
+      "optimizer_step": 397,
+      "lr": 3.2911643474473646e-06,
+      "lm_loss": 0.004406274762004614,
+      "order_loss": 0.44921875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04932814836502075
+    },
+    {
+      "global_step": 3200,
+      "optimizer_step": 400,
+      "lr": 3.035151364093214e-06,
+      "lm_loss": 0.00038232127553783357,
+      "order_loss": 0.60546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06092919781804085
+    },
+    {
+      "global_step": 3220,
+      "optimizer_step": 402,
+      "lr": 2.869869624374699e-06,
+      "lm_loss": 0.054574642330408096,
+      "order_loss": 0.498046875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.1043793261051178
+    },
+    {
+      "global_step": 3240,
+      "optimizer_step": 405,
+      "lr": 2.6301186630158485e-06,
+      "lm_loss": 0.006128022447228432,
+      "order_loss": 0.435546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04958505183458328
+    },
+    {
+      "global_step": 3260,
+      "optimizer_step": 407,
+      "lr": 2.475778302439524e-06,
+      "lm_loss": 0.044944874942302704,
+      "order_loss": 0.498046875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0947495624423027
+    },
+    {
+      "global_step": 3280,
+      "optimizer_step": 410,
+      "lr": 2.2525840762775863e-06,
+      "lm_loss": 0.2107994258403778,
+      "order_loss": 0.609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.2718345820903778
+    },
+    {
+      "global_step": 3300,
+      "optimizer_step": 412,
+      "lr": 2.1093748173018848e-06,
+      "lm_loss": 0.0034342966973781586,
+      "order_loss": 0.482421875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05177414044737816
+    },
+    {
+      "global_step": 3320,
+      "optimizer_step": 415,
+      "lr": 1.9030116872178316e-06,
+      "lm_loss": 0.016720784828066826,
+      "order_loss": 0.48828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06554891169071198
+    },
+    {
+      "global_step": 3340,
+      "optimizer_step": 417,
+      "lr": 1.771109569425547e-06,
+      "lm_loss": 0.021915944293141365,
+      "order_loss": 0.578125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.07977727055549622
+    },
+    {
+      "global_step": 3360,
+      "optimizer_step": 420,
+      "lr": 1.5818312067223672e-06,
+      "lm_loss": 0.10221873968839645,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.15446484088897705
+    },
+    {
+      "global_step": 3380,
+      "optimizer_step": 422,
+      "lr": 1.4613983704244826e-06,
+      "lm_loss": 0.03383534774184227,
+      "order_loss": 0.52734375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.08656972646713257
+    },
+    {
+      "global_step": 3400,
+      "optimizer_step": 425,
+      "lr": 1.2894374450024338e-06,
+      "lm_loss": 0.007806207053363323,
+      "order_loss": 0.6640625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.0742124542593956
+    },
+    {
+      "global_step": 3420,
+      "optimizer_step": 427,
+      "lr": 1.180621931927592e-06,
+      "lm_loss": 0.008136852644383907,
+      "order_loss": 0.54296875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06233607232570648
+    },
+    {
+      "global_step": 3440,
+      "optimizer_step": 430,
+      "lr": 1.0261898262753812e-06,
+      "lm_loss": 0.0025549507699906826,
+      "order_loss": 0.447265625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04723268374800682
+    },
+    {
+      "global_step": 3460,
+      "optimizer_step": 432,
+      "lr": 9.291253975900139e-07,
+      "lm_loss": 0.005603352095931768,
+      "order_loss": 0.48828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.054431475698947906
+    },
+    {
+      "global_step": 3480,
+      "optimizer_step": 435,
+      "lr": 7.924119469434665e-07,
+      "lm_loss": 0.013628451153635979,
+      "order_loss": 0.48828125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06245657801628113
+    },
+    {
+      "global_step": 3500,
+      "optimizer_step": 437,
+      "lr": 7.072179188262251e-07,
+      "lm_loss": 0.16960863769054413,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.22429613769054413
+    },
+    {
+      "global_step": 3520,
+      "optimizer_step": 440,
+      "lr": 5.883911778138324e-07,
+      "lm_loss": 0.09196390211582184,
+      "order_loss": 0.53515625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.14543069899082184
+    },
+    {
+      "global_step": 3540,
+      "optimizer_step": 442,
+      "lr": 5.151722747865434e-07,
+      "lm_loss": 0.6089231371879578,
+      "order_loss": 0.4609375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.6550657153129578
+    },
+    {
+      "global_step": 3560,
+      "optimizer_step": 445,
+      "lr": 4.1437831084878974e-07,
+      "lm_loss": 0.17815802991390228,
+      "order_loss": 0.66796875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.24505256116390228
+    },
+    {
+      "global_step": 3580,
+      "optimizer_step": 447,
+      "lr": 3.5322453704410286e-07,
+      "lm_loss": 0.0015013264492154121,
+      "order_loss": 0.546875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05618882551789284
+    },
+    {
+      "global_step": 3600,
+      "optimizer_step": 450,
+      "lr": 2.705872508804747e-07,
+      "lm_loss": 0.0026061830576509237,
+      "order_loss": 0.5234375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05485227704048157
+    },
+    {
+      "global_step": 3620,
+      "optimizer_step": 452,
+      "lr": 2.215737794045436e-07,
+      "lm_loss": 0.06758156418800354,
+      "order_loss": 0.51171875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.11885109543800354
+    },
+    {
+      "global_step": 3640,
+      "optimizer_step": 455,
+      "lr": 1.571947526689349e-07,
+      "lm_loss": 0.009578689932823181,
+      "order_loss": 0.53125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.06280134618282318
+    },
+    {
+      "global_step": 3660,
+      "optimizer_step": 457,
+      "lr": 1.2038183319507955e-07,
+      "lm_loss": 0.005645351950079203,
+      "order_loss": 0.48046875,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.053741056472063065
+    },
+    {
+      "global_step": 3680,
+      "optimizer_step": 460,
+      "lr": 7.434020362682704e-08,
+      "lm_loss": 0.04718263819813728,
+      "order_loss": 0.6015625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.10724122822284698
+    },
+    {
+      "global_step": 3700,
+      "optimizer_step": 462,
+      "lr": 4.977308833379324e-08,
+      "lm_loss": 0.0005944116273894906,
+      "order_loss": 0.59375,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.05992058292031288
+    },
+    {
+      "global_step": 3720,
+      "optimizer_step": 465,
+      "lr": 2.2125452477828047e-08,
+      "lm_loss": 0.0038628741167485714,
+      "order_loss": 0.39453125,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.04341365396976471
+    },
+    {
+      "global_step": 3740,
+      "optimizer_step": 467,
+      "lr": 9.834340423678368e-09,
+      "lm_loss": 0.07419663667678833,
+      "order_loss": 0.50390625,
+      "ret_loss": 0.0,
+      "pretrain_loss": 0.0,
+      "total_loss": 0.12448960542678833
+    }
+  ]
+}

cls_gemma_orderkd_last_l02/checkpoint-200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b87b7fe3f2833be63c93988b277af6205a4ec88beec08107553eb0252e039e4
+size 52231312

cls_gemma_orderkd_last_l02/checkpoint-200/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
+size 4689074

cls_gemma_orderkd_last_l02/checkpoint-400/README.md

@@ -0,0 +1,208 @@
+---
+base_model: unsloth/gemma-3-1b-it
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/gemma-3-1b-it
+- lora
+- transformers
+- unsloth
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.19.1