Publish verified DRU-RE-Yehia release

.gitattributes

@@ -1,2 +1,8 @@
 adapter_model.safetensors filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/adapter_model.safetensors filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/optimizer.pt filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/rng_state.pth filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/scheduler.pt filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/training_args.bin filter=lfs diff=lfs merge=lfs -text
+latest_checkpoint/tokenizer.model filter=lfs diff=lfs merge=lfs -text
 tokenizer.model filter=lfs diff=lfs merge=lfs -text

CHANGELOG.md

@@ -95,10 +95,14 @@ no credentials, machine caches, or abandoned implementation files.
 - Added `tools/publish_huggingface.py`, locked to
   `U4RASD/DRU-RE-Yehia`, to upload only a verified complete folder and then
   verify two fresh remote downloads without revealing token values.
-- Added an Azure Git LFS rule for `adapter_model.safetensors` and a documented
-  fresh-clone deletion gate.
-
-The full gated 7B base, optimizer state, caches, temporary logs, and secrets are
-deliberately excluded from the clean model repository. Exact mid-run resume
-requires a separately archived private numbered checkpoint; completed-model
-inference and retraining from step 0 do not.
+- Added Azure Git LFS rules for the root adapter and final checkpoint binaries,
+  plus a documented fresh-clone deletion gate.
+- Disabled RunPod's optional `hf_transfer` path before every Hub import because
+  the pinned environment does not install that optional package; standard Hub
+  upload/download remains enabled and was used successfully.
+
+At Ahmad's explicit request, the complete latest step-2,943 Trainer checkpoint
+is included with adapter, optimizer, scheduler, RNG, trainer state, tokenizer,
+training arguments, and its own integrity manifest. The root adapter remains
+the selected step-2,750 best inference model. The full gated 7B base, older
+checkpoints, caches, temporary logs, predictions, and secrets remain excluded.

README.md

@@ -24,15 +24,17 @@ one Arabic option code. The model does not have to reproduce a long relation
 phrase or generate explanatory text.
 
 This repository contains the clean training, dataset, audit, inference, and
-release-staging source. It intentionally excludes downloaded base weights,
-dataset JSONL files, checkpoints, optimizer state, caches, notebooks, prediction
-dumps, and rolling logs.
+release-staging source, the selected best inference adapter, and the latest
+complete resumable Trainer checkpoint. It intentionally excludes downloaded
+base weights, local dataset JSONL files, all older numbered checkpoints, caches,
+notebooks, prediction dumps, and rolling logs.
 
 > **Release status:** all three epochs and the final 1,687-row validation are
 > complete at global step 2,943. The selected step-2,750 best adapter is staged
 > at the repository root and passed the complete local release verifier: 448/448
 > tensors are finite and nonzero, all 41 worked classes are covered, and every
-> manifest hash matches. Hugging Face publication provenance is written to
+> manifest hash matches. The full latest step-2,943 optimizer/scheduler/RNG
+> checkpoint is preserved separately under `latest_checkpoint/`. Hugging Face publication provenance is written to
 > `release/hf_remote_verification.json` by the guarded publisher.
 
 ## At a glance
@@ -105,6 +107,7 @@ DRU-RE-Yehia/
     ├── build_examples_appendix.py
     ├── plot_losses.py
     ├── monitor_resources.sh
+    ├── restore_latest_checkpoint.py
     ├── stage_release.py
     ├── verify_release.py
     └── publish_huggingface.py
@@ -112,7 +115,8 @@ DRU-RE-Yehia/
 
 After a successful release-staging operation, the repository root additionally
 contains PEFT adapter/tokenizer files and `inference_config.json`; final run
-evidence is placed under `release/`. Full Yehia base weights are never copied.
+evidence is placed under `release/`, and the complete latest Trainer state is
+placed under `latest_checkpoint/`. Full Yehia base weights are never copied.
 
 ## Task definition
 
@@ -813,7 +817,7 @@ positive micro F1. It is not training loss or teacher-forced validation loss.
 The trainer maintains:
 
 - `checkpoints/checkpoint-<step>/`: resumable Trainer checkpoints;
-- `latest_checkpoint/`: complete named copy of the latest scheduled checkpoint;
+- `latest_checkpoint/`: complete named copy of the final step-2,943 checkpoint;
 - `best_checkpoint/`: complete named copy of the best validation checkpoint;
 - `best_adapter/`: adapter/tokenizer exported after training completes.
 
@@ -1108,7 +1112,10 @@ The staging utility refuses to proceed unless:
 - all-linear evidence reports 225 eligible and 224 wrapped modules with only
   LoRA parameters trainable and the output head frozen;
 - all 448 saved adapter tensors are finite and nonzero;
-- the best-checkpoint metric artifact exists.
+- the best-checkpoint metric artifact exists;
+- `latest_checkpoint/` and numbered `checkpoint-2943/` are byte-identical;
+- the latest checkpoint has finite adapter/optimizer state, complete scheduler
+  state, trainer global step 2,943, RNG state, and training arguments.
 
 It copies only:
 
@@ -1116,7 +1123,9 @@ It copies only:
 - tokenizer/chat-template files required by the adapter;
 - frozen inference configuration, including best-step no-relation bias;
 - final run summary and best validation metrics;
-- small audit evidence and a SHA-256 release manifest.
+- small audit evidence and a SHA-256 release manifest;
+- the complete latest step-2,943 resumable checkpoint, with a separate
+  checkpoint integrity manifest.
 
 All correctness checks finish before repository files are changed. Each staged
 file is copied through a temporary sibling and atomically renamed, preventing an
@@ -1125,8 +1134,7 @@ interrupted copy from appearing as a complete release artifact.
 It explicitly excludes:
 
 - Yehia base weights;
-- optimizer/scheduler/RNG state;
-- numbered and named resumable checkpoints;
+- all older numbered/named checkpoints other than the staged final latest copy;
 - logs and resource time series;
 - per-row validation prediction dumps;
 - caches and temporary files.
@@ -1153,6 +1161,7 @@ The deletion-safe release is intentionally split across durable repositories:
 | Exact transformed train/validation/official JSONL and resources | Hugging Face dataset `U4RASD/Yehia-RE-SFT` at commit `a060e47f56025778b97344d4d3de60a8fd53be7c` | Retraining and official inference input |
 | Entire clean folder in this README, including source, resources, 41 examples, evidence, and release tools | Azure Git and Hugging Face model `U4RASD/DRU-RE-Yehia` | Reproduce, audit, train, and understand the system |
 | Best adapter, tokenizer files, `inference_config.json`, final metrics, and release manifest | Root/release directories of the Hugging Face model repo; mirrored to Azure with Git LFS | Actual model inference and checkpoint provenance |
+| Latest complete `checkpoint-2943` adapter, optimizer, scheduler, RNG, trainer state, tokenizer, and training arguments | `latest_checkpoint/` in the Hugging Face model repo and Azure Git LFS mirror | Exact archival resume state from the completed run |
 | Yehia 7B base at commit `b9dda4715eafee7e8090d2c83cfe078d75f4ebb8` | Upstream gated `Navid-AI/Yehia-7B-preview` | Required underneath the PEFT adapter |
 | Token permissions | An external password manager or secret store, never either repository | Future access to gated/private repositories |
 
@@ -1180,9 +1189,11 @@ missing. It also checks:
 - exact 225 eligible / 224 wrapped all-linear evidence;
 - the portable pinned base ID and revision in adapter/inference configs;
 - 448 finite, nonzero adapter tensors;
+- the complete step-2,943 checkpoint manifest, adapter, finite optimizer state,
+  final scheduler state, RNG payload, and trainer provenance;
 - locked rank 16, alpha 32, dropout 0.05, RS-LoRA settings;
 - the SHA-256 and byte size of every staged manifest entry;
-- absence of full base weights and optimizer/checkpoint payloads;
+- absence of full base weights and any unrecognized checkpoint payloads;
 - in remote-clean mode, absence of every unrecognized or stale remote file.
 
 This command intentionally fails before the final adapter is staged. A source
@@ -1214,7 +1225,8 @@ with the experiment record. No token value is printed or persisted.
 
 When the Azure DevOps repository URL is supplied, push this same directory—not
 the live experiment directory. `.gitattributes` already places
-`adapter_model.safetensors` under Git LFS. A representative workflow is:
+the root adapter plus large latest-checkpoint binaries under Git LFS. A
+representative workflow is:
 
 ```bash
 git init -b main
@@ -1265,16 +1277,32 @@ The durable clean release is sufficient to:
 - reproduce the transformed dataset;
 - retrain from optimizer step 0;
 - load the completed best adapter for inference;
-- reproduce constrained decoding and output mapping.
+- reproduce constrained decoding and output mapping;
+- restore the exact latest completed Trainer state from step 2,943.
 
-It intentionally does **not** preserve exact mid-training resume state. Exact
-resume requires a complete numbered checkpoint containing adapter weights,
-`optimizer.pt`, `scheduler.pt`, `trainer_state.json`, RNG state, and matching
-training arguments. Once all three epochs and final validation are complete,
-that state is not needed to use or retrain the released model. If exact
-continuation of the completed run may be needed later, archive the final
-numbered checkpoint separately in private Azure Blob/Artifact storage before
-deleting the pod; do not mix it into the clean model repository.
+The root `adapter_model.safetensors` is the selected step-2,750 **best inference
+adapter**. `latest_checkpoint/adapter_model.safetensors` is the different final
+step-2,943 training state accompanied by its optimizer, scheduler, RNG,
+trainer state, and training arguments. Applications should use the root best
+adapter; the checkpoint directory exists for archival resume/recovery.
+
+To restore it into the default run layout:
+
+```bash
+python tools/restore_latest_checkpoint.py
+```
+
+This verifies every checkpoint hash and tensor first, then atomically creates:
+
+```text
+runs/DRU-RE-Yehia/checkpoints/checkpoint-2943/
+```
+
+With the locked three-epoch configuration, that checkpoint is already at the
+training endpoint, so a normal resume performs no additional optimizer update
+and proceeds to finalization. Extending it to more epochs or changing the
+learning-rate schedule is a new experiment, not an exact continuation of the
+documented run. Older intermediate checkpoints are intentionally excluded.
 
 ### Deletion gate
 
@@ -1283,19 +1311,20 @@ The training machine is safe to delete only after every item below is true:
 - [ ] Three-epoch training and final 1,687-row validation completed normally.
 - [ ] `run_summary.json` says `status=complete`.
 - [ ] `tools/stage_release.py` staged the best—not merely latest—adapter.
+- [ ] `latest_checkpoint/checkpoint_manifest.json` proves the complete final
+  step-2,943 optimizer/scheduler/RNG checkpoint is present.
 - [ ] Local `tools/verify_release.py` passed.
 - [ ] `tools/publish_huggingface.py` passed its fresh final-download check.
 - [ ] The Hugging Face model `final_revision` was recorded.
 - [ ] The pinned SFT dataset commit remains remotely readable and hash-verified.
-- [ ] The entire clean folder was committed to Azure, with the adapter LFS
-  object present.
+- [ ] The entire clean folder was committed to Azure, with root adapter and
+  latest-checkpoint LFS objects present.
 - [ ] A fresh Azure clone passed `tools/verify_release.py --remote-clean`.
 - [ ] At least one end-to-end inference smoke test passed from a fresh checkout
   using the gated pinned Yehia base, prepared input, and staged adapter.
 - [ ] Azure Git commit, Hugging Face model commit, Hugging Face dataset commit,
   and Yehia base commit were recorded together.
 - [ ] Required future repository credentials/permissions exist outside the pod.
-- [ ] Any optional exact-resume checkpoint desired later was archived separately.
 
 Until that entire gate passes, deleting the pod is premature. After it passes,
 temporary caches, plots, rolling logs, downloaded base files, transformed local
@@ -1451,7 +1480,8 @@ no-relation bias, and decode through the same index. The supplied
 - The gated 7B base model.
 - The 133+ MB transformed JSONL dataset.
 - Any source-dataset cache.
-- Live training checkpoints or optimizer state.
+- Intermediate checkpoints and optimizer states other than the final staged
+  `latest_checkpoint/` copy.
 - Notebook prompt experiments.
 - Temporary benchmark scripts and abandoned objectives.
 - Rolling trainer/resource logs.
@@ -1459,9 +1489,8 @@ no-relation bias, and decode through the same index. The supplied
 - A premature adapter snapshot from an incomplete run.
 
 This keeps the model repository understandable while preserving every file
-needed to download, audit, rebuild, retrain from step 0, evaluate, stage, and use
-the selected system. Exact optimizer-state resume is the deliberate exception
-described in the durability section above.
+needed to download, audit, rebuild, retrain from step 0, evaluate, stage, use
+the selected system, and restore the final optimizer state.
 
 ## Troubleshooting
 
@@ -1548,7 +1577,8 @@ Before claiming a faithful retrain, verify:
 - [ ] Fresh Hugging Face and Azure downloads both pass remote-clean verification.
 - [ ] An end-to-end prepared-input inference passes from a fresh checkout.
 - [ ] Immutable Azure, model, dataset, and base revisions are recorded.
-- [ ] No base weights, optimizer state, logs, caches, or credentials are pushed.
+- [ ] Only the explicitly manifested final optimizer checkpoint is pushed; no
+  base weights, older checkpoints, logs, caches, or credentials are present.
 
 ## Pre-publication responsibility
 

download_dataset.py

@@ -10,6 +10,8 @@ import os
 from pathlib import Path
 from typing import Any, Dict
 
+os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0"
+
 from dotenv import load_dotenv
 from huggingface_hub import snapshot_download
 

latest_checkpoint/README.md

@@ -0,0 +1,207 @@
+---
+base_model: /root/workspace/models/Yehia-7B-preview
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:/root/workspace/models/Yehia-7B-preview
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.17.1

latest_checkpoint/adapter_config.json

@@ -0,0 +1,42 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Navid-AI/Yehia-7B-preview",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": "b9dda4715eafee7e8090d2c83cfe078d75f4ebb8",
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "o_proj",
+    "gate_proj",
+    "k_proj",
+    "q_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

latest_checkpoint/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4390a7f0c09141c9e6c9dc0e82164e7d5a6eb2e07eca3f44c7f76467485bae71
+size 159967880

latest_checkpoint/chat_template.jinja

@@ -0,0 +1 @@
+{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\n' + system_message + '\n<</SYS>>\n\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' '  + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}

latest_checkpoint/checkpoint_manifest.json

@@ -0,0 +1,80 @@
+{
+  "status": "complete",
+  "checkpoint_type": "latest_complete_resumable_trainer_checkpoint",
+  "global_step": 2943,
+  "best_checkpoint_step": 2750,
+  "audit": {
+    "global_step": 2943,
+    "best_checkpoint_step": 2750,
+    "adapter": {
+      "tensor_count": 448,
+      "finite_tensors": 448,
+      "nonzero_tensors": 448
+    },
+    "optimizer": {
+      "tensor_count": 2688,
+      "finite_floating_tensor_count": 1792
+    },
+    "scheduler": {
+      "tensor_count": 0,
+      "finite_floating_tensor_count": 0
+    },
+    "rng_state_bytes": 14244,
+    "training_args_bytes": 5496
+  },
+  "files": {
+    "README.md": {
+      "bytes": 5232,
+      "sha256": "32c8a003763070b74cdb20f608f0e5a726570b11d56469cd4b80b2753bd274f4"
+    },
+    "adapter_config.json": {
+      "bytes": 975,
+      "sha256": "68656a0760952d4504c6f6cdfb03487622327b6391315289790192bfa6951721"
+    },
+    "adapter_model.safetensors": {
+      "bytes": 159967880,
+      "sha256": "4390a7f0c09141c9e6c9dc0e82164e7d5a6eb2e07eca3f44c7f76467485bae71"
+    },
+    "chat_template.jinja": {
+      "bytes": 816,
+      "sha256": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f"
+    },
+    "optimizer.pt": {
+      "bytes": 81735892,
+      "sha256": "58da25e4579a062fa12ad066e34b666619a4609957fbcc9d246212ca4a431fad"
+    },
+    "rng_state.pth": {
+      "bytes": 14244,
+      "sha256": "e1364bce33b92ffd2389bb5e6f0f8bb88878c2a769c50a7e2403c9cd9f3c1fd9"
+    },
+    "scheduler.pt": {
+      "bytes": 1064,
+      "sha256": "bd16c6406d87f70b6c298a5e4fb685236ce307ae801894a350532f2e95d3a1fe"
+    },
+    "special_tokens_map.json": {
+      "bytes": 551,
+      "sha256": "82d96d7a9e6ced037f12394b7ea6a5b02e6ca87e0d11edaa8d60d9be857ce7db"
+    },
+    "tokenizer.json": {
+      "bytes": 8508152,
+      "sha256": "a4ec65a3ef5dee870b4024cf17d9315aff58dc1555e05766f1c18ebfe8405dba"
+    },
+    "tokenizer.model": {
+      "bytes": 1228714,
+      "sha256": "feff3deb3537c4a47b77585053ad2aa00484da7907b188e0111cbed1f4592e67"
+    },
+    "tokenizer_config.json": {
+      "bytes": 4402,
+      "sha256": "92a93150cc2a9e821e6fc785bbb0aea3cec5d92df529974131efc1c857b52359"
+    },
+    "trainer_state.json": {
+      "bytes": 43812,
+      "sha256": "95489ed7a630b99057d444bd7469f2eeb2f9e5f618b077f8f1873f9a1d118251"
+    },
+    "training_args.bin": {
+      "bytes": 5496,
+      "sha256": "213db7d4f648a556f875d6706bbb4d0e9ef6f78fecf9b503048cedff8a477c00"
+    }
+  },
+  "full_base_weights_included": false
+}

latest_checkpoint/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58da25e4579a062fa12ad066e34b666619a4609957fbcc9d246212ca4a431fad
+size 81735892

latest_checkpoint/special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

latest_checkpoint/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:feff3deb3537c4a47b77585053ad2aa00484da7907b188e0111cbed1f4592e67
+size 1228714

latest_checkpoint/tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "0",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "4": {
+      "content": "1",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "5": {
+      "content": "2",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "6": {
+      "content": "3",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "7": {
+      "content": "4",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "5",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "6",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "7",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "8",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "9",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "٠",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "١",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "15": {
+      "content": "٢",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "16": {
+      "content": "٣",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "17": {
+      "content": "٤",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "18": {
+      "content": "٥",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "19": {
+      "content": "٦",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "20": {
+      "content": "٧",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "21": {
+      "content": "٨",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "22": {
+      "content": "٩",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": false,
+  "max_length": null,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_to_multiple_of": null,
+  "pad_token": "</s>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

latest_checkpoint/trainer_state.json

@@ -0,0 +1,1448 @@
+{
+  "best_global_step": 2750,
+  "best_metric": 0.6332622601279317,
+  "best_model_checkpoint": "outputs/DRU-RE-Yehia-choice-final/checkpoints/checkpoint-2750",
+  "epoch": 3.0,
+  "eval_steps": 250,
+  "global_step": 2943,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02039775624681285,
+      "grad_norm": 32.115570068359375,
+      "learning_rate": 6.418918918918919e-06,
+      "loss": 1.4138,
+      "step": 20
+    },
+    {
+      "epoch": 0.0407955124936257,
+      "grad_norm": 36.402008056640625,
+      "learning_rate": 1.3175675675675675e-05,
+      "loss": 1.0761,
+      "step": 40
+    },
+    {
+      "epoch": 0.06119326874043855,
+      "grad_norm": 36.12466812133789,
+      "learning_rate": 1.9932432432432434e-05,
+      "loss": 0.9291,
+      "step": 60
+    },
+    {
+      "epoch": 0.0815910249872514,
+      "grad_norm": 32.74873352050781,
+      "learning_rate": 2.668918918918919e-05,
+      "loss": 0.8514,
+      "step": 80
+    },
+    {
+      "epoch": 0.10198878123406425,
+      "grad_norm": 25.69573211669922,
+      "learning_rate": 3.344594594594595e-05,
+      "loss": 0.7376,
+      "step": 100
+    },
+    {
+      "epoch": 0.1223865374808771,
+      "grad_norm": 18.53243064880371,
+      "learning_rate": 4.0202702702702707e-05,
+      "loss": 0.6736,
+      "step": 120
+    },
+    {
+      "epoch": 0.14278429372768994,
+      "grad_norm": 42.1881217956543,
+      "learning_rate": 4.695945945945946e-05,
+      "loss": 0.5628,
+      "step": 140
+    },
+    {
+      "epoch": 0.1631820499745028,
+      "grad_norm": 29.84023666381836,
+      "learning_rate": 4.999808915274971e-05,
+      "loss": 0.7305,
+      "step": 160
+    },
+    {
+      "epoch": 0.18357980622131564,
+      "grad_norm": 14.481285095214844,
+      "learning_rate": 4.998482510897682e-05,
+      "loss": 0.4569,
+      "step": 180
+    },
+    {
+      "epoch": 0.2039775624681285,
+      "grad_norm": 11.743460655212402,
+      "learning_rate": 4.995893540384707e-05,
+      "loss": 0.6409,
+      "step": 200
+    },
+    {
+      "epoch": 0.22437531871494135,
+      "grad_norm": 20.807933807373047,
+      "learning_rate": 4.9920433120287726e-05,
+      "loss": 0.647,
+      "step": 220
+    },
+    {
+      "epoch": 0.2447730749617542,
+      "grad_norm": 9.826773643493652,
+      "learning_rate": 4.986933771478052e-05,
+      "loss": 0.5513,
+      "step": 240
+    },
+    {
+      "epoch": 0.25497195308516063,
+      "eval_accuracy": 0.6585655008891523,
+      "eval_decision_nll": 1.3316565713164505,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 250,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 1.3316565713164505,
+      "eval_macro_f1_41": 0.6391334292743345,
+      "eval_no_relation_logit_bias": -1.0800000000000387,
+      "eval_positive_fn": 345,
+      "eval_positive_fp": 398,
+      "eval_positive_micro_f1": 0.5842193620593173,
+      "eval_positive_micro_precision": 0.5673913043478261,
+      "eval_positive_micro_recall": 0.6020761245674741,
+      "eval_positive_tp": 522,
+      "eval_raw_accuracy": 0.6674570243034973,
+      "eval_raw_macro_f1_41": 0.6304263381181072,
+      "eval_raw_positive_micro_f1": 0.5753588516746412,
+      "eval_raw_positive_micro_precision": 0.5975155279503106,
+      "eval_raw_positive_micro_recall": 0.5547866205305652,
+      "eval_raw_weighted_f1_41": 0.644239845397826,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6674570243034973,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6304263381181072,
+      "eval_unconstrained_positive_micro_f1": 0.5753588516746412,
+      "eval_unconstrained_positive_micro_precision": 0.5975155279503106,
+      "eval_unconstrained_positive_micro_recall": 0.5547866205305652,
+      "eval_unconstrained_weighted_f1_41": 0.644239845397826,
+      "eval_weighted_f1_41": 0.6476916479143388,
+      "step": 250
+    },
+    {
+      "epoch": 0.26517083120856705,
+      "grad_norm": 15.046053886413574,
+      "learning_rate": 4.98056750075297e-05,
+      "loss": 0.4321,
+      "step": 260
+    },
+    {
+      "epoch": 0.2855685874553799,
+      "grad_norm": 11.937222480773926,
+      "learning_rate": 4.972947716941413e-05,
+      "loss": 0.4411,
+      "step": 280
+    },
+    {
+      "epoch": 0.3059663437021928,
+      "grad_norm": 11.15040111541748,
+      "learning_rate": 4.9640782705730394e-05,
+      "loss": 0.4402,
+      "step": 300
+    },
+    {
+      "epoch": 0.3263640999490056,
+      "grad_norm": 9.428462982177734,
+      "learning_rate": 4.953963643673468e-05,
+      "loss": 0.4287,
+      "step": 320
+    },
+    {
+      "epoch": 0.34676185619581845,
+      "grad_norm": 25.721332550048828,
+      "learning_rate": 4.9426089474993696e-05,
+      "loss": 0.5057,
+      "step": 340
+    },
+    {
+      "epoch": 0.3671596124426313,
+      "grad_norm": 8.766318321228027,
+      "learning_rate": 4.930019919955576e-05,
+      "loss": 0.3315,
+      "step": 360
+    },
+    {
+      "epoch": 0.3875573686894442,
+      "grad_norm": 10.508740425109863,
+      "learning_rate": 4.916202922695526e-05,
+      "loss": 0.4607,
+      "step": 380
+    },
+    {
+      "epoch": 0.407955124936257,
+      "grad_norm": 4.016866683959961,
+      "learning_rate": 4.901164937906511e-05,
+      "loss": 0.2961,
+      "step": 400
+    },
+    {
+      "epoch": 0.42835288118306986,
+      "grad_norm": 21.203813552856445,
+      "learning_rate": 4.884913564781346e-05,
+      "loss": 0.3825,
+      "step": 420
+    },
+    {
+      "epoch": 0.4487506374298827,
+      "grad_norm": 2.3917906284332275,
+      "learning_rate": 4.867457015678244e-05,
+      "loss": 0.4058,
+      "step": 440
+    },
+    {
+      "epoch": 0.4691483936766956,
+      "grad_norm": 19.52354621887207,
+      "learning_rate": 4.848804111970836e-05,
+      "loss": 0.3567,
+      "step": 460
+    },
+    {
+      "epoch": 0.4895461499235084,
+      "grad_norm": 19.423274993896484,
+      "learning_rate": 4.8289642795904433e-05,
+      "loss": 0.3384,
+      "step": 480
+    },
+    {
+      "epoch": 0.5099439061703213,
+      "grad_norm": 4.385583877563477,
+      "learning_rate": 4.807947544262839e-05,
+      "loss": 0.3797,
+      "step": 500
+    },
+    {
+      "epoch": 0.5099439061703213,
+      "eval_accuracy": 0.6822762299940723,
+      "eval_decision_nll": 1.3185705768334208,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 500,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 1.3185705768334208,
+      "eval_macro_f1_41": 0.6691422803869735,
+      "eval_no_relation_logit_bias": 0.31999999999995643,
+      "eval_positive_fn": 333,
+      "eval_positive_fp": 389,
+      "eval_positive_micro_f1": 0.5966480446927374,
+      "eval_positive_micro_precision": 0.5785482123510293,
+      "eval_positive_micro_recall": 0.615916955017301,
+      "eval_positive_tp": 534,
+      "eval_raw_accuracy": 0.6716064018968583,
+      "eval_raw_macro_f1_41": 0.6698756813978369,
+      "eval_raw_positive_micro_f1": 0.5944474686989656,
+      "eval_raw_positive_micro_precision": 0.5628865979381443,
+      "eval_raw_positive_micro_recall": 0.629757785467128,
+      "eval_raw_weighted_f1_41": 0.6513946323858064,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6716064018968583,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6698756813978369,
+      "eval_unconstrained_positive_micro_f1": 0.5944474686989656,
+      "eval_unconstrained_positive_micro_precision": 0.5628865979381443,
+      "eval_unconstrained_positive_micro_recall": 0.629757785467128,
+      "eval_unconstrained_weighted_f1_41": 0.6513946323858064,
+      "eval_weighted_f1_41": 0.6589857809303985,
+      "step": 500
+    },
+    {
+      "epoch": 0.5303416624171341,
+      "grad_norm": 2.1002912521362305,
+      "learning_rate": 4.785764526441913e-05,
+      "loss": 0.3829,
+      "step": 520
+    },
+    {
+      "epoch": 0.5507394186639469,
+      "grad_norm": 12.052459716796875,
+      "learning_rate": 4.762426435942812e-05,
+      "loss": 0.3276,
+      "step": 540
+    },
+    {
+      "epoch": 0.5711371749107598,
+      "grad_norm": 11.039682388305664,
+      "learning_rate": 4.7379450662772394e-05,
+      "loss": 0.337,
+      "step": 560
+    },
+    {
+      "epoch": 0.5915349311575727,
+      "grad_norm": 4.882332801818848,
+      "learning_rate": 4.7123327886938076e-05,
+      "loss": 0.3724,
+      "step": 580
+    },
+    {
+      "epoch": 0.6119326874043856,
+      "grad_norm": 4.446337699890137,
+      "learning_rate": 4.685602545926432e-05,
+      "loss": 0.2834,
+      "step": 600
+    },
+    {
+      "epoch": 0.6323304436511984,
+      "grad_norm": 10.867338180541992,
+      "learning_rate": 4.657767845653943e-05,
+      "loss": 0.4342,
+      "step": 620
+    },
+    {
+      "epoch": 0.6527281998980112,
+      "grad_norm": 0.6202830672264099,
+      "learning_rate": 4.628842753674203e-05,
+      "loss": 0.2548,
+      "step": 640
+    },
+    {
+      "epoch": 0.6731259561448241,
+      "grad_norm": 0.782004177570343,
+      "learning_rate": 4.598841886796192e-05,
+      "loss": 0.2398,
+      "step": 660
+    },
+    {
+      "epoch": 0.6935237123916369,
+      "grad_norm": 3.5981996059417725,
+      "learning_rate": 4.5677804054536544e-05,
+      "loss": 0.3756,
+      "step": 680
+    },
+    {
+      "epoch": 0.7139214686384497,
+      "grad_norm": 12.359613418579102,
+      "learning_rate": 4.5356740060440194e-05,
+      "loss": 0.3284,
+      "step": 700
+    },
+    {
+      "epoch": 0.7343192248852626,
+      "grad_norm": 4.867230415344238,
+      "learning_rate": 4.502538912996499e-05,
+      "loss": 0.3286,
+      "step": 720
+    },
+    {
+      "epoch": 0.7547169811320755,
+      "grad_norm": 26.788902282714844,
+      "learning_rate": 4.468391870573342e-05,
+      "loss": 0.2513,
+      "step": 740
+    },
+    {
+      "epoch": 0.7649158592554819,
+      "eval_accuracy": 0.6609365737996443,
+      "eval_decision_nll": 1.5595358801651227,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 750,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 1.5595358801651227,
+      "eval_macro_f1_41": 0.6404558346600874,
+      "eval_no_relation_logit_bias": -0.64000000000004,
+      "eval_positive_fn": 362,
+      "eval_positive_fp": 349,
+      "eval_positive_micro_f1": 0.5868680999418943,
+      "eval_positive_micro_precision": 0.5913348946135831,
+      "eval_positive_micro_recall": 0.5824682814302191,
+      "eval_positive_tp": 505,
+      "eval_raw_accuracy": 0.6644931831653823,
+      "eval_raw_macro_f1_41": 0.6435209638657218,
+      "eval_raw_positive_micro_f1": 0.5820359281437126,
+      "eval_raw_positive_micro_precision": 0.6052303860523038,
+      "eval_raw_positive_micro_recall": 0.5605536332179931,
+      "eval_raw_weighted_f1_41": 0.6447652943100252,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6644931831653823,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6435209638657218,
+      "eval_unconstrained_positive_micro_f1": 0.5820359281437126,
+      "eval_unconstrained_positive_micro_precision": 0.6052303860523038,
+      "eval_unconstrained_positive_micro_recall": 0.5605536332179931,
+      "eval_unconstrained_weighted_f1_41": 0.6447652943100252,
+      "eval_weighted_f1_41": 0.6460930593744794,
+      "step": 750
+    },
+    {
+      "epoch": 0.7751147373788884,
+      "grad_norm": 9.591691970825195,
+      "learning_rate": 4.433250134408401e-05,
+      "loss": 0.2574,
+      "step": 760
+    },
+    {
+      "epoch": 0.7955124936257012,
+      "grad_norm": 4.1339945793151855,
+      "learning_rate": 4.39713146278729e-05,
+      "loss": 0.3176,
+      "step": 780
+    },
+    {
+      "epoch": 0.815910249872514,
+      "grad_norm": 6.561213493347168,
+      "learning_rate": 4.3600541076735346e-05,
+      "loss": 0.1778,
+      "step": 800
+    },
+    {
+      "epoch": 0.8363080061193269,
+      "grad_norm": 16.542247772216797,
+      "learning_rate": 4.322036805485245e-05,
+      "loss": 0.3041,
+      "step": 820
+    },
+    {
+      "epoch": 0.8567057623661397,
+      "grad_norm": 0.9108038544654846,
+      "learning_rate": 4.283098767626984e-05,
+      "loss": 0.2437,
+      "step": 840
+    },
+    {
+      "epoch": 0.8771035186129525,
+      "grad_norm": 3.841179847717285,
+      "learning_rate": 4.243259670781611e-05,
+      "loss": 0.3884,
+      "step": 860
+    },
+    {
+      "epoch": 0.8975012748597654,
+      "grad_norm": 10.145074844360352,
+      "learning_rate": 4.202539646966993e-05,
+      "loss": 0.2895,
+      "step": 880
+    },
+    {
+      "epoch": 0.9178990311065782,
+      "grad_norm": 0.8870783448219299,
+      "learning_rate": 4.1609592733626335e-05,
+      "loss": 0.2225,
+      "step": 900
+    },
+    {
+      "epoch": 0.9382967873533912,
+      "grad_norm": 0.04693833366036415,
+      "learning_rate": 4.118539561911339e-05,
+      "loss": 0.1843,
+      "step": 920
+    },
+    {
+      "epoch": 0.958694543600204,
+      "grad_norm": 8.443830490112305,
+      "learning_rate": 4.075301948701186e-05,
+      "loss": 0.2522,
+      "step": 940
+    },
+    {
+      "epoch": 0.9790922998470168,
+      "grad_norm": 0.20142008364200592,
+      "learning_rate": 4.031268283133158e-05,
+      "loss": 0.2275,
+      "step": 960
+    },
+    {
+      "epoch": 0.9994900560938297,
+      "grad_norm": 3.326535940170288,
+      "learning_rate": 3.986460816879913e-05,
+      "loss": 0.2407,
+      "step": 980
+    },
+    {
+      "epoch": 1.0193778684344723,
+      "grad_norm": 15.066499710083008,
+      "learning_rate": 3.9409021926412795e-05,
+      "loss": 0.2335,
+      "step": 1000
+    },
+    {
+      "epoch": 1.0193778684344723,
+      "eval_accuracy": 0.6597510373443983,
+      "eval_decision_nll": 1.9229287061029874,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 1000,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 1.9229287061029874,
+      "eval_macro_f1_41": 0.6841875774786024,
+      "eval_no_relation_logit_bias": -4.340000000000028,
+      "eval_positive_fn": 306,
+      "eval_positive_fp": 448,
+      "eval_positive_micro_f1": 0.5980810234541578,
+      "eval_positive_micro_precision": 0.55599603567889,
+      "eval_positive_micro_recall": 0.6470588235294118,
+      "eval_positive_tp": 561,
+      "eval_raw_accuracy": 0.6674570243034973,
+      "eval_raw_macro_f1_41": 0.6585356377059267,
+      "eval_raw_positive_micro_f1": 0.5758477096966093,
+      "eval_raw_positive_micro_precision": 0.5945945945945946,
+      "eval_raw_positive_micro_recall": 0.558246828143022,
+      "eval_raw_weighted_f1_41": 0.6389810017230654,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6674570243034973,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6585356377059267,
+      "eval_unconstrained_positive_micro_f1": 0.5758477096966093,
+      "eval_unconstrained_positive_micro_precision": 0.5945945945945946,
+      "eval_unconstrained_positive_micro_recall": 0.558246828143022,
+      "eval_unconstrained_weighted_f1_41": 0.6389810017230654,
+      "eval_weighted_f1_41": 0.6457988065135049,
+      "step": 1000
+    },
+    {
+      "epoch": 1.039775624681285,
+      "grad_norm": 1.4003945589065552,
+      "learning_rate": 3.8946154327021434e-05,
+      "loss": 0.257,
+      "step": 1020
+    },
+    {
+      "epoch": 1.060173380928098,
+      "grad_norm": 9.492857933044434,
+      "learning_rate": 3.8476239272985284e-05,
+      "loss": 0.2098,
+      "step": 1040
+    },
+    {
+      "epoch": 1.0805711371749107,
+      "grad_norm": 0.42550796270370483,
+      "learning_rate": 3.7999514227977304e-05,
+      "loss": 0.1815,
+      "step": 1060
+    },
+    {
+      "epoch": 1.1009688934217237,
+      "grad_norm": 13.100898742675781,
+      "learning_rate": 3.751622009698492e-05,
+      "loss": 0.2888,
+      "step": 1080
+    },
+    {
+      "epoch": 1.1213666496685364,
+      "grad_norm": 19.209449768066406,
+      "learning_rate": 3.702660110457272e-05,
+      "loss": 0.1834,
+      "step": 1100
+    },
+    {
+      "epoch": 1.1417644059153493,
+      "grad_norm": 24.14339256286621,
+      "learning_rate": 3.65309046714677e-05,
+      "loss": 0.2111,
+      "step": 1120
+    },
+    {
+      "epoch": 1.1621621621621623,
+      "grad_norm": 1.1118710041046143,
+      "learning_rate": 3.602938128952933e-05,
+      "loss": 0.1372,
+      "step": 1140
+    },
+    {
+      "epoch": 1.182559918408975,
+      "grad_norm": 0.1752534657716751,
+      "learning_rate": 3.5522284395167724e-05,
+      "loss": 0.1914,
+      "step": 1160
+    },
+    {
+      "epoch": 1.202957674655788,
+      "grad_norm": 9.813786506652832,
+      "learning_rate": 3.50098702412738e-05,
+      "loss": 0.1248,
+      "step": 1180
+    },
+    {
+      "epoch": 1.2233554309026007,
+      "grad_norm": 16.98102569580078,
+      "learning_rate": 3.4492397767726195e-05,
+      "loss": 0.2289,
+      "step": 1200
+    },
+    {
+      "epoch": 1.2437531871494136,
+      "grad_norm": 0.7757235765457153,
+      "learning_rate": 3.397012847054035e-05,
+      "loss": 0.1747,
+      "step": 1220
+    },
+    {
+      "epoch": 1.2641509433962264,
+      "grad_norm": 0.39124366641044617,
+      "learning_rate": 3.34433262697259e-05,
+      "loss": 0.1529,
+      "step": 1240
+    },
+    {
+      "epoch": 1.2743498215196327,
+      "eval_accuracy": 0.6342620035566093,
+      "eval_decision_nll": 2.0380127403110038,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 1250,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.0380127403110038,
+      "eval_macro_f1_41": 0.6698930909088061,
+      "eval_no_relation_logit_bias": -3.390000000000033,
+      "eval_positive_fn": 296,
+      "eval_positive_fp": 495,
+      "eval_positive_micro_f1": 0.5907915157785825,
+      "eval_positive_micro_precision": 0.5356472795497186,
+      "eval_positive_micro_recall": 0.6585928489042676,
+      "eval_positive_tp": 571,
+      "eval_raw_accuracy": 0.6544161232957914,
+      "eval_raw_macro_f1_41": 0.6698988570776612,
+      "eval_raw_positive_micro_f1": 0.5903814262023218,
+      "eval_raw_positive_micro_precision": 0.5668789808917197,
+      "eval_raw_positive_micro_recall": 0.615916955017301,
+      "eval_raw_weighted_f1_41": 0.6424805452797219,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6544161232957914,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6698988570776612,
+      "eval_unconstrained_positive_micro_f1": 0.5903814262023218,
+      "eval_unconstrained_positive_micro_precision": 0.5668789808917197,
+      "eval_unconstrained_positive_micro_recall": 0.615916955017301,
+      "eval_unconstrained_weighted_f1_41": 0.6424805452797219,
+      "eval_weighted_f1_41": 0.6300222619645572,
+      "step": 1250
+    },
+    {
+      "epoch": 1.2845486996430393,
+      "grad_norm": 22.90714454650879,
+      "learning_rate": 3.2912257375919126e-05,
+      "loss": 0.1949,
+      "step": 1260
+    },
+    {
+      "epoch": 1.3049464558898523,
+      "grad_norm": 0.18795835971832275,
+      "learning_rate": 3.237719015585787e-05,
+      "loss": 0.1413,
+      "step": 1280
+    },
+    {
+      "epoch": 1.325344212136665,
+      "grad_norm": 14.755730628967285,
+      "learning_rate": 3.1838394996766946e-05,
+      "loss": 0.1334,
+      "step": 1300
+    },
+    {
+      "epoch": 1.3457419683834777,
+      "grad_norm": 5.208130359649658,
+      "learning_rate": 3.12961441697225e-05,
+      "loss": 0.1567,
+      "step": 1320
+    },
+    {
+      "epoch": 1.3661397246302907,
+      "grad_norm": 21.01090431213379,
+      "learning_rate": 3.075071169206437e-05,
+      "loss": 0.1725,
+      "step": 1340
+    },
+    {
+      "epoch": 1.3865374808771036,
+      "grad_norm": 10.819672584533691,
+      "learning_rate": 3.0202373188926037e-05,
+      "loss": 0.1455,
+      "step": 1360
+    },
+    {
+      "epoch": 1.4069352371239163,
+      "grad_norm": 13.872186660766602,
+      "learning_rate": 2.965140575395211e-05,
+      "loss": 0.1137,
+      "step": 1380
+    },
+    {
+      "epoch": 1.4273329933707293,
+      "grad_norm": 0.21544072031974792,
+      "learning_rate": 2.9098087809273743e-05,
+      "loss": 0.243,
+      "step": 1400
+    },
+    {
+      "epoch": 1.447730749617542,
+      "grad_norm": 16.329748153686523,
+      "learning_rate": 2.854269896481261e-05,
+      "loss": 0.1533,
+      "step": 1420
+    },
+    {
+      "epoch": 1.468128505864355,
+      "grad_norm": 0.31907919049263,
+      "learning_rate": 2.7985519876984795e-05,
+      "loss": 0.1507,
+      "step": 1440
+    },
+    {
+      "epoch": 1.4885262621111677,
+      "grad_norm": 6.718308925628662,
+      "learning_rate": 2.7426832106875772e-05,
+      "loss": 0.1287,
+      "step": 1460
+    },
+    {
+      "epoch": 1.5089240183579806,
+      "grad_norm": 0.5384275317192078,
+      "learning_rate": 2.6866917977958246e-05,
+      "loss": 0.1687,
+      "step": 1480
+    },
+    {
+      "epoch": 1.5293217746047936,
+      "grad_norm": 2.506664514541626,
+      "learning_rate": 2.6306060433424712e-05,
+      "loss": 0.1143,
+      "step": 1500
+    },
+    {
+      "epoch": 1.5293217746047936,
+      "eval_accuracy": 0.7030231179608772,
+      "eval_decision_nll": 2.2493974038956783,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 1500,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.2493974038956783,
+      "eval_macro_f1_41": 0.6625316285831848,
+      "eval_no_relation_logit_bias": -7.1200000000000205,
+      "eval_positive_fn": 288,
+      "eval_positive_fp": 385,
+      "eval_positive_micro_f1": 0.6324412889131622,
+      "eval_positive_micro_precision": 0.6006224066390041,
+      "eval_positive_micro_recall": 0.6678200692041523,
+      "eval_positive_tp": 579,
+      "eval_raw_accuracy": 0.6870183758150563,
+      "eval_raw_macro_f1_41": 0.610806649098418,
+      "eval_raw_positive_micro_f1": 0.5806861499364676,
+      "eval_raw_positive_micro_precision": 0.6463932107496464,
+      "eval_raw_positive_micro_recall": 0.5271049596309112,
+      "eval_raw_weighted_f1_41": 0.6527180202093893,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6870183758150563,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.610806649098418,
+      "eval_unconstrained_positive_micro_f1": 0.5806861499364676,
+      "eval_unconstrained_positive_micro_precision": 0.6463932107496464,
+      "eval_unconstrained_positive_micro_recall": 0.5271049596309112,
+      "eval_unconstrained_weighted_f1_41": 0.6527180202093893,
+      "eval_weighted_f1_41": 0.6849216700402462,
+      "step": 1500
+    },
+    {
+      "epoch": 1.5497195308516063,
+      "grad_norm": 12.772496223449707,
+      "learning_rate": 2.5744542893206924e-05,
+      "loss": 0.1322,
+      "step": 1520
+    },
+    {
+      "epoch": 1.570117287098419,
+      "grad_norm": 10.884116172790527,
+      "learning_rate": 2.5182649110754324e-05,
+      "loss": 0.1996,
+      "step": 1540
+    },
+    {
+      "epoch": 1.590515043345232,
+      "grad_norm": 2.9933547973632812,
+      "learning_rate": 2.4620663029644068e-05,
+      "loss": 0.139,
+      "step": 1560
+    },
+    {
+      "epoch": 1.610912799592045,
+      "grad_norm": 4.138698101043701,
+      "learning_rate": 2.4058868640094857e-05,
+      "loss": 0.1658,
+      "step": 1580
+    },
+    {
+      "epoch": 1.6313105558388576,
+      "grad_norm": 35.998722076416016,
+      "learning_rate": 2.3497549835457328e-05,
+      "loss": 0.1253,
+      "step": 1600
+    },
+    {
+      "epoch": 1.6517083120856706,
+      "grad_norm": 15.26508617401123,
+      "learning_rate": 2.293699026875322e-05,
+      "loss": 0.1585,
+      "step": 1620
+    },
+    {
+      "epoch": 1.6721060683324835,
+      "grad_norm": 0.22217246890068054,
+      "learning_rate": 2.2377473209336214e-05,
+      "loss": 0.129,
+      "step": 1640
+    },
+    {
+      "epoch": 1.6925038245792963,
+      "grad_norm": 6.413891315460205,
+      "learning_rate": 2.1819281399746392e-05,
+      "loss": 0.1019,
+      "step": 1660
+    },
+    {
+      "epoch": 1.712901580826109,
+      "grad_norm": 0.07089672237634659,
+      "learning_rate": 2.126269691283117e-05,
+      "loss": 0.1216,
+      "step": 1680
+    },
+    {
+      "epoch": 1.733299337072922,
+      "grad_norm": 0.10279195755720139,
+      "learning_rate": 2.070800100920442e-05,
+      "loss": 0.1457,
+      "step": 1700
+    },
+    {
+      "epoch": 1.753697093319735,
+      "grad_norm": 4.6941633224487305,
+      "learning_rate": 2.015547399511624e-05,
+      "loss": 0.1023,
+      "step": 1720
+    },
+    {
+      "epoch": 1.7740948495665476,
+      "grad_norm": 0.09467876702547073,
+      "learning_rate": 1.960539508080485e-05,
+      "loss": 0.0874,
+      "step": 1740
+    },
+    {
+      "epoch": 1.784293727689954,
+      "eval_accuracy": 0.6882039122703023,
+      "eval_decision_nll": 2.1201932841528786,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 1750,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.1201932841528786,
+      "eval_macro_f1_41": 0.6429354759970077,
+      "eval_no_relation_logit_bias": -3.970000000000031,
+      "eval_positive_fn": 329,
+      "eval_positive_fp": 352,
+      "eval_positive_micro_f1": 0.6124075128059191,
+      "eval_positive_micro_precision": 0.604494382022472,
+      "eval_positive_micro_recall": 0.6205305651672434,
+      "eval_positive_tp": 538,
+      "eval_raw_accuracy": 0.6899822169531713,
+      "eval_raw_macro_f1_41": 0.6036357001053071,
+      "eval_raw_positive_micro_f1": 0.5940353012781497,
+      "eval_raw_positive_micro_precision": 0.6288659793814433,
+      "eval_raw_positive_micro_recall": 0.5628604382929643,
+      "eval_raw_weighted_f1_41": 0.6622892735295407,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6899822169531713,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6036357001053071,
+      "eval_unconstrained_positive_micro_f1": 0.5940353012781497,
+      "eval_unconstrained_positive_micro_precision": 0.6288659793814433,
+      "eval_unconstrained_positive_micro_recall": 0.5628604382929643,
+      "eval_unconstrained_weighted_f1_41": 0.6622892735295407,
+      "eval_weighted_f1_41": 0.6686562598836959,
+      "step": 1750
+    },
+    {
+      "epoch": 1.7944926058133606,
+      "grad_norm": 9.511631965637207,
+      "learning_rate": 1.9058042239402378e-05,
+      "loss": 0.1082,
+      "step": 1760
+    },
+    {
+      "epoch": 1.8148903620601735,
+      "grad_norm": 26.438539505004883,
+      "learning_rate": 1.8513692066465866e-05,
+      "loss": 0.1342,
+      "step": 1780
+    },
+    {
+      "epoch": 1.8352881183069862,
+      "grad_norm": 0.6027206778526306,
+      "learning_rate": 1.7972619640204296e-05,
+      "loss": 0.1355,
+      "step": 1800
+    },
+    {
+      "epoch": 1.855685874553799,
+      "grad_norm": 35.500850677490234,
+      "learning_rate": 1.7435098382472486e-05,
+      "loss": 0.1114,
+      "step": 1820
+    },
+    {
+      "epoch": 1.876083630800612,
+      "grad_norm": 0.8295138478279114,
+      "learning_rate": 1.690139992060195e-05,
+      "loss": 0.1182,
+      "step": 1840
+    },
+    {
+      "epoch": 1.8964813870474249,
+      "grad_norm": 0.09546128660440445,
+      "learning_rate": 1.637179395013853e-05,
+      "loss": 0.1547,
+      "step": 1860
+    },
+    {
+      "epoch": 1.9168791432942376,
+      "grad_norm": 4.773397445678711,
+      "learning_rate": 1.584654809855639e-05,
+      "loss": 0.0366,
+      "step": 1880
+    },
+    {
+      "epoch": 1.9372768995410505,
+      "grad_norm": 0.06276191771030426,
+      "learning_rate": 1.5325927790016858e-05,
+      "loss": 0.1077,
+      "step": 1900
+    },
+    {
+      "epoch": 1.9576746557878635,
+      "grad_norm": 0.0431281216442585,
+      "learning_rate": 1.4810196111240898e-05,
+      "loss": 0.0775,
+      "step": 1920
+    },
+    {
+      "epoch": 1.9780724120346762,
+      "grad_norm": 0.09372670948505402,
+      "learning_rate": 1.429961367856256e-05,
+      "loss": 0.1194,
+      "step": 1940
+    },
+    {
+      "epoch": 1.998470168281489,
+      "grad_norm": 0.04784310609102249,
+      "learning_rate": 1.3794438506230967e-05,
+      "loss": 0.0708,
+      "step": 1960
+    },
+    {
+      "epoch": 2.0183579806221315,
+      "grad_norm": 0.1206798180937767,
+      "learning_rate": 1.3294925876027112e-05,
+      "loss": 0.0389,
+      "step": 1980
+    },
+    {
+      "epoch": 2.0387557368689446,
+      "grad_norm": 0.05475665628910065,
+      "learning_rate": 1.2801328208261526e-05,
+      "loss": 0.0852,
+      "step": 2000
+    },
+    {
+      "epoch": 2.0387557368689446,
+      "eval_accuracy": 0.7012448132780082,
+      "eval_decision_nll": 2.2389320018521834,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 2000,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.2389320018521834,
+      "eval_macro_f1_41": 0.6462471485102084,
+      "eval_no_relation_logit_bias": -4.370000000000029,
+      "eval_positive_fn": 321,
+      "eval_positive_fp": 346,
+      "eval_positive_micro_f1": 0.6208072768618533,
+      "eval_positive_micro_precision": 0.6121076233183856,
+      "eval_positive_micro_recall": 0.629757785467128,
+      "eval_positive_tp": 546,
+      "eval_raw_accuracy": 0.6941315945465323,
+      "eval_raw_macro_f1_41": 0.6097002065413685,
+      "eval_raw_positive_micro_f1": 0.5943627450980392,
+      "eval_raw_positive_micro_precision": 0.6339869281045751,
+      "eval_raw_positive_micro_recall": 0.5594002306805075,
+      "eval_raw_weighted_f1_41": 0.6648000936336502,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6941315945465323,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6097002065413685,
+      "eval_unconstrained_positive_micro_f1": 0.5943627450980392,
+      "eval_unconstrained_positive_micro_precision": 0.6339869281045751,
+      "eval_unconstrained_positive_micro_recall": 0.5594002306805075,
+      "eval_unconstrained_weighted_f1_41": 0.6648000936336502,
+      "eval_weighted_f1_41": 0.6814423147512597,
+      "step": 2000
+    },
+    {
+      "epoch": 2.0591534931157573,
+      "grad_norm": 27.940425872802734,
+      "learning_rate": 1.2313894934217907e-05,
+      "loss": 0.1449,
+      "step": 2020
+    },
+    {
+      "epoch": 2.07955124936257,
+      "grad_norm": 0.05362895876169205,
+      "learning_rate": 1.183287237010719e-05,
+      "loss": 0.08,
+      "step": 2040
+    },
+    {
+      "epoch": 2.099949005609383,
+      "grad_norm": 0.28913530707359314,
+      "learning_rate": 1.1358503592595837e-05,
+      "loss": 0.0454,
+      "step": 2060
+    },
+    {
+      "epoch": 2.120346761856196,
+      "grad_norm": 0.01951238326728344,
+      "learning_rate": 1.0891028315971105e-05,
+      "loss": 0.097,
+      "step": 2080
+    },
+    {
+      "epoch": 2.1407445181030087,
+      "grad_norm": 0.04203539341688156,
+      "learning_rate": 1.0430682771005456e-05,
+      "loss": 0.0559,
+      "step": 2100
+    },
+    {
+      "epoch": 2.1611422743498214,
+      "grad_norm": 0.2714548110961914,
+      "learning_rate": 9.977699585581324e-06,
+      "loss": 0.0539,
+      "step": 2120
+    },
+    {
+      "epoch": 2.1815400305966346,
+      "grad_norm": 22.596508026123047,
+      "learning_rate": 9.532307667136493e-06,
+      "loss": 0.12,
+      "step": 2140
+    },
+    {
+      "epoch": 2.2019377868434473,
+      "grad_norm": 24.418514251708984,
+      "learning_rate": 9.094732086989608e-06,
+      "loss": 0.1384,
+      "step": 2160
+    },
+    {
+      "epoch": 2.22233554309026,
+      "grad_norm": 9.244231224060059,
+      "learning_rate": 8.665193966604157e-06,
+      "loss": 0.0604,
+      "step": 2180
+    },
+    {
+      "epoch": 2.2427332993370728,
+      "grad_norm": 0.3614042401313782,
+      "learning_rate": 8.243910365848448e-06,
+      "loss": 0.0793,
+      "step": 2200
+    },
+    {
+      "epoch": 2.263131055583886,
+      "grad_norm": 2.0732710361480713,
+      "learning_rate": 7.831094173308056e-06,
+      "loss": 0.1103,
+      "step": 2220
+    },
+    {
+      "epoch": 2.2835288118306987,
+      "grad_norm": 0.01613030396401882,
+      "learning_rate": 7.4269539987061625e-06,
+      "loss": 0.0436,
+      "step": 2240
+    },
+    {
+      "epoch": 2.293727689954105,
+      "eval_accuracy": 0.6935388263189093,
+      "eval_decision_nll": 2.150244537584161,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 2250,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.150244537584161,
+      "eval_macro_f1_41": 0.6849303140069825,
+      "eval_no_relation_logit_bias": -5.940000000000023,
+      "eval_positive_fn": 273,
+      "eval_positive_fp": 426,
+      "eval_positive_micro_f1": 0.629570747217806,
+      "eval_positive_micro_precision": 0.5823529411764706,
+      "eval_positive_micro_recall": 0.6851211072664359,
+      "eval_positive_tp": 594,
+      "eval_raw_accuracy": 0.6905749851807943,
+      "eval_raw_macro_f1_41": 0.6524529256483073,
+      "eval_raw_positive_micro_f1": 0.5938069216757741,
+      "eval_raw_positive_micro_precision": 0.6269230769230769,
+      "eval_raw_positive_micro_recall": 0.5640138408304498,
+      "eval_raw_weighted_f1_41": 0.660259626008938,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6905749851807943,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6524529256483073,
+      "eval_unconstrained_positive_micro_f1": 0.5938069216757741,
+      "eval_unconstrained_positive_micro_precision": 0.6269230769230769,
+      "eval_unconstrained_positive_micro_recall": 0.5640138408304498,
+      "eval_unconstrained_weighted_f1_41": 0.660259626008938,
+      "eval_weighted_f1_41": 0.6790266875399904,
+      "step": 2250
+    },
+    {
+      "epoch": 2.3039265680775114,
+      "grad_norm": 0.03432675823569298,
+      "learning_rate": 7.031694067486136e-06,
+      "loss": 0.0656,
+      "step": 2260
+    },
+    {
+      "epoch": 2.3243243243243246,
+      "grad_norm": 0.025095023214817047,
+      "learning_rate": 6.645514117609616e-06,
+      "loss": 0.0861,
+      "step": 2280
+    },
+    {
+      "epoch": 2.3447220805711373,
+      "grad_norm": 16.071243286132812,
+      "learning_rate": 6.268609298622327e-06,
+      "loss": 0.0985,
+      "step": 2300
+    },
+    {
+      "epoch": 2.36511983681795,
+      "grad_norm": 0.029141830280423164,
+      "learning_rate": 5.901170073038523e-06,
+      "loss": 0.0491,
+      "step": 2320
+    },
+    {
+      "epoch": 2.3855175930647627,
+      "grad_norm": 35.70805358886719,
+      "learning_rate": 5.543382120093946e-06,
+      "loss": 0.0867,
+      "step": 2340
+    },
+    {
+      "epoch": 2.405915349311576,
+      "grad_norm": 0.045629482716321945,
+      "learning_rate": 5.195426241915963e-06,
+      "loss": 0.021,
+      "step": 2360
+    },
+    {
+      "epoch": 2.4263131055583886,
+      "grad_norm": 0.0359937809407711,
+      "learning_rate": 4.857478272158217e-06,
+      "loss": 0.0821,
+      "step": 2380
+    },
+    {
+      "epoch": 2.4467108618052014,
+      "grad_norm": 7.41368293762207,
+      "learning_rate": 4.529708987146114e-06,
+      "loss": 0.1018,
+      "step": 2400
+    },
+    {
+      "epoch": 2.467108618052014,
+      "grad_norm": 0.17788079380989075,
+      "learning_rate": 4.212284019577792e-06,
+      "loss": 0.0669,
+      "step": 2420
+    },
+    {
+      "epoch": 2.4875063742988273,
+      "grad_norm": 0.026721414178609848,
+      "learning_rate": 3.905363774824492e-06,
+      "loss": 0.0176,
+      "step": 2440
+    },
+    {
+      "epoch": 2.50790413054564,
+      "grad_norm": 0.28086280822753906,
+      "learning_rate": 3.609103349872342e-06,
+      "loss": 0.0158,
+      "step": 2460
+    },
+    {
+      "epoch": 2.5283018867924527,
+      "grad_norm": 0.16154006123542786,
+      "learning_rate": 3.323652454946774e-06,
+      "loss": 0.0125,
+      "step": 2480
+    },
+    {
+      "epoch": 2.5486996430392654,
+      "grad_norm": 27.4371337890625,
+      "learning_rate": 3.0491553378589084e-06,
+      "loss": 0.0363,
+      "step": 2500
+    },
+    {
+      "epoch": 2.5486996430392654,
+      "eval_accuracy": 0.6923532898636633,
+      "eval_decision_nll": 2.3230580910466165,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 2500,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.3230580910466165,
+      "eval_macro_f1_41": 0.6831361552053256,
+      "eval_no_relation_logit_bias": -5.8600000000000225,
+      "eval_positive_fn": 288,
+      "eval_positive_fp": 402,
+      "eval_positive_micro_f1": 0.6266233766233766,
+      "eval_positive_micro_precision": 0.5902140672782875,
+      "eval_positive_micro_recall": 0.6678200692041523,
+      "eval_positive_tp": 579,
+      "eval_raw_accuracy": 0.6923532898636633,
+      "eval_raw_macro_f1_41": 0.6521268255671399,
+      "eval_raw_positive_micro_f1": 0.599880023995201,
+      "eval_raw_positive_micro_precision": 0.625,
+      "eval_raw_positive_micro_recall": 0.5767012687427913,
+      "eval_raw_weighted_f1_41": 0.663769934218916,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6923532898636633,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6521268255671399,
+      "eval_unconstrained_positive_micro_f1": 0.599880023995201,
+      "eval_unconstrained_positive_micro_precision": 0.625,
+      "eval_unconstrained_positive_micro_recall": 0.5767012687427913,
+      "eval_unconstrained_weighted_f1_41": 0.663769934218916,
+      "eval_weighted_f1_41": 0.6767481730025282,
+      "step": 2500
+    },
+    {
+      "epoch": 2.5690973992860786,
+      "grad_norm": 7.680412769317627,
+      "learning_rate": 2.7857507111123755e-06,
+      "loss": 0.0525,
+      "step": 2520
+    },
+    {
+      "epoch": 2.5894951555328913,
+      "grad_norm": 17.607656478881836,
+      "learning_rate": 2.533571681807295e-06,
+      "loss": 0.0413,
+      "step": 2540
+    },
+    {
+      "epoch": 2.6098929117797045,
+      "grad_norm": 0.10886464267969131,
+      "learning_rate": 2.2927456843768206e-06,
+      "loss": 0.104,
+      "step": 2560
+    },
+    {
+      "epoch": 2.6302906680265172,
+      "grad_norm": 0.00933705735951662,
+      "learning_rate": 2.0633944161903144e-06,
+      "loss": 0.0434,
+      "step": 2580
+    },
+    {
+      "epoch": 2.65068842427333,
+      "grad_norm": 0.08042579889297485,
+      "learning_rate": 1.8456337760555915e-06,
+      "loss": 0.0449,
+      "step": 2600
+    },
+    {
+      "epoch": 2.6710861805201427,
+      "grad_norm": 0.31250545382499695,
+      "learning_rate": 1.639573805651437e-06,
+      "loss": 0.0805,
+      "step": 2620
+    },
+    {
+      "epoch": 2.6914839367669554,
+      "grad_norm": 0.017284924164414406,
+      "learning_rate": 1.4453186339199037e-06,
+      "loss": 0.065,
+      "step": 2640
+    },
+    {
+      "epoch": 2.7118816930137686,
+      "grad_norm": 0.13521786034107208,
+      "learning_rate": 1.2629664244464463e-06,
+      "loss": 0.0517,
+      "step": 2660
+    },
+    {
+      "epoch": 2.7322794492605813,
+      "grad_norm": 55.31725311279297,
+      "learning_rate": 1.0926093258546655e-06,
+      "loss": 0.0267,
+      "step": 2680
+    },
+    {
+      "epoch": 2.752677205507394,
+      "grad_norm": 0.017036089673638344,
+      "learning_rate": 9.343334252405133e-07,
+      "loss": 0.0397,
+      "step": 2700
+    },
+    {
+      "epoch": 2.773074961754207,
+      "grad_norm": 0.18307919800281525,
+      "learning_rate": 7.88218704669641e-07,
+      "loss": 0.0858,
+      "step": 2720
+    },
+    {
+      "epoch": 2.79347271800102,
+      "grad_norm": 12.35401725769043,
+      "learning_rate": 6.543390007598016e-07,
+      "loss": 0.1114,
+      "step": 2740
+    },
+    {
+      "epoch": 2.8036715961244263,
+      "eval_accuracy": 0.6970954356846473,
+      "eval_decision_nll": 2.2821931980316217,
+      "eval_evaluated_rows": 1687,
+      "eval_global_step": 2750,
+      "eval_invalid_output_rate": 0.0,
+      "eval_invalid_outputs": 0,
+      "eval_loss": 2.2821931980316217,
+      "eval_macro_f1_41": 0.6888990793956042,
+      "eval_no_relation_logit_bias": -6.6600000000000215,
+      "eval_positive_fn": 273,
+      "eval_positive_fp": 415,
+      "eval_positive_micro_f1": 0.6332622601279317,
+      "eval_positive_micro_precision": 0.5887016848364718,
+      "eval_positive_micro_recall": 0.6851211072664359,
+      "eval_positive_tp": 594,
+      "eval_raw_accuracy": 0.6917605216360403,
+      "eval_raw_macro_f1_41": 0.6508407746399698,
+      "eval_raw_positive_micro_f1": 0.5953827460510328,
+      "eval_raw_positive_micro_precision": 0.6290115532734275,
+      "eval_raw_positive_micro_recall": 0.5651672433679354,
+      "eval_raw_weighted_f1_41": 0.6620183629552824,
+      "eval_rows": 1687,
+      "eval_unconstrained_accuracy": 0.6917605216360403,
+      "eval_unconstrained_invalid_output_rate": 0.0,
+      "eval_unconstrained_invalid_outputs": 0,
+      "eval_unconstrained_macro_f1_41": 0.6508407746399698,
+      "eval_unconstrained_positive_micro_f1": 0.5953827460510328,
+      "eval_unconstrained_positive_micro_precision": 0.6290115532734275,
+      "eval_unconstrained_positive_micro_recall": 0.5651672433679354,
+      "eval_unconstrained_weighted_f1_41": 0.6620183629552824,
+      "eval_weighted_f1_41": 0.6828491038015017,
+      "step": 2750
+    },
+    {
+      "epoch": 2.8138704742478327,
+      "grad_norm": 0.687584638595581,
+      "learning_rate": 5.327619673687528e-07,
+      "loss": 0.0261,
+      "step": 2760
+    },
+    {
+      "epoch": 2.8342682304946454,
+      "grad_norm": 0.028698189184069633,
+      "learning_rate": 4.235490414065263e-07,
+      "loss": 0.0681,
+      "step": 2780
+    },
+    {
+      "epoch": 2.8546659867414585,
+      "grad_norm": 0.0782444030046463,
+      "learning_rate": 3.2675541178931456e-07,
+      "loss": 0.0228,
+      "step": 2800
+    },
+    {
+      "epoch": 2.8750637429882713,
+      "grad_norm": 0.04141950607299805,
+      "learning_rate": 2.424299915506828e-07,
+      "loss": 0.0459,
+      "step": 2820
+    },
+    {
+      "epoch": 2.895461499235084,
+      "grad_norm": 24.31253433227539,
+      "learning_rate": 1.7061539312417108e-07,
+      "loss": 0.0676,
+      "step": 2840
+    },
+    {
+      "epoch": 2.915859255481897,
+      "grad_norm": 0.10220429301261902,
+      "learning_rate": 1.1134790680984996e-07,
+      "loss": 0.0602,
+      "step": 2860
+    },
+    {
+      "epoch": 2.93625701172871,
+      "grad_norm": 4.274705410003662,
+      "learning_rate": 6.465748243559556e-08,
+      "loss": 0.0713,
+      "step": 2880
+    },
+    {
+      "epoch": 2.9566547679755226,
+      "grad_norm": 0.014307630248367786,
+      "learning_rate": 3.056771422247362e-08,
+      "loss": 0.0698,
+      "step": 2900
+    },
+    {
+      "epoch": 2.9770525242223354,
+      "grad_norm": 0.08406448364257812,
+      "learning_rate": 9.095828861771516e-09,
+      "loss": 0.137,
+      "step": 2920
+    },
+    {
+      "epoch": 2.9974502804691485,
+      "grad_norm": 0.06355658918619156,
+      "learning_rate": 2.526768098060073e-10,
+      "loss": 0.0307,
+      "step": 2940
+    }
+  ],
+  "logging_steps": 20,
+  "max_steps": 2943,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 250,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.100502627428434e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

model.py

@@ -24,6 +24,7 @@ from pathlib import Path
 from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Tuple
 
 os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0"
 
 import bitsandbytes as bnb
 import numpy as np

predict.py

@@ -10,6 +10,7 @@ from pathlib import Path
 from typing import Any, Dict, Iterable, List, Mapping
 
 os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0"
 
 import torch
 from dotenv import load_dotenv

release/release_manifest.json

@@ -8,6 +8,25 @@
     "finite_tensors": 448,
     "nonzero_tensors": 448
   },
+  "latest_checkpoint": {
+    "global_step": 2943,
+    "best_checkpoint_step": 2750,
+    "adapter": {
+      "tensor_count": 448,
+      "finite_tensors": 448,
+      "nonzero_tensors": 448
+    },
+    "optimizer": {
+      "tensor_count": 2688,
+      "finite_floating_tensor_count": 1792
+    },
+    "scheduler": {
+      "tensor_count": 0,
+      "finite_floating_tensor_count": 0
+    },
+    "rng_state_bytes": 14244,
+    "training_args_bytes": 5496
+  },
   "files": {
     "adapter_config.json": {
       "bytes": 975,
@@ -61,6 +80,62 @@
       "bytes": 504,
       "sha256": "d00476c1b9072f1b30b5fed98fbf484f2ca53f4fcef5baa96918343f39f786c8"
     },
+    "latest_checkpoint/README.md": {
+      "bytes": 5232,
+      "sha256": "32c8a003763070b74cdb20f608f0e5a726570b11d56469cd4b80b2753bd274f4"
+    },
+    "latest_checkpoint/adapter_config.json": {
+      "bytes": 975,
+      "sha256": "68656a0760952d4504c6f6cdfb03487622327b6391315289790192bfa6951721"
+    },
+    "latest_checkpoint/adapter_model.safetensors": {
+      "bytes": 159967880,
+      "sha256": "4390a7f0c09141c9e6c9dc0e82164e7d5a6eb2e07eca3f44c7f76467485bae71"
+    },
+    "latest_checkpoint/chat_template.jinja": {
+      "bytes": 816,
+      "sha256": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f"
+    },
+    "latest_checkpoint/checkpoint_manifest.json": {
+      "bytes": 2447,
+      "sha256": "5094be1262aa9ac08dd2f3671151ae7166ac699805a734967d2194dc095a554f"
+    },
+    "latest_checkpoint/optimizer.pt": {
+      "bytes": 81735892,
+      "sha256": "58da25e4579a062fa12ad066e34b666619a4609957fbcc9d246212ca4a431fad"
+    },
+    "latest_checkpoint/rng_state.pth": {
+      "bytes": 14244,
+      "sha256": "e1364bce33b92ffd2389bb5e6f0f8bb88878c2a769c50a7e2403c9cd9f3c1fd9"
+    },
+    "latest_checkpoint/scheduler.pt": {
+      "bytes": 1064,
+      "sha256": "bd16c6406d87f70b6c298a5e4fb685236ce307ae801894a350532f2e95d3a1fe"
+    },
+    "latest_checkpoint/special_tokens_map.json": {
+      "bytes": 551,
+      "sha256": "82d96d7a9e6ced037f12394b7ea6a5b02e6ca87e0d11edaa8d60d9be857ce7db"
+    },
+    "latest_checkpoint/tokenizer.json": {
+      "bytes": 8508152,
+      "sha256": "a4ec65a3ef5dee870b4024cf17d9315aff58dc1555e05766f1c18ebfe8405dba"
+    },
+    "latest_checkpoint/tokenizer.model": {
+      "bytes": 1228714,
+      "sha256": "feff3deb3537c4a47b77585053ad2aa00484da7907b188e0111cbed1f4592e67"
+    },
+    "latest_checkpoint/tokenizer_config.json": {
+      "bytes": 4402,
+      "sha256": "92a93150cc2a9e821e6fc785bbb0aea3cec5d92df529974131efc1c857b52359"
+    },
+    "latest_checkpoint/trainer_state.json": {
+      "bytes": 43812,
+      "sha256": "95489ed7a630b99057d444bd7469f2eeb2f9e5f618b077f8f1873f9a1d118251"
+    },
+    "latest_checkpoint/training_args.bin": {
+      "bytes": 5496,
+      "sha256": "213db7d4f648a556f875d6706bbb4d0e9ef6f78fecf9b503048cedff8a477c00"
+    },
     "release/best_validation_metrics.json": {
       "bytes": 1369,
       "sha256": "a2d588206ad639c5b02cdee82e3698ac96919b11e8ea1783df5fd540b9e1ea54"
@@ -96,10 +171,7 @@
   },
   "excluded": [
     "base model weights",
-    "optimizer state",
-    "scheduler state",
-    "RNG state",
-    "numbered checkpoints",
+    "numbered checkpoints other than the staged latest step-2943 copy",
     "training logs",
     "prediction dumps",
     "caches"

setup_env.sh

@@ -54,6 +54,7 @@ python -m py_compile \
   tools/build_examples_appendix.py \
   tools/plot_losses.py \
   tools/publish_huggingface.py \
+  tools/restore_latest_checkpoint.py \
   tools/stage_release.py \
   tools/verify_release.py
 python re_sft_common.py

tools/publish_huggingface.py

@@ -15,6 +15,8 @@ import tempfile
 from datetime import datetime, timezone
 from pathlib import Path
 
+os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "0"
+
 from dotenv import load_dotenv
 from huggingface_hub import HfApi, snapshot_download
 
@@ -100,6 +102,7 @@ def main() -> None:
         "local_release": {
             "manifest": local["manifest"],
             "adapter": local["adapter"],
+            "latest_checkpoint": local["latest_checkpoint"],
             "worked_examples": local["worked_examples"],
         },
         "fresh_snapshot_verification": {
@@ -107,6 +110,7 @@ def main() -> None:
             "revision": remote["revision"],
             "manifest": remote["manifest"],
             "adapter": remote["adapter"],
+            "latest_checkpoint": remote["latest_checkpoint"],
             "worked_examples": remote["worked_examples"],
             "remote_clean_mode": remote["remote_clean_mode"],
         },
@@ -134,6 +138,10 @@ def main() -> None:
                 "final_revision": final_revision,
                 "fresh_final_download_verified": final_remote["status"] == "passed",
                 "adapter_tensors": final_remote["adapter"]["tensors"],
+                "latest_checkpoint_global_step": final_remote["latest_checkpoint"][
+                    "global_step"
+                ],
+                "latest_checkpoint_files": final_remote["latest_checkpoint"]["files"],
                 "worked_example_classes": final_remote["worked_examples"]["classes"],
                 "token_values_printed": False,
             },

tools/restore_latest_checkpoint.py

@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+"""Restore the published step-2943 Trainer checkpoint into a run directory."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shutil
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+from verify_release import verify_latest_checkpoint
+
+
+ROOT = Path(__file__).resolve().parents[1]
+load_dotenv(ROOT / ".env")
+
+
+def resolve(value: str) -> Path:
+    path = Path(value)
+    return path if path.is_absolute() else ROOT / path
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--output-dir",
+        default=os.environ.get("CHOICE_OUTPUT_DIR", "runs/DRU-RE-Yehia"),
+        help="Trainer run root that will receive checkpoints/checkpoint-2943.",
+    )
+    parser.add_argument("--overwrite", action="store_true")
+    args = parser.parse_args()
+
+    audit = verify_latest_checkpoint(ROOT)
+    source = ROOT / "latest_checkpoint"
+    output = resolve(args.output_dir)
+    destination = output / "checkpoints/checkpoint-2943"
+    temporary = output / "checkpoints/.checkpoint-2943.restoring"
+    if destination.exists() and not args.overwrite:
+        raise RuntimeError(
+            f"Destination already exists: {destination}. Use --overwrite only intentionally."
+        )
+    if temporary.exists():
+        shutil.rmtree(temporary)
+    temporary.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copytree(source, temporary)
+    if destination.exists():
+        shutil.rmtree(destination)
+    os.replace(temporary, destination)
+    print(
+        json.dumps(
+            {
+                "status": "passed",
+                "source": str(source),
+                "destination": str(destination),
+                "global_step": audit["global_step"],
+                "checkpoint_files": audit["files"],
+                "full_base_weights_included": False,
+            },
+            ensure_ascii=False,
+            indent=2,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

tools/stage_release.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Stage only the completed best adapter and durable evidence for Hub upload."""
+"""Stage the completed best adapter, latest resumable state, and evidence."""
 
 from __future__ import annotations
 
@@ -29,6 +29,14 @@ ADAPTER_FILES = {
     "tokenizer.model",
     "tokenizer_config.json",
 }
+LATEST_CHECKPOINT_FILES = ADAPTER_FILES | {
+    "README.md",
+    "optimizer.pt",
+    "rng_state.pth",
+    "scheduler.pt",
+    "trainer_state.json",
+    "training_args.bin",
+}
 EVIDENCE_FILES = {
     "all_linear_eligible_modules.json",
     "decision_token_inventory.json",
@@ -94,6 +102,115 @@ def validate_adapter_tensors(path: Path) -> Dict[str, Any]:
     return {"tensor_count": tensor_count, "finite_tensors": tensor_count, "nonzero_tensors": nonzero_count}
 
 
+def validate_tensor_tree(
+    value: Any, label: str, require_tensor: bool = True
+) -> Dict[str, int]:
+    tensor_count = 0
+    floating_tensor_count = 0
+    stack = [value]
+    while stack:
+        current = stack.pop()
+        if isinstance(current, dict):
+            stack.extend(current.values())
+        elif isinstance(current, (list, tuple)):
+            stack.extend(current)
+        elif isinstance(current, torch.Tensor):
+            tensor_count += 1
+            if current.is_floating_point() or current.is_complex():
+                floating_tensor_count += 1
+                if not torch.isfinite(current).all().item():
+                    raise RuntimeError(f"Refusing release: non-finite tensor in {label}")
+    if require_tensor and tensor_count == 0:
+        raise RuntimeError(f"Refusing release: no tensors found in {label}")
+    return {
+        "tensor_count": tensor_count,
+        "finite_floating_tensor_count": floating_tensor_count,
+    }
+
+
+def validate_latest_checkpoint(run_dir: Path) -> Dict[str, Any]:
+    latest = run_dir / "latest_checkpoint"
+    numbered = run_dir / "checkpoints" / "checkpoint-2943"
+    for directory in (latest, numbered):
+        if not directory.is_dir():
+            raise FileNotFoundError(directory)
+        for name in sorted(LATEST_CHECKPOINT_FILES):
+            if not (directory / name).is_file():
+                raise FileNotFoundError(directory / name)
+    for name in sorted(LATEST_CHECKPOINT_FILES):
+        if sha256(latest / name) != sha256(numbered / name):
+            raise RuntimeError(f"Refusing release: latest checkpoint diverges for {name}")
+
+    state = read_json(latest / "trainer_state.json")
+    if state.get("global_step") != 2943:
+        raise RuntimeError("Refusing release: latest checkpoint is not global step 2943")
+    if not str(state.get("best_model_checkpoint", "")).endswith("checkpoint-2750"):
+        raise RuntimeError("Refusing release: latest checkpoint has wrong best-checkpoint provenance")
+    adapter = validate_adapter_tensors(latest / "adapter_model.safetensors")
+    optimizer = torch.load(latest / "optimizer.pt", map_location="cpu", weights_only=True)
+    scheduler = torch.load(latest / "scheduler.pt", map_location="cpu", weights_only=True)
+    if scheduler.get("last_epoch") != 2943 or scheduler.get("_step_count") != 2944:
+        raise RuntimeError("Refusing release: latest scheduler state is incomplete")
+    return {
+        "global_step": 2943,
+        "best_checkpoint_step": 2750,
+        "adapter": adapter,
+        "optimizer": validate_tensor_tree(optimizer, "latest_checkpoint/optimizer.pt"),
+        "scheduler": validate_tensor_tree(
+            scheduler, "latest_checkpoint/scheduler.pt", require_tensor=False
+        ),
+        "rng_state_bytes": (latest / "rng_state.pth").stat().st_size,
+        "training_args_bytes": (latest / "training_args.bin").stat().st_size,
+    }
+
+
+def stage_latest_checkpoint(
+    source: Path,
+    repo_dir: Path,
+    base_model_id: str,
+    base_model_revision: str,
+    audit: Dict[str, Any],
+) -> list[Path]:
+    temporary = repo_dir / ".latest_checkpoint.staging"
+    destination = repo_dir / "latest_checkpoint"
+    if temporary.exists():
+        shutil.rmtree(temporary)
+    temporary.mkdir(parents=True)
+    for name in sorted(LATEST_CHECKPOINT_FILES):
+        shutil.copy2(source / name, temporary / name)
+
+    adapter_config = read_json(temporary / "adapter_config.json")
+    adapter_config["base_model_name_or_path"] = base_model_id
+    adapter_config["revision"] = base_model_revision
+    write_json(temporary / "adapter_config.json", adapter_config)
+    manifest_files = {
+        name: {
+            "bytes": (temporary / name).stat().st_size,
+            "sha256": sha256(temporary / name),
+        }
+        for name in sorted(LATEST_CHECKPOINT_FILES)
+    }
+    write_json(
+        temporary / "checkpoint_manifest.json",
+        {
+            "status": "complete",
+            "checkpoint_type": "latest_complete_resumable_trainer_checkpoint",
+            "global_step": 2943,
+            "best_checkpoint_step": 2750,
+            "audit": audit,
+            "files": manifest_files,
+            "full_base_weights_included": False,
+        },
+    )
+    if destination.exists():
+        shutil.rmtree(destination)
+    os.replace(temporary, destination)
+    return [
+        destination / name
+        for name in sorted(LATEST_CHECKPOINT_FILES | {"checkpoint_manifest.json"})
+    ]
+
+
 def metric_step(path: Path) -> int:
     match = re.search(r"step_(\d+)\.json$", path.name)
     if not match:
@@ -205,6 +322,7 @@ def main() -> None:
                 f"Refusing release: LoRA evidence {key}={lora.get(key)!r}, expected {expected!r}"
             )
     tensor_audit = validate_adapter_tensors(source / "adapter_model.safetensors")
+    latest_checkpoint_audit = validate_latest_checkpoint(run_dir)
 
     # All correctness checks above complete before the repository is modified.
     copied = []
@@ -221,6 +339,16 @@ def main() -> None:
     staged_adapter_config["revision"] = base_model_revision
     write_json(repo_dir / "adapter_config.json", staged_adapter_config)
 
+    copied.extend(
+        stage_latest_checkpoint(
+            run_dir / "latest_checkpoint",
+            repo_dir,
+            base_model_id,
+            base_model_revision,
+            latest_checkpoint_audit,
+        )
+    )
+
     release_dir = repo_dir / "release"
     release_dir.mkdir(parents=True, exist_ok=True)
     for source_path, destination in (
@@ -289,6 +417,7 @@ def main() -> None:
         "selection_metric": "positive_micro_f1",
         "selection_metric_value": best_metrics["positive_micro_f1"],
         "adapter_tensor_audit": tensor_audit,
+        "latest_checkpoint": latest_checkpoint_audit,
         "files": {
             str(path.relative_to(repo_dir)): {
                 "bytes": path.stat().st_size,
@@ -298,10 +427,7 @@ def main() -> None:
         },
         "excluded": [
             "base model weights",
-            "optimizer state",
-            "scheduler state",
-            "RNG state",
-            "numbered checkpoints",
+            "numbered checkpoints other than the staged latest step-2943 copy",
             "training logs",
             "prediction dumps",
             "caches",

tools/verify_release.py

@@ -50,6 +50,7 @@ REQUIRED_SOURCE = {
     "tools/monitor_resources.sh",
     "tools/plot_losses.py",
     "tools/publish_huggingface.py",
+    "tools/restore_latest_checkpoint.py",
     "tools/stage_release.py",
     "tools/verify_release.py",
     "resources/wojood_ontology.json",
@@ -88,6 +89,25 @@ REQUIRED_RELEASE = {
     "release/validation_history.json",
     "release/release_manifest.json",
 }
+LATEST_CHECKPOINT_NAMES = {
+    "README.md",
+    "adapter_config.json",
+    "adapter_model.safetensors",
+    "chat_template.jinja",
+    "checkpoint_manifest.json",
+    "optimizer.pt",
+    "rng_state.pth",
+    "scheduler.pt",
+    "special_tokens_map.json",
+    "tokenizer.json",
+    "tokenizer.model",
+    "tokenizer_config.json",
+    "trainer_state.json",
+    "training_args.bin",
+}
+REQUIRED_LATEST_CHECKPOINT = {
+    f"latest_checkpoint/{name}" for name in LATEST_CHECKPOINT_NAMES
+}
 OPTIONAL_CLEAN_FILES = {
     "evidence/README.md",
     "evidence/train_class_counts.json",
@@ -195,8 +215,7 @@ def verify_lora_evidence(root: Path) -> Dict[str, Any]:
     return dict(expected)
 
 
-def verify_adapter(root: Path) -> Dict[str, Any]:
-    config = read_json(root / "adapter_config.json")
+def verify_adapter_config(config: Mapping[str, Any], label: str) -> None:
     locked: Mapping[str, Any] = {
         "r": 16,
         "lora_alpha": 32,
@@ -209,7 +228,9 @@ def verify_adapter(root: Path) -> Dict[str, Any]:
     }
     for key, value in locked.items():
         if config.get(key) != value:
-            raise RuntimeError(f"Adapter config {key}={config.get(key)!r}, expected {value!r}")
+            raise RuntimeError(
+                f"{label} adapter config {key}={config.get(key)!r}, expected {value!r}"
+            )
     expected_resolved_targets = {
         "q_proj",
         "k_proj",
@@ -220,27 +241,111 @@ def verify_adapter(root: Path) -> Dict[str, Any]:
         "down_proj",
     }
     if set(config.get("target_modules") or []) != expected_resolved_targets:
-        raise RuntimeError("Adapter config has an unexpected resolved target-module set")
+        raise RuntimeError(f"{label} has an unexpected resolved target-module set")
     if config.get("modules_to_save") is not None:
-        raise RuntimeError("Adapter unexpectedly saves additional non-LoRA modules")
+        raise RuntimeError(f"{label} unexpectedly saves additional non-LoRA modules")
+
+
+def verify_adapter_safetensors(path: Path, label: str) -> Dict[str, Any]:
 
     tensor_count = 0
     nonzero_count = 0
-    with safe_open(root / "adapter_model.safetensors", framework="pt", device="cpu") as handle:
+    with safe_open(path, framework="pt", device="cpu") as handle:
         for name in handle.keys():
             tensor = handle.get_tensor(name)
             tensor_count += 1
             if not torch.isfinite(tensor).all().item():
-                raise RuntimeError(f"Adapter contains non-finite tensor {name}")
+                raise RuntimeError(f"{label} contains non-finite tensor {name}")
             if torch.count_nonzero(tensor).item() > 0:
                 nonzero_count += 1
     if tensor_count != 448 or nonzero_count != 448:
         raise RuntimeError(
-            f"Expected 448 finite/nonzero adapter tensors, found {tensor_count}/{nonzero_count}"
+            f"Expected 448 finite/nonzero {label} tensors, found {tensor_count}/{nonzero_count}"
         )
     return {"tensors": tensor_count, "finite": tensor_count, "nonzero": nonzero_count}
 
 
+def verify_adapter(root: Path) -> Dict[str, Any]:
+    verify_adapter_config(read_json(root / "adapter_config.json"), "root adapter")
+    return verify_adapter_safetensors(
+        root / "adapter_model.safetensors", "root adapter"
+    )
+
+
+def verify_tensor_tree(
+    value: Any, label: str, require_tensor: bool = True
+) -> Dict[str, int]:
+    tensor_count = 0
+    floating_tensor_count = 0
+    stack = [value]
+    while stack:
+        current = stack.pop()
+        if isinstance(current, dict):
+            stack.extend(current.values())
+        elif isinstance(current, (list, tuple)):
+            stack.extend(current)
+        elif isinstance(current, torch.Tensor):
+            tensor_count += 1
+            if current.is_floating_point() or current.is_complex():
+                floating_tensor_count += 1
+                if not torch.isfinite(current).all().item():
+                    raise RuntimeError(f"Non-finite tensor in {label}")
+    if require_tensor and tensor_count == 0:
+        raise RuntimeError(f"No tensors found in {label}")
+    return {
+        "tensor_count": tensor_count,
+        "finite_floating_tensor_count": floating_tensor_count,
+    }
+
+
+def verify_latest_checkpoint(root: Path) -> Dict[str, Any]:
+    directory = root / "latest_checkpoint"
+    state = read_json(directory / "trainer_state.json")
+    if state.get("global_step") != 2943:
+        raise RuntimeError("Latest checkpoint is not global step 2943")
+    if not str(state.get("best_model_checkpoint", "")).endswith("checkpoint-2750"):
+        raise RuntimeError("Latest checkpoint has incorrect best-checkpoint provenance")
+    verify_adapter_config(
+        read_json(directory / "adapter_config.json"), "latest checkpoint"
+    )
+    adapter = verify_adapter_safetensors(
+        directory / "adapter_model.safetensors", "latest checkpoint adapter"
+    )
+    optimizer = torch.load(
+        directory / "optimizer.pt", map_location="cpu", weights_only=True
+    )
+    scheduler = torch.load(
+        directory / "scheduler.pt", map_location="cpu", weights_only=True
+    )
+    if scheduler.get("last_epoch") != 2943 or scheduler.get("_step_count") != 2944:
+        raise RuntimeError("Latest checkpoint scheduler state is incomplete")
+    optimizer_audit = verify_tensor_tree(optimizer, "latest checkpoint optimizer")
+    scheduler_audit = verify_tensor_tree(
+        scheduler, "latest checkpoint scheduler", require_tensor=False
+    )
+
+    manifest = read_json(directory / "checkpoint_manifest.json")
+    if manifest.get("status") != "complete" or manifest.get("global_step") != 2943:
+        raise RuntimeError("Latest checkpoint manifest is incomplete")
+    files = manifest.get("files") or {}
+    expected_names = LATEST_CHECKPOINT_NAMES - {"checkpoint_manifest.json"}
+    if set(files) != expected_names:
+        raise RuntimeError("Latest checkpoint manifest has the wrong file inventory")
+    for name, expected in files.items():
+        path = directory / name
+        if path.stat().st_size != expected.get("bytes") or sha256(path) != expected.get("sha256"):
+            raise RuntimeError(f"Latest checkpoint manifest mismatch for {name}")
+    return {
+        "status": "complete",
+        "global_step": 2943,
+        "best_checkpoint_step": 2750,
+        "adapter": adapter,
+        "optimizer": optimizer_audit,
+        "scheduler": scheduler_audit,
+        "files": len(files) + 1,
+    }
+
+
 def verify_inference_config(root: Path) -> Dict[str, Any]:
     config = read_json(root / "inference_config.json")
     expected = {
@@ -347,7 +452,9 @@ def verify_manifest(root: Path) -> Dict[str, Any]:
         "release/run_summary.json",
         "release/run_config.json",
         "release/best_validation_metrics.json",
+        "latest_checkpoint/checkpoint_manifest.json",
     }
+    required_manifested.update(REQUIRED_LATEST_CHECKPOINT)
     if not required_manifested.issubset(files):
         raise RuntimeError("Release manifest omits one or more critical release artifacts")
     return {"manifested_files": len(files), "best_checkpoint_step": manifest.get("best_checkpoint_step")}
@@ -389,14 +496,20 @@ def reject_accidental_payloads(root: Path, require_remote_clean: bool) -> None:
             # publisher explicitly ignores them. Do not inspect their payloads
             # as if they were part of the clean release tree.
             continue
-        if path.is_file() and path.name in forbidden_names:
+        relative_name = relative.as_posix()
+        if path.is_file() and path.name in forbidden_names and relative_name not in REQUIRED_LATEST_CHECKPOINT:
             raise RuntimeError(f"Training/base payload must not be published: {relative}")
         if path.is_file() and path.suffix == ".safetensors" and path.name != "adapter_model.safetensors":
             raise RuntimeError(f"Unexpected safetensors payload: {relative}")
     if require_remote_clean and (root / ".env").exists():
         raise RuntimeError("A remote release checkout must never contain .env")
     if require_remote_clean:
-        allowed = REQUIRED_SOURCE | REQUIRED_RELEASE | OPTIONAL_CLEAN_FILES
+        allowed = (
+            REQUIRED_SOURCE
+            | REQUIRED_RELEASE
+            | REQUIRED_LATEST_CHECKPOINT
+            | OPTIONAL_CLEAN_FILES
+        )
         unexpected = []
         for path in root.rglob("*"):
             if not path.is_file():
@@ -417,6 +530,7 @@ def verify_release(root: Path, require_remote_clean: bool = False) -> Dict[str,
     root = root.resolve()
     require_files(root, REQUIRED_SOURCE)
     require_files(root, REQUIRED_RELEASE)
+    require_files(root, REQUIRED_LATEST_CHECKPOINT)
     reject_accidental_payloads(root, require_remote_clean=require_remote_clean)
     inference = verify_inference_config(root)
     result = {
@@ -428,11 +542,12 @@ def verify_release(root: Path, require_remote_clean: bool = False) -> Dict[str,
         "base_model": verify_base_evidence(root),
         "lora": verify_lora_evidence(root),
         "adapter": verify_adapter(root),
+        "latest_checkpoint": verify_latest_checkpoint(root),
         "inference": inference,
         "final_run": verify_final_run(root, inference),
         "manifest": verify_manifest(root),
         "remote_clean_mode": require_remote_clean,
-        "gated_base_access_tested": False,
+        "gated_base_access_evidence_verified": True,
     }
     return result