Add files using upload-large-folder tool

deepseek-coder-6.7b/base/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: deepseek-ai/deepseek-coder-6.7b-instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:deepseek-ai/deepseek-coder-6.7b-instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

deepseek-coder-6.7b/base/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/base/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.957184,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.968208,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/base/codecarbon.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
+2026-03-17T21:46:21,codedp-deepseek-coder-6.7b-cpt-base,064d823c-8d0a-48ad-b578-92d6f569557c,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,1810.5507336058654,0.09709380205154217,5.362666742741399e-05,72.0230906963752,4629.388481318127,54.0,0.03488049743955748,2.3248590518302024,0.026150659639004155,2.3858902089087644,0.0,Sweden,SWE,östergötland county,,,Linux-6.8.0-94-generic-x86_64-with-glibc2.39,3.11.0,3.2.3,256,AMD EPYC 9554 64-Core Processor,8,8 x NVIDIA H200,16.1885,58.594,1511.49019241333,machine,3.485983379501395,91.87222991689751,5.226869806094248,78.96254361435317,N,1.0,0.0

deepseek-coder-6.7b/base/epochs/epoch_001/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: deepseek-ai/deepseek-coder-6.7b-instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:deepseek-ai/deepseek-coder-6.7b-instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

deepseek-coder-6.7b/base/epochs/epoch_001/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/base/epochs/epoch_001/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.923752,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.916456,
+    "empirical_epsilon": {
+      "0.05": 3.4791953936219215,
+      "0.01": 3.023197554051876
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 3.4791953936219215,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 3.023197554051876,
+        "num_guesses": 100,
+        "correct_guesses": 100,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/base/epochs/epoch_002/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/base/metrics.jsonl

@@ -0,0 +1,55 @@
+{"timestamp": 1773782272.0493639, "event": "train_step", "step": 10, "epoch": 1, "metrics": {"train/step_loss": 14.555815444273108, "train/step_real_loss": 14.770240783691406, "train/lr": 4.761904761904762e-05, "train/step_canary_loss": 11.125, "perf/step_duration_sec": 4.0931806759908795, "perf/samples_per_sec": 8.30649870879918, "perf/tokens_per_sec": 5843.6218416402235, "perf/logical_batch_size": 34.0, "perf/logical_token_count": 23919.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1168704032898}}
+{"timestamp": 1773782309.7872338, "event": "train_step", "step": 20, "epoch": 1, "metrics": {"train/step_loss": 12.852383613586426, "train/step_real_loss": 12.852383613586426, "train/lr": 9.523809523809524e-05, "perf/step_duration_sec": 3.573033411987126, "perf/samples_per_sec": 8.955975584399406, "perf/tokens_per_sec": 7755.035233378848, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27709.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1168704032898}}
+{"timestamp": 1773782347.6002963, "event": "train_step", "step": 30, "epoch": 1, "metrics": {"train/step_loss": 9.603991746902466, "train/step_real_loss": 9.603991746902466, "train/lr": 9.98706541985615e-05, "perf/step_duration_sec": 3.6900156908668578, "perf/samples_per_sec": 8.672049844992005, "perf/tokens_per_sec": 7146.58207694638, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26371.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1168704032898}}
+{"timestamp": 1773782386.0648503, "event": "train_step", "step": 40, "epoch": 1, "metrics": {"train/step_loss": 7.74934458732605, "train/step_real_loss": 7.74934458732605, "train/lr": 9.942439201095397e-05, "perf/step_duration_sec": 4.0733352322131395, "perf/samples_per_sec": 7.855970151176003, "perf/tokens_per_sec": 7048.523719075445, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 28711.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1168704032898}}
+{"timestamp": 1773782423.7351854, "event": "train_step", "step": 50, "epoch": 1, "metrics": {"train/step_loss": 7.176188588142395, "train/step_real_loss": 7.176188588142395, "train/lr": 9.866246608261724e-05, "perf/step_duration_sec": 3.6839785198681056, "perf/samples_per_sec": 8.686261287198187, "perf/tokens_per_sec": 7128.163168807011, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26260.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1168704032898}}
+{"timestamp": 1773782438.1861553, "event": "eval_step", "step": 50, "epoch": 1, "metrics": {"eval/loss": 6.95498745685274, "eval/duration_sec": 14.44864141382277}}
+{"timestamp": 1773782475.968484, "event": "train_step", "step": 60, "epoch": 1, "metrics": {"train/step_loss": 6.844001889228821, "train/step_real_loss": 6.844001889228821, "train/lr": 9.7589742682592e-05, "perf/step_duration_sec": 3.950888440012932, "perf/samples_per_sec": 8.099444083492081, "perf/tokens_per_sec": 7022.471128015244, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27745.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782514.3752964, "event": "train_step", "step": 70, "epoch": 1, "metrics": {"train/step_loss": 6.707658648490906, "train/step_real_loss": 6.707658648490906, "train/lr": 9.621307308142384e-05, "perf/step_duration_sec": 4.038279882632196, "perf/samples_per_sec": 7.924165964232782, "perf/tokens_per_sec": 6047.624411827906, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 24422.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782552.1368628, "event": "train_step", "step": 80, "epoch": 1, "metrics": {"train/step_loss": 7.028713240342982, "train/step_real_loss": 6.692617297172546, "train/lr": 9.454124979346391e-05, "train/step_canary_loss": 12.40625, "perf/step_duration_sec": 3.830359708983451, "perf/samples_per_sec": 8.876450929728307, "perf/tokens_per_sec": 6924.153869360416, "perf/logical_batch_size": 34.0, "perf/logical_token_count": 26522.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.167274475097656, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782590.0828424, "event": "train_step", "step": 90, "epoch": 1, "metrics": {"train/step_loss": 6.5469324588775635, "train/step_real_loss": 6.5469324588775635, "train/lr": 9.258495042083221e-05, "perf/step_duration_sec": 3.57225156808272, "perf/samples_per_sec": 8.957935741679826, "perf/tokens_per_sec": 7531.66441030924, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26905.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782627.521805, "event": "train_step", "step": 100, "epoch": 1, "metrics": {"train/step_loss": 6.427393674850464, "train/step_real_loss": 6.427393674850464, "train/lr": 9.035666945770107e-05, "perf/step_duration_sec": 3.5687404102645814, "perf/samples_per_sec": 8.966749138704534, "perf/tokens_per_sec": 8349.164291776258, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 29796.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782642.0074606, "event": "eval_step", "step": 100, "epoch": 1, "metrics": {"eval/loss": 6.393013093959201, "eval/duration_sec": 14.483735092915595}}
+{"timestamp": 1773782680.1220238, "event": "train_step", "step": 110, "epoch": 1, "metrics": {"train/step_loss": 6.492933461160371, "train/step_real_loss": 6.355993866920471, "train/lr": 8.787063849045118e-05, "train/step_canary_loss": 10.875, "perf/step_duration_sec": 3.834493800997734, "perf/samples_per_sec": 8.606090324468228, "perf/tokens_per_sec": 7034.044491865366, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 26972.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782717.354572, "event": "train_step", "step": 120, "epoch": 1, "metrics": {"train/step_loss": 6.159674048423767, "train/step_real_loss": 6.159674048423767, "train/lr": 8.5142735303366e-05, "perf/step_duration_sec": 3.8241200419142842, "perf/samples_per_sec": 8.367938152898931, "perf/tokens_per_sec": 7346.00370597615, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 28092.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.1198000907898}}
+{"timestamp": 1773782755.7000217, "event": "train_step", "step": 130, "epoch": 1, "metrics": {"train/step_loss": 6.002715587615967, "train/step_real_loss": 6.002715587615967, "train/lr": 8.219038247038819e-05, "perf/step_duration_sec": 3.574957752134651, "perf/samples_per_sec": 8.951154732078276, "perf/tokens_per_sec": 7391.13629642826, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26423.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782794.0098693, "event": "train_step", "step": 140, "epoch": 1, "metrics": {"train/step_loss": 5.99733579158783, "train/step_real_loss": 5.99733579158783, "train/lr": 7.903243608061246e-05, "perf/step_duration_sec": 3.9556330870836973, "perf/samples_per_sec": 8.089729076361857, "perf/tokens_per_sec": 7194.044385188417, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 28457.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782832.684074, "event": "train_step", "step": 150, "epoch": 1, "metrics": {"train/step_loss": 6.004466891288757, "train/step_real_loss": 6.004466891288757, "train/lr": 7.568906530820282e-05, "perf/step_duration_sec": 3.8291429793462157, "perf/samples_per_sec": 8.356961380811027, "perf/tokens_per_sec": 6940.456426763558, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26576.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782847.266101, "event": "eval_step", "step": 150, "epoch": 1, "metrics": {"eval/loss": 5.90363540568135, "eval/duration_sec": 14.580172989983112}}
+{"timestamp": 1773782884.7874682, "event": "train_step", "step": 160, "epoch": 1, "metrics": {"train/step_loss": 5.9455406665802, "train/step_real_loss": 5.9455406665802, "train/lr": 7.21816235958972e-05, "perf/step_duration_sec": 3.6964078596793115, "perf/samples_per_sec": 8.65705333793339, "perf/tokens_per_sec": 6998.416024968714, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 25869.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782922.1683276, "event": "train_step", "step": 170, "epoch": 1, "metrics": {"train/step_loss": 5.978999730312463, "train/step_real_loss": 5.847484111785889, "train/lr": 6.853251227482479e-05, "train/step_canary_loss": 10.1875, "perf/step_duration_sec": 3.8291215640492737, "perf/samples_per_sec": 8.618164622880943, "perf/tokens_per_sec": 7364.352248503628, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 28199.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.167274475097656, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782960.5932062, "event": "train_step", "step": 180, "epoch": 1, "metrics": {"train/step_loss": 5.780862244692716, "train/step_real_loss": 5.639248490333557, "train/lr": 6.476503749166904e-05, "train/step_canary_loss": 10.3125, "perf/step_duration_sec": 3.831100419629365, "perf/samples_per_sec": 8.61371313341678, "perf/tokens_per_sec": 7198.453963435392, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 27578.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773782999.2272496, "event": "train_step", "step": 190, "epoch": 1, "metrics": {"train/step_loss": 5.710304021835327, "train/step_real_loss": 5.710304021835327, "train/lr": 6.090326135695403e-05, "perf/step_duration_sec": 3.831862016580999, "perf/samples_per_sec": 8.351031394536536, "perf/tokens_per_sec": 6408.111746651394, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 24555.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773783037.1072268, "event": "train_step", "step": 200, "epoch": 1, "metrics": {"train/step_loss": 5.810285813880689, "train/step_real_loss": 5.673497915267944, "train/lr": 5.697184826514057e-05, "train/step_canary_loss": 10.1875, "perf/step_duration_sec": 3.822298335842788, "perf/samples_per_sec": 8.633549006509913, "perf/tokens_per_sec": 5939.881716478821, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 22704.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 90.77418279647827}}
+{"timestamp": 1773783051.6295593, "event": "eval_step", "step": 200, "epoch": 1, "metrics": {"eval/loss": 5.488488009030169, "eval/duration_sec": 14.520337663125247}}
+{"timestamp": 1773783092.753959, "event": "train_epoch", "step": 207, "epoch": 1, "metrics": {"train/epoch_loss": 7.408701919082488, "train/epoch_real_loss": 7.374264005589601, "train/epoch_canary_loss": 11.38703087321013, "perf/epoch_duration_sec": 846.9085897100158, "perf/epoch_samples_per_sec": 63.12959940376397, "perf/epoch_tokens_per_sec": 51777.628108618774, "perf/epoch_samples": 53465.0, "perf/epoch_tokens": 43850918.0, "system/cuda_epoch_peak_memory_gb": 90.77418279647827, "eval/loss": 5.447803701866757, "eval/duration_sec": 14.591437232214957}}
+{"timestamp": 1773783101.5745099, "event": "audit_epoch", "step": 207, "epoch": 1, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.923752, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.916456, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0, "perf/audit_duration_sec": 6.246651589404792}}
+{"timestamp": 1773783113.496209, "event": "train_step", "step": 210, "epoch": 2, "metrics": {"train/step_loss": 5.665921092033386, "train/step_real_loss": 5.665921092033386, "train/lr": 5.29959073680547e-05, "perf/step_duration_sec": 3.699097161181271, "perf/samples_per_sec": 8.650759524732544, "perf/tokens_per_sec": 6618.371708890693, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 24482.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 77.32090139389038}}
+{"timestamp": 1773783152.2505124, "event": "train_step", "step": 220, "epoch": 2, "metrics": {"train/step_loss": 5.316019058227539, "train/step_real_loss": 5.316019058227539, "train/lr": 4.9000832207739676e-05, "perf/step_duration_sec": 3.963131622876972, "perf/samples_per_sec": 8.074422715430812, "perf/tokens_per_sec": 6834.746502902325, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27087.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 77.32090139389038}}
+{"timestamp": 1773783190.084273, "event": "train_step", "step": 230, "epoch": 2, "metrics": {"train/step_loss": 5.287669539451599, "train/step_real_loss": 5.287669539451599, "train/lr": 4.501213853296425e-05, "perf/step_duration_sec": 3.7102178959175944, "perf/samples_per_sec": 8.624830373226882, "perf/tokens_per_sec": 7261.298596409544, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26941.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 77.32090139389038}}
+{"timestamp": 1773783228.4977129, "event": "train_step", "step": 240, "epoch": 2, "metrics": {"train/step_loss": 5.252316236495972, "train/step_real_loss": 5.252316236495972, "train/lr": 4.1055301335220955e-05, "perf/step_duration_sec": 3.832394160795957, "perf/samples_per_sec": 8.349871818339755, "perf/tokens_per_sec": 6669.981981887527, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 25562.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783266.2993257, "event": "train_step", "step": 250, "epoch": 2, "metrics": {"train/step_loss": 5.295487284660339, "train/step_real_loss": 5.295487284660339, "train/lr": 3.715559214503298e-05, "perf/step_duration_sec": 3.704376870766282, "perf/samples_per_sec": 8.63842992124625, "perf/tokens_per_sec": 6620.006779960055, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 24523.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783280.9439437, "event": "eval_step", "step": 250, "epoch": 2, "metrics": {"eval/loss": 5.170890462669458, "eval/duration_sec": 14.642520225141197}}
+{"timestamp": 1773783318.7159903, "event": "train_step", "step": 260, "epoch": 2, "metrics": {"train/step_loss": 5.102391600608826, "train/step_real_loss": 5.102391600608826, "train/lr": 3.33379176277258e-05, "perf/step_duration_sec": 3.9554404942318797, "perf/samples_per_sec": 8.090122970289858, "perf/tokens_per_sec": 6473.109641603173, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 25604.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783356.4853346, "event": "train_step", "step": 270, "epoch": 2, "metrics": {"train/step_loss": 5.203338623046875, "train/step_real_loss": 5.203338623046875, "train/lr": 2.962666050951997e-05, "perf/step_duration_sec": 3.9644229151308537, "perf/samples_per_sec": 8.071792713604516, "perf/tokens_per_sec": 7155.896484132703, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 28369.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783394.5975869, "event": "train_step", "step": 280, "epoch": 2, "metrics": {"train/step_loss": 4.966692328453064, "train/step_real_loss": 4.966692328453064, "train/lr": 2.604552384991855e-05, "perf/step_duration_sec": 3.573012210894376, "perf/samples_per_sec": 8.956028726246625, "perf/tokens_per_sec": 7867.31148421227, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 28110.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783432.839679, "event": "train_step", "step": 290, "epoch": 2, "metrics": {"train/step_loss": 5.069774425390995, "train/step_real_loss": 4.921564221382141, "train/lr": 2.2617379654990623e-05, "train/step_canary_loss": 9.8125, "perf/step_duration_sec": 3.8366584139876068, "perf/samples_per_sec": 8.601234834899378, "perf/tokens_per_sec": 7269.607296369047, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 27891.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783471.2908254, "event": "train_step", "step": 300, "epoch": 2, "metrics": {"train/step_loss": 5.246372468543775, "train/step_real_loss": 5.078290581703186, "train/lr": 1.936412279842705e-05, "train/step_canary_loss": 10.625, "perf/step_duration_sec": 4.068281149957329, "perf/samples_per_sec": 8.111533786288621, "perf/tokens_per_sec": 6243.177170846809, "perf/logical_batch_size": 33.0, "perf/logical_token_count": 25399.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783485.8945208, "event": "eval_step", "step": 300, "epoch": 2, "metrics": {"eval/loss": 4.965634505857121, "eval/duration_sec": 14.601576885208488}}
+{"timestamp": 1773783524.0127125, "event": "train_step", "step": 310, "epoch": 2, "metrics": {"train/step_loss": 5.0469924211502075, "train/step_real_loss": 5.0469924211502075, "train/lr": 1.6306531183346385e-05, "perf/step_duration_sec": 3.7034485950134695, "perf/samples_per_sec": 8.64059515854671, "perf/tokens_per_sec": 6898.975196902139, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 25550.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783561.666821, "event": "train_step", "step": 320, "epoch": 2, "metrics": {"train/step_loss": 5.018897533416748, "train/step_real_loss": 5.018897533416748, "train/lr": 1.3464133037968912e-05, "perf/step_duration_sec": 3.5697252051904798, "perf/samples_per_sec": 8.964275444359446, "perf/tokens_per_sec": 7121.5565733283065, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 25422.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783600.0391417, "event": "train_step", "step": 330, "epoch": 2, "metrics": {"train/step_loss": 5.0458667278289795, "train/step_real_loss": 5.0458667278289795, "train/lr": 1.0855082192715294e-05, "perf/step_duration_sec": 3.95492945285514, "perf/samples_per_sec": 8.091168346099975, "perf/tokens_per_sec": 6913.6505027316125, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27343.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783637.8100953, "event": "train_step", "step": 340, "epoch": 2, "metrics": {"train/step_loss": 5.095871806144714, "train/step_real_loss": 5.095871806144714, "train/lr": 8.49604213531004e-06, "perf/step_duration_sec": 3.57716670492664, "perf/samples_per_sec": 8.945627263031415, "perf/tokens_per_sec": 7446.396043917744, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26637.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783676.205935, "event": "train_step", "step": 350, "epoch": 2, "metrics": {"train/step_loss": 5.241442084312439, "train/step_real_loss": 5.241442084312439, "train/lr": 6.402079584406673e-06, "perf/step_duration_sec": 3.8292608251795173, "perf/samples_per_sec": 8.356704194601273, "perf/tokens_per_sec": 5819.13873650938, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 22283.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783690.815634, "event": "eval_step", "step": 350, "epoch": 2, "metrics": {"eval/loss": 4.8690267882563845, "eval/duration_sec": 14.607622059062123}}
+{"timestamp": 1773783728.3614442, "event": "train_step", "step": 360, "epoch": 2, "metrics": {"train/step_loss": 4.913779258728027, "train/step_real_loss": 4.913779258728027, "train/lr": 4.586568261458729e-06, "perf/step_duration_sec": 3.8354132031090558, "perf/samples_per_sec": 8.343299223682136, "perf/tokens_per_sec": 7633.075877266194, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 29276.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783766.5169628, "event": "train_step", "step": 370, "epoch": 2, "metrics": {"train/step_loss": 5.0288437604904175, "train/step_real_loss": 5.0288437604904175, "train/lr": 3.06110347542643e-06, "perf/step_duration_sec": 3.820353894960135, "perf/samples_per_sec": 8.376187358510125, "perf/tokens_per_sec": 7220.011747180773, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27583.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783804.5456383, "event": "train_step", "step": 380, "epoch": 2, "metrics": {"train/step_loss": 4.826860070228577, "train/step_real_loss": 4.826860070228577, "train/lr": 1.8354280658494649e-06, "perf/step_duration_sec": 3.825985827948898, "perf/samples_per_sec": 8.363857431525073, "perf/tokens_per_sec": 7291.454086476718, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27897.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783843.007424, "event": "train_step", "step": 390, "epoch": 2, "metrics": {"train/step_loss": 4.915419220924377, "train/step_real_loss": 4.915419220924377, "train/lr": 9.17370177272775e-07, "perf/step_duration_sec": 3.7134576980024576, "perf/samples_per_sec": 8.617305649452646, "perf/tokens_per_sec": 7082.61737144544, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 26301.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783881.5532339, "event": "train_step", "step": 400, "epoch": 2, "metrics": {"train/step_loss": 4.916181445121765, "train/step_real_loss": 4.916181445121765, "train/lr": 3.127932624475638e-07, "perf/step_duration_sec": 3.831934977322817, "perf/samples_per_sec": 8.350872389373583, "perf/tokens_per_sec": 7147.302906255116, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 27388.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783896.2133112, "event": "eval_step", "step": 400, "epoch": 2, "metrics": {"eval/loss": 4.840054991570385, "eval/duration_sec": 14.658054957631975}}
+{"timestamp": 1773783934.3512156, "event": "train_step", "step": 410, "epoch": 2, "metrics": {"train/step_loss": 4.948451399803162, "train/step_real_loss": 4.948451399803162, "train/lr": 2.5558633627303928e-08, "perf/step_duration_sec": 3.8317640791647136, "perf/samples_per_sec": 8.35124484150801, "perf/tokens_per_sec": 6492.570911469883, "perf/logical_batch_size": 32.0, "perf/logical_token_count": 24878.0, "perf/gradient_accumulation_steps": 4.0, "system/cuda_memory_allocated_gb": 15.10503625869751, "system/cuda_max_memory_allocated_gb": 84.09655332565308}}
+{"timestamp": 1773783964.370691, "event": "train_epoch", "step": 414, "epoch": 2, "metrics": {"train/epoch_loss": 5.128793139947108, "train/epoch_real_loss": 5.082210323228928, "train/epoch_canary_loss": 10.210301459293396, "perf/epoch_duration_sec": 848.1400936129503, "perf/epoch_samples_per_sec": 63.06976925490229, "perf/epoch_tokens_per_sec": 51703.841535419684, "perf/epoch_samples": 53492.0, "perf/epoch_tokens": 43852101.0, "system/cuda_epoch_peak_memory_gb": 84.09655332565308, "eval/loss": 4.839725652878935, "eval/duration_sec": 14.596055098809302}}
+{"timestamp": 1773783972.8350441, "event": "audit_epoch", "step": 414, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.957184, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.968208, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0, "perf/audit_duration_sec": 5.8983861412853}}
+{"timestamp": 1773783981.1943111, "event": "audit_final", "step": 414, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.957184, "audit/loss/empirical_epsilon/0.05": 3.4791953936219215, "audit/loss/empirical_epsilon/0.01": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0, "audit/embedding/auc": 0.968208, "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215, "audit/embedding/empirical_epsilon/0.01": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0}}
+{"timestamp": 1773783981.7373376, "event": "energy_final", "step": 414, "epoch": null, "metrics": {"energy/codecarbon/duration": 1810.5507336058654, "energy/codecarbon/emissions": 0.09709380205154217, "energy/codecarbon/emissions_rate": 5.362666742741399e-05, "energy/codecarbon/cpu_power": 72.0230906963752, "energy/codecarbon/gpu_power": 4629.388481318127, "energy/codecarbon/ram_power": 54.0, "energy/codecarbon/cpu_energy": 0.03488049743955748, "energy/codecarbon/gpu_energy": 2.3248590518302024, "energy/codecarbon/ram_energy": 0.026150659639004155, "energy/codecarbon/energy_consumed": 2.3858902089087644, "energy/codecarbon/water_consumed": 0.0, "energy/codecarbon/cpu_count": 256.0, "energy/codecarbon/gpu_count": 8.0, "energy/codecarbon/longitude": 16.1885, "energy/codecarbon/latitude": 58.594, "energy/codecarbon/ram_total_size": 1511.49019241333, "energy/codecarbon/cpu_utilization_percent": 3.485983379501395, "energy/codecarbon/gpu_utilization_percent": 91.87222991689751, "energy/codecarbon/ram_utilization_percent": 5.226869806094248, "energy/codecarbon/ram_used_gb": 78.96254361435317, "energy/codecarbon/pue": 1.0, "energy/codecarbon/wue": 0.0}}

deepseek-coder-6.7b/base/resolved_config.yaml

@@ -0,0 +1,100 @@
+model:
+  name: deepseek-ai/deepseek-coder-6.7b-instruct
+  tokenizer_name: deepseek-ai/deepseek-coder-6.7b-instruct
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0001
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 4
+  num_workers: 4
+  output_dir: runs/cpt/deepseek-coder-6.7b/base
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: ghost
+  secure_mode: false
+  enabled: false
+  target_epsilon: 8.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: deepseek-coder-6.7b-cpt-base
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-deepseek-coder-6.7b-cpt-base

deepseek-coder-6.7b/base/scalars.csv

@@ -0,0 +1,591 @@
+timestamp,event,step,epoch,key,value
+1773782272.0493639,train_step,10,1,train/step_loss,14.555815444273108
+1773782272.0493639,train_step,10,1,train/step_real_loss,14.770240783691406
+1773782272.0493639,train_step,10,1,train/lr,4.761904761904762e-05
+1773782272.0493639,train_step,10,1,train/step_canary_loss,11.125
+1773782272.0493639,train_step,10,1,perf/step_duration_sec,4.0931806759908795
+1773782272.0493639,train_step,10,1,perf/samples_per_sec,8.30649870879918
+1773782272.0493639,train_step,10,1,perf/tokens_per_sec,5843.6218416402235
+1773782272.0493639,train_step,10,1,perf/logical_batch_size,34.0
+1773782272.0493639,train_step,10,1,perf/logical_token_count,23919.0
+1773782272.0493639,train_step,10,1,perf/gradient_accumulation_steps,4.0
+1773782272.0493639,train_step,10,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782272.0493639,train_step,10,1,system/cuda_max_memory_allocated_gb,84.1168704032898
+1773782309.7872338,train_step,20,1,train/step_loss,12.852383613586426
+1773782309.7872338,train_step,20,1,train/step_real_loss,12.852383613586426
+1773782309.7872338,train_step,20,1,train/lr,9.523809523809524e-05
+1773782309.7872338,train_step,20,1,perf/step_duration_sec,3.573033411987126
+1773782309.7872338,train_step,20,1,perf/samples_per_sec,8.955975584399406
+1773782309.7872338,train_step,20,1,perf/tokens_per_sec,7755.035233378848
+1773782309.7872338,train_step,20,1,perf/logical_batch_size,32.0
+1773782309.7872338,train_step,20,1,perf/logical_token_count,27709.0
+1773782309.7872338,train_step,20,1,perf/gradient_accumulation_steps,4.0
+1773782309.7872338,train_step,20,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782309.7872338,train_step,20,1,system/cuda_max_memory_allocated_gb,84.1168704032898
+1773782347.6002963,train_step,30,1,train/step_loss,9.603991746902466
+1773782347.6002963,train_step,30,1,train/step_real_loss,9.603991746902466
+1773782347.6002963,train_step,30,1,train/lr,9.98706541985615e-05
+1773782347.6002963,train_step,30,1,perf/step_duration_sec,3.6900156908668578
+1773782347.6002963,train_step,30,1,perf/samples_per_sec,8.672049844992005
+1773782347.6002963,train_step,30,1,perf/tokens_per_sec,7146.58207694638
+1773782347.6002963,train_step,30,1,perf/logical_batch_size,32.0
+1773782347.6002963,train_step,30,1,perf/logical_token_count,26371.0
+1773782347.6002963,train_step,30,1,perf/gradient_accumulation_steps,4.0
+1773782347.6002963,train_step,30,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782347.6002963,train_step,30,1,system/cuda_max_memory_allocated_gb,84.1168704032898
+1773782386.0648503,train_step,40,1,train/step_loss,7.74934458732605
+1773782386.0648503,train_step,40,1,train/step_real_loss,7.74934458732605
+1773782386.0648503,train_step,40,1,train/lr,9.942439201095397e-05
+1773782386.0648503,train_step,40,1,perf/step_duration_sec,4.0733352322131395
+1773782386.0648503,train_step,40,1,perf/samples_per_sec,7.855970151176003
+1773782386.0648503,train_step,40,1,perf/tokens_per_sec,7048.523719075445
+1773782386.0648503,train_step,40,1,perf/logical_batch_size,32.0
+1773782386.0648503,train_step,40,1,perf/logical_token_count,28711.0
+1773782386.0648503,train_step,40,1,perf/gradient_accumulation_steps,4.0
+1773782386.0648503,train_step,40,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782386.0648503,train_step,40,1,system/cuda_max_memory_allocated_gb,84.1168704032898
+1773782423.7351854,train_step,50,1,train/step_loss,7.176188588142395
+1773782423.7351854,train_step,50,1,train/step_real_loss,7.176188588142395
+1773782423.7351854,train_step,50,1,train/lr,9.866246608261724e-05
+1773782423.7351854,train_step,50,1,perf/step_duration_sec,3.6839785198681056
+1773782423.7351854,train_step,50,1,perf/samples_per_sec,8.686261287198187
+1773782423.7351854,train_step,50,1,perf/tokens_per_sec,7128.163168807011
+1773782423.7351854,train_step,50,1,perf/logical_batch_size,32.0
+1773782423.7351854,train_step,50,1,perf/logical_token_count,26260.0
+1773782423.7351854,train_step,50,1,perf/gradient_accumulation_steps,4.0
+1773782423.7351854,train_step,50,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782423.7351854,train_step,50,1,system/cuda_max_memory_allocated_gb,84.1168704032898
+1773782438.1861553,eval_step,50,1,eval/loss,6.95498745685274
+1773782438.1861553,eval_step,50,1,eval/duration_sec,14.44864141382277
+1773782475.968484,train_step,60,1,train/step_loss,6.844001889228821
+1773782475.968484,train_step,60,1,train/step_real_loss,6.844001889228821
+1773782475.968484,train_step,60,1,train/lr,9.7589742682592e-05
+1773782475.968484,train_step,60,1,perf/step_duration_sec,3.950888440012932
+1773782475.968484,train_step,60,1,perf/samples_per_sec,8.099444083492081
+1773782475.968484,train_step,60,1,perf/tokens_per_sec,7022.471128015244
+1773782475.968484,train_step,60,1,perf/logical_batch_size,32.0
+1773782475.968484,train_step,60,1,perf/logical_token_count,27745.0
+1773782475.968484,train_step,60,1,perf/gradient_accumulation_steps,4.0
+1773782475.968484,train_step,60,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782475.968484,train_step,60,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782514.3752964,train_step,70,1,train/step_loss,6.707658648490906
+1773782514.3752964,train_step,70,1,train/step_real_loss,6.707658648490906
+1773782514.3752964,train_step,70,1,train/lr,9.621307308142384e-05
+1773782514.3752964,train_step,70,1,perf/step_duration_sec,4.038279882632196
+1773782514.3752964,train_step,70,1,perf/samples_per_sec,7.924165964232782
+1773782514.3752964,train_step,70,1,perf/tokens_per_sec,6047.624411827906
+1773782514.3752964,train_step,70,1,perf/logical_batch_size,32.0
+1773782514.3752964,train_step,70,1,perf/logical_token_count,24422.0
+1773782514.3752964,train_step,70,1,perf/gradient_accumulation_steps,4.0
+1773782514.3752964,train_step,70,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782514.3752964,train_step,70,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782552.1368628,train_step,80,1,train/step_loss,7.028713240342982
+1773782552.1368628,train_step,80,1,train/step_real_loss,6.692617297172546
+1773782552.1368628,train_step,80,1,train/lr,9.454124979346391e-05
+1773782552.1368628,train_step,80,1,train/step_canary_loss,12.40625
+1773782552.1368628,train_step,80,1,perf/step_duration_sec,3.830359708983451
+1773782552.1368628,train_step,80,1,perf/samples_per_sec,8.876450929728307
+1773782552.1368628,train_step,80,1,perf/tokens_per_sec,6924.153869360416
+1773782552.1368628,train_step,80,1,perf/logical_batch_size,34.0
+1773782552.1368628,train_step,80,1,perf/logical_token_count,26522.0
+1773782552.1368628,train_step,80,1,perf/gradient_accumulation_steps,4.0
+1773782552.1368628,train_step,80,1,system/cuda_memory_allocated_gb,15.167274475097656
+1773782552.1368628,train_step,80,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782590.0828424,train_step,90,1,train/step_loss,6.5469324588775635
+1773782590.0828424,train_step,90,1,train/step_real_loss,6.5469324588775635
+1773782590.0828424,train_step,90,1,train/lr,9.258495042083221e-05
+1773782590.0828424,train_step,90,1,perf/step_duration_sec,3.57225156808272
+1773782590.0828424,train_step,90,1,perf/samples_per_sec,8.957935741679826
+1773782590.0828424,train_step,90,1,perf/tokens_per_sec,7531.66441030924
+1773782590.0828424,train_step,90,1,perf/logical_batch_size,32.0
+1773782590.0828424,train_step,90,1,perf/logical_token_count,26905.0
+1773782590.0828424,train_step,90,1,perf/gradient_accumulation_steps,4.0
+1773782590.0828424,train_step,90,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782590.0828424,train_step,90,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782627.521805,train_step,100,1,train/step_loss,6.427393674850464
+1773782627.521805,train_step,100,1,train/step_real_loss,6.427393674850464
+1773782627.521805,train_step,100,1,train/lr,9.035666945770107e-05
+1773782627.521805,train_step,100,1,perf/step_duration_sec,3.5687404102645814
+1773782627.521805,train_step,100,1,perf/samples_per_sec,8.966749138704534
+1773782627.521805,train_step,100,1,perf/tokens_per_sec,8349.164291776258
+1773782627.521805,train_step,100,1,perf/logical_batch_size,32.0
+1773782627.521805,train_step,100,1,perf/logical_token_count,29796.0
+1773782627.521805,train_step,100,1,perf/gradient_accumulation_steps,4.0
+1773782627.521805,train_step,100,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782627.521805,train_step,100,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782642.0074606,eval_step,100,1,eval/loss,6.393013093959201
+1773782642.0074606,eval_step,100,1,eval/duration_sec,14.483735092915595
+1773782680.1220238,train_step,110,1,train/step_loss,6.492933461160371
+1773782680.1220238,train_step,110,1,train/step_real_loss,6.355993866920471
+1773782680.1220238,train_step,110,1,train/lr,8.787063849045118e-05
+1773782680.1220238,train_step,110,1,train/step_canary_loss,10.875
+1773782680.1220238,train_step,110,1,perf/step_duration_sec,3.834493800997734
+1773782680.1220238,train_step,110,1,perf/samples_per_sec,8.606090324468228
+1773782680.1220238,train_step,110,1,perf/tokens_per_sec,7034.044491865366
+1773782680.1220238,train_step,110,1,perf/logical_batch_size,33.0
+1773782680.1220238,train_step,110,1,perf/logical_token_count,26972.0
+1773782680.1220238,train_step,110,1,perf/gradient_accumulation_steps,4.0
+1773782680.1220238,train_step,110,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782680.1220238,train_step,110,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782717.354572,train_step,120,1,train/step_loss,6.159674048423767
+1773782717.354572,train_step,120,1,train/step_real_loss,6.159674048423767
+1773782717.354572,train_step,120,1,train/lr,8.5142735303366e-05
+1773782717.354572,train_step,120,1,perf/step_duration_sec,3.8241200419142842
+1773782717.354572,train_step,120,1,perf/samples_per_sec,8.367938152898931
+1773782717.354572,train_step,120,1,perf/tokens_per_sec,7346.00370597615
+1773782717.354572,train_step,120,1,perf/logical_batch_size,32.0
+1773782717.354572,train_step,120,1,perf/logical_token_count,28092.0
+1773782717.354572,train_step,120,1,perf/gradient_accumulation_steps,4.0
+1773782717.354572,train_step,120,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782717.354572,train_step,120,1,system/cuda_max_memory_allocated_gb,84.1198000907898
+1773782755.7000217,train_step,130,1,train/step_loss,6.002715587615967
+1773782755.7000217,train_step,130,1,train/step_real_loss,6.002715587615967
+1773782755.7000217,train_step,130,1,train/lr,8.219038247038819e-05
+1773782755.7000217,train_step,130,1,perf/step_duration_sec,3.574957752134651
+1773782755.7000217,train_step,130,1,perf/samples_per_sec,8.951154732078276
+1773782755.7000217,train_step,130,1,perf/tokens_per_sec,7391.13629642826
+1773782755.7000217,train_step,130,1,perf/logical_batch_size,32.0
+1773782755.7000217,train_step,130,1,perf/logical_token_count,26423.0
+1773782755.7000217,train_step,130,1,perf/gradient_accumulation_steps,4.0
+1773782755.7000217,train_step,130,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782755.7000217,train_step,130,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782794.0098693,train_step,140,1,train/step_loss,5.99733579158783
+1773782794.0098693,train_step,140,1,train/step_real_loss,5.99733579158783
+1773782794.0098693,train_step,140,1,train/lr,7.903243608061246e-05
+1773782794.0098693,train_step,140,1,perf/step_duration_sec,3.9556330870836973
+1773782794.0098693,train_step,140,1,perf/samples_per_sec,8.089729076361857
+1773782794.0098693,train_step,140,1,perf/tokens_per_sec,7194.044385188417
+1773782794.0098693,train_step,140,1,perf/logical_batch_size,32.0
+1773782794.0098693,train_step,140,1,perf/logical_token_count,28457.0
+1773782794.0098693,train_step,140,1,perf/gradient_accumulation_steps,4.0
+1773782794.0098693,train_step,140,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782794.0098693,train_step,140,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782832.684074,train_step,150,1,train/step_loss,6.004466891288757
+1773782832.684074,train_step,150,1,train/step_real_loss,6.004466891288757
+1773782832.684074,train_step,150,1,train/lr,7.568906530820282e-05
+1773782832.684074,train_step,150,1,perf/step_duration_sec,3.8291429793462157
+1773782832.684074,train_step,150,1,perf/samples_per_sec,8.356961380811027
+1773782832.684074,train_step,150,1,perf/tokens_per_sec,6940.456426763558
+1773782832.684074,train_step,150,1,perf/logical_batch_size,32.0
+1773782832.684074,train_step,150,1,perf/logical_token_count,26576.0
+1773782832.684074,train_step,150,1,perf/gradient_accumulation_steps,4.0
+1773782832.684074,train_step,150,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782832.684074,train_step,150,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782847.266101,eval_step,150,1,eval/loss,5.90363540568135
+1773782847.266101,eval_step,150,1,eval/duration_sec,14.580172989983112
+1773782884.7874682,train_step,160,1,train/step_loss,5.9455406665802
+1773782884.7874682,train_step,160,1,train/step_real_loss,5.9455406665802
+1773782884.7874682,train_step,160,1,train/lr,7.21816235958972e-05
+1773782884.7874682,train_step,160,1,perf/step_duration_sec,3.6964078596793115
+1773782884.7874682,train_step,160,1,perf/samples_per_sec,8.65705333793339
+1773782884.7874682,train_step,160,1,perf/tokens_per_sec,6998.416024968714
+1773782884.7874682,train_step,160,1,perf/logical_batch_size,32.0
+1773782884.7874682,train_step,160,1,perf/logical_token_count,25869.0
+1773782884.7874682,train_step,160,1,perf/gradient_accumulation_steps,4.0
+1773782884.7874682,train_step,160,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782884.7874682,train_step,160,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782922.1683276,train_step,170,1,train/step_loss,5.978999730312463
+1773782922.1683276,train_step,170,1,train/step_real_loss,5.847484111785889
+1773782922.1683276,train_step,170,1,train/lr,6.853251227482479e-05
+1773782922.1683276,train_step,170,1,train/step_canary_loss,10.1875
+1773782922.1683276,train_step,170,1,perf/step_duration_sec,3.8291215640492737
+1773782922.1683276,train_step,170,1,perf/samples_per_sec,8.618164622880943
+1773782922.1683276,train_step,170,1,perf/tokens_per_sec,7364.352248503628
+1773782922.1683276,train_step,170,1,perf/logical_batch_size,33.0
+1773782922.1683276,train_step,170,1,perf/logical_token_count,28199.0
+1773782922.1683276,train_step,170,1,perf/gradient_accumulation_steps,4.0
+1773782922.1683276,train_step,170,1,system/cuda_memory_allocated_gb,15.167274475097656
+1773782922.1683276,train_step,170,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782960.5932062,train_step,180,1,train/step_loss,5.780862244692716
+1773782960.5932062,train_step,180,1,train/step_real_loss,5.639248490333557
+1773782960.5932062,train_step,180,1,train/lr,6.476503749166904e-05
+1773782960.5932062,train_step,180,1,train/step_canary_loss,10.3125
+1773782960.5932062,train_step,180,1,perf/step_duration_sec,3.831100419629365
+1773782960.5932062,train_step,180,1,perf/samples_per_sec,8.61371313341678
+1773782960.5932062,train_step,180,1,perf/tokens_per_sec,7198.453963435392
+1773782960.5932062,train_step,180,1,perf/logical_batch_size,33.0
+1773782960.5932062,train_step,180,1,perf/logical_token_count,27578.0
+1773782960.5932062,train_step,180,1,perf/gradient_accumulation_steps,4.0
+1773782960.5932062,train_step,180,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782960.5932062,train_step,180,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773782999.2272496,train_step,190,1,train/step_loss,5.710304021835327
+1773782999.2272496,train_step,190,1,train/step_real_loss,5.710304021835327
+1773782999.2272496,train_step,190,1,train/lr,6.090326135695403e-05
+1773782999.2272496,train_step,190,1,perf/step_duration_sec,3.831862016580999
+1773782999.2272496,train_step,190,1,perf/samples_per_sec,8.351031394536536
+1773782999.2272496,train_step,190,1,perf/tokens_per_sec,6408.111746651394
+1773782999.2272496,train_step,190,1,perf/logical_batch_size,32.0
+1773782999.2272496,train_step,190,1,perf/logical_token_count,24555.0
+1773782999.2272496,train_step,190,1,perf/gradient_accumulation_steps,4.0
+1773782999.2272496,train_step,190,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773782999.2272496,train_step,190,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773783037.1072268,train_step,200,1,train/step_loss,5.810285813880689
+1773783037.1072268,train_step,200,1,train/step_real_loss,5.673497915267944
+1773783037.1072268,train_step,200,1,train/lr,5.697184826514057e-05
+1773783037.1072268,train_step,200,1,train/step_canary_loss,10.1875
+1773783037.1072268,train_step,200,1,perf/step_duration_sec,3.822298335842788
+1773783037.1072268,train_step,200,1,perf/samples_per_sec,8.633549006509913
+1773783037.1072268,train_step,200,1,perf/tokens_per_sec,5939.881716478821
+1773783037.1072268,train_step,200,1,perf/logical_batch_size,33.0
+1773783037.1072268,train_step,200,1,perf/logical_token_count,22704.0
+1773783037.1072268,train_step,200,1,perf/gradient_accumulation_steps,4.0
+1773783037.1072268,train_step,200,1,system/cuda_memory_allocated_gb,15.10503625869751
+1773783037.1072268,train_step,200,1,system/cuda_max_memory_allocated_gb,90.77418279647827
+1773783051.6295593,eval_step,200,1,eval/loss,5.488488009030169
+1773783051.6295593,eval_step,200,1,eval/duration_sec,14.520337663125247
+1773783092.753959,train_epoch,207,1,train/epoch_loss,7.408701919082488
+1773783092.753959,train_epoch,207,1,train/epoch_real_loss,7.374264005589601
+1773783092.753959,train_epoch,207,1,train/epoch_canary_loss,11.38703087321013
+1773783092.753959,train_epoch,207,1,perf/epoch_duration_sec,846.9085897100158
+1773783092.753959,train_epoch,207,1,perf/epoch_samples_per_sec,63.12959940376397
+1773783092.753959,train_epoch,207,1,perf/epoch_tokens_per_sec,51777.628108618774
+1773783092.753959,train_epoch,207,1,perf/epoch_samples,53465.0
+1773783092.753959,train_epoch,207,1,perf/epoch_tokens,43850918.0
+1773783092.753959,train_epoch,207,1,system/cuda_epoch_peak_memory_gb,90.77418279647827
+1773783092.753959,train_epoch,207,1,eval/loss,5.447803701866757
+1773783092.753959,train_epoch,207,1,eval/duration_sec,14.591437232214957
+1773783101.5745099,audit_epoch,207,1,audit/delta,1e-05
+1773783101.5745099,audit_epoch,207,1,audit/num_canaries,500.0
+1773783101.5745099,audit_epoch,207,1,audit/num_members,250.0
+1773783101.5745099,audit_epoch,207,1,audit/paper_guess_fraction,0.2
+1773783101.5745099,audit_epoch,207,1,audit/paper_guess_steps,20.0
+1773783101.5745099,audit_epoch,207,1,audit/loss/auc,0.923752
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/embedding/auc,0.916456
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783101.5745099,audit_epoch,207,1,perf/audit_duration_sec,6.246651589404792
+1773783113.496209,train_step,210,2,train/step_loss,5.665921092033386
+1773783113.496209,train_step,210,2,train/step_real_loss,5.665921092033386
+1773783113.496209,train_step,210,2,train/lr,5.29959073680547e-05
+1773783113.496209,train_step,210,2,perf/step_duration_sec,3.699097161181271
+1773783113.496209,train_step,210,2,perf/samples_per_sec,8.650759524732544
+1773783113.496209,train_step,210,2,perf/tokens_per_sec,6618.371708890693
+1773783113.496209,train_step,210,2,perf/logical_batch_size,32.0
+1773783113.496209,train_step,210,2,perf/logical_token_count,24482.0
+1773783113.496209,train_step,210,2,perf/gradient_accumulation_steps,4.0
+1773783113.496209,train_step,210,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783113.496209,train_step,210,2,system/cuda_max_memory_allocated_gb,77.32090139389038
+1773783152.2505124,train_step,220,2,train/step_loss,5.316019058227539
+1773783152.2505124,train_step,220,2,train/step_real_loss,5.316019058227539
+1773783152.2505124,train_step,220,2,train/lr,4.9000832207739676e-05
+1773783152.2505124,train_step,220,2,perf/step_duration_sec,3.963131622876972
+1773783152.2505124,train_step,220,2,perf/samples_per_sec,8.074422715430812
+1773783152.2505124,train_step,220,2,perf/tokens_per_sec,6834.746502902325
+1773783152.2505124,train_step,220,2,perf/logical_batch_size,32.0
+1773783152.2505124,train_step,220,2,perf/logical_token_count,27087.0
+1773783152.2505124,train_step,220,2,perf/gradient_accumulation_steps,4.0
+1773783152.2505124,train_step,220,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783152.2505124,train_step,220,2,system/cuda_max_memory_allocated_gb,77.32090139389038
+1773783190.084273,train_step,230,2,train/step_loss,5.287669539451599
+1773783190.084273,train_step,230,2,train/step_real_loss,5.287669539451599
+1773783190.084273,train_step,230,2,train/lr,4.501213853296425e-05
+1773783190.084273,train_step,230,2,perf/step_duration_sec,3.7102178959175944
+1773783190.084273,train_step,230,2,perf/samples_per_sec,8.624830373226882
+1773783190.084273,train_step,230,2,perf/tokens_per_sec,7261.298596409544
+1773783190.084273,train_step,230,2,perf/logical_batch_size,32.0
+1773783190.084273,train_step,230,2,perf/logical_token_count,26941.0
+1773783190.084273,train_step,230,2,perf/gradient_accumulation_steps,4.0
+1773783190.084273,train_step,230,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783190.084273,train_step,230,2,system/cuda_max_memory_allocated_gb,77.32090139389038
+1773783228.4977129,train_step,240,2,train/step_loss,5.252316236495972
+1773783228.4977129,train_step,240,2,train/step_real_loss,5.252316236495972
+1773783228.4977129,train_step,240,2,train/lr,4.1055301335220955e-05
+1773783228.4977129,train_step,240,2,perf/step_duration_sec,3.832394160795957
+1773783228.4977129,train_step,240,2,perf/samples_per_sec,8.349871818339755
+1773783228.4977129,train_step,240,2,perf/tokens_per_sec,6669.981981887527
+1773783228.4977129,train_step,240,2,perf/logical_batch_size,32.0
+1773783228.4977129,train_step,240,2,perf/logical_token_count,25562.0
+1773783228.4977129,train_step,240,2,perf/gradient_accumulation_steps,4.0
+1773783228.4977129,train_step,240,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783228.4977129,train_step,240,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783266.2993257,train_step,250,2,train/step_loss,5.295487284660339
+1773783266.2993257,train_step,250,2,train/step_real_loss,5.295487284660339
+1773783266.2993257,train_step,250,2,train/lr,3.715559214503298e-05
+1773783266.2993257,train_step,250,2,perf/step_duration_sec,3.704376870766282
+1773783266.2993257,train_step,250,2,perf/samples_per_sec,8.63842992124625
+1773783266.2993257,train_step,250,2,perf/tokens_per_sec,6620.006779960055
+1773783266.2993257,train_step,250,2,perf/logical_batch_size,32.0
+1773783266.2993257,train_step,250,2,perf/logical_token_count,24523.0
+1773783266.2993257,train_step,250,2,perf/gradient_accumulation_steps,4.0
+1773783266.2993257,train_step,250,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783266.2993257,train_step,250,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783280.9439437,eval_step,250,2,eval/loss,5.170890462669458
+1773783280.9439437,eval_step,250,2,eval/duration_sec,14.642520225141197
+1773783318.7159903,train_step,260,2,train/step_loss,5.102391600608826
+1773783318.7159903,train_step,260,2,train/step_real_loss,5.102391600608826
+1773783318.7159903,train_step,260,2,train/lr,3.33379176277258e-05
+1773783318.7159903,train_step,260,2,perf/step_duration_sec,3.9554404942318797
+1773783318.7159903,train_step,260,2,perf/samples_per_sec,8.090122970289858
+1773783318.7159903,train_step,260,2,perf/tokens_per_sec,6473.109641603173
+1773783318.7159903,train_step,260,2,perf/logical_batch_size,32.0
+1773783318.7159903,train_step,260,2,perf/logical_token_count,25604.0
+1773783318.7159903,train_step,260,2,perf/gradient_accumulation_steps,4.0
+1773783318.7159903,train_step,260,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783318.7159903,train_step,260,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783356.4853346,train_step,270,2,train/step_loss,5.203338623046875
+1773783356.4853346,train_step,270,2,train/step_real_loss,5.203338623046875
+1773783356.4853346,train_step,270,2,train/lr,2.962666050951997e-05
+1773783356.4853346,train_step,270,2,perf/step_duration_sec,3.9644229151308537
+1773783356.4853346,train_step,270,2,perf/samples_per_sec,8.071792713604516
+1773783356.4853346,train_step,270,2,perf/tokens_per_sec,7155.896484132703
+1773783356.4853346,train_step,270,2,perf/logical_batch_size,32.0
+1773783356.4853346,train_step,270,2,perf/logical_token_count,28369.0
+1773783356.4853346,train_step,270,2,perf/gradient_accumulation_steps,4.0
+1773783356.4853346,train_step,270,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783356.4853346,train_step,270,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783394.5975869,train_step,280,2,train/step_loss,4.966692328453064
+1773783394.5975869,train_step,280,2,train/step_real_loss,4.966692328453064
+1773783394.5975869,train_step,280,2,train/lr,2.604552384991855e-05
+1773783394.5975869,train_step,280,2,perf/step_duration_sec,3.573012210894376
+1773783394.5975869,train_step,280,2,perf/samples_per_sec,8.956028726246625
+1773783394.5975869,train_step,280,2,perf/tokens_per_sec,7867.31148421227
+1773783394.5975869,train_step,280,2,perf/logical_batch_size,32.0
+1773783394.5975869,train_step,280,2,perf/logical_token_count,28110.0
+1773783394.5975869,train_step,280,2,perf/gradient_accumulation_steps,4.0
+1773783394.5975869,train_step,280,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783394.5975869,train_step,280,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783432.839679,train_step,290,2,train/step_loss,5.069774425390995
+1773783432.839679,train_step,290,2,train/step_real_loss,4.921564221382141
+1773783432.839679,train_step,290,2,train/lr,2.2617379654990623e-05
+1773783432.839679,train_step,290,2,train/step_canary_loss,9.8125
+1773783432.839679,train_step,290,2,perf/step_duration_sec,3.8366584139876068
+1773783432.839679,train_step,290,2,perf/samples_per_sec,8.601234834899378
+1773783432.839679,train_step,290,2,perf/tokens_per_sec,7269.607296369047
+1773783432.839679,train_step,290,2,perf/logical_batch_size,33.0
+1773783432.839679,train_step,290,2,perf/logical_token_count,27891.0
+1773783432.839679,train_step,290,2,perf/gradient_accumulation_steps,4.0
+1773783432.839679,train_step,290,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783432.839679,train_step,290,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783471.2908254,train_step,300,2,train/step_loss,5.246372468543775
+1773783471.2908254,train_step,300,2,train/step_real_loss,5.078290581703186
+1773783471.2908254,train_step,300,2,train/lr,1.936412279842705e-05
+1773783471.2908254,train_step,300,2,train/step_canary_loss,10.625
+1773783471.2908254,train_step,300,2,perf/step_duration_sec,4.068281149957329
+1773783471.2908254,train_step,300,2,perf/samples_per_sec,8.111533786288621
+1773783471.2908254,train_step,300,2,perf/tokens_per_sec,6243.177170846809
+1773783471.2908254,train_step,300,2,perf/logical_batch_size,33.0
+1773783471.2908254,train_step,300,2,perf/logical_token_count,25399.0
+1773783471.2908254,train_step,300,2,perf/gradient_accumulation_steps,4.0
+1773783471.2908254,train_step,300,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783471.2908254,train_step,300,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783485.8945208,eval_step,300,2,eval/loss,4.965634505857121
+1773783485.8945208,eval_step,300,2,eval/duration_sec,14.601576885208488
+1773783524.0127125,train_step,310,2,train/step_loss,5.0469924211502075
+1773783524.0127125,train_step,310,2,train/step_real_loss,5.0469924211502075
+1773783524.0127125,train_step,310,2,train/lr,1.6306531183346385e-05
+1773783524.0127125,train_step,310,2,perf/step_duration_sec,3.7034485950134695
+1773783524.0127125,train_step,310,2,perf/samples_per_sec,8.64059515854671
+1773783524.0127125,train_step,310,2,perf/tokens_per_sec,6898.975196902139
+1773783524.0127125,train_step,310,2,perf/logical_batch_size,32.0
+1773783524.0127125,train_step,310,2,perf/logical_token_count,25550.0
+1773783524.0127125,train_step,310,2,perf/gradient_accumulation_steps,4.0
+1773783524.0127125,train_step,310,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783524.0127125,train_step,310,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783561.666821,train_step,320,2,train/step_loss,5.018897533416748
+1773783561.666821,train_step,320,2,train/step_real_loss,5.018897533416748
+1773783561.666821,train_step,320,2,train/lr,1.3464133037968912e-05
+1773783561.666821,train_step,320,2,perf/step_duration_sec,3.5697252051904798
+1773783561.666821,train_step,320,2,perf/samples_per_sec,8.964275444359446
+1773783561.666821,train_step,320,2,perf/tokens_per_sec,7121.5565733283065
+1773783561.666821,train_step,320,2,perf/logical_batch_size,32.0
+1773783561.666821,train_step,320,2,perf/logical_token_count,25422.0
+1773783561.666821,train_step,320,2,perf/gradient_accumulation_steps,4.0
+1773783561.666821,train_step,320,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783561.666821,train_step,320,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783600.0391417,train_step,330,2,train/step_loss,5.0458667278289795
+1773783600.0391417,train_step,330,2,train/step_real_loss,5.0458667278289795
+1773783600.0391417,train_step,330,2,train/lr,1.0855082192715294e-05
+1773783600.0391417,train_step,330,2,perf/step_duration_sec,3.95492945285514
+1773783600.0391417,train_step,330,2,perf/samples_per_sec,8.091168346099975
+1773783600.0391417,train_step,330,2,perf/tokens_per_sec,6913.6505027316125
+1773783600.0391417,train_step,330,2,perf/logical_batch_size,32.0
+1773783600.0391417,train_step,330,2,perf/logical_token_count,27343.0
+1773783600.0391417,train_step,330,2,perf/gradient_accumulation_steps,4.0
+1773783600.0391417,train_step,330,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783600.0391417,train_step,330,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783637.8100953,train_step,340,2,train/step_loss,5.095871806144714
+1773783637.8100953,train_step,340,2,train/step_real_loss,5.095871806144714
+1773783637.8100953,train_step,340,2,train/lr,8.49604213531004e-06
+1773783637.8100953,train_step,340,2,perf/step_duration_sec,3.57716670492664
+1773783637.8100953,train_step,340,2,perf/samples_per_sec,8.945627263031415
+1773783637.8100953,train_step,340,2,perf/tokens_per_sec,7446.396043917744
+1773783637.8100953,train_step,340,2,perf/logical_batch_size,32.0
+1773783637.8100953,train_step,340,2,perf/logical_token_count,26637.0
+1773783637.8100953,train_step,340,2,perf/gradient_accumulation_steps,4.0
+1773783637.8100953,train_step,340,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783637.8100953,train_step,340,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783676.205935,train_step,350,2,train/step_loss,5.241442084312439
+1773783676.205935,train_step,350,2,train/step_real_loss,5.241442084312439
+1773783676.205935,train_step,350,2,train/lr,6.402079584406673e-06
+1773783676.205935,train_step,350,2,perf/step_duration_sec,3.8292608251795173
+1773783676.205935,train_step,350,2,perf/samples_per_sec,8.356704194601273
+1773783676.205935,train_step,350,2,perf/tokens_per_sec,5819.13873650938
+1773783676.205935,train_step,350,2,perf/logical_batch_size,32.0
+1773783676.205935,train_step,350,2,perf/logical_token_count,22283.0
+1773783676.205935,train_step,350,2,perf/gradient_accumulation_steps,4.0
+1773783676.205935,train_step,350,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783676.205935,train_step,350,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783690.815634,eval_step,350,2,eval/loss,4.8690267882563845
+1773783690.815634,eval_step,350,2,eval/duration_sec,14.607622059062123
+1773783728.3614442,train_step,360,2,train/step_loss,4.913779258728027
+1773783728.3614442,train_step,360,2,train/step_real_loss,4.913779258728027
+1773783728.3614442,train_step,360,2,train/lr,4.586568261458729e-06
+1773783728.3614442,train_step,360,2,perf/step_duration_sec,3.8354132031090558
+1773783728.3614442,train_step,360,2,perf/samples_per_sec,8.343299223682136
+1773783728.3614442,train_step,360,2,perf/tokens_per_sec,7633.075877266194
+1773783728.3614442,train_step,360,2,perf/logical_batch_size,32.0
+1773783728.3614442,train_step,360,2,perf/logical_token_count,29276.0
+1773783728.3614442,train_step,360,2,perf/gradient_accumulation_steps,4.0
+1773783728.3614442,train_step,360,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783728.3614442,train_step,360,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783766.5169628,train_step,370,2,train/step_loss,5.0288437604904175
+1773783766.5169628,train_step,370,2,train/step_real_loss,5.0288437604904175
+1773783766.5169628,train_step,370,2,train/lr,3.06110347542643e-06
+1773783766.5169628,train_step,370,2,perf/step_duration_sec,3.820353894960135
+1773783766.5169628,train_step,370,2,perf/samples_per_sec,8.376187358510125
+1773783766.5169628,train_step,370,2,perf/tokens_per_sec,7220.011747180773
+1773783766.5169628,train_step,370,2,perf/logical_batch_size,32.0
+1773783766.5169628,train_step,370,2,perf/logical_token_count,27583.0
+1773783766.5169628,train_step,370,2,perf/gradient_accumulation_steps,4.0
+1773783766.5169628,train_step,370,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783766.5169628,train_step,370,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783804.5456383,train_step,380,2,train/step_loss,4.826860070228577
+1773783804.5456383,train_step,380,2,train/step_real_loss,4.826860070228577
+1773783804.5456383,train_step,380,2,train/lr,1.8354280658494649e-06
+1773783804.5456383,train_step,380,2,perf/step_duration_sec,3.825985827948898
+1773783804.5456383,train_step,380,2,perf/samples_per_sec,8.363857431525073
+1773783804.5456383,train_step,380,2,perf/tokens_per_sec,7291.454086476718
+1773783804.5456383,train_step,380,2,perf/logical_batch_size,32.0
+1773783804.5456383,train_step,380,2,perf/logical_token_count,27897.0
+1773783804.5456383,train_step,380,2,perf/gradient_accumulation_steps,4.0
+1773783804.5456383,train_step,380,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783804.5456383,train_step,380,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783843.007424,train_step,390,2,train/step_loss,4.915419220924377
+1773783843.007424,train_step,390,2,train/step_real_loss,4.915419220924377
+1773783843.007424,train_step,390,2,train/lr,9.17370177272775e-07
+1773783843.007424,train_step,390,2,perf/step_duration_sec,3.7134576980024576
+1773783843.007424,train_step,390,2,perf/samples_per_sec,8.617305649452646
+1773783843.007424,train_step,390,2,perf/tokens_per_sec,7082.61737144544
+1773783843.007424,train_step,390,2,perf/logical_batch_size,32.0
+1773783843.007424,train_step,390,2,perf/logical_token_count,26301.0
+1773783843.007424,train_step,390,2,perf/gradient_accumulation_steps,4.0
+1773783843.007424,train_step,390,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783843.007424,train_step,390,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783881.5532339,train_step,400,2,train/step_loss,4.916181445121765
+1773783881.5532339,train_step,400,2,train/step_real_loss,4.916181445121765
+1773783881.5532339,train_step,400,2,train/lr,3.127932624475638e-07
+1773783881.5532339,train_step,400,2,perf/step_duration_sec,3.831934977322817
+1773783881.5532339,train_step,400,2,perf/samples_per_sec,8.350872389373583
+1773783881.5532339,train_step,400,2,perf/tokens_per_sec,7147.302906255116
+1773783881.5532339,train_step,400,2,perf/logical_batch_size,32.0
+1773783881.5532339,train_step,400,2,perf/logical_token_count,27388.0
+1773783881.5532339,train_step,400,2,perf/gradient_accumulation_steps,4.0
+1773783881.5532339,train_step,400,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783881.5532339,train_step,400,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783896.2133112,eval_step,400,2,eval/loss,4.840054991570385
+1773783896.2133112,eval_step,400,2,eval/duration_sec,14.658054957631975
+1773783934.3512156,train_step,410,2,train/step_loss,4.948451399803162
+1773783934.3512156,train_step,410,2,train/step_real_loss,4.948451399803162
+1773783934.3512156,train_step,410,2,train/lr,2.5558633627303928e-08
+1773783934.3512156,train_step,410,2,perf/step_duration_sec,3.8317640791647136
+1773783934.3512156,train_step,410,2,perf/samples_per_sec,8.35124484150801
+1773783934.3512156,train_step,410,2,perf/tokens_per_sec,6492.570911469883
+1773783934.3512156,train_step,410,2,perf/logical_batch_size,32.0
+1773783934.3512156,train_step,410,2,perf/logical_token_count,24878.0
+1773783934.3512156,train_step,410,2,perf/gradient_accumulation_steps,4.0
+1773783934.3512156,train_step,410,2,system/cuda_memory_allocated_gb,15.10503625869751
+1773783934.3512156,train_step,410,2,system/cuda_max_memory_allocated_gb,84.09655332565308
+1773783964.370691,train_epoch,414,2,train/epoch_loss,5.128793139947108
+1773783964.370691,train_epoch,414,2,train/epoch_real_loss,5.082210323228928
+1773783964.370691,train_epoch,414,2,train/epoch_canary_loss,10.210301459293396
+1773783964.370691,train_epoch,414,2,perf/epoch_duration_sec,848.1400936129503
+1773783964.370691,train_epoch,414,2,perf/epoch_samples_per_sec,63.06976925490229
+1773783964.370691,train_epoch,414,2,perf/epoch_tokens_per_sec,51703.841535419684
+1773783964.370691,train_epoch,414,2,perf/epoch_samples,53492.0
+1773783964.370691,train_epoch,414,2,perf/epoch_tokens,43852101.0
+1773783964.370691,train_epoch,414,2,system/cuda_epoch_peak_memory_gb,84.09655332565308
+1773783964.370691,train_epoch,414,2,eval/loss,4.839725652878935
+1773783964.370691,train_epoch,414,2,eval/duration_sec,14.596055098809302
+1773783972.8350441,audit_epoch,414,2,audit/delta,1e-05
+1773783972.8350441,audit_epoch,414,2,audit/num_canaries,500.0
+1773783972.8350441,audit_epoch,414,2,audit/num_members,250.0
+1773783972.8350441,audit_epoch,414,2,audit/paper_guess_fraction,0.2
+1773783972.8350441,audit_epoch,414,2,audit/paper_guess_steps,20.0
+1773783972.8350441,audit_epoch,414,2,audit/loss/auc,0.957184
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/embedding/auc,0.968208
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783972.8350441,audit_epoch,414,2,perf/audit_duration_sec,5.8983861412853
+1773783981.1943111,audit_final,414,2,audit/delta,1e-05
+1773783981.1943111,audit_final,414,2,audit/num_canaries,500.0
+1773783981.1943111,audit_final,414,2,audit/num_members,250.0
+1773783981.1943111,audit_final,414,2,audit/paper_guess_fraction,0.2
+1773783981.1943111,audit_final,414,2,audit/paper_guess_steps,20.0
+1773783981.1943111,audit_final,414,2,audit/loss/auc,0.957184
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon/0.05,3.4791953936219215
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon/0.01,3.023197554051876
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/embedding/auc,0.968208
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon/0.05,3.4791953936219215
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon/0.01,3.023197554051876
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,3.4791953936219215
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,3.023197554051876
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,100.0
+1773783981.1943111,audit_final,414,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,100.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/duration,1810.5507336058654
+1773783981.7373376,energy_final,414,,energy/codecarbon/emissions,0.09709380205154217
+1773783981.7373376,energy_final,414,,energy/codecarbon/emissions_rate,5.362666742741399e-05
+1773783981.7373376,energy_final,414,,energy/codecarbon/cpu_power,72.0230906963752
+1773783981.7373376,energy_final,414,,energy/codecarbon/gpu_power,4629.388481318127
+1773783981.7373376,energy_final,414,,energy/codecarbon/ram_power,54.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/cpu_energy,0.03488049743955748
+1773783981.7373376,energy_final,414,,energy/codecarbon/gpu_energy,2.3248590518302024
+1773783981.7373376,energy_final,414,,energy/codecarbon/ram_energy,0.026150659639004155
+1773783981.7373376,energy_final,414,,energy/codecarbon/energy_consumed,2.3858902089087644
+1773783981.7373376,energy_final,414,,energy/codecarbon/water_consumed,0.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/cpu_count,256.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/gpu_count,8.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/longitude,16.1885
+1773783981.7373376,energy_final,414,,energy/codecarbon/latitude,58.594
+1773783981.7373376,energy_final,414,,energy/codecarbon/ram_total_size,1511.49019241333
+1773783981.7373376,energy_final,414,,energy/codecarbon/cpu_utilization_percent,3.485983379501395
+1773783981.7373376,energy_final,414,,energy/codecarbon/gpu_utilization_percent,91.87222991689751
+1773783981.7373376,energy_final,414,,energy/codecarbon/ram_utilization_percent,5.226869806094248
+1773783981.7373376,energy_final,414,,energy/codecarbon/ram_used_gb,78.96254361435317
+1773783981.7373376,energy_final,414,,energy/codecarbon/pue,1.0
+1773783981.7373376,energy_final,414,,energy/codecarbon/wue,0.0

deepseek-coder-6.7b/base/summary.json

@@ -0,0 +1,71 @@
+{
+  "audit/delta": 1e-05,
+  "audit/embedding/auc": 0.968208,
+  "audit/embedding/empirical_epsilon/0.01": 3.023197554051876,
+  "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215,
+  "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
+  "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0,
+  "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
+  "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0,
+  "audit/loss/auc": 0.957184,
+  "audit/loss/empirical_epsilon/0.01": 3.023197554051876,
+  "audit/loss/empirical_epsilon/0.05": 3.4791953936219215,
+  "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
+  "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0,
+  "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
+  "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0,
+  "audit/num_canaries": 500.0,
+  "audit/num_members": 250.0,
+  "audit/paper_guess_fraction": 0.2,
+  "audit/paper_guess_steps": 20.0,
+  "energy/codecarbon/cpu_count": 256.0,
+  "energy/codecarbon/cpu_energy": 0.03488049743955748,
+  "energy/codecarbon/cpu_power": 72.0230906963752,
+  "energy/codecarbon/cpu_utilization_percent": 3.485983379501395,
+  "energy/codecarbon/duration": 1810.5507336058654,
+  "energy/codecarbon/emissions": 0.09709380205154217,
+  "energy/codecarbon/emissions_rate": 5.362666742741399e-05,
+  "energy/codecarbon/energy_consumed": 2.3858902089087644,
+  "energy/codecarbon/gpu_count": 8.0,
+  "energy/codecarbon/gpu_energy": 2.3248590518302024,
+  "energy/codecarbon/gpu_power": 4629.388481318127,
+  "energy/codecarbon/gpu_utilization_percent": 91.87222991689751,
+  "energy/codecarbon/latitude": 58.594,
+  "energy/codecarbon/longitude": 16.1885,
+  "energy/codecarbon/pue": 1.0,
+  "energy/codecarbon/ram_energy": 0.026150659639004155,
+  "energy/codecarbon/ram_power": 54.0,
+  "energy/codecarbon/ram_total_size": 1511.49019241333,
+  "energy/codecarbon/ram_used_gb": 78.96254361435317,
+  "energy/codecarbon/ram_utilization_percent": 5.226869806094248,
+  "energy/codecarbon/water_consumed": 0.0,
+  "energy/codecarbon/wue": 0.0,
+  "eval/duration_sec": 14.596055098809302,
+  "eval/loss": 4.839725652878935,
+  "perf/audit_duration_sec": 5.8983861412853,
+  "perf/epoch_duration_sec": 848.1400936129503,
+  "perf/epoch_samples": 53492.0,
+  "perf/epoch_samples_per_sec": 63.06976925490229,
+  "perf/epoch_tokens": 43852101.0,
+  "perf/epoch_tokens_per_sec": 51703.841535419684,
+  "perf/gradient_accumulation_steps": 4.0,
+  "perf/logical_batch_size": 32.0,
+  "perf/logical_token_count": 24878.0,
+  "perf/samples_per_sec": 8.35124484150801,
+  "perf/step_duration_sec": 3.8317640791647136,
+  "perf/tokens_per_sec": 6492.570911469883,
+  "system/cuda_epoch_peak_memory_gb": 84.09655332565308,
+  "system/cuda_max_memory_allocated_gb": 84.09655332565308,
+  "system/cuda_memory_allocated_gb": 15.10503625869751,
+  "train/epoch_canary_loss": 10.210301459293396,
+  "train/epoch_loss": 5.128793139947108,
+  "train/epoch_real_loss": 5.082210323228928,
+  "train/lr": 2.5558633627303928e-08,
+  "train/step_canary_loss": 10.625,
+  "train/step_loss": 4.948451399803162,
+  "train/step_real_loss": 4.948451399803162
+}

deepseek-coder-6.7b/base/tokenizer/chat_template.jinja

@@ -0,0 +1,26 @@
+{% if not add_generation_prompt is defined %}
+{% set add_generation_prompt = false %}
+{% endif %}
+{%- set ns = namespace(found=false) -%}
+{%- for message in messages -%}
+    {%- if message['role'] == 'system' -%}
+        {%- set ns.found = true -%}
+    {%- endif -%}
+{%- endfor -%}
+{{bos_token}}{%- if not ns.found -%}
+{{'You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer\n'}}
+{%- endif %}
+{%- for message in messages %}
+    {%- if message['role'] == 'system' %}
+{{ message['content'] }}
+    {%- else %}
+        {%- if message['role'] == 'user' %}
+{{'### Instruction:\n' + message['content'] + '\n'}}
+        {%- else %}
+{{'### Response:\n' + message['content'] + '\n<|EOT|>\n'}}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{% if add_generation_prompt %}
+{{'### Response:'}}
+{% endif %}

deepseek-coder-6.7b/base/tokenizer/tokenizer_config.json

@@ -0,0 +1,516 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|EOT|>",
+  "extra_special_tokens": [
+    "865331112869",
+    "569765693871",
+    "485177821815",
+    "135441121756",
+    "367459894796",
+    "877482678543",
+    "457919547633",
+    "765474393376",
+    "114848338811",
+    "746285987371",
+    "649291669397",
+    "927914615679",
+    "445925149649",
+    "691587454538",
+    "143777992227",
+    "997981281989",
+    "425949483533",
+    "982993456429",
+    "718726519731",
+    "172599315861",
+    "643489267333",
+    "282322838685",
+    "781653545886",
+    "796415361892",
+    "841991688488",
+    "211411365397",
+    "698218415444",
+    "355977139358",
+    "682564697312",
+    "383837596997",
+    "689362171782",
+    "749966767285",
+    "753159165157",
+    "795693824762",
+    "669689115557",
+    "327491773134",
+    "983569279932",
+    "612128769512",
+    "374327157578",
+    "311632789559",
+    "523918658846",
+    "765981581453",
+    "794825141891",
+    "873898736873",
+    "447445629421",
+    "473822473819",
+    "181439694557",
+    "592538279337",
+    "668134915514",
+    "643692393748",
+    "696651276628",
+    "853859348234",
+    "778466723723",
+    "929826356991",
+    "272362973463",
+    "694235616268",
+    "281673864127",
+    "479676316326",
+    "646979124677",
+    "922327493433",
+    "883685933161",
+    "264259917554",
+    "836746273134",
+    "658481324922",
+    "481884157827",
+    "587787496812",
+    "579184949249",
+    "912193598348",
+    "529679678956",
+    "795838284624",
+    "159337222655",
+    "173781362446",
+    "773687856563",
+    "535787224917",
+    "351885857332",
+    "578827344666",
+    "198462689911",
+    "722618266242",
+    "952872416512",
+    "517778845323",
+    "749665846687",
+    "661436365453",
+    "259666844669",
+    "242851284913",
+    "514532995959",
+    "161588262349",
+    "742765629356",
+    "225164373623",
+    "676539973863",
+    "826214551218",
+    "182345464792",
+    "232776999554",
+    "337326533813",
+    "676676697292",
+    "929185622831",
+    "545512344383",
+    "499444466686",
+    "314697386682",
+    "517379856925",
+    "379557332953",
+    "614797267726",
+    "429781429464",
+    "922466849763",
+    "721737645236",
+    "479227349997",
+    "136931728327",
+    "259533577263",
+    "488538864842",
+    "937495658852",
+    "489991411364",
+    "499148455254",
+    "441373944925",
+    "899151413682",
+    "467893531755",
+    "527117488925",
+    "928335588653",
+    "374439448821",
+    "879425227932",
+    "867678158885",
+    "399749397872",
+    "129693547287",
+    "689285841825",
+    "771619544974",
+    "724883568652",
+    "516968424863",
+    "733737988257",
+    "852347289392",
+    "296953381169",
+    "377273562477",
+    "262296912232",
+    "547149832394",
+    "298464134954",
+    "216667245274",
+    "843998562287",
+    "572154333646",
+    "124589118494",
+    "841824384614",
+    "232896526252",
+    "295448593321",
+    "123741461297",
+    "653573457168",
+    "196735786156",
+    "377338713663",
+    "964342468552",
+    "586855179568",
+    "484773717614",
+    "894885246797",
+    "677896358599",
+    "848845611563",
+    "851852651677",
+    "398549545767",
+    "454244839926",
+    "799364566435",
+    "967114116556",
+    "817378986438",
+    "233795848681",
+    "824387273757",
+    "916198946615",
+    "563117729724",
+    "951794811935",
+    "374598961236",
+    "922867396683",
+    "765737843639",
+    "175469284871",
+    "231853711778",
+    "662426712668",
+    "711412347158",
+    "753466987363",
+    "513361312532",
+    "712992815957",
+    "971621888444",
+    "829235161526",
+    "585544633356",
+    "582471228164",
+    "678666359123",
+    "557533689478",
+    "632962475133",
+    "484489193824",
+    "489562189822",
+    "589547936288",
+    "363214487524",
+    "244885399387",
+    "431751228368",
+    "433581868192",
+    "486391569221",
+    "185438575221",
+    "126574388585",
+    "741757479784",
+    "529854679937",
+    "996116119839",
+    "616248973917",
+    "763531783491",
+    "955456118295",
+    "364196983365",
+    "195792996468",
+    "151859598873",
+    "399223169721",
+    "938488813964",
+    "961981959227",
+    "183368827562",
+    "533417736566",
+    "786391632558",
+    "665661658354",
+    "693281533643",
+    "475794684356",
+    "652154162978",
+    "753233719644",
+    "668514843129",
+    "819162623892",
+    "941169431859",
+    "877385381798",
+    "752644929761",
+    "881136466196",
+    "275597777299",
+    "731681792655",
+    "961133895172",
+    "864718285734",
+    "963852916563",
+    "319584985416",
+    "563365646341",
+    "811371928234",
+    "837131396371",
+    "267514771964",
+    "944513428457",
+    "117298239631",
+    "158142752582",
+    "252867443568",
+    "839269684865",
+    "612788593128",
+    "145669731981",
+    "121557291859",
+    "245416776926",
+    "799417897197",
+    "997958836435",
+    "892336777248",
+    "158929292238",
+    "581976444672",
+    "897784492783",
+    "492373714791",
+    "512659818733",
+    "881112998642",
+    "619454958782",
+    "431149748713",
+    "624221476921",
+    "125866399464",
+    "339882449689",
+    "186198784585",
+    "943193294691",
+    "955668961269",
+    "232787996724",
+    "215671314196",
+    "286173241916",
+    "745977673725",
+    "556976448182",
+    "599961512792",
+    "766294538337",
+    "934912591213",
+    "295118729589",
+    "529455466433",
+    "196119929397",
+    "379571934299",
+    "251789649997",
+    "564544131355",
+    "244371196654",
+    "384598329253",
+    "887753195844",
+    "364947325679",
+    "655517954651",
+    "673948786567",
+    "857231548835",
+    "816115936673",
+    "644234165531",
+    "182782912224",
+    "234316622259",
+    "421369185549",
+    "434632855397",
+    "921889371893",
+    "415956914763",
+    "598916996413",
+    "773671349113",
+    "952465217972",
+    "117657531962",
+    "729825168745",
+    "691315125346",
+    "768461952319",
+    "664847713559",
+    "953267689786",
+    "886464195129",
+    "824488329416",
+    "837873762491",
+    "532833541879",
+    "669183782449",
+    "941976537588",
+    "739394546916",
+    "267954879268",
+    "637551427887",
+    "217756494954",
+    "524444658383",
+    "117783274348",
+    "138218735276",
+    "814611949491",
+    "711641973413",
+    "499156317423",
+    "515856611931",
+    "454164859837",
+    "345271433112",
+    "462294118988",
+    "511785788222",
+    "497294727353",
+    "866519986723",
+    "334513529294",
+    "549946382131",
+    "284445431422",
+    "396521188476",
+    "421435255895",
+    "133373659361",
+    "322683334381",
+    "228358422847",
+    "291762694874",
+    "143182978129",
+    "511923256573",
+    "327158398268",
+    "879764613759",
+    "564395222747",
+    "451161679736",
+    "538631466654",
+    "221762325616",
+    "218391991184",
+    "322589379462",
+    "876537814263",
+    "152676556624",
+    "332522971941",
+    "884354318946",
+    "513349618943",
+    "116639746413",
+    "635185846287",
+    "993832498489",
+    "813981174797",
+    "438745114173",
+    "983493951323",
+    "724492262421",
+    "622553389126",
+    "889965243135",
+    "364492359246",
+    "154962668224",
+    "179564995814",
+    "418412875665",
+    "718951851413",
+    "699446724178",
+    "624266421831",
+    "815458725125",
+    "455423278865",
+    "393741199486",
+    "328552864359",
+    "211662639865",
+    "218784516525",
+    "762486672996",
+    "142799718159",
+    "858146415154",
+    "767858144912",
+    "571317457151",
+    "635127952696",
+    "116427191984",
+    "268921994538",
+    "523937669294",
+    "165429152138",
+    "739246183345",
+    "591464355756",
+    "212985874612",
+    "191887635211",
+    "967214577653",
+    "119342152414",
+    "946444632795",
+    "618423867817",
+    "228565148417",
+    "729116422489",
+    "527874729936",
+    "739784153482",
+    "387763951128",
+    "331369926711",
+    "562716493614",
+    "739667844957",
+    "562389434565",
+    "256497188281",
+    "859927364588",
+    "417668946583",
+    "357621613582",
+    "438435178228",
+    "485692541169",
+    "825815739116",
+    "342221452223",
+    "697747991249",
+    "716763689965",
+    "141499982867",
+    "818479319499",
+    "336813343298",
+    "594688742928",
+    "472129283475",
+    "514354144759",
+    "349249721685",
+    "546276298359",
+    "353755529131",
+    "315534574435",
+    "523723475786",
+    "215826764872",
+    "367968398551",
+    "569853653352",
+    "389715484387",
+    "293847485454",
+    "714738141818",
+    "178478368922",
+    "581493616981",
+    "589439538674",
+    "846657726193",
+    "722339992679",
+    "138154781148",
+    "757785319772",
+    "492516914298",
+    "919181521716",
+    "985781138935",
+    "476969195485",
+    "313145133463",
+    "758963111966",
+    "147541537162",
+    "557163366873",
+    "144373897488",
+    "522515164754",
+    "724964923582",
+    "284776712475",
+    "375429755114",
+    "181233596124",
+    "948585673431",
+    "243165586174",
+    "396847976144",
+    "997724962668",
+    "558837194455",
+    "163165456396",
+    "378749551722",
+    "161238482259",
+    "754978243758",
+    "195388849133",
+    "229775525672",
+    "262437452884",
+    "441377892146",
+    "451885565366",
+    "981277526855",
+    "762495822823",
+    "368763327262",
+    "757422791351",
+    "636324136426",
+    "214193645583",
+    "412843856172",
+    "179386156569",
+    "756916173536",
+    "892697125149",
+    "625334487352",
+    "941861857715",
+    "887417525236",
+    "649516938598",
+    "717628619782",
+    "438124184139",
+    "547563892268",
+    "856317483891",
+    "313313831273",
+    "371496153876",
+    "587541149322",
+    "265847332563",
+    "449549215429",
+    "163497196769",
+    "861342291298",
+    "268433315926",
+    "774679513717",
+    "851254219729",
+    "583527834464",
+    "488496781997",
+    "556814553861",
+    "482829231639",
+    "618878266619",
+    "147444452794",
+    "949235426629",
+    "357299947518",
+    "175528632226",
+    "645527857972",
+    "186872457894",
+    "552738847828",
+    "626748382482",
+    "921894985642",
+    "943878645871",
+    "859289776479",
+    "614583493135",
+    "933775286797",
+    "332234613346",
+    "325196781219",
+    "142526557681",
+    "356722692178",
+    "449318681694",
+    "687284547244",
+    "947262995132",
+    "893974619684",
+    "797238311233"
+  ],
+  "is_local": false,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false
+}

deepseek-coder-6.7b/base/train.log

@@ -0,0 +1,49 @@
+2026-03-17 21:17:52,048 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=14.8217
+2026-03-17 21:18:29,786 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=13.9264
+2026-03-17 21:19:07,599 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=11.1213
+2026-03-17 21:19:46,064 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=8.4720
+2026-03-17 21:20:23,734 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=7.3598
+2026-03-17 21:20:38,185 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=6.9550 duration_sec=14.45
+2026-03-17 21:21:15,967 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=6.9297
+2026-03-17 21:21:54,374 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=6.7893
+2026-03-17 21:22:32,136 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=6.7460
+2026-03-17 21:23:10,082 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=6.6407
+2026-03-17 21:23:47,521 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=6.5150
+2026-03-17 21:24:02,007 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=6.3930 duration_sec=14.48
+2026-03-17 21:24:40,121 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=6.4560
+2026-03-17 21:25:17,354 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=6.2952
+2026-03-17 21:25:55,699 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=6.2335
+2026-03-17 21:26:34,009 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=6.1285
+2026-03-17 21:27:12,683 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=6.0463
+2026-03-17 21:27:27,265 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=5.9036 duration_sec=14.58
+2026-03-17 21:28:04,787 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=5.9513
+2026-03-17 21:28:42,167 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=5.8611
+2026-03-17 21:29:20,592 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=5.8208
+2026-03-17 21:29:59,226 [INFO] new_opacus_codex.train_steps: epoch=1 step=190 loss=5.8057
+2026-03-17 21:30:37,106 [INFO] new_opacus_codex.train_steps: epoch=1 step=200 loss=5.6750
+2026-03-17 21:30:51,629 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=200 eval_loss=5.4885 duration_sec=14.52
+2026-03-17 21:31:53,495 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=5.6623
+2026-03-17 21:32:32,250 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=5.4695
+2026-03-17 21:33:10,083 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=5.4676
+2026-03-17 21:33:48,497 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=5.3727
+2026-03-17 21:34:26,298 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=5.3441
+2026-03-17 21:34:40,943 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=5.1709 duration_sec=14.64
+2026-03-17 21:35:18,715 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=5.2466
+2026-03-17 21:35:56,484 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=5.1862
+2026-03-17 21:36:34,597 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=5.1289
+2026-03-17 21:37:12,839 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=5.1346
+2026-03-17 21:37:51,290 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=5.1520
+2026-03-17 21:38:05,894 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=4.9656 duration_sec=14.60
+2026-03-17 21:38:44,012 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=5.0725
+2026-03-17 21:39:21,666 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=5.0865
+2026-03-17 21:40:00,038 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=5.0690
+2026-03-17 21:40:37,809 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=5.0108
+2026-03-17 21:41:16,205 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=5.0537
+2026-03-17 21:41:30,815 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=4.8690 duration_sec=14.61
+2026-03-17 21:42:08,360 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=4.9427
+2026-03-17 21:42:46,516 [INFO] new_opacus_codex.train_steps: epoch=2 step=370 loss=4.9867
+2026-03-17 21:43:24,545 [INFO] new_opacus_codex.train_steps: epoch=2 step=380 loss=4.9544
+2026-03-17 21:44:03,007 [INFO] new_opacus_codex.train_steps: epoch=2 step=390 loss=5.0119
+2026-03-17 21:44:41,552 [INFO] new_opacus_codex.train_steps: epoch=2 step=400 loss=4.9797
+2026-03-17 21:44:56,213 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=400 eval_loss=4.8401 duration_sec=14.66
+2026-03-17 21:45:34,350 [INFO] new_opacus_codex.train_steps: epoch=2 step=410 loss=4.9406

deepseek-coder-6.7b/dp3/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "v_proj",
+    "q_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/dp3/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.52164,
+    "empirical_epsilon": {
+      "0.05": 0.019017613492906094,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.019017613492906094,
+        "num_guesses": 85,
+        "correct_guesses": 51,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.543272,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/dp3/codecarbon.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
+2026-03-17T23:04:51,codedp-deepseek-coder-6.7b-cpt-dp3,0be3f13a-f9c0-4f72-8f3f-769479cfe611,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,2107.2360566542484,0.10251235144047784,4.864777779250855e-05,72.02572178945746,4183.395055207508,54.0,0.040608159184490934,2.447988524778083,0.030443774090695673,2.5190404580532704,0.0,Sweden,SWE,östergötland county,,,Linux-6.8.0-94-generic-x86_64-with-glibc2.39,3.11.0,3.2.3,256,AMD EPYC 9554 64-Core Processor,8,8 x NVIDIA H200,16.1885,58.594,1511.49019241333,machine,3.5188995215311123,80.96321770334929,5.3144497607653856,80.42555782920437,N,1.0,0.0

deepseek-coder-6.7b/dp3/metrics.jsonl

@@ -0,0 +1,30 @@
+{"timestamp": 1773786740.0321612, "event": "train_step", "step": 10, "epoch": 1, "metrics": {"train/step_loss": 14.807524247602982, "train/step_real_loss": 14.926509380340576, "train/lr": 0.00018181818181818183, "train/step_canary_loss": 11.0, "perf/step_duration_sec": 9.13194552809, "perf/samples_per_sec": 7.227375568216326, "perf/tokens_per_sec": 5865.562802059686, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 53564.0, "perf/physical_batches": 9.0, "privacy/epsilon": 0.7131647471248268, "system/cuda_memory_allocated_gb": 14.177838802337646, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773786833.0391247, "event": "train_step", "step": 20, "epoch": 1, "metrics": {"train/step_loss": 14.44931131250718, "train/step_real_loss": 14.67465889453888, "train/lr": 0.00019897180218885507, "train/step_canary_loss": 10.84375, "perf/step_duration_sec": 9.05874504102394, "perf/samples_per_sec": 7.50655854558787, "perf/tokens_per_sec": 6064.305789733377, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 54935.0, "perf/physical_batches": 9.0, "privacy/epsilon": 0.9542480237478311, "system/cuda_memory_allocated_gb": 14.301010608673096, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773786926.3488383, "event": "train_step", "step": 30, "epoch": 1, "metrics": {"train/step_loss": 13.955331860166607, "train/step_real_loss": 14.391435980796814, "train/lr": 0.00019544467510209388, "train/step_canary_loss": 0.0, "perf/step_duration_sec": 9.485276577994227, "perf/samples_per_sec": 6.74729929841788, "perf/tokens_per_sec": 5592.9839856307335, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 53051.0, "perf/physical_batches": 10.0, "privacy/epsilon": 1.145336977893831, "system/cuda_memory_allocated_gb": 14.547617435455322, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773787018.6373072, "event": "train_step", "step": 40, "epoch": 1, "metrics": {"train/step_loss": 14.15506328235973, "train/step_real_loss": 14.120846509933472, "train/lr": 0.00018949541262593762, "train/step_canary_loss": 15.25, "perf/step_duration_sec": 8.954908549785614, "perf/samples_per_sec": 7.370259521140512, "perf/tokens_per_sec": 5682.023408404114, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 50882.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.311193108872294, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787111.6987886, "event": "train_step", "step": 50, "epoch": 1, "metrics": {"train/step_loss": 13.791429372934195, "train/step_real_loss": 13.741295456886292, "train/lr": 0.00018127499143005268, "train/step_canary_loss": 17.0, "perf/step_duration_sec": 9.314833164215088, "perf/samples_per_sec": 6.97811746642026, "perf/tokens_per_sec": 5992.48521320173, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55819.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.4583641061524852, "system/cuda_memory_allocated_gb": 14.1148681640625, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787126.3622315, "event": "eval_step", "step": 50, "epoch": 1, "metrics": {"eval/loss": 13.72979099913077, "eval/duration_sec": 14.660556460730731}}
+{"timestamp": 1773787219.2685702, "event": "train_step", "step": 60, "epoch": 1, "metrics": {"train/step_loss": 13.225338772365026, "train/step_real_loss": 13.441776752471924, "train/lr": 0.0001709920242324663, "train/step_canary_loss": 10.916666984558105, "perf/step_duration_sec": 9.050328846089542, "perf/samples_per_sec": 7.734525583591975, "perf/tokens_per_sec": 6221.431392996139, "perf/logical_batch_size": 70.0, "perf/logical_token_count": 56306.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.5929146459722179, "system/cuda_memory_allocated_gb": 14.425142288208008, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787312.4062448, "event": "train_step", "step": 70, "epoch": 1, "metrics": {"train/step_loss": 12.945215898401598, "train/step_real_loss": 13.073627829551697, "train/lr": 0.00015890746575622231, "train/step_canary_loss": 10.890625, "perf/step_duration_sec": 9.778116607107222, "perf/samples_per_sec": 6.9543044670355245, "perf/tokens_per_sec": 5356.041669817419, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 52372.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.7182200620091768, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787406.2219386, "event": "train_step", "step": 80, "epoch": 1, "metrics": {"train/step_loss": 12.60925786635455, "train/step_real_loss": 12.712766170501709, "train/lr": 0.00014532799038330385, "train/step_canary_loss": 10.953125, "perf/step_duration_sec": 9.040645513217896, "perf/samples_per_sec": 7.521586804900208, "perf/tokens_per_sec": 5688.089409635122, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 51424.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.8360751937320303, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787500.4468713, "event": "train_step", "step": 90, "epoch": 1, "metrics": {"train/step_loss": 12.172602767374977, "train/step_real_loss": 12.225615382194519, "train/lr": 0.00013059820956358998, "train/step_canary_loss": 11.041666984558105, "perf/step_duration_sec": 9.056960263755172, "perf/samples_per_sec": 7.397625477957065, "perf/tokens_per_sec": 6038.781048744861, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 54693.0, "perf/physical_batches": 9.0, "privacy/epsilon": 1.9476934830264792, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787593.4491894, "event": "train_step", "step": 100, "epoch": 1, "metrics": {"train/step_loss": 11.868438344606211, "train/step_real_loss": 11.924361228942871, "train/lr": 0.00011509192648058249, "train/step_canary_loss": 11.35714340209961, "perf/step_duration_sec": 9.823987863957882, "perf/samples_per_sec": 7.227207625172652, "perf/tokens_per_sec": 4852.001107908164, "perf/logical_batch_size": 71.0, "perf/logical_token_count": 47666.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.054057754637441, "system/cuda_memory_allocated_gb": 14.487197399139404, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787607.9792094, "event": "eval_step", "step": 100, "epoch": 1, "metrics": {"eval/loss": 11.81135509501804, "eval/duration_sec": 14.526865308173}}
+{"timestamp": 1773787658.5497303, "event": "train_epoch", "step": 104, "epoch": 1, "metrics": {"train/epoch_loss": 13.449225129287315, "train/epoch_real_loss": 13.57216826545721, "train/epoch_canary_loss": 10.867904580308076, "perf/epoch_duration_sec": 997.745806640014, "perf/epoch_samples_per_sec": 55.996226321558325, "perf/epoch_tokens_per_sec": 44201.770337193724, "perf/epoch_samples": 55870.0, "perf/epoch_tokens": 44102131.0, "system/cuda_epoch_peak_memory_gb": 73.2897834777832, "eval/loss": 11.659296675161881, "eval/duration_sec": 14.518914998974651, "privacy/epsilon": 2.0952814257505974}}
+{"timestamp": 1773787666.4096277, "event": "audit_epoch", "step": 104, "epoch": 1, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.536216, "audit/loss/empirical_epsilon/0.05": 0.0, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.542752, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.343048084992915}}
+{"timestamp": 1773787722.7301056, "event": "train_step", "step": 110, "epoch": 2, "metrics": {"train/step_loss": 11.533822839910334, "train/step_real_loss": 11.543668866157532, "train/lr": 9.920264990753837e-05, "train/step_canary_loss": 11.21875, "perf/step_duration_sec": 9.264618248213083, "perf/samples_per_sec": 7.123876908012889, "perf/tokens_per_sec": 5974.126350071172, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 55348.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.155895236118489, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.28975915908813}}
+{"timestamp": 1773787815.270247, "event": "train_step", "step": 120, "epoch": 2, "metrics": {"train/step_loss": 11.204551952988353, "train/step_real_loss": 11.20437467098236, "train/lr": 8.333360798744496e-05, "train/step_canary_loss": 11.208333969116211, "perf/step_duration_sec": 9.1695022219792, "perf/samples_per_sec": 7.306830662999536, "perf/tokens_per_sec": 5355.143475760137, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 49104.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.2538737991591966, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.28975915908813}}
+{"timestamp": 1773787907.84406, "event": "train_step", "step": 130, "epoch": 2, "metrics": {"train/step_loss": 10.907811731532, "train/step_real_loss": 10.883031368255615, "train/lr": 6.788751536089739e-05, "train/step_canary_loss": 11.225000381469727, "perf/step_duration_sec": 9.083537110127509, "perf/samples_per_sec": 7.59615986189673, "perf/tokens_per_sec": 6119.20217048794, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 55584.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.34829265801524, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773787999.754929, "event": "train_step", "step": 140, "epoch": 2, "metrics": {"train/step_loss": 10.733073462301226, "train/step_real_loss": 10.715678453445435, "train/lr": 5.325635332531864e-05, "train/step_canary_loss": 11.104166984558105, "perf/step_duration_sec": 9.54831570899114, "perf/samples_per_sec": 7.016944353537626, "perf/tokens_per_sec": 5555.115856721534, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 53042.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.4397183333948855, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773788092.0733988, "event": "train_step", "step": 150, "epoch": 2, "metrics": {"train/step_loss": 10.644248768903207, "train/step_real_loss": 10.618408799171448, "train/lr": 3.981142237826332e-05, "train/step_canary_loss": 10.975000381469727, "perf/step_duration_sec": 9.050548555329442, "perf/samples_per_sec": 7.623847281540647, "perf/tokens_per_sec": 6154.654567009547, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 55703.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.5283910517887054, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773788106.5788815, "event": "eval_step", "step": 150, "epoch": 2, "metrics": {"eval/loss": 10.528999336741187, "eval/duration_sec": 14.50253726914525}}
+{"timestamp": 1773788198.8056324, "event": "train_step", "step": 160, "epoch": 2, "metrics": {"train/step_loss": 10.509951504794033, "train/step_real_loss": 10.495614051818848, "train/lr": 2.789391958515183e-05, "train/step_canary_loss": 10.96875, "perf/step_duration_sec": 8.9364311741665, "perf/samples_per_sec": 7.38549860830275, "perf/tokens_per_sec": 6128.061519492174, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 54763.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.6145698431381854, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773788292.7653294, "event": "train_step", "step": 170, "epoch": 2, "metrics": {"train/step_loss": 10.498227048276076, "train/step_real_loss": 10.467870473861694, "train/lr": 1.7806279893114875e-05, "train/step_canary_loss": 11.145833969116211, "perf/step_duration_sec": 9.595276300795376, "perf/samples_per_sec": 6.982602470180687, "perf/tokens_per_sec": 5185.6766225559795, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 49758.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.6985357679843074, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773788385.7951636, "event": "train_step", "step": 180, "epoch": 2, "metrics": {"train/step_loss": 10.444608576157513, "train/step_real_loss": 10.407943487167358, "train/lr": 9.804501125681243e-06, "train/step_canary_loss": 11.03125, "perf/step_duration_sec": 9.077591832727194, "perf/samples_per_sec": 7.490973515116802, "perf/tokens_per_sec": 5571.85219736901, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 50579.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.780402267783889, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773788479.450152, "event": "train_step", "step": 190, "epoch": 2, "metrics": {"train/step_loss": 10.40689816682235, "train/step_real_loss": 10.370327711105347, "train/lr": 4.091647429802869e-06, "train/step_canary_loss": 10.875, "perf/step_duration_sec": 9.14594066562131, "perf/samples_per_sec": 7.5443305967820455, "perf/tokens_per_sec": 5864.897003063607, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 53640.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.860377969759561, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.29015684127808}}
+{"timestamp": 1773788572.2790332, "event": "train_step", "step": 200, "epoch": 2, "metrics": {"train/step_loss": 10.382469764122597, "train/step_real_loss": 10.374773979187012, "train/lr": 8.126960406835249e-07, "train/step_canary_loss": 10.875, "perf/step_duration_sec": 9.668661074247211, "perf/samples_per_sec": 6.722750906341063, "perf/tokens_per_sec": 5157.073933722741, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 49862.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.938565800812133, "system/cuda_memory_allocated_gb": 14.1148681640625, "system/cuda_max_memory_allocated_gb": 73.29015684127808}}
+{"timestamp": 1773788586.7941608, "event": "eval_step", "step": 200, "epoch": 2, "metrics": {"eval/loss": 10.326958019625057, "eval/duration_sec": 14.51232642121613}}
+{"timestamp": 1773788674.2453506, "event": "train_epoch", "step": 208, "epoch": 2, "metrics": {"train/epoch_loss": 10.700755941458253, "train/epoch_real_loss": 10.694620649826392, "train/epoch_canary_loss": 10.469284867902852, "perf/epoch_duration_sec": 993.2564477077685, "perf/epoch_samples_per_sec": 56.02984015702426, "perf/epoch_tokens_per_sec": 44390.36172556743, "perf/epoch_samples": 55652.0, "perf/epoch_tokens": 44091013.0, "system/cuda_epoch_peak_memory_gb": 73.29015684127808, "eval/loss": 10.326239856806668, "eval/duration_sec": 14.522059130016714, "privacy/epsilon": 2.9999680995370417}}
+{"timestamp": 1773788682.610862, "event": "audit_epoch", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.52164, "audit/loss/empirical_epsilon/0.05": 0.019017613492906094, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.019017613492906094, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 85.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 51.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.543272, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.763615260832012}}
+{"timestamp": 1773788690.7880108, "event": "audit_final", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.52164, "audit/loss/empirical_epsilon/0.05": 0.019017613492906094, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.019017613492906094, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 85.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 51.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.543272, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0}}
+{"timestamp": 1773788691.3375037, "event": "energy_final", "step": 208, "epoch": null, "metrics": {"energy/codecarbon/duration": 2107.2360566542484, "energy/codecarbon/emissions": 0.10251235144047784, "energy/codecarbon/emissions_rate": 4.864777779250855e-05, "energy/codecarbon/cpu_power": 72.02572178945746, "energy/codecarbon/gpu_power": 4183.395055207508, "energy/codecarbon/ram_power": 54.0, "energy/codecarbon/cpu_energy": 0.040608159184490934, "energy/codecarbon/gpu_energy": 2.447988524778083, "energy/codecarbon/ram_energy": 0.030443774090695673, "energy/codecarbon/energy_consumed": 2.5190404580532704, "energy/codecarbon/water_consumed": 0.0, "energy/codecarbon/cpu_count": 256.0, "energy/codecarbon/gpu_count": 8.0, "energy/codecarbon/longitude": 16.1885, "energy/codecarbon/latitude": 58.594, "energy/codecarbon/ram_total_size": 1511.49019241333, "energy/codecarbon/cpu_utilization_percent": 3.5188995215311123, "energy/codecarbon/gpu_utilization_percent": 80.96321770334929, "energy/codecarbon/ram_utilization_percent": 5.3144497607653856, "energy/codecarbon/ram_used_gb": 80.42555782920437, "energy/codecarbon/pue": 1.0, "energy/codecarbon/wue": 0.0}}

deepseek-coder-6.7b/dp3/resolved_config.yaml

@@ -0,0 +1,101 @@
+model:
+  name: deepseek-ai/deepseek-coder-6.7b-instruct
+  tokenizer_name: deepseek-ai/deepseek-coder-6.7b-instruct
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0002
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 8
+  num_workers: 4
+  output_dir: runs/cpt/deepseek-coder-6.7b/dp3
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: hooks
+  clipping: flat
+  secure_mode: false
+  enabled: true
+  target_epsilon: 3.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: deepseek-coder-6.7b-cpt-dp3
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-deepseek-coder-6.7b-cpt-dp3

deepseek-coder-6.7b/dp3/scalars.csv

@@ -0,0 +1,386 @@
+timestamp,event,step,epoch,key,value
+1773786740.0321612,train_step,10,1,train/step_loss,14.807524247602982
+1773786740.0321612,train_step,10,1,train/step_real_loss,14.926509380340576
+1773786740.0321612,train_step,10,1,train/lr,0.00018181818181818183
+1773786740.0321612,train_step,10,1,train/step_canary_loss,11.0
+1773786740.0321612,train_step,10,1,perf/step_duration_sec,9.13194552809
+1773786740.0321612,train_step,10,1,perf/samples_per_sec,7.227375568216326
+1773786740.0321612,train_step,10,1,perf/tokens_per_sec,5865.562802059686
+1773786740.0321612,train_step,10,1,perf/logical_batch_size,66.0
+1773786740.0321612,train_step,10,1,perf/logical_token_count,53564.0
+1773786740.0321612,train_step,10,1,perf/physical_batches,9.0
+1773786740.0321612,train_step,10,1,privacy/epsilon,0.7131647471248268
+1773786740.0321612,train_step,10,1,system/cuda_memory_allocated_gb,14.177838802337646
+1773786740.0321612,train_step,10,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773786833.0391247,train_step,20,1,train/step_loss,14.44931131250718
+1773786833.0391247,train_step,20,1,train/step_real_loss,14.67465889453888
+1773786833.0391247,train_step,20,1,train/lr,0.00019897180218885507
+1773786833.0391247,train_step,20,1,train/step_canary_loss,10.84375
+1773786833.0391247,train_step,20,1,perf/step_duration_sec,9.05874504102394
+1773786833.0391247,train_step,20,1,perf/samples_per_sec,7.50655854558787
+1773786833.0391247,train_step,20,1,perf/tokens_per_sec,6064.305789733377
+1773786833.0391247,train_step,20,1,perf/logical_batch_size,68.0
+1773786833.0391247,train_step,20,1,perf/logical_token_count,54935.0
+1773786833.0391247,train_step,20,1,perf/physical_batches,9.0
+1773786833.0391247,train_step,20,1,privacy/epsilon,0.9542480237478311
+1773786833.0391247,train_step,20,1,system/cuda_memory_allocated_gb,14.301010608673096
+1773786833.0391247,train_step,20,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773786926.3488383,train_step,30,1,train/step_loss,13.955331860166607
+1773786926.3488383,train_step,30,1,train/step_real_loss,14.391435980796814
+1773786926.3488383,train_step,30,1,train/lr,0.00019544467510209388
+1773786926.3488383,train_step,30,1,train/step_canary_loss,0.0
+1773786926.3488383,train_step,30,1,perf/step_duration_sec,9.485276577994227
+1773786926.3488383,train_step,30,1,perf/samples_per_sec,6.74729929841788
+1773786926.3488383,train_step,30,1,perf/tokens_per_sec,5592.9839856307335
+1773786926.3488383,train_step,30,1,perf/logical_batch_size,64.0
+1773786926.3488383,train_step,30,1,perf/logical_token_count,53051.0
+1773786926.3488383,train_step,30,1,perf/physical_batches,10.0
+1773786926.3488383,train_step,30,1,privacy/epsilon,1.145336977893831
+1773786926.3488383,train_step,30,1,system/cuda_memory_allocated_gb,14.547617435455322
+1773786926.3488383,train_step,30,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773787018.6373072,train_step,40,1,train/step_loss,14.15506328235973
+1773787018.6373072,train_step,40,1,train/step_real_loss,14.120846509933472
+1773787018.6373072,train_step,40,1,train/lr,0.00018949541262593762
+1773787018.6373072,train_step,40,1,train/step_canary_loss,15.25
+1773787018.6373072,train_step,40,1,perf/step_duration_sec,8.954908549785614
+1773787018.6373072,train_step,40,1,perf/samples_per_sec,7.370259521140512
+1773787018.6373072,train_step,40,1,perf/tokens_per_sec,5682.023408404114
+1773787018.6373072,train_step,40,1,perf/logical_batch_size,66.0
+1773787018.6373072,train_step,40,1,perf/logical_token_count,50882.0
+1773787018.6373072,train_step,40,1,perf/physical_batches,9.0
+1773787018.6373072,train_step,40,1,privacy/epsilon,1.311193108872294
+1773787018.6373072,train_step,40,1,system/cuda_memory_allocated_gb,14.17786169052124
+1773787018.6373072,train_step,40,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787111.6987886,train_step,50,1,train/step_loss,13.791429372934195
+1773787111.6987886,train_step,50,1,train/step_real_loss,13.741295456886292
+1773787111.6987886,train_step,50,1,train/lr,0.00018127499143005268
+1773787111.6987886,train_step,50,1,train/step_canary_loss,17.0
+1773787111.6987886,train_step,50,1,perf/step_duration_sec,9.314833164215088
+1773787111.6987886,train_step,50,1,perf/samples_per_sec,6.97811746642026
+1773787111.6987886,train_step,50,1,perf/tokens_per_sec,5992.48521320173
+1773787111.6987886,train_step,50,1,perf/logical_batch_size,65.0
+1773787111.6987886,train_step,50,1,perf/logical_token_count,55819.0
+1773787111.6987886,train_step,50,1,perf/physical_batches,9.0
+1773787111.6987886,train_step,50,1,privacy/epsilon,1.4583641061524852
+1773787111.6987886,train_step,50,1,system/cuda_memory_allocated_gb,14.1148681640625
+1773787111.6987886,train_step,50,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787126.3622315,eval_step,50,1,eval/loss,13.72979099913077
+1773787126.3622315,eval_step,50,1,eval/duration_sec,14.660556460730731
+1773787219.2685702,train_step,60,1,train/step_loss,13.225338772365026
+1773787219.2685702,train_step,60,1,train/step_real_loss,13.441776752471924
+1773787219.2685702,train_step,60,1,train/lr,0.0001709920242324663
+1773787219.2685702,train_step,60,1,train/step_canary_loss,10.916666984558105
+1773787219.2685702,train_step,60,1,perf/step_duration_sec,9.050328846089542
+1773787219.2685702,train_step,60,1,perf/samples_per_sec,7.734525583591975
+1773787219.2685702,train_step,60,1,perf/tokens_per_sec,6221.431392996139
+1773787219.2685702,train_step,60,1,perf/logical_batch_size,70.0
+1773787219.2685702,train_step,60,1,perf/logical_token_count,56306.0
+1773787219.2685702,train_step,60,1,perf/physical_batches,9.0
+1773787219.2685702,train_step,60,1,privacy/epsilon,1.5929146459722179
+1773787219.2685702,train_step,60,1,system/cuda_memory_allocated_gb,14.425142288208008
+1773787219.2685702,train_step,60,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787312.4062448,train_step,70,1,train/step_loss,12.945215898401598
+1773787312.4062448,train_step,70,1,train/step_real_loss,13.073627829551697
+1773787312.4062448,train_step,70,1,train/lr,0.00015890746575622231
+1773787312.4062448,train_step,70,1,train/step_canary_loss,10.890625
+1773787312.4062448,train_step,70,1,perf/step_duration_sec,9.778116607107222
+1773787312.4062448,train_step,70,1,perf/samples_per_sec,6.9543044670355245
+1773787312.4062448,train_step,70,1,perf/tokens_per_sec,5356.041669817419
+1773787312.4062448,train_step,70,1,perf/logical_batch_size,68.0
+1773787312.4062448,train_step,70,1,perf/logical_token_count,52372.0
+1773787312.4062448,train_step,70,1,perf/physical_batches,9.0
+1773787312.4062448,train_step,70,1,privacy/epsilon,1.7182200620091768
+1773787312.4062448,train_step,70,1,system/cuda_memory_allocated_gb,14.30103349685669
+1773787312.4062448,train_step,70,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787406.2219386,train_step,80,1,train/step_loss,12.60925786635455
+1773787406.2219386,train_step,80,1,train/step_real_loss,12.712766170501709
+1773787406.2219386,train_step,80,1,train/lr,0.00014532799038330385
+1773787406.2219386,train_step,80,1,train/step_canary_loss,10.953125
+1773787406.2219386,train_step,80,1,perf/step_duration_sec,9.040645513217896
+1773787406.2219386,train_step,80,1,perf/samples_per_sec,7.521586804900208
+1773787406.2219386,train_step,80,1,perf/tokens_per_sec,5688.089409635122
+1773787406.2219386,train_step,80,1,perf/logical_batch_size,68.0
+1773787406.2219386,train_step,80,1,perf/logical_token_count,51424.0
+1773787406.2219386,train_step,80,1,perf/physical_batches,9.0
+1773787406.2219386,train_step,80,1,privacy/epsilon,1.8360751937320303
+1773787406.2219386,train_step,80,1,system/cuda_memory_allocated_gb,14.30103349685669
+1773787406.2219386,train_step,80,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787500.4468713,train_step,90,1,train/step_loss,12.172602767374977
+1773787500.4468713,train_step,90,1,train/step_real_loss,12.225615382194519
+1773787500.4468713,train_step,90,1,train/lr,0.00013059820956358998
+1773787500.4468713,train_step,90,1,train/step_canary_loss,11.041666984558105
+1773787500.4468713,train_step,90,1,perf/step_duration_sec,9.056960263755172
+1773787500.4468713,train_step,90,1,perf/samples_per_sec,7.397625477957065
+1773787500.4468713,train_step,90,1,perf/tokens_per_sec,6038.781048744861
+1773787500.4468713,train_step,90,1,perf/logical_batch_size,67.0
+1773787500.4468713,train_step,90,1,perf/logical_token_count,54693.0
+1773787500.4468713,train_step,90,1,perf/physical_batches,9.0
+1773787500.4468713,train_step,90,1,privacy/epsilon,1.9476934830264792
+1773787500.4468713,train_step,90,1,system/cuda_memory_allocated_gb,14.238978385925293
+1773787500.4468713,train_step,90,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787593.4491894,train_step,100,1,train/step_loss,11.868438344606211
+1773787593.4491894,train_step,100,1,train/step_real_loss,11.924361228942871
+1773787593.4491894,train_step,100,1,train/lr,0.00011509192648058249
+1773787593.4491894,train_step,100,1,train/step_canary_loss,11.35714340209961
+1773787593.4491894,train_step,100,1,perf/step_duration_sec,9.823987863957882
+1773787593.4491894,train_step,100,1,perf/samples_per_sec,7.227207625172652
+1773787593.4491894,train_step,100,1,perf/tokens_per_sec,4852.001107908164
+1773787593.4491894,train_step,100,1,perf/logical_batch_size,71.0
+1773787593.4491894,train_step,100,1,perf/logical_token_count,47666.0
+1773787593.4491894,train_step,100,1,perf/physical_batches,9.0
+1773787593.4491894,train_step,100,1,privacy/epsilon,2.054057754637441
+1773787593.4491894,train_step,100,1,system/cuda_memory_allocated_gb,14.487197399139404
+1773787593.4491894,train_step,100,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787607.9792094,eval_step,100,1,eval/loss,11.81135509501804
+1773787607.9792094,eval_step,100,1,eval/duration_sec,14.526865308173
+1773787658.5497303,train_epoch,104,1,train/epoch_loss,13.449225129287315
+1773787658.5497303,train_epoch,104,1,train/epoch_real_loss,13.57216826545721
+1773787658.5497303,train_epoch,104,1,train/epoch_canary_loss,10.867904580308076
+1773787658.5497303,train_epoch,104,1,perf/epoch_duration_sec,997.745806640014
+1773787658.5497303,train_epoch,104,1,perf/epoch_samples_per_sec,55.996226321558325
+1773787658.5497303,train_epoch,104,1,perf/epoch_tokens_per_sec,44201.770337193724
+1773787658.5497303,train_epoch,104,1,perf/epoch_samples,55870.0
+1773787658.5497303,train_epoch,104,1,perf/epoch_tokens,44102131.0
+1773787658.5497303,train_epoch,104,1,system/cuda_epoch_peak_memory_gb,73.2897834777832
+1773787658.5497303,train_epoch,104,1,eval/loss,11.659296675161881
+1773787658.5497303,train_epoch,104,1,eval/duration_sec,14.518914998974651
+1773787658.5497303,train_epoch,104,1,privacy/epsilon,2.0952814257505974
+1773787666.4096277,audit_epoch,104,1,audit/delta,1e-05
+1773787666.4096277,audit_epoch,104,1,audit/num_canaries,500.0
+1773787666.4096277,audit_epoch,104,1,audit/num_members,250.0
+1773787666.4096277,audit_epoch,104,1,audit/paper_guess_fraction,0.2
+1773787666.4096277,audit_epoch,104,1,audit/paper_guess_steps,20.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/auc,0.536216
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon/0.05,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon/0.01,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/epsilon,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/num_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/auc,0.542752
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.05,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.01,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773787666.4096277,audit_epoch,104,1,perf/audit_duration_sec,5.343048084992915
+1773787722.7301056,train_step,110,2,train/step_loss,11.533822839910334
+1773787722.7301056,train_step,110,2,train/step_real_loss,11.543668866157532
+1773787722.7301056,train_step,110,2,train/lr,9.920264990753837e-05
+1773787722.7301056,train_step,110,2,train/step_canary_loss,11.21875
+1773787722.7301056,train_step,110,2,perf/step_duration_sec,9.264618248213083
+1773787722.7301056,train_step,110,2,perf/samples_per_sec,7.123876908012889
+1773787722.7301056,train_step,110,2,perf/tokens_per_sec,5974.126350071172
+1773787722.7301056,train_step,110,2,perf/logical_batch_size,66.0
+1773787722.7301056,train_step,110,2,perf/logical_token_count,55348.0
+1773787722.7301056,train_step,110,2,perf/physical_batches,9.0
+1773787722.7301056,train_step,110,2,privacy/epsilon,2.155895236118489
+1773787722.7301056,train_step,110,2,system/cuda_memory_allocated_gb,14.17786169052124
+1773787722.7301056,train_step,110,2,system/cuda_max_memory_allocated_gb,73.28975915908813
+1773787815.270247,train_step,120,2,train/step_loss,11.204551952988353
+1773787815.270247,train_step,120,2,train/step_real_loss,11.20437467098236
+1773787815.270247,train_step,120,2,train/lr,8.333360798744496e-05
+1773787815.270247,train_step,120,2,train/step_canary_loss,11.208333969116211
+1773787815.270247,train_step,120,2,perf/step_duration_sec,9.1695022219792
+1773787815.270247,train_step,120,2,perf/samples_per_sec,7.306830662999536
+1773787815.270247,train_step,120,2,perf/tokens_per_sec,5355.143475760137
+1773787815.270247,train_step,120,2,perf/logical_batch_size,67.0
+1773787815.270247,train_step,120,2,perf/logical_token_count,49104.0
+1773787815.270247,train_step,120,2,perf/physical_batches,9.0
+1773787815.270247,train_step,120,2,privacy/epsilon,2.2538737991591966
+1773787815.270247,train_step,120,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773787815.270247,train_step,120,2,system/cuda_max_memory_allocated_gb,73.28975915908813
+1773787907.84406,train_step,130,2,train/step_loss,10.907811731532
+1773787907.84406,train_step,130,2,train/step_real_loss,10.883031368255615
+1773787907.84406,train_step,130,2,train/lr,6.788751536089739e-05
+1773787907.84406,train_step,130,2,train/step_canary_loss,11.225000381469727
+1773787907.84406,train_step,130,2,perf/step_duration_sec,9.083537110127509
+1773787907.84406,train_step,130,2,perf/samples_per_sec,7.59615986189673
+1773787907.84406,train_step,130,2,perf/tokens_per_sec,6119.20217048794
+1773787907.84406,train_step,130,2,perf/logical_batch_size,69.0
+1773787907.84406,train_step,130,2,perf/logical_token_count,55584.0
+1773787907.84406,train_step,130,2,perf/physical_batches,9.0
+1773787907.84406,train_step,130,2,privacy/epsilon,2.34829265801524
+1773787907.84406,train_step,130,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773787907.84406,train_step,130,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773787999.754929,train_step,140,2,train/step_loss,10.733073462301226
+1773787999.754929,train_step,140,2,train/step_real_loss,10.715678453445435
+1773787999.754929,train_step,140,2,train/lr,5.325635332531864e-05
+1773787999.754929,train_step,140,2,train/step_canary_loss,11.104166984558105
+1773787999.754929,train_step,140,2,perf/step_duration_sec,9.54831570899114
+1773787999.754929,train_step,140,2,perf/samples_per_sec,7.016944353537626
+1773787999.754929,train_step,140,2,perf/tokens_per_sec,5555.115856721534
+1773787999.754929,train_step,140,2,perf/logical_batch_size,67.0
+1773787999.754929,train_step,140,2,perf/logical_token_count,53042.0
+1773787999.754929,train_step,140,2,perf/physical_batches,9.0
+1773787999.754929,train_step,140,2,privacy/epsilon,2.4397183333948855
+1773787999.754929,train_step,140,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773787999.754929,train_step,140,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773788092.0733988,train_step,150,2,train/step_loss,10.644248768903207
+1773788092.0733988,train_step,150,2,train/step_real_loss,10.618408799171448
+1773788092.0733988,train_step,150,2,train/lr,3.981142237826332e-05
+1773788092.0733988,train_step,150,2,train/step_canary_loss,10.975000381469727
+1773788092.0733988,train_step,150,2,perf/step_duration_sec,9.050548555329442
+1773788092.0733988,train_step,150,2,perf/samples_per_sec,7.623847281540647
+1773788092.0733988,train_step,150,2,perf/tokens_per_sec,6154.654567009547
+1773788092.0733988,train_step,150,2,perf/logical_batch_size,69.0
+1773788092.0733988,train_step,150,2,perf/logical_token_count,55703.0
+1773788092.0733988,train_step,150,2,perf/physical_batches,9.0
+1773788092.0733988,train_step,150,2,privacy/epsilon,2.5283910517887054
+1773788092.0733988,train_step,150,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773788092.0733988,train_step,150,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773788106.5788815,eval_step,150,2,eval/loss,10.528999336741187
+1773788106.5788815,eval_step,150,2,eval/duration_sec,14.50253726914525
+1773788198.8056324,train_step,160,2,train/step_loss,10.509951504794033
+1773788198.8056324,train_step,160,2,train/step_real_loss,10.495614051818848
+1773788198.8056324,train_step,160,2,train/lr,2.789391958515183e-05
+1773788198.8056324,train_step,160,2,train/step_canary_loss,10.96875
+1773788198.8056324,train_step,160,2,perf/step_duration_sec,8.9364311741665
+1773788198.8056324,train_step,160,2,perf/samples_per_sec,7.38549860830275
+1773788198.8056324,train_step,160,2,perf/tokens_per_sec,6128.061519492174
+1773788198.8056324,train_step,160,2,perf/logical_batch_size,66.0
+1773788198.8056324,train_step,160,2,perf/logical_token_count,54763.0
+1773788198.8056324,train_step,160,2,perf/physical_batches,9.0
+1773788198.8056324,train_step,160,2,privacy/epsilon,2.6145698431381854
+1773788198.8056324,train_step,160,2,system/cuda_memory_allocated_gb,14.17786169052124
+1773788198.8056324,train_step,160,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773788292.7653294,train_step,170,2,train/step_loss,10.498227048276076
+1773788292.7653294,train_step,170,2,train/step_real_loss,10.467870473861694
+1773788292.7653294,train_step,170,2,train/lr,1.7806279893114875e-05
+1773788292.7653294,train_step,170,2,train/step_canary_loss,11.145833969116211
+1773788292.7653294,train_step,170,2,perf/step_duration_sec,9.595276300795376
+1773788292.7653294,train_step,170,2,perf/samples_per_sec,6.982602470180687
+1773788292.7653294,train_step,170,2,perf/tokens_per_sec,5185.6766225559795
+1773788292.7653294,train_step,170,2,perf/logical_batch_size,67.0
+1773788292.7653294,train_step,170,2,perf/logical_token_count,49758.0
+1773788292.7653294,train_step,170,2,perf/physical_batches,9.0
+1773788292.7653294,train_step,170,2,privacy/epsilon,2.6985357679843074
+1773788292.7653294,train_step,170,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773788292.7653294,train_step,170,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773788385.7951636,train_step,180,2,train/step_loss,10.444608576157513
+1773788385.7951636,train_step,180,2,train/step_real_loss,10.407943487167358
+1773788385.7951636,train_step,180,2,train/lr,9.804501125681243e-06
+1773788385.7951636,train_step,180,2,train/step_canary_loss,11.03125
+1773788385.7951636,train_step,180,2,perf/step_duration_sec,9.077591832727194
+1773788385.7951636,train_step,180,2,perf/samples_per_sec,7.490973515116802
+1773788385.7951636,train_step,180,2,perf/tokens_per_sec,5571.85219736901
+1773788385.7951636,train_step,180,2,perf/logical_batch_size,68.0
+1773788385.7951636,train_step,180,2,perf/logical_token_count,50579.0
+1773788385.7951636,train_step,180,2,perf/physical_batches,9.0
+1773788385.7951636,train_step,180,2,privacy/epsilon,2.780402267783889
+1773788385.7951636,train_step,180,2,system/cuda_memory_allocated_gb,14.30103349685669
+1773788385.7951636,train_step,180,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773788479.450152,train_step,190,2,train/step_loss,10.40689816682235
+1773788479.450152,train_step,190,2,train/step_real_loss,10.370327711105347
+1773788479.450152,train_step,190,2,train/lr,4.091647429802869e-06
+1773788479.450152,train_step,190,2,train/step_canary_loss,10.875
+1773788479.450152,train_step,190,2,perf/step_duration_sec,9.14594066562131
+1773788479.450152,train_step,190,2,perf/samples_per_sec,7.5443305967820455
+1773788479.450152,train_step,190,2,perf/tokens_per_sec,5864.897003063607
+1773788479.450152,train_step,190,2,perf/logical_batch_size,69.0
+1773788479.450152,train_step,190,2,perf/logical_token_count,53640.0
+1773788479.450152,train_step,190,2,perf/physical_batches,9.0
+1773788479.450152,train_step,190,2,privacy/epsilon,2.860377969759561
+1773788479.450152,train_step,190,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773788479.450152,train_step,190,2,system/cuda_max_memory_allocated_gb,73.29015684127808
+1773788572.2790332,train_step,200,2,train/step_loss,10.382469764122597
+1773788572.2790332,train_step,200,2,train/step_real_loss,10.374773979187012
+1773788572.2790332,train_step,200,2,train/lr,8.126960406835249e-07
+1773788572.2790332,train_step,200,2,train/step_canary_loss,10.875
+1773788572.2790332,train_step,200,2,perf/step_duration_sec,9.668661074247211
+1773788572.2790332,train_step,200,2,perf/samples_per_sec,6.722750906341063
+1773788572.2790332,train_step,200,2,perf/tokens_per_sec,5157.073933722741
+1773788572.2790332,train_step,200,2,perf/logical_batch_size,65.0
+1773788572.2790332,train_step,200,2,perf/logical_token_count,49862.0
+1773788572.2790332,train_step,200,2,perf/physical_batches,9.0
+1773788572.2790332,train_step,200,2,privacy/epsilon,2.938565800812133
+1773788572.2790332,train_step,200,2,system/cuda_memory_allocated_gb,14.1148681640625
+1773788572.2790332,train_step,200,2,system/cuda_max_memory_allocated_gb,73.29015684127808
+1773788586.7941608,eval_step,200,2,eval/loss,10.326958019625057
+1773788586.7941608,eval_step,200,2,eval/duration_sec,14.51232642121613
+1773788674.2453506,train_epoch,208,2,train/epoch_loss,10.700755941458253
+1773788674.2453506,train_epoch,208,2,train/epoch_real_loss,10.694620649826392
+1773788674.2453506,train_epoch,208,2,train/epoch_canary_loss,10.469284867902852
+1773788674.2453506,train_epoch,208,2,perf/epoch_duration_sec,993.2564477077685
+1773788674.2453506,train_epoch,208,2,perf/epoch_samples_per_sec,56.02984015702426
+1773788674.2453506,train_epoch,208,2,perf/epoch_tokens_per_sec,44390.36172556743
+1773788674.2453506,train_epoch,208,2,perf/epoch_samples,55652.0
+1773788674.2453506,train_epoch,208,2,perf/epoch_tokens,44091013.0
+1773788674.2453506,train_epoch,208,2,system/cuda_epoch_peak_memory_gb,73.29015684127808
+1773788674.2453506,train_epoch,208,2,eval/loss,10.326239856806668
+1773788674.2453506,train_epoch,208,2,eval/duration_sec,14.522059130016714
+1773788674.2453506,train_epoch,208,2,privacy/epsilon,2.9999680995370417
+1773788682.610862,audit_epoch,208,2,audit/delta,1e-05
+1773788682.610862,audit_epoch,208,2,audit/num_canaries,500.0
+1773788682.610862,audit_epoch,208,2,audit/num_members,250.0
+1773788682.610862,audit_epoch,208,2,audit/paper_guess_fraction,0.2
+1773788682.610862,audit_epoch,208,2,audit/paper_guess_steps,20.0
+1773788682.610862,audit_epoch,208,2,audit/loss/auc,0.52164
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon/0.05,0.019017613492906094
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.019017613492906094
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,85.0
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,51.0
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773788682.610862,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/auc,0.543272
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773788682.610862,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773788682.610862,audit_epoch,208,2,perf/audit_duration_sec,5.763615260832012
+1773788690.7880108,audit_final,208,2,audit/delta,1e-05
+1773788690.7880108,audit_final,208,2,audit/num_canaries,500.0
+1773788690.7880108,audit_final,208,2,audit/num_members,250.0
+1773788690.7880108,audit_final,208,2,audit/paper_guess_fraction,0.2
+1773788690.7880108,audit_final,208,2,audit/paper_guess_steps,20.0
+1773788690.7880108,audit_final,208,2,audit/loss/auc,0.52164
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon/0.05,0.019017613492906094
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.019017613492906094
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,85.0
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,51.0
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773788690.7880108,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/auc,0.543272
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773788690.7880108,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/duration,2107.2360566542484
+1773788691.3375037,energy_final,208,,energy/codecarbon/emissions,0.10251235144047784
+1773788691.3375037,energy_final,208,,energy/codecarbon/emissions_rate,4.864777779250855e-05
+1773788691.3375037,energy_final,208,,energy/codecarbon/cpu_power,72.02572178945746
+1773788691.3375037,energy_final,208,,energy/codecarbon/gpu_power,4183.395055207508
+1773788691.3375037,energy_final,208,,energy/codecarbon/ram_power,54.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/cpu_energy,0.040608159184490934
+1773788691.3375037,energy_final,208,,energy/codecarbon/gpu_energy,2.447988524778083
+1773788691.3375037,energy_final,208,,energy/codecarbon/ram_energy,0.030443774090695673
+1773788691.3375037,energy_final,208,,energy/codecarbon/energy_consumed,2.5190404580532704
+1773788691.3375037,energy_final,208,,energy/codecarbon/water_consumed,0.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/cpu_count,256.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/gpu_count,8.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/longitude,16.1885
+1773788691.3375037,energy_final,208,,energy/codecarbon/latitude,58.594
+1773788691.3375037,energy_final,208,,energy/codecarbon/ram_total_size,1511.49019241333
+1773788691.3375037,energy_final,208,,energy/codecarbon/cpu_utilization_percent,3.5188995215311123
+1773788691.3375037,energy_final,208,,energy/codecarbon/gpu_utilization_percent,80.96321770334929
+1773788691.3375037,energy_final,208,,energy/codecarbon/ram_utilization_percent,5.3144497607653856
+1773788691.3375037,energy_final,208,,energy/codecarbon/ram_used_gb,80.42555782920437
+1773788691.3375037,energy_final,208,,energy/codecarbon/pue,1.0
+1773788691.3375037,energy_final,208,,energy/codecarbon/wue,0.0

deepseek-coder-6.7b/dp3/summary.json

@@ -0,0 +1,72 @@
+{
+  "audit/delta": 1e-05,
+  "audit/embedding/auc": 0.543272,
+  "audit/embedding/empirical_epsilon/0.01": 0.0,
+  "audit/embedding/empirical_epsilon/0.05": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0,
+  "audit/loss/auc": 0.52164,
+  "audit/loss/empirical_epsilon/0.01": 0.0,
+  "audit/loss/empirical_epsilon/0.05": 0.019017613492906094,
+  "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 51.0,
+  "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.019017613492906094,
+  "audit/loss/empirical_epsilon_details/0.05/num_guesses": 85.0,
+  "audit/num_canaries": 500.0,
+  "audit/num_members": 250.0,
+  "audit/paper_guess_fraction": 0.2,
+  "audit/paper_guess_steps": 20.0,
+  "energy/codecarbon/cpu_count": 256.0,
+  "energy/codecarbon/cpu_energy": 0.040608159184490934,
+  "energy/codecarbon/cpu_power": 72.02572178945746,
+  "energy/codecarbon/cpu_utilization_percent": 3.5188995215311123,
+  "energy/codecarbon/duration": 2107.2360566542484,
+  "energy/codecarbon/emissions": 0.10251235144047784,
+  "energy/codecarbon/emissions_rate": 4.864777779250855e-05,
+  "energy/codecarbon/energy_consumed": 2.5190404580532704,
+  "energy/codecarbon/gpu_count": 8.0,
+  "energy/codecarbon/gpu_energy": 2.447988524778083,
+  "energy/codecarbon/gpu_power": 4183.395055207508,
+  "energy/codecarbon/gpu_utilization_percent": 80.96321770334929,
+  "energy/codecarbon/latitude": 58.594,
+  "energy/codecarbon/longitude": 16.1885,
+  "energy/codecarbon/pue": 1.0,
+  "energy/codecarbon/ram_energy": 0.030443774090695673,
+  "energy/codecarbon/ram_power": 54.0,
+  "energy/codecarbon/ram_total_size": 1511.49019241333,
+  "energy/codecarbon/ram_used_gb": 80.42555782920437,
+  "energy/codecarbon/ram_utilization_percent": 5.3144497607653856,
+  "energy/codecarbon/water_consumed": 0.0,
+  "energy/codecarbon/wue": 0.0,
+  "eval/duration_sec": 14.522059130016714,
+  "eval/loss": 10.326239856806668,
+  "perf/audit_duration_sec": 5.763615260832012,
+  "perf/epoch_duration_sec": 993.2564477077685,
+  "perf/epoch_samples": 55652.0,
+  "perf/epoch_samples_per_sec": 56.02984015702426,
+  "perf/epoch_tokens": 44091013.0,
+  "perf/epoch_tokens_per_sec": 44390.36172556743,
+  "perf/logical_batch_size": 65.0,
+  "perf/logical_token_count": 49862.0,
+  "perf/physical_batches": 9.0,
+  "perf/samples_per_sec": 6.722750906341063,
+  "perf/step_duration_sec": 9.668661074247211,
+  "perf/tokens_per_sec": 5157.073933722741,
+  "privacy/epsilon": 2.9999680995370417,
+  "system/cuda_epoch_peak_memory_gb": 73.29015684127808,
+  "system/cuda_max_memory_allocated_gb": 73.29015684127808,
+  "system/cuda_memory_allocated_gb": 14.1148681640625,
+  "train/epoch_canary_loss": 10.469284867902852,
+  "train/epoch_loss": 10.700755941458253,
+  "train/epoch_real_loss": 10.694620649826392,
+  "train/lr": 8.126960406835249e-07,
+  "train/step_canary_loss": 10.875,
+  "train/step_loss": 10.382469764122597,
+  "train/step_real_loss": 10.374773979187012
+}

deepseek-coder-6.7b/dp3/tokenizer/tokenizer_config.json

@@ -0,0 +1,516 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|EOT|>",
+  "extra_special_tokens": [
+    "865331112869",
+    "569765693871",
+    "485177821815",
+    "135441121756",
+    "367459894796",
+    "877482678543",
+    "457919547633",
+    "765474393376",
+    "114848338811",
+    "746285987371",
+    "649291669397",
+    "927914615679",
+    "445925149649",
+    "691587454538",
+    "143777992227",
+    "997981281989",
+    "425949483533",
+    "982993456429",
+    "718726519731",
+    "172599315861",
+    "643489267333",
+    "282322838685",
+    "781653545886",
+    "796415361892",
+    "841991688488",
+    "211411365397",
+    "698218415444",
+    "355977139358",
+    "682564697312",
+    "383837596997",
+    "689362171782",
+    "749966767285",
+    "753159165157",
+    "795693824762",
+    "669689115557",
+    "327491773134",
+    "983569279932",
+    "612128769512",
+    "374327157578",
+    "311632789559",
+    "523918658846",
+    "765981581453",
+    "794825141891",
+    "873898736873",
+    "447445629421",
+    "473822473819",
+    "181439694557",
+    "592538279337",
+    "668134915514",
+    "643692393748",
+    "696651276628",
+    "853859348234",
+    "778466723723",
+    "929826356991",
+    "272362973463",
+    "694235616268",
+    "281673864127",
+    "479676316326",
+    "646979124677",
+    "922327493433",
+    "883685933161",
+    "264259917554",
+    "836746273134",
+    "658481324922",
+    "481884157827",
+    "587787496812",
+    "579184949249",
+    "912193598348",
+    "529679678956",
+    "795838284624",
+    "159337222655",
+    "173781362446",
+    "773687856563",
+    "535787224917",
+    "351885857332",
+    "578827344666",
+    "198462689911",
+    "722618266242",
+    "952872416512",
+    "517778845323",
+    "749665846687",
+    "661436365453",
+    "259666844669",
+    "242851284913",
+    "514532995959",
+    "161588262349",
+    "742765629356",
+    "225164373623",
+    "676539973863",
+    "826214551218",
+    "182345464792",
+    "232776999554",
+    "337326533813",
+    "676676697292",
+    "929185622831",
+    "545512344383",
+    "499444466686",
+    "314697386682",
+    "517379856925",
+    "379557332953",
+    "614797267726",
+    "429781429464",
+    "922466849763",
+    "721737645236",
+    "479227349997",
+    "136931728327",
+    "259533577263",
+    "488538864842",
+    "937495658852",
+    "489991411364",
+    "499148455254",
+    "441373944925",
+    "899151413682",
+    "467893531755",
+    "527117488925",
+    "928335588653",
+    "374439448821",
+    "879425227932",
+    "867678158885",
+    "399749397872",
+    "129693547287",
+    "689285841825",
+    "771619544974",
+    "724883568652",
+    "516968424863",
+    "733737988257",
+    "852347289392",
+    "296953381169",
+    "377273562477",
+    "262296912232",
+    "547149832394",
+    "298464134954",
+    "216667245274",
+    "843998562287",
+    "572154333646",
+    "124589118494",
+    "841824384614",
+    "232896526252",
+    "295448593321",
+    "123741461297",
+    "653573457168",
+    "196735786156",
+    "377338713663",
+    "964342468552",
+    "586855179568",
+    "484773717614",
+    "894885246797",
+    "677896358599",
+    "848845611563",
+    "851852651677",
+    "398549545767",
+    "454244839926",
+    "799364566435",
+    "967114116556",
+    "817378986438",
+    "233795848681",
+    "824387273757",
+    "916198946615",
+    "563117729724",
+    "951794811935",
+    "374598961236",
+    "922867396683",
+    "765737843639",
+    "175469284871",
+    "231853711778",
+    "662426712668",
+    "711412347158",
+    "753466987363",
+    "513361312532",
+    "712992815957",
+    "971621888444",
+    "829235161526",
+    "585544633356",
+    "582471228164",
+    "678666359123",
+    "557533689478",
+    "632962475133",
+    "484489193824",
+    "489562189822",
+    "589547936288",
+    "363214487524",
+    "244885399387",
+    "431751228368",
+    "433581868192",
+    "486391569221",
+    "185438575221",
+    "126574388585",
+    "741757479784",
+    "529854679937",
+    "996116119839",
+    "616248973917",
+    "763531783491",
+    "955456118295",
+    "364196983365",
+    "195792996468",
+    "151859598873",
+    "399223169721",
+    "938488813964",
+    "961981959227",
+    "183368827562",
+    "533417736566",
+    "786391632558",
+    "665661658354",
+    "693281533643",
+    "475794684356",
+    "652154162978",
+    "753233719644",
+    "668514843129",
+    "819162623892",
+    "941169431859",
+    "877385381798",
+    "752644929761",
+    "881136466196",
+    "275597777299",
+    "731681792655",
+    "961133895172",
+    "864718285734",
+    "963852916563",
+    "319584985416",
+    "563365646341",
+    "811371928234",
+    "837131396371",
+    "267514771964",
+    "944513428457",
+    "117298239631",
+    "158142752582",
+    "252867443568",
+    "839269684865",
+    "612788593128",
+    "145669731981",
+    "121557291859",
+    "245416776926",
+    "799417897197",
+    "997958836435",
+    "892336777248",
+    "158929292238",
+    "581976444672",
+    "897784492783",
+    "492373714791",
+    "512659818733",
+    "881112998642",
+    "619454958782",
+    "431149748713",
+    "624221476921",
+    "125866399464",
+    "339882449689",
+    "186198784585",
+    "943193294691",
+    "955668961269",
+    "232787996724",
+    "215671314196",
+    "286173241916",
+    "745977673725",
+    "556976448182",
+    "599961512792",
+    "766294538337",
+    "934912591213",
+    "295118729589",
+    "529455466433",
+    "196119929397",
+    "379571934299",
+    "251789649997",
+    "564544131355",
+    "244371196654",
+    "384598329253",
+    "887753195844",
+    "364947325679",
+    "655517954651",
+    "673948786567",
+    "857231548835",
+    "816115936673",
+    "644234165531",
+    "182782912224",
+    "234316622259",
+    "421369185549",
+    "434632855397",
+    "921889371893",
+    "415956914763",
+    "598916996413",
+    "773671349113",
+    "952465217972",
+    "117657531962",
+    "729825168745",
+    "691315125346",
+    "768461952319",
+    "664847713559",
+    "953267689786",
+    "886464195129",
+    "824488329416",
+    "837873762491",
+    "532833541879",
+    "669183782449",
+    "941976537588",
+    "739394546916",
+    "267954879268",
+    "637551427887",
+    "217756494954",
+    "524444658383",
+    "117783274348",
+    "138218735276",
+    "814611949491",
+    "711641973413",
+    "499156317423",
+    "515856611931",
+    "454164859837",
+    "345271433112",
+    "462294118988",
+    "511785788222",
+    "497294727353",
+    "866519986723",
+    "334513529294",
+    "549946382131",
+    "284445431422",
+    "396521188476",
+    "421435255895",
+    "133373659361",
+    "322683334381",
+    "228358422847",
+    "291762694874",
+    "143182978129",
+    "511923256573",
+    "327158398268",
+    "879764613759",
+    "564395222747",
+    "451161679736",
+    "538631466654",
+    "221762325616",
+    "218391991184",
+    "322589379462",
+    "876537814263",
+    "152676556624",
+    "332522971941",
+    "884354318946",
+    "513349618943",
+    "116639746413",
+    "635185846287",
+    "993832498489",
+    "813981174797",
+    "438745114173",
+    "983493951323",
+    "724492262421",
+    "622553389126",
+    "889965243135",
+    "364492359246",
+    "154962668224",
+    "179564995814",
+    "418412875665",
+    "718951851413",
+    "699446724178",
+    "624266421831",
+    "815458725125",
+    "455423278865",
+    "393741199486",
+    "328552864359",
+    "211662639865",
+    "218784516525",
+    "762486672996",
+    "142799718159",
+    "858146415154",
+    "767858144912",
+    "571317457151",
+    "635127952696",
+    "116427191984",
+    "268921994538",
+    "523937669294",
+    "165429152138",
+    "739246183345",
+    "591464355756",
+    "212985874612",
+    "191887635211",
+    "967214577653",
+    "119342152414",
+    "946444632795",
+    "618423867817",
+    "228565148417",
+    "729116422489",
+    "527874729936",
+    "739784153482",
+    "387763951128",
+    "331369926711",
+    "562716493614",
+    "739667844957",
+    "562389434565",
+    "256497188281",
+    "859927364588",
+    "417668946583",
+    "357621613582",
+    "438435178228",
+    "485692541169",
+    "825815739116",
+    "342221452223",
+    "697747991249",
+    "716763689965",
+    "141499982867",
+    "818479319499",
+    "336813343298",
+    "594688742928",
+    "472129283475",
+    "514354144759",
+    "349249721685",
+    "546276298359",
+    "353755529131",
+    "315534574435",
+    "523723475786",
+    "215826764872",
+    "367968398551",
+    "569853653352",
+    "389715484387",
+    "293847485454",
+    "714738141818",
+    "178478368922",
+    "581493616981",
+    "589439538674",
+    "846657726193",
+    "722339992679",
+    "138154781148",
+    "757785319772",
+    "492516914298",
+    "919181521716",
+    "985781138935",
+    "476969195485",
+    "313145133463",
+    "758963111966",
+    "147541537162",
+    "557163366873",
+    "144373897488",
+    "522515164754",
+    "724964923582",
+    "284776712475",
+    "375429755114",
+    "181233596124",
+    "948585673431",
+    "243165586174",
+    "396847976144",
+    "997724962668",
+    "558837194455",
+    "163165456396",
+    "378749551722",
+    "161238482259",
+    "754978243758",
+    "195388849133",
+    "229775525672",
+    "262437452884",
+    "441377892146",
+    "451885565366",
+    "981277526855",
+    "762495822823",
+    "368763327262",
+    "757422791351",
+    "636324136426",
+    "214193645583",
+    "412843856172",
+    "179386156569",
+    "756916173536",
+    "892697125149",
+    "625334487352",
+    "941861857715",
+    "887417525236",
+    "649516938598",
+    "717628619782",
+    "438124184139",
+    "547563892268",
+    "856317483891",
+    "313313831273",
+    "371496153876",
+    "587541149322",
+    "265847332563",
+    "449549215429",
+    "163497196769",
+    "861342291298",
+    "268433315926",
+    "774679513717",
+    "851254219729",
+    "583527834464",
+    "488496781997",
+    "556814553861",
+    "482829231639",
+    "618878266619",
+    "147444452794",
+    "949235426629",
+    "357299947518",
+    "175528632226",
+    "645527857972",
+    "186872457894",
+    "552738847828",
+    "626748382482",
+    "921894985642",
+    "943878645871",
+    "859289776479",
+    "614583493135",
+    "933775286797",
+    "332234613346",
+    "325196781219",
+    "142526557681",
+    "356722692178",
+    "449318681694",
+    "687284547244",
+    "947262995132",
+    "893974619684",
+    "797238311233"
+  ],
+  "is_local": false,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": null,
+  "use_default_system_prompt": false
+}

deepseek-coder-6.7b/dp3/train.log

@@ -0,0 +1,24 @@
+2026-03-17 22:32:20,031 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=14.7503
+2026-03-17 22:33:53,038 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=14.6211
+2026-03-17 22:35:26,347 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=14.3372
+2026-03-17 22:36:58,636 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=14.1539
+2026-03-17 22:38:31,697 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=13.8100
+2026-03-17 22:38:46,362 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=13.7298 duration_sec=14.66
+2026-03-17 22:40:19,267 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=13.4669
+2026-03-17 22:41:52,405 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=13.1042
+2026-03-17 22:43:26,221 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=12.7799
+2026-03-17 22:45:00,446 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=12.3359
+2026-03-17 22:46:33,448 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=12.0393
+2026-03-17 22:46:47,978 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=11.8114 duration_sec=14.53
+2026-03-17 22:48:42,729 [INFO] new_opacus_codex.train_steps: epoch=2 step=110 loss=11.6187
+2026-03-17 22:50:15,269 [INFO] new_opacus_codex.train_steps: epoch=2 step=120 loss=11.3041
+2026-03-17 22:51:47,843 [INFO] new_opacus_codex.train_steps: epoch=2 step=130 loss=10.9837
+2026-03-17 22:53:19,754 [INFO] new_opacus_codex.train_steps: epoch=2 step=140 loss=10.7969
+2026-03-17 22:54:52,072 [INFO] new_opacus_codex.train_steps: epoch=2 step=150 loss=10.6606
+2026-03-17 22:55:06,578 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=150 eval_loss=10.5290 duration_sec=14.50
+2026-03-17 22:56:38,804 [INFO] new_opacus_codex.train_steps: epoch=2 step=160 loss=10.5269
+2026-03-17 22:58:12,764 [INFO] new_opacus_codex.train_steps: epoch=2 step=170 loss=10.5162
+2026-03-17 22:59:45,794 [INFO] new_opacus_codex.train_steps: epoch=2 step=180 loss=10.4517
+2026-03-17 23:01:19,449 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=10.4031
+2026-03-17 23:02:52,278 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=10.3874
+2026-03-17 23:03:06,793 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=10.3270 duration_sec=14.51

deepseek-coder-6.7b/dp8/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.533352,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.544592,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/dp8/codecarbon.csv

@@ -0,0 +1,2 @@
+timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
+2026-03-17T22:24:53,codedp-deepseek-coder-6.7b-cpt-dp8,468761a7-8e0a-40d1-b862-00771db2f603,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,2107.286476707086,0.1023234868151428,4.85569892590194e-05,72.02878771140225,4175.089097615359,54.0,0.04057807438109415,2.443401743608831,0.030419661433661826,2.5143994794235853,0.0,Sweden,SWE,östergötland county,,,Linux-6.8.0-94-generic-x86_64-with-glibc2.39,3.11.0,3.2.3,256,AMD EPYC 9554 64-Core Processor,8,8 x NVIDIA H200,16.1885,58.594,1511.49019241333,machine,3.622827125119416,80.66583094555874,5.312368672397158,80.40963100407845,N,1.0,0.0

deepseek-coder-6.7b/dp8/epochs/epoch_001/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: deepseek-ai/deepseek-coder-6.7b-instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:deepseek-ai/deepseek-coder-6.7b-instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

deepseek-coder-6.7b/dp8/epochs/epoch_001/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/dp8/epochs/epoch_001/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.5184,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.53976,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/dp8/epochs/epoch_002/adapter/README.md

@@ -0,0 +1,207 @@
+---
+base_model: deepseek-ai/deepseek-coder-6.7b-instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:deepseek-ai/deepseek-coder-6.7b-instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

deepseek-coder-6.7b/dp8/epochs/epoch_002/adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "deepseek-ai/deepseek-coder-6.7b-instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": true,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "lm_head",
+    "embed_tokens"
+  ],
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

deepseek-coder-6.7b/dp8/epochs/epoch_002/audit_results.json

@@ -0,0 +1,137 @@
+{
+  "delta": 1e-05,
+  "num_canaries": 500,
+  "num_members": 250,
+  "paper_guess_fraction": 0.2,
+  "paper_guess_steps": 20,
+  "loss": {
+    "auc": 0.533352,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  },
+  "embedding": {
+    "auc": 0.544592,
+    "empirical_epsilon": {
+      "0.05": 0.0,
+      "0.01": 0.0
+    },
+    "empirical_epsilon_details": {
+      "0.05": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      },
+      "0.01": {
+        "epsilon": 0.0,
+        "num_guesses": 0,
+        "correct_guesses": 0,
+        "candidate_num_guesses": [
+          5,
+          10,
+          15,
+          20,
+          25,
+          30,
+          35,
+          40,
+          45,
+          50,
+          55,
+          60,
+          65,
+          70,
+          75,
+          80,
+          85,
+          90,
+          95,
+          100
+        ],
+        "direction": "lower"
+      }
+    }
+  }
+}

deepseek-coder-6.7b/dp8/metrics.jsonl

@@ -0,0 +1,30 @@
+{"timestamp": 1773784344.144435, "event": "train_step", "step": 10, "epoch": 1, "metrics": {"train/step_loss": 14.774163795239998, "train/step_real_loss": 14.892106413841248, "train/lr": 0.00018181818181818183, "train/step_canary_loss": 11.0, "perf/step_duration_sec": 9.184927014168352, "perf/samples_per_sec": 7.185685841399793, "perf/tokens_per_sec": 5831.728430435432, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 53564.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.347414329116491, "system/cuda_memory_allocated_gb": 14.177838802337646, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773784437.5361402, "event": "train_step", "step": 20, "epoch": 1, "metrics": {"train/step_loss": 14.265574847950655, "train/step_real_loss": 14.47943890094757, "train/lr": 0.00019897180218885507, "train/step_canary_loss": 10.84375, "perf/step_duration_sec": 9.109625170007348, "perf/samples_per_sec": 7.46463204917411, "perf/tokens_per_sec": 6030.434729726173, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 54935.0, "perf/physical_batches": 9.0, "privacy/epsilon": 2.9091472084164676, "system/cuda_memory_allocated_gb": 14.301010608673096, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773784530.4163961, "event": "train_step", "step": 30, "epoch": 1, "metrics": {"train/step_loss": 13.518484404592803, "train/step_real_loss": 13.940937042236328, "train/lr": 0.00019544467510209388, "train/step_canary_loss": 0.0, "perf/step_duration_sec": 9.479914616327733, "perf/samples_per_sec": 6.751115657705354, "perf/tokens_per_sec": 5596.14744932698, "perf/logical_batch_size": 64.0, "perf/logical_token_count": 53051.0, "perf/physical_batches": 10.0, "privacy/epsilon": 3.362310066272621, "system/cuda_memory_allocated_gb": 14.547617435455322, "system/cuda_max_memory_allocated_gb": 73.28973627090454}}
+{"timestamp": 1773784622.9840994, "event": "train_step", "step": 40, "epoch": 1, "metrics": {"train/step_loss": 13.427360419071082, "train/step_real_loss": 13.384074807167053, "train/lr": 0.00018949541262593762, "train/step_canary_loss": 14.8125, "perf/step_duration_sec": 9.461411211173981, "perf/samples_per_sec": 6.975703573908047, "perf/tokens_per_sec": 5377.844685569534, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 50882.0, "perf/physical_batches": 9.0, "privacy/epsilon": 3.7564957912533217, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773784716.151862, "event": "train_step", "step": 50, "epoch": 1, "metrics": {"train/step_loss": 12.773963928222656, "train/step_real_loss": 12.735275864601135, "train/lr": 0.00018127499143005268, "train/step_canary_loss": 15.25, "perf/step_duration_sec": 9.381330342032015, "perf/samples_per_sec": 6.928654852795735, "perf/tokens_per_sec": 5950.009003510849, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 55819.0, "perf/physical_batches": 9.0, "privacy/epsilon": 4.113499817895439, "system/cuda_memory_allocated_gb": 14.1148681640625, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773784730.6923964, "event": "eval_step", "step": 50, "epoch": 1, "metrics": {"eval/loss": 12.672339336438613, "eval/duration_sec": 14.538249942008406}}
+{"timestamp": 1773784823.310222, "event": "train_step", "step": 60, "epoch": 1, "metrics": {"train/step_loss": 11.997090639386858, "train/step_real_loss": 12.100333452224731, "train/lr": 0.0001709920242324663, "train/step_canary_loss": 10.895833969116211, "perf/step_duration_sec": 9.064209748990834, "perf/samples_per_sec": 7.722680954927535, "perf/tokens_per_sec": 6211.903912116425, "perf/logical_batch_size": 70.0, "perf/logical_token_count": 56306.0, "perf/physical_batches": 9.0, "privacy/epsilon": 4.44246859480526, "system/cuda_memory_allocated_gb": 14.425142288208008, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773784915.821449, "event": "train_step", "step": 70, "epoch": 1, "metrics": {"train/step_loss": 11.382361131555895, "train/step_real_loss": 11.408211827278137, "train/lr": 0.00015890746575622231, "train/step_canary_loss": 10.96875, "perf/step_duration_sec": 9.208213192876428, "perf/samples_per_sec": 7.384711732413572, "perf/tokens_per_sec": 5687.531218381818, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 52372.0, "perf/physical_batches": 9.0, "privacy/epsilon": 4.751375515961613, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785010.2598872, "event": "train_step", "step": 80, "epoch": 1, "metrics": {"train/step_loss": 10.664647719439339, "train/step_real_loss": 10.645641326904297, "train/lr": 0.00014532799038330385, "train/step_canary_loss": 10.96875, "perf/step_duration_sec": 9.077163262758404, "perf/samples_per_sec": 7.491327194586108, "perf/tokens_per_sec": 5665.206024329354, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 51424.0, "perf/physical_batches": 9.0, "privacy/epsilon": 5.041931555886533, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785104.2668839, "event": "train_step", "step": 90, "epoch": 1, "metrics": {"train/step_loss": 10.064055400108224, "train/step_real_loss": 10.030925154685974, "train/lr": 0.00013059820956358998, "train/step_canary_loss": 10.770833969116211, "perf/step_duration_sec": 9.01999548682943, "perf/samples_per_sec": 7.427941632324565, "perf/tokens_per_sec": 6063.528532786976, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 54693.0, "perf/physical_batches": 9.0, "privacy/epsilon": 5.3191578546045415, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785196.997746, "event": "train_step", "step": 100, "epoch": 1, "metrics": {"train/step_loss": 9.603301625856211, "train/step_real_loss": 9.47983455657959, "train/lr": 0.00011509192648058249, "train/step_canary_loss": 10.73214340209961, "perf/step_duration_sec": 9.839758691843599, "perf/samples_per_sec": 7.215624104568085, "perf/tokens_per_sec": 4844.224486878061, "perf/logical_batch_size": 71.0, "perf/logical_token_count": 47666.0, "perf/physical_batches": 9.0, "privacy/epsilon": 5.584672549637297, "system/cuda_memory_allocated_gb": 14.487197399139404, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785211.3789763, "event": "eval_step", "step": 100, "epoch": 1, "metrics": {"eval/loss": 9.217194291678343, "eval/duration_sec": 14.378479943610728}}
+{"timestamp": 1773785262.4304821, "event": "train_epoch", "step": 104, "epoch": 1, "metrics": {"train/epoch_loss": 12.341963201017878, "train/epoch_real_loss": 12.414478586751462, "train/epoch_canary_loss": 10.728888959316615, "perf/epoch_duration_sec": 998.3496765969321, "perf/epoch_samples_per_sec": 55.96235598577414, "perf/epoch_tokens_per_sec": 44175.03409259433, "perf/epoch_samples": 55870.0, "perf/epoch_tokens": 44102131.0, "system/cuda_epoch_peak_memory_gb": 73.2897834777832, "eval/loss": 8.922484595667232, "eval/duration_sec": 14.388765855692327, "privacy/epsilon": 5.688895252675942}}
+{"timestamp": 1773785270.4813497, "event": "audit_epoch", "step": 104, "epoch": 1, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.5184, "audit/loss/empirical_epsilon/0.05": 0.0, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.53976, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.523622438777238}}
+{"timestamp": 1773785326.5610142, "event": "train_step", "step": 110, "epoch": 2, "metrics": {"train/step_loss": 8.857723698471531, "train/step_real_loss": 8.790777564048767, "train/lr": 9.920264990753837e-05, "train/step_canary_loss": 11.0, "perf/step_duration_sec": 9.301705885212868, "perf/samples_per_sec": 7.095472681513365, "perf/tokens_per_sec": 5950.3063935818445, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 55348.0, "perf/physical_batches": 9.0, "privacy/epsilon": 5.840075127464654, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.28975915908813}}
+{"timestamp": 1773785418.9144747, "event": "train_step", "step": 120, "epoch": 2, "metrics": {"train/step_loss": 8.510492253659377, "train/step_real_loss": 8.394773125648499, "train/lr": 8.333360798744496e-05, "train/step_canary_loss": 10.979166984558105, "perf/step_duration_sec": 9.138508805073798, "perf/samples_per_sec": 7.331611910556007, "perf/tokens_per_sec": 5373.305541133465, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 49104.0, "perf/physical_batches": 9.0, "privacy/epsilon": 6.086971689386329, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.28975915908813}}
+{"timestamp": 1773785511.284102, "event": "train_step", "step": 130, "epoch": 2, "metrics": {"train/step_loss": 8.294911356939785, "train/step_real_loss": 8.076740324497223, "train/lr": 6.788751536089739e-05, "train/step_canary_loss": 11.08750057220459, "perf/step_duration_sec": 9.095777447801083, "perf/samples_per_sec": 7.5859375843327, "perf/tokens_per_sec": 6110.967459239838, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 55584.0, "perf/physical_batches": 9.0, "privacy/epsilon": 6.327001481453001, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785603.6243958, "event": "train_step", "step": 140, "epoch": 2, "metrics": {"train/step_loss": 8.052544166792684, "train/step_real_loss": 7.91242903470993, "train/lr": 5.325635332531864e-05, "train/step_canary_loss": 11.041666984558105, "perf/step_duration_sec": 9.209645525086671, "perf/samples_per_sec": 7.274981411336075, "perf/tokens_per_sec": 5759.396477911762, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 53042.0, "perf/physical_batches": 9.0, "privacy/epsilon": 6.557571491578655, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785696.4521918, "event": "train_step", "step": 150, "epoch": 2, "metrics": {"train/step_loss": 8.013610591059146, "train/step_real_loss": 7.757837951183319, "train/lr": 3.981142237826332e-05, "train/step_canary_loss": 11.287500381469727, "perf/step_duration_sec": 9.033272176980972, "perf/samples_per_sec": 7.638428096501862, "perf/tokens_per_sec": 6166.425511006423, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 55703.0, "perf/physical_batches": 9.0, "privacy/epsilon": 6.783113430476456, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785710.7493467, "event": "eval_step", "step": 150, "epoch": 2, "metrics": {"eval/loss": 7.667719158259305, "eval/duration_sec": 14.295333066955209}}
+{"timestamp": 1773785803.0583243, "event": "train_step", "step": 160, "epoch": 2, "metrics": {"train/step_loss": 7.74585654518821, "train/step_real_loss": 7.636352062225342, "train/lr": 2.789391958515183e-05, "train/step_canary_loss": 11.25, "perf/step_duration_sec": 8.934944829903543, "perf/samples_per_sec": 7.386727199379082, "perf/tokens_per_sec": 6129.080933630252, "perf/logical_batch_size": 66.0, "perf/logical_token_count": 54763.0, "perf/physical_batches": 9.0, "privacy/epsilon": 7.0043611451158165, "system/cuda_memory_allocated_gb": 14.17786169052124, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785896.8129106, "event": "train_step", "step": 170, "epoch": 2, "metrics": {"train/step_loss": 7.764082396208351, "train/step_real_loss": 7.601656556129456, "train/lr": 1.7806279893114875e-05, "train/step_canary_loss": 11.229166984558105, "perf/step_duration_sec": 9.581119411159307, "perf/samples_per_sec": 6.992919837943347, "perf/tokens_per_sec": 5193.338885020673, "perf/logical_batch_size": 67.0, "perf/logical_token_count": 49758.0, "perf/physical_batches": 9.0, "privacy/epsilon": 7.2178133573416465, "system/cuda_memory_allocated_gb": 14.238978385925293, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773785989.7971883, "event": "train_step", "step": 180, "epoch": 2, "metrics": {"train/step_loss": 7.800547375398524, "train/step_real_loss": 7.582026898860931, "train/lr": 9.804501125681243e-06, "train/step_canary_loss": 11.296875, "perf/step_duration_sec": 9.495344399940223, "perf/samples_per_sec": 7.161404277282253, "perf/tokens_per_sec": 5326.71569030381, "perf/logical_batch_size": 68.0, "perf/logical_token_count": 50579.0, "perf/physical_batches": 9.0, "privacy/epsilon": 7.430040084733278, "system/cuda_memory_allocated_gb": 14.30103349685669, "system/cuda_max_memory_allocated_gb": 73.2897834777832}}
+{"timestamp": 1773786082.775681, "event": "train_step", "step": 190, "epoch": 2, "metrics": {"train/step_loss": 7.886315041694088, "train/step_real_loss": 7.6274334192276, "train/lr": 4.091647429802869e-06, "train/step_canary_loss": 11.199999809265137, "perf/step_duration_sec": 9.168436062987894, "perf/samples_per_sec": 7.525820055455963, "perf/tokens_per_sec": 5850.50706919794, "perf/logical_batch_size": 69.0, "perf/logical_token_count": 53640.0, "perf/physical_batches": 9.0, "privacy/epsilon": 7.633877151395027, "system/cuda_memory_allocated_gb": 14.363087177276611, "system/cuda_max_memory_allocated_gb": 73.29015684127808}}
+{"timestamp": 1773786175.474541, "event": "train_step", "step": 200, "epoch": 2, "metrics": {"train/step_loss": 7.722104996901292, "train/step_real_loss": 7.670887887477875, "train/lr": 8.126960406835249e-07, "train/step_canary_loss": 11.0, "perf/step_duration_sec": 9.673653797712177, "perf/samples_per_sec": 6.719281189840856, "perf/tokens_per_sec": 5154.412287505304, "perf/logical_batch_size": 65.0, "perf/logical_token_count": 49862.0, "perf/physical_batches": 9.0, "privacy/epsilon": 7.837365803470095, "system/cuda_memory_allocated_gb": 14.1148681640625, "system/cuda_max_memory_allocated_gb": 73.29015684127808}}
+{"timestamp": 1773786189.8028603, "event": "eval_step", "step": 200, "epoch": 2, "metrics": {"eval/loss": 7.523608349940993, "eval/duration_sec": 14.325929747894406}}
+{"timestamp": 1773786276.8565547, "event": "train_epoch", "step": 208, "epoch": 2, "metrics": {"train/epoch_loss": 8.035395088671393, "train/epoch_real_loss": 7.897920552785785, "train/epoch_canary_loss": 10.571939474888266, "perf/epoch_duration_sec": 991.9095743251964, "perf/epoch_samples_per_sec": 56.10592078200321, "perf/epoch_tokens_per_sec": 44450.63757953486, "perf/epoch_samples": 55652.0, "perf/epoch_tokens": 44091013.0, "system/cuda_epoch_peak_memory_gb": 73.29015684127808, "eval/loss": 7.523212566971779, "eval/duration_sec": 14.324650165159255, "privacy/epsilon": 7.995186040237391}}
+{"timestamp": 1773786284.9576344, "event": "audit_epoch", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.533352, "audit/loss/empirical_epsilon/0.05": 0.0, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.544592, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0, "perf/audit_duration_sec": 5.583974160254002}}
+{"timestamp": 1773786292.845513, "event": "audit_final", "step": 208, "epoch": 2, "metrics": {"audit/delta": 1e-05, "audit/num_canaries": 500.0, "audit/num_members": 250.0, "audit/paper_guess_fraction": 0.2, "audit/paper_guess_steps": 20.0, "audit/loss/auc": 0.533352, "audit/loss/empirical_epsilon/0.05": 0.0, "audit/loss/empirical_epsilon/0.01": 0.0, "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0, "audit/embedding/auc": 0.544592, "audit/embedding/empirical_epsilon/0.05": 0.0, "audit/embedding/empirical_epsilon/0.01": 0.0, "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0, "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0, "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0}}
+{"timestamp": 1773786293.3942149, "event": "energy_final", "step": 208, "epoch": null, "metrics": {"energy/codecarbon/duration": 2107.286476707086, "energy/codecarbon/emissions": 0.1023234868151428, "energy/codecarbon/emissions_rate": 4.85569892590194e-05, "energy/codecarbon/cpu_power": 72.02878771140225, "energy/codecarbon/gpu_power": 4175.089097615359, "energy/codecarbon/ram_power": 54.0, "energy/codecarbon/cpu_energy": 0.04057807438109415, "energy/codecarbon/gpu_energy": 2.443401743608831, "energy/codecarbon/ram_energy": 0.030419661433661826, "energy/codecarbon/energy_consumed": 2.5143994794235853, "energy/codecarbon/water_consumed": 0.0, "energy/codecarbon/cpu_count": 256.0, "energy/codecarbon/gpu_count": 8.0, "energy/codecarbon/longitude": 16.1885, "energy/codecarbon/latitude": 58.594, "energy/codecarbon/ram_total_size": 1511.49019241333, "energy/codecarbon/cpu_utilization_percent": 3.622827125119416, "energy/codecarbon/gpu_utilization_percent": 80.66583094555874, "energy/codecarbon/ram_utilization_percent": 5.312368672397158, "energy/codecarbon/ram_used_gb": 80.40963100407845, "energy/codecarbon/pue": 1.0, "energy/codecarbon/wue": 0.0}}

deepseek-coder-6.7b/dp8/resolved_config.yaml

@@ -0,0 +1,101 @@
+model:
+  name: deepseek-ai/deepseek-coder-6.7b-instruct
+  tokenizer_name: deepseek-ai/deepseek-coder-6.7b-instruct
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0002
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 8
+  num_workers: 4
+  output_dir: runs/cpt/deepseek-coder-6.7b/dp8
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: hooks
+  clipping: flat
+  secure_mode: false
+  enabled: true
+  target_epsilon: 8.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: deepseek-coder-6.7b-cpt-dp8
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-deepseek-coder-6.7b-cpt-dp8

deepseek-coder-6.7b/dp8/scalars.csv

@@ -0,0 +1,386 @@
+timestamp,event,step,epoch,key,value
+1773784344.144435,train_step,10,1,train/step_loss,14.774163795239998
+1773784344.144435,train_step,10,1,train/step_real_loss,14.892106413841248
+1773784344.144435,train_step,10,1,train/lr,0.00018181818181818183
+1773784344.144435,train_step,10,1,train/step_canary_loss,11.0
+1773784344.144435,train_step,10,1,perf/step_duration_sec,9.184927014168352
+1773784344.144435,train_step,10,1,perf/samples_per_sec,7.185685841399793
+1773784344.144435,train_step,10,1,perf/tokens_per_sec,5831.728430435432
+1773784344.144435,train_step,10,1,perf/logical_batch_size,66.0
+1773784344.144435,train_step,10,1,perf/logical_token_count,53564.0
+1773784344.144435,train_step,10,1,perf/physical_batches,9.0
+1773784344.144435,train_step,10,1,privacy/epsilon,2.347414329116491
+1773784344.144435,train_step,10,1,system/cuda_memory_allocated_gb,14.177838802337646
+1773784344.144435,train_step,10,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773784437.5361402,train_step,20,1,train/step_loss,14.265574847950655
+1773784437.5361402,train_step,20,1,train/step_real_loss,14.47943890094757
+1773784437.5361402,train_step,20,1,train/lr,0.00019897180218885507
+1773784437.5361402,train_step,20,1,train/step_canary_loss,10.84375
+1773784437.5361402,train_step,20,1,perf/step_duration_sec,9.109625170007348
+1773784437.5361402,train_step,20,1,perf/samples_per_sec,7.46463204917411
+1773784437.5361402,train_step,20,1,perf/tokens_per_sec,6030.434729726173
+1773784437.5361402,train_step,20,1,perf/logical_batch_size,68.0
+1773784437.5361402,train_step,20,1,perf/logical_token_count,54935.0
+1773784437.5361402,train_step,20,1,perf/physical_batches,9.0
+1773784437.5361402,train_step,20,1,privacy/epsilon,2.9091472084164676
+1773784437.5361402,train_step,20,1,system/cuda_memory_allocated_gb,14.301010608673096
+1773784437.5361402,train_step,20,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773784530.4163961,train_step,30,1,train/step_loss,13.518484404592803
+1773784530.4163961,train_step,30,1,train/step_real_loss,13.940937042236328
+1773784530.4163961,train_step,30,1,train/lr,0.00019544467510209388
+1773784530.4163961,train_step,30,1,train/step_canary_loss,0.0
+1773784530.4163961,train_step,30,1,perf/step_duration_sec,9.479914616327733
+1773784530.4163961,train_step,30,1,perf/samples_per_sec,6.751115657705354
+1773784530.4163961,train_step,30,1,perf/tokens_per_sec,5596.14744932698
+1773784530.4163961,train_step,30,1,perf/logical_batch_size,64.0
+1773784530.4163961,train_step,30,1,perf/logical_token_count,53051.0
+1773784530.4163961,train_step,30,1,perf/physical_batches,10.0
+1773784530.4163961,train_step,30,1,privacy/epsilon,3.362310066272621
+1773784530.4163961,train_step,30,1,system/cuda_memory_allocated_gb,14.547617435455322
+1773784530.4163961,train_step,30,1,system/cuda_max_memory_allocated_gb,73.28973627090454
+1773784622.9840994,train_step,40,1,train/step_loss,13.427360419071082
+1773784622.9840994,train_step,40,1,train/step_real_loss,13.384074807167053
+1773784622.9840994,train_step,40,1,train/lr,0.00018949541262593762
+1773784622.9840994,train_step,40,1,train/step_canary_loss,14.8125
+1773784622.9840994,train_step,40,1,perf/step_duration_sec,9.461411211173981
+1773784622.9840994,train_step,40,1,perf/samples_per_sec,6.975703573908047
+1773784622.9840994,train_step,40,1,perf/tokens_per_sec,5377.844685569534
+1773784622.9840994,train_step,40,1,perf/logical_batch_size,66.0
+1773784622.9840994,train_step,40,1,perf/logical_token_count,50882.0
+1773784622.9840994,train_step,40,1,perf/physical_batches,9.0
+1773784622.9840994,train_step,40,1,privacy/epsilon,3.7564957912533217
+1773784622.9840994,train_step,40,1,system/cuda_memory_allocated_gb,14.17786169052124
+1773784622.9840994,train_step,40,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773784716.151862,train_step,50,1,train/step_loss,12.773963928222656
+1773784716.151862,train_step,50,1,train/step_real_loss,12.735275864601135
+1773784716.151862,train_step,50,1,train/lr,0.00018127499143005268
+1773784716.151862,train_step,50,1,train/step_canary_loss,15.25
+1773784716.151862,train_step,50,1,perf/step_duration_sec,9.381330342032015
+1773784716.151862,train_step,50,1,perf/samples_per_sec,6.928654852795735
+1773784716.151862,train_step,50,1,perf/tokens_per_sec,5950.009003510849
+1773784716.151862,train_step,50,1,perf/logical_batch_size,65.0
+1773784716.151862,train_step,50,1,perf/logical_token_count,55819.0
+1773784716.151862,train_step,50,1,perf/physical_batches,9.0
+1773784716.151862,train_step,50,1,privacy/epsilon,4.113499817895439
+1773784716.151862,train_step,50,1,system/cuda_memory_allocated_gb,14.1148681640625
+1773784716.151862,train_step,50,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773784730.6923964,eval_step,50,1,eval/loss,12.672339336438613
+1773784730.6923964,eval_step,50,1,eval/duration_sec,14.538249942008406
+1773784823.310222,train_step,60,1,train/step_loss,11.997090639386858
+1773784823.310222,train_step,60,1,train/step_real_loss,12.100333452224731
+1773784823.310222,train_step,60,1,train/lr,0.0001709920242324663
+1773784823.310222,train_step,60,1,train/step_canary_loss,10.895833969116211
+1773784823.310222,train_step,60,1,perf/step_duration_sec,9.064209748990834
+1773784823.310222,train_step,60,1,perf/samples_per_sec,7.722680954927535
+1773784823.310222,train_step,60,1,perf/tokens_per_sec,6211.903912116425
+1773784823.310222,train_step,60,1,perf/logical_batch_size,70.0
+1773784823.310222,train_step,60,1,perf/logical_token_count,56306.0
+1773784823.310222,train_step,60,1,perf/physical_batches,9.0
+1773784823.310222,train_step,60,1,privacy/epsilon,4.44246859480526
+1773784823.310222,train_step,60,1,system/cuda_memory_allocated_gb,14.425142288208008
+1773784823.310222,train_step,60,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773784915.821449,train_step,70,1,train/step_loss,11.382361131555895
+1773784915.821449,train_step,70,1,train/step_real_loss,11.408211827278137
+1773784915.821449,train_step,70,1,train/lr,0.00015890746575622231
+1773784915.821449,train_step,70,1,train/step_canary_loss,10.96875
+1773784915.821449,train_step,70,1,perf/step_duration_sec,9.208213192876428
+1773784915.821449,train_step,70,1,perf/samples_per_sec,7.384711732413572
+1773784915.821449,train_step,70,1,perf/tokens_per_sec,5687.531218381818
+1773784915.821449,train_step,70,1,perf/logical_batch_size,68.0
+1773784915.821449,train_step,70,1,perf/logical_token_count,52372.0
+1773784915.821449,train_step,70,1,perf/physical_batches,9.0
+1773784915.821449,train_step,70,1,privacy/epsilon,4.751375515961613
+1773784915.821449,train_step,70,1,system/cuda_memory_allocated_gb,14.30103349685669
+1773784915.821449,train_step,70,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785010.2598872,train_step,80,1,train/step_loss,10.664647719439339
+1773785010.2598872,train_step,80,1,train/step_real_loss,10.645641326904297
+1773785010.2598872,train_step,80,1,train/lr,0.00014532799038330385
+1773785010.2598872,train_step,80,1,train/step_canary_loss,10.96875
+1773785010.2598872,train_step,80,1,perf/step_duration_sec,9.077163262758404
+1773785010.2598872,train_step,80,1,perf/samples_per_sec,7.491327194586108
+1773785010.2598872,train_step,80,1,perf/tokens_per_sec,5665.206024329354
+1773785010.2598872,train_step,80,1,perf/logical_batch_size,68.0
+1773785010.2598872,train_step,80,1,perf/logical_token_count,51424.0
+1773785010.2598872,train_step,80,1,perf/physical_batches,9.0
+1773785010.2598872,train_step,80,1,privacy/epsilon,5.041931555886533
+1773785010.2598872,train_step,80,1,system/cuda_memory_allocated_gb,14.30103349685669
+1773785010.2598872,train_step,80,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785104.2668839,train_step,90,1,train/step_loss,10.064055400108224
+1773785104.2668839,train_step,90,1,train/step_real_loss,10.030925154685974
+1773785104.2668839,train_step,90,1,train/lr,0.00013059820956358998
+1773785104.2668839,train_step,90,1,train/step_canary_loss,10.770833969116211
+1773785104.2668839,train_step,90,1,perf/step_duration_sec,9.01999548682943
+1773785104.2668839,train_step,90,1,perf/samples_per_sec,7.427941632324565
+1773785104.2668839,train_step,90,1,perf/tokens_per_sec,6063.528532786976
+1773785104.2668839,train_step,90,1,perf/logical_batch_size,67.0
+1773785104.2668839,train_step,90,1,perf/logical_token_count,54693.0
+1773785104.2668839,train_step,90,1,perf/physical_batches,9.0
+1773785104.2668839,train_step,90,1,privacy/epsilon,5.3191578546045415
+1773785104.2668839,train_step,90,1,system/cuda_memory_allocated_gb,14.238978385925293
+1773785104.2668839,train_step,90,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785196.997746,train_step,100,1,train/step_loss,9.603301625856211
+1773785196.997746,train_step,100,1,train/step_real_loss,9.47983455657959
+1773785196.997746,train_step,100,1,train/lr,0.00011509192648058249
+1773785196.997746,train_step,100,1,train/step_canary_loss,10.73214340209961
+1773785196.997746,train_step,100,1,perf/step_duration_sec,9.839758691843599
+1773785196.997746,train_step,100,1,perf/samples_per_sec,7.215624104568085
+1773785196.997746,train_step,100,1,perf/tokens_per_sec,4844.224486878061
+1773785196.997746,train_step,100,1,perf/logical_batch_size,71.0
+1773785196.997746,train_step,100,1,perf/logical_token_count,47666.0
+1773785196.997746,train_step,100,1,perf/physical_batches,9.0
+1773785196.997746,train_step,100,1,privacy/epsilon,5.584672549637297
+1773785196.997746,train_step,100,1,system/cuda_memory_allocated_gb,14.487197399139404
+1773785196.997746,train_step,100,1,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785211.3789763,eval_step,100,1,eval/loss,9.217194291678343
+1773785211.3789763,eval_step,100,1,eval/duration_sec,14.378479943610728
+1773785262.4304821,train_epoch,104,1,train/epoch_loss,12.341963201017878
+1773785262.4304821,train_epoch,104,1,train/epoch_real_loss,12.414478586751462
+1773785262.4304821,train_epoch,104,1,train/epoch_canary_loss,10.728888959316615
+1773785262.4304821,train_epoch,104,1,perf/epoch_duration_sec,998.3496765969321
+1773785262.4304821,train_epoch,104,1,perf/epoch_samples_per_sec,55.96235598577414
+1773785262.4304821,train_epoch,104,1,perf/epoch_tokens_per_sec,44175.03409259433
+1773785262.4304821,train_epoch,104,1,perf/epoch_samples,55870.0
+1773785262.4304821,train_epoch,104,1,perf/epoch_tokens,44102131.0
+1773785262.4304821,train_epoch,104,1,system/cuda_epoch_peak_memory_gb,73.2897834777832
+1773785262.4304821,train_epoch,104,1,eval/loss,8.922484595667232
+1773785262.4304821,train_epoch,104,1,eval/duration_sec,14.388765855692327
+1773785262.4304821,train_epoch,104,1,privacy/epsilon,5.688895252675942
+1773785270.4813497,audit_epoch,104,1,audit/delta,1e-05
+1773785270.4813497,audit_epoch,104,1,audit/num_canaries,500.0
+1773785270.4813497,audit_epoch,104,1,audit/num_members,250.0
+1773785270.4813497,audit_epoch,104,1,audit/paper_guess_fraction,0.2
+1773785270.4813497,audit_epoch,104,1,audit/paper_guess_steps,20.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/auc,0.5184
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon/0.05,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon/0.01,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/epsilon,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/num_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/auc,0.53976
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.05,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon/0.01,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773785270.4813497,audit_epoch,104,1,perf/audit_duration_sec,5.523622438777238
+1773785326.5610142,train_step,110,2,train/step_loss,8.857723698471531
+1773785326.5610142,train_step,110,2,train/step_real_loss,8.790777564048767
+1773785326.5610142,train_step,110,2,train/lr,9.920264990753837e-05
+1773785326.5610142,train_step,110,2,train/step_canary_loss,11.0
+1773785326.5610142,train_step,110,2,perf/step_duration_sec,9.301705885212868
+1773785326.5610142,train_step,110,2,perf/samples_per_sec,7.095472681513365
+1773785326.5610142,train_step,110,2,perf/tokens_per_sec,5950.3063935818445
+1773785326.5610142,train_step,110,2,perf/logical_batch_size,66.0
+1773785326.5610142,train_step,110,2,perf/logical_token_count,55348.0
+1773785326.5610142,train_step,110,2,perf/physical_batches,9.0
+1773785326.5610142,train_step,110,2,privacy/epsilon,5.840075127464654
+1773785326.5610142,train_step,110,2,system/cuda_memory_allocated_gb,14.17786169052124
+1773785326.5610142,train_step,110,2,system/cuda_max_memory_allocated_gb,73.28975915908813
+1773785418.9144747,train_step,120,2,train/step_loss,8.510492253659377
+1773785418.9144747,train_step,120,2,train/step_real_loss,8.394773125648499
+1773785418.9144747,train_step,120,2,train/lr,8.333360798744496e-05
+1773785418.9144747,train_step,120,2,train/step_canary_loss,10.979166984558105
+1773785418.9144747,train_step,120,2,perf/step_duration_sec,9.138508805073798
+1773785418.9144747,train_step,120,2,perf/samples_per_sec,7.331611910556007
+1773785418.9144747,train_step,120,2,perf/tokens_per_sec,5373.305541133465
+1773785418.9144747,train_step,120,2,perf/logical_batch_size,67.0
+1773785418.9144747,train_step,120,2,perf/logical_token_count,49104.0
+1773785418.9144747,train_step,120,2,perf/physical_batches,9.0
+1773785418.9144747,train_step,120,2,privacy/epsilon,6.086971689386329
+1773785418.9144747,train_step,120,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773785418.9144747,train_step,120,2,system/cuda_max_memory_allocated_gb,73.28975915908813
+1773785511.284102,train_step,130,2,train/step_loss,8.294911356939785
+1773785511.284102,train_step,130,2,train/step_real_loss,8.076740324497223
+1773785511.284102,train_step,130,2,train/lr,6.788751536089739e-05
+1773785511.284102,train_step,130,2,train/step_canary_loss,11.08750057220459
+1773785511.284102,train_step,130,2,perf/step_duration_sec,9.095777447801083
+1773785511.284102,train_step,130,2,perf/samples_per_sec,7.5859375843327
+1773785511.284102,train_step,130,2,perf/tokens_per_sec,6110.967459239838
+1773785511.284102,train_step,130,2,perf/logical_batch_size,69.0
+1773785511.284102,train_step,130,2,perf/logical_token_count,55584.0
+1773785511.284102,train_step,130,2,perf/physical_batches,9.0
+1773785511.284102,train_step,130,2,privacy/epsilon,6.327001481453001
+1773785511.284102,train_step,130,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773785511.284102,train_step,130,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785603.6243958,train_step,140,2,train/step_loss,8.052544166792684
+1773785603.6243958,train_step,140,2,train/step_real_loss,7.91242903470993
+1773785603.6243958,train_step,140,2,train/lr,5.325635332531864e-05
+1773785603.6243958,train_step,140,2,train/step_canary_loss,11.041666984558105
+1773785603.6243958,train_step,140,2,perf/step_duration_sec,9.209645525086671
+1773785603.6243958,train_step,140,2,perf/samples_per_sec,7.274981411336075
+1773785603.6243958,train_step,140,2,perf/tokens_per_sec,5759.396477911762
+1773785603.6243958,train_step,140,2,perf/logical_batch_size,67.0
+1773785603.6243958,train_step,140,2,perf/logical_token_count,53042.0
+1773785603.6243958,train_step,140,2,perf/physical_batches,9.0
+1773785603.6243958,train_step,140,2,privacy/epsilon,6.557571491578655
+1773785603.6243958,train_step,140,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773785603.6243958,train_step,140,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785696.4521918,train_step,150,2,train/step_loss,8.013610591059146
+1773785696.4521918,train_step,150,2,train/step_real_loss,7.757837951183319
+1773785696.4521918,train_step,150,2,train/lr,3.981142237826332e-05
+1773785696.4521918,train_step,150,2,train/step_canary_loss,11.287500381469727
+1773785696.4521918,train_step,150,2,perf/step_duration_sec,9.033272176980972
+1773785696.4521918,train_step,150,2,perf/samples_per_sec,7.638428096501862
+1773785696.4521918,train_step,150,2,perf/tokens_per_sec,6166.425511006423
+1773785696.4521918,train_step,150,2,perf/logical_batch_size,69.0
+1773785696.4521918,train_step,150,2,perf/logical_token_count,55703.0
+1773785696.4521918,train_step,150,2,perf/physical_batches,9.0
+1773785696.4521918,train_step,150,2,privacy/epsilon,6.783113430476456
+1773785696.4521918,train_step,150,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773785696.4521918,train_step,150,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785710.7493467,eval_step,150,2,eval/loss,7.667719158259305
+1773785710.7493467,eval_step,150,2,eval/duration_sec,14.295333066955209
+1773785803.0583243,train_step,160,2,train/step_loss,7.74585654518821
+1773785803.0583243,train_step,160,2,train/step_real_loss,7.636352062225342
+1773785803.0583243,train_step,160,2,train/lr,2.789391958515183e-05
+1773785803.0583243,train_step,160,2,train/step_canary_loss,11.25
+1773785803.0583243,train_step,160,2,perf/step_duration_sec,8.934944829903543
+1773785803.0583243,train_step,160,2,perf/samples_per_sec,7.386727199379082
+1773785803.0583243,train_step,160,2,perf/tokens_per_sec,6129.080933630252
+1773785803.0583243,train_step,160,2,perf/logical_batch_size,66.0
+1773785803.0583243,train_step,160,2,perf/logical_token_count,54763.0
+1773785803.0583243,train_step,160,2,perf/physical_batches,9.0
+1773785803.0583243,train_step,160,2,privacy/epsilon,7.0043611451158165
+1773785803.0583243,train_step,160,2,system/cuda_memory_allocated_gb,14.17786169052124
+1773785803.0583243,train_step,160,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785896.8129106,train_step,170,2,train/step_loss,7.764082396208351
+1773785896.8129106,train_step,170,2,train/step_real_loss,7.601656556129456
+1773785896.8129106,train_step,170,2,train/lr,1.7806279893114875e-05
+1773785896.8129106,train_step,170,2,train/step_canary_loss,11.229166984558105
+1773785896.8129106,train_step,170,2,perf/step_duration_sec,9.581119411159307
+1773785896.8129106,train_step,170,2,perf/samples_per_sec,6.992919837943347
+1773785896.8129106,train_step,170,2,perf/tokens_per_sec,5193.338885020673
+1773785896.8129106,train_step,170,2,perf/logical_batch_size,67.0
+1773785896.8129106,train_step,170,2,perf/logical_token_count,49758.0
+1773785896.8129106,train_step,170,2,perf/physical_batches,9.0
+1773785896.8129106,train_step,170,2,privacy/epsilon,7.2178133573416465
+1773785896.8129106,train_step,170,2,system/cuda_memory_allocated_gb,14.238978385925293
+1773785896.8129106,train_step,170,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773785989.7971883,train_step,180,2,train/step_loss,7.800547375398524
+1773785989.7971883,train_step,180,2,train/step_real_loss,7.582026898860931
+1773785989.7971883,train_step,180,2,train/lr,9.804501125681243e-06
+1773785989.7971883,train_step,180,2,train/step_canary_loss,11.296875
+1773785989.7971883,train_step,180,2,perf/step_duration_sec,9.495344399940223
+1773785989.7971883,train_step,180,2,perf/samples_per_sec,7.161404277282253
+1773785989.7971883,train_step,180,2,perf/tokens_per_sec,5326.71569030381
+1773785989.7971883,train_step,180,2,perf/logical_batch_size,68.0
+1773785989.7971883,train_step,180,2,perf/logical_token_count,50579.0
+1773785989.7971883,train_step,180,2,perf/physical_batches,9.0
+1773785989.7971883,train_step,180,2,privacy/epsilon,7.430040084733278
+1773785989.7971883,train_step,180,2,system/cuda_memory_allocated_gb,14.30103349685669
+1773785989.7971883,train_step,180,2,system/cuda_max_memory_allocated_gb,73.2897834777832
+1773786082.775681,train_step,190,2,train/step_loss,7.886315041694088
+1773786082.775681,train_step,190,2,train/step_real_loss,7.6274334192276
+1773786082.775681,train_step,190,2,train/lr,4.091647429802869e-06
+1773786082.775681,train_step,190,2,train/step_canary_loss,11.199999809265137
+1773786082.775681,train_step,190,2,perf/step_duration_sec,9.168436062987894
+1773786082.775681,train_step,190,2,perf/samples_per_sec,7.525820055455963
+1773786082.775681,train_step,190,2,perf/tokens_per_sec,5850.50706919794
+1773786082.775681,train_step,190,2,perf/logical_batch_size,69.0
+1773786082.775681,train_step,190,2,perf/logical_token_count,53640.0
+1773786082.775681,train_step,190,2,perf/physical_batches,9.0
+1773786082.775681,train_step,190,2,privacy/epsilon,7.633877151395027
+1773786082.775681,train_step,190,2,system/cuda_memory_allocated_gb,14.363087177276611
+1773786082.775681,train_step,190,2,system/cuda_max_memory_allocated_gb,73.29015684127808
+1773786175.474541,train_step,200,2,train/step_loss,7.722104996901292
+1773786175.474541,train_step,200,2,train/step_real_loss,7.670887887477875
+1773786175.474541,train_step,200,2,train/lr,8.126960406835249e-07
+1773786175.474541,train_step,200,2,train/step_canary_loss,11.0
+1773786175.474541,train_step,200,2,perf/step_duration_sec,9.673653797712177
+1773786175.474541,train_step,200,2,perf/samples_per_sec,6.719281189840856
+1773786175.474541,train_step,200,2,perf/tokens_per_sec,5154.412287505304
+1773786175.474541,train_step,200,2,perf/logical_batch_size,65.0
+1773786175.474541,train_step,200,2,perf/logical_token_count,49862.0
+1773786175.474541,train_step,200,2,perf/physical_batches,9.0
+1773786175.474541,train_step,200,2,privacy/epsilon,7.837365803470095
+1773786175.474541,train_step,200,2,system/cuda_memory_allocated_gb,14.1148681640625
+1773786175.474541,train_step,200,2,system/cuda_max_memory_allocated_gb,73.29015684127808
+1773786189.8028603,eval_step,200,2,eval/loss,7.523608349940993
+1773786189.8028603,eval_step,200,2,eval/duration_sec,14.325929747894406
+1773786276.8565547,train_epoch,208,2,train/epoch_loss,8.035395088671393
+1773786276.8565547,train_epoch,208,2,train/epoch_real_loss,7.897920552785785
+1773786276.8565547,train_epoch,208,2,train/epoch_canary_loss,10.571939474888266
+1773786276.8565547,train_epoch,208,2,perf/epoch_duration_sec,991.9095743251964
+1773786276.8565547,train_epoch,208,2,perf/epoch_samples_per_sec,56.10592078200321
+1773786276.8565547,train_epoch,208,2,perf/epoch_tokens_per_sec,44450.63757953486
+1773786276.8565547,train_epoch,208,2,perf/epoch_samples,55652.0
+1773786276.8565547,train_epoch,208,2,perf/epoch_tokens,44091013.0
+1773786276.8565547,train_epoch,208,2,system/cuda_epoch_peak_memory_gb,73.29015684127808
+1773786276.8565547,train_epoch,208,2,eval/loss,7.523212566971779
+1773786276.8565547,train_epoch,208,2,eval/duration_sec,14.324650165159255
+1773786276.8565547,train_epoch,208,2,privacy/epsilon,7.995186040237391
+1773786284.9576344,audit_epoch,208,2,audit/delta,1e-05
+1773786284.9576344,audit_epoch,208,2,audit/num_canaries,500.0
+1773786284.9576344,audit_epoch,208,2,audit/num_members,250.0
+1773786284.9576344,audit_epoch,208,2,audit/paper_guess_fraction,0.2
+1773786284.9576344,audit_epoch,208,2,audit/paper_guess_steps,20.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/auc,0.533352
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon/0.05,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/auc,0.544592
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773786284.9576344,audit_epoch,208,2,perf/audit_duration_sec,5.583974160254002
+1773786292.845513,audit_final,208,2,audit/delta,1e-05
+1773786292.845513,audit_final,208,2,audit/num_canaries,500.0
+1773786292.845513,audit_final,208,2,audit/num_members,250.0
+1773786292.845513,audit_final,208,2,audit/paper_guess_fraction,0.2
+1773786292.845513,audit_final,208,2,audit/paper_guess_steps,20.0
+1773786292.845513,audit_final,208,2,audit/loss/auc,0.533352
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon/0.05,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon/0.01,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/epsilon,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/num_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/epsilon,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/num_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/loss/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/auc,0.544592
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon/0.05,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon/0.01,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/epsilon,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/num_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.05/correct_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/epsilon,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/num_guesses,0.0
+1773786292.845513,audit_final,208,2,audit/embedding/empirical_epsilon_details/0.01/correct_guesses,0.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/duration,2107.286476707086
+1773786293.3942149,energy_final,208,,energy/codecarbon/emissions,0.1023234868151428
+1773786293.3942149,energy_final,208,,energy/codecarbon/emissions_rate,4.85569892590194e-05
+1773786293.3942149,energy_final,208,,energy/codecarbon/cpu_power,72.02878771140225
+1773786293.3942149,energy_final,208,,energy/codecarbon/gpu_power,4175.089097615359
+1773786293.3942149,energy_final,208,,energy/codecarbon/ram_power,54.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/cpu_energy,0.04057807438109415
+1773786293.3942149,energy_final,208,,energy/codecarbon/gpu_energy,2.443401743608831
+1773786293.3942149,energy_final,208,,energy/codecarbon/ram_energy,0.030419661433661826
+1773786293.3942149,energy_final,208,,energy/codecarbon/energy_consumed,2.5143994794235853
+1773786293.3942149,energy_final,208,,energy/codecarbon/water_consumed,0.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/cpu_count,256.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/gpu_count,8.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/longitude,16.1885
+1773786293.3942149,energy_final,208,,energy/codecarbon/latitude,58.594
+1773786293.3942149,energy_final,208,,energy/codecarbon/ram_total_size,1511.49019241333
+1773786293.3942149,energy_final,208,,energy/codecarbon/cpu_utilization_percent,3.622827125119416
+1773786293.3942149,energy_final,208,,energy/codecarbon/gpu_utilization_percent,80.66583094555874
+1773786293.3942149,energy_final,208,,energy/codecarbon/ram_utilization_percent,5.312368672397158
+1773786293.3942149,energy_final,208,,energy/codecarbon/ram_used_gb,80.40963100407845
+1773786293.3942149,energy_final,208,,energy/codecarbon/pue,1.0
+1773786293.3942149,energy_final,208,,energy/codecarbon/wue,0.0

deepseek-coder-6.7b/dp8/summary.json

@@ -0,0 +1,72 @@
+{
+  "audit/delta": 1e-05,
+  "audit/embedding/auc": 0.544592,
+  "audit/embedding/empirical_epsilon/0.01": 0.0,
+  "audit/embedding/empirical_epsilon/0.05": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.0,
+  "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 0.0,
+  "audit/loss/auc": 0.533352,
+  "audit/loss/empirical_epsilon/0.01": 0.0,
+  "audit/loss/empirical_epsilon/0.05": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0,
+  "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0,
+  "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0,
+  "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0,
+  "audit/num_canaries": 500.0,
+  "audit/num_members": 250.0,
+  "audit/paper_guess_fraction": 0.2,
+  "audit/paper_guess_steps": 20.0,
+  "energy/codecarbon/cpu_count": 256.0,
+  "energy/codecarbon/cpu_energy": 0.04057807438109415,
+  "energy/codecarbon/cpu_power": 72.02878771140225,
+  "energy/codecarbon/cpu_utilization_percent": 3.622827125119416,
+  "energy/codecarbon/duration": 2107.286476707086,
+  "energy/codecarbon/emissions": 0.1023234868151428,
+  "energy/codecarbon/emissions_rate": 4.85569892590194e-05,
+  "energy/codecarbon/energy_consumed": 2.5143994794235853,
+  "energy/codecarbon/gpu_count": 8.0,
+  "energy/codecarbon/gpu_energy": 2.443401743608831,
+  "energy/codecarbon/gpu_power": 4175.089097615359,
+  "energy/codecarbon/gpu_utilization_percent": 80.66583094555874,
+  "energy/codecarbon/latitude": 58.594,
+  "energy/codecarbon/longitude": 16.1885,
+  "energy/codecarbon/pue": 1.0,
+  "energy/codecarbon/ram_energy": 0.030419661433661826,
+  "energy/codecarbon/ram_power": 54.0,
+  "energy/codecarbon/ram_total_size": 1511.49019241333,
+  "energy/codecarbon/ram_used_gb": 80.40963100407845,
+  "energy/codecarbon/ram_utilization_percent": 5.312368672397158,
+  "energy/codecarbon/water_consumed": 0.0,
+  "energy/codecarbon/wue": 0.0,
+  "eval/duration_sec": 14.324650165159255,
+  "eval/loss": 7.523212566971779,
+  "perf/audit_duration_sec": 5.583974160254002,
+  "perf/epoch_duration_sec": 991.9095743251964,
+  "perf/epoch_samples": 55652.0,
+  "perf/epoch_samples_per_sec": 56.10592078200321,
+  "perf/epoch_tokens": 44091013.0,
+  "perf/epoch_tokens_per_sec": 44450.63757953486,
+  "perf/logical_batch_size": 65.0,
+  "perf/logical_token_count": 49862.0,
+  "perf/physical_batches": 9.0,
+  "perf/samples_per_sec": 6.719281189840856,
+  "perf/step_duration_sec": 9.673653797712177,
+  "perf/tokens_per_sec": 5154.412287505304,
+  "privacy/epsilon": 7.995186040237391,
+  "system/cuda_epoch_peak_memory_gb": 73.29015684127808,
+  "system/cuda_max_memory_allocated_gb": 73.29015684127808,
+  "system/cuda_memory_allocated_gb": 14.1148681640625,
+  "train/epoch_canary_loss": 10.571939474888266,
+  "train/epoch_loss": 8.035395088671393,
+  "train/epoch_real_loss": 7.897920552785785,
+  "train/lr": 8.126960406835249e-07,
+  "train/step_canary_loss": 11.0,
+  "train/step_loss": 7.722104996901292,
+  "train/step_real_loss": 7.670887887477875
+}

deepseek-coder-6.7b/dp8/train.log

@@ -0,0 +1,24 @@
+2026-03-17 21:52:24,143 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=14.7407
+2026-03-17 21:53:57,535 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=14.5101
+2026-03-17 21:55:30,416 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=14.0244
+2026-03-17 21:57:02,983 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=13.5601
+2026-03-17 21:58:36,151 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=12.9428
+2026-03-17 21:58:50,692 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=12.6723 duration_sec=14.54
+2026-03-17 22:00:23,309 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=12.3388
+2026-03-17 22:01:55,821 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=11.6780
+2026-03-17 22:03:30,259 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=10.9401
+2026-03-17 22:05:04,266 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=10.3183
+2026-03-17 22:06:36,997 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=9.7867
+2026-03-17 22:06:51,378 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=9.2172 duration_sec=14.38
+2026-03-17 22:08:46,560 [INFO] new_opacus_codex.train_steps: epoch=2 step=110 loss=9.0149
+2026-03-17 22:10:18,914 [INFO] new_opacus_codex.train_steps: epoch=2 step=120 loss=8.6390
+2026-03-17 22:11:51,283 [INFO] new_opacus_codex.train_steps: epoch=2 step=130 loss=8.3007
+2026-03-17 22:13:23,623 [INFO] new_opacus_codex.train_steps: epoch=2 step=140 loss=8.1182
+2026-03-17 22:14:56,451 [INFO] new_opacus_codex.train_steps: epoch=2 step=150 loss=7.9517
+2026-03-17 22:15:10,749 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=150 eval_loss=7.6677 duration_sec=14.30
+2026-03-17 22:16:43,057 [INFO] new_opacus_codex.train_steps: epoch=2 step=160 loss=7.8052
+2026-03-17 22:18:16,812 [INFO] new_opacus_codex.train_steps: epoch=2 step=170 loss=7.8663
+2026-03-17 22:19:49,796 [INFO] new_opacus_codex.train_steps: epoch=2 step=180 loss=7.8266
+2026-03-17 22:21:22,775 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=7.7819
+2026-03-17 22:22:55,474 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=7.7509
+2026-03-17 22:23:09,802 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=7.5236 duration_sec=14.33

granite-4.0-h-tiny/base/resolved_config.yaml

@@ -0,0 +1,110 @@
+model:
+  name: ibm-granite/granite-4.0-h-tiny
+  tokenizer_name: ibm-granite/granite-4.0-h-tiny
+  max_length: 1024
+  dtype: bfloat16
+  trust_remote_code: true
+  use_fast_tokenizer: true
+  cache_dir: null
+  local_files_only: false
+  low_cpu_mem_usage: true
+  tie_word_embeddings: true
+  gradient_checkpointing: false
+  use_chat_template: false
+dataset:
+  name: melihcatal/codedp-cpt
+  split: train
+  mode: cpt
+  text_column: text
+  validation_ratio: 0.05
+  max_samples: -1
+lora:
+  enabled: true
+  r: 16
+  alpha: 32
+  dropout: 0.05
+  target_modules:
+  - q_proj
+  - k_proj
+  - v_proj
+  - o_proj
+  modules_to_save:
+  - lm_head
+  bias: none
+training:
+  seed: 42
+  epochs: 2
+  warmup_steps: null
+  warmup_ratio: 0.05
+  mixed_precision: false
+  mixed_precision_dtype: bfloat16
+  batch_size: 8
+  eval_batch_size: 8
+  eval_every_steps: 50
+  eval_every_epochs: 1
+  learning_rate: 0.0001
+  optimizer: adamw
+  lr_scheduler: cosine
+  adam_beta1: 0.9
+  adam_beta2: 0.999
+  adam_epsilon: 1.0e-08
+  sgd_momentum: 0.9
+  weight_decay: 0.01
+  max_grad_norm: 1.0
+  log_every: 10
+  gradient_accumulation_steps: 4
+  num_workers: 4
+  output_dir: runs/cpt/granite-4.0-h-tiny/base
+distributed:
+  strategy: dpddp
+  backend: nccl
+  devices: null
+dp:
+  module_validator: auto
+  target_delta: 1.0e-05
+  noise_multiplier: null
+  max_grad_norm: 1.0
+  grad_sample_mode: hooks
+  clipping: flat
+  secure_mode: false
+  enabled: false
+  target_epsilon: 8.0
+audit:
+  enabled: true
+  run_every_epoch: true
+  epoch_device: cuda
+  q_canary: auto
+  num_canaries: 500
+  prefix_length: 49
+  num_digits: 12
+  batch_size: 32
+  delta: 1.0e-05
+  p_values:
+  - 0.05
+  - 0.01
+  paper_guess_fraction: 0.2
+  paper_guess_steps: 20
+  enable_holdout_empirical_epsilon: false
+  holdout_seed: 42
+  tie_seed: 42
+tracking:
+  enabled: true
+  tensorboard: true
+  wandb: false
+  wandb_project: codedp-finetune-h200-audit
+  wandb_run_name: granite-4.0-h-tiny-cpt-base
+  wandb_mode: online
+  codecarbon: true
+  codecarbon_output_file: codecarbon.csv
+  codecarbon_measure_power_secs: 15
+  codecarbon_country_iso_code: null
+  codecarbon_project_name: codedp-granite-4.0-h-tiny-cpt-base
+moe:
+  output_router_logits: false
+  router_aux_loss_coef: 0.0
+  freeze_router: true
+  profile:
+    enabled: false
+    num_batches: 8
+    top_experts: 8
+    output_file: moe_expert_profile.json

granite-4.0-h-tiny/base/scalars.csv

@@ -0,0 +1,35 @@
+timestamp,event,step,epoch,key,value
+1773822187.1417096,train_step,10,1,train/step_loss,4.135385597453398
+1773822187.1417096,train_step,10,1,train/step_real_loss,3.4641597270965576
+1773822187.1417096,train_step,10,1,train/lr,4.545454545454546e-05
+1773822187.1417096,train_step,10,1,train/step_canary_loss,14.875
+1773822187.1417096,train_step,10,1,perf/step_duration_sec,4.6632686029188335
+1773822187.1417096,train_step,10,1,perf/samples_per_sec,7.291023291842704
+1773822187.1417096,train_step,10,1,perf/tokens_per_sec,5700.722446774896
+1773822187.1417096,train_step,10,1,perf/logical_batch_size,34.0
+1773822187.1417096,train_step,10,1,perf/logical_token_count,26584.0
+1773822187.1417096,train_step,10,1,perf/gradient_accumulation_steps,4.0
+1773822187.1417096,train_step,10,1,system/cuda_memory_allocated_gb,16.85233783721924
+1773822187.1417096,train_step,10,1,system/cuda_max_memory_allocated_gb,60.90630769729614
+1773822228.5490818,train_step,20,1,train/step_loss,2.740965247154236
+1773822228.5490818,train_step,20,1,train/step_real_loss,2.740965247154236
+1773822228.5490818,train_step,20,1,train/lr,9.090909090909092e-05
+1773822228.5490818,train_step,20,1,perf/step_duration_sec,3.847110118251294
+1773822228.5490818,train_step,20,1,perf/samples_per_sec,8.317931906390456
+1773822228.5490818,train_step,20,1,perf/tokens_per_sec,6799.649398102124
+1773822228.5490818,train_step,20,1,perf/logical_batch_size,32.0
+1773822228.5490818,train_step,20,1,perf/logical_token_count,26159.0
+1773822228.5490818,train_step,20,1,perf/gradient_accumulation_steps,4.0
+1773822228.5490818,train_step,20,1,system/cuda_memory_allocated_gb,16.85233783721924
+1773822228.5490818,train_step,20,1,system/cuda_max_memory_allocated_gb,60.90630769729614
+1773822269.2440736,train_step,30,1,train/step_loss,1.4690485894680023
+1773822269.2440736,train_step,30,1,train/step_real_loss,1.4690485894680023
+1773822269.2440736,train_step,30,1,train/lr,9.990789447882137e-05
+1773822269.2440736,train_step,30,1,perf/step_duration_sec,3.921983283944428
+1773822269.2440736,train_step,30,1,perf/samples_per_sec,8.159137273990844
+1773822269.2440736,train_step,30,1,perf/tokens_per_sec,6951.839930480011
+1773822269.2440736,train_step,30,1,perf/logical_batch_size,32.0
+1773822269.2440736,train_step,30,1,perf/logical_token_count,27265.0
+1773822269.2440736,train_step,30,1,perf/gradient_accumulation_steps,4.0
+1773822269.2440736,train_step,30,1,system/cuda_memory_allocated_gb,16.85233783721924
+1773822269.2440736,train_step,30,1,system/cuda_max_memory_allocated_gb,60.90630769729614

granite-4.0-h-tiny/base/summary.json

@@ -0,0 +1,14 @@
+{
+  "perf/gradient_accumulation_steps": 4.0,
+  "perf/logical_batch_size": 32.0,
+  "perf/logical_token_count": 27265.0,
+  "perf/samples_per_sec": 8.159137273990844,
+  "perf/step_duration_sec": 3.921983283944428,
+  "perf/tokens_per_sec": 6951.839930480011,
+  "system/cuda_max_memory_allocated_gb": 60.90630769729614,
+  "system/cuda_memory_allocated_gb": 16.85233783721924,
+  "train/lr": 9.990789447882137e-05,
+  "train/step_canary_loss": 14.875,
+  "train/step_loss": 1.4690485894680023,
+  "train/step_real_loss": 1.4690485894680023
+}

granite-4.0-h-tiny/base/tokenizer/chat_template.jinja

@@ -0,0 +1,118 @@
+{%- set tools_system_message_prefix = 'You are a helpful assistant with access to the following tools. You may call one or more tools to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>'  %}
+{%- set tools_system_message_suffix = '\n</tools>\n\nFor each tool call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.' %}
+{%- set documents_system_message_prefix = 'You are a helpful assistant with access to the following documents. You may use one or more documents to assist with the user query.\n\nYou are given a list of documents within <documents></documents> XML tags:\n<documents>' %}
+{%- set documents_system_message_suffix = '\n</documents>\n\nWrite the response to the user\'s input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.' %}
+{%- set g4_default_system_message = 'You are a helpful assistant. Please ensure responses are professional, accurate, and safe.' %}
+{%- if available_tools is defined and available_tools %}
+    {%- set tools = available_tools %}
+{%- endif %}
+{%- set ns = namespace(tools_system_message=tools_system_message_prefix,
+                       documents_system_message=documents_system_message_prefix,
+                       default_system_message=g4_default_system_message,
+                       system_message=''
+                       ) %}
+{%- if tools %}
+    {%- for tool in tools %}
+        {%- set ns.tools_system_message = ns.tools_system_message + '\n' + (tool | tojson) %}
+    {%- endfor %}
+    {%- set ns.tools_system_message = ns.tools_system_message + tools_system_message_suffix %}
+{%- else %}
+    {%- set ns.tools_system_message = '' %}
+{%- endif %}
+{%- if documents %}
+    {%- for document in documents %}
+        {%- set ns.documents_system_message = ns.documents_system_message + '\n' + (document | tojson) %}
+    {%- endfor %}
+    {%- set ns.documents_system_message = ns.documents_system_message + documents_system_message_suffix %}
+{%- else %}
+    {%- set ns.documents_system_message = '' %}
+{%- endif %}
+{%- if messages[0].role == 'system' %}
+    {%- if messages[0].content is string %}
+        {%- set ns.system_message = messages[0].content %}
+    {%- elif messages[0].content is iterable %}
+        {%- for entry in messages[0].content %}
+            {%- if entry.type== 'text' %}
+                {%- if ns.system_message != '' %}
+                    {%- set ns.system_message = ns.system_message + '\n' %}
+                {%- endif %}
+                {%- set ns.system_message = ns.system_message + entry.text %}
+            {%- endif %}
+        {%- endfor %}
+    {%- endif %}
+    {%- if tools and documents %}
+        {%- set ns.system_message = ns.system_message + '\n\n' +  ns.tools_system_message + '\n\n' + ns.documents_system_message %}
+    {%- elif tools %}
+        {%- set ns.system_message = ns.system_message + '\n\n' + ns.tools_system_message %}
+    {%- elif documents %}
+        {%- set ns.system_message = ns.system_message + '\n\n' + ns.documents_system_message %}
+    {%- endif %}
+{%- else %}
+    {%- if tools and documents %}
+        {%- set ns.system_message = ns.tools_system_message + '\n\n' + ns.documents_system_message  %}
+    {%- elif tools %}
+        {%- set ns.system_message = ns.tools_system_message %}
+    {%- elif documents %}
+        {%- set ns.system_message = ns.documents_system_message %}
+    {%- endif %}
+{%- endif %}
+{%- if ns.system_message %}
+    {{- '<|start_of_role|>system<|end_of_role|>' + ns.system_message + '<|end_of_text|>\n' }}
+{%- else %}
+    {{- '<|start_of_role|>system<|end_of_role|>' + ns.default_system_message + '<|end_of_text|>\n' }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = namespace(val='') %}
+    {%- if message.content is string %}
+        {%- set content.val = message.content %}
+    {%- else %}
+        {%- if message.content is iterable %}
+            {%- for entry in message.content %}
+                {%- if entry.type== 'text' %}
+                    {%- if content.val != '' %}
+                        {%- set content.val = content.val + '\n' %}
+                    {%- endif %}
+                    {%- set content.val = content.val + entry.text %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+    {%- if (message.role == 'user') or (message.role == 'system' and not loop.first) %}
+        {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val + '<|end_of_text|>\n' }}
+    {%- elif message.role == 'assistant' %}
+        {{- '<|start_of_role|>' + message.role + '<|end_of_role|>' + content.val }}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content.val) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|end_of_text|>\n' }}
+    {%- elif message.role == 'tool' %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != 'tool') %}
+            {{- '<|start_of_role|>user<|end_of_role|>' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content.val }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != 'tool') %}
+            {{- '<|end_of_text|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_of_role|>assistant<|end_of_role|>' }}
+{%- endif %}

granite-4.0-h-tiny/base/tokenizer/tokenizer_config.json

@@ -0,0 +1,516 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|end_of_text|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|end_of_text|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "865331112869",
+    "569765693871",
+    "485177821815",
+    "135441121756",
+    "367459894796",
+    "877482678543",
+    "457919547633",
+    "765474393376",
+    "114848338811",
+    "746285987371",
+    "649291669397",
+    "927914615679",
+    "445925149649",
+    "691587454538",
+    "143777992227",
+    "997981281989",
+    "425949483533",
+    "982993456429",
+    "718726519731",
+    "172599315861",
+    "643489267333",
+    "282322838685",
+    "781653545886",
+    "796415361892",
+    "841991688488",
+    "211411365397",
+    "698218415444",
+    "355977139358",
+    "682564697312",
+    "383837596997",
+    "689362171782",
+    "749966767285",
+    "753159165157",
+    "795693824762",
+    "669689115557",
+    "327491773134",
+    "983569279932",
+    "612128769512",
+    "374327157578",
+    "311632789559",
+    "523918658846",
+    "765981581453",
+    "794825141891",
+    "873898736873",
+    "447445629421",
+    "473822473819",
+    "181439694557",
+    "592538279337",
+    "668134915514",
+    "643692393748",
+    "696651276628",
+    "853859348234",
+    "778466723723",
+    "929826356991",
+    "272362973463",
+    "694235616268",
+    "281673864127",
+    "479676316326",
+    "646979124677",
+    "922327493433",
+    "883685933161",
+    "264259917554",
+    "836746273134",
+    "658481324922",
+    "481884157827",
+    "587787496812",
+    "579184949249",
+    "912193598348",
+    "529679678956",
+    "795838284624",
+    "159337222655",
+    "173781362446",
+    "773687856563",
+    "535787224917",
+    "351885857332",
+    "578827344666",
+    "198462689911",
+    "722618266242",
+    "952872416512",
+    "517778845323",
+    "749665846687",
+    "661436365453",
+    "259666844669",
+    "242851284913",
+    "514532995959",
+    "161588262349",
+    "742765629356",
+    "225164373623",
+    "676539973863",
+    "826214551218",
+    "182345464792",
+    "232776999554",
+    "337326533813",
+    "676676697292",
+    "929185622831",
+    "545512344383",
+    "499444466686",
+    "314697386682",
+    "517379856925",
+    "379557332953",
+    "614797267726",
+    "429781429464",
+    "922466849763",
+    "721737645236",
+    "479227349997",
+    "136931728327",
+    "259533577263",
+    "488538864842",
+    "937495658852",
+    "489991411364",
+    "499148455254",
+    "441373944925",
+    "899151413682",
+    "467893531755",
+    "527117488925",
+    "928335588653",
+    "374439448821",
+    "879425227932",
+    "867678158885",
+    "399749397872",
+    "129693547287",
+    "689285841825",
+    "771619544974",
+    "724883568652",
+    "516968424863",
+    "733737988257",
+    "852347289392",
+    "296953381169",
+    "377273562477",
+    "262296912232",
+    "547149832394",
+    "298464134954",
+    "216667245274",
+    "843998562287",
+    "572154333646",
+    "124589118494",
+    "841824384614",
+    "232896526252",
+    "295448593321",
+    "123741461297",
+    "653573457168",
+    "196735786156",
+    "377338713663",
+    "964342468552",
+    "586855179568",
+    "484773717614",
+    "894885246797",
+    "677896358599",
+    "848845611563",
+    "851852651677",
+    "398549545767",
+    "454244839926",
+    "799364566435",
+    "967114116556",
+    "817378986438",
+    "233795848681",
+    "824387273757",
+    "916198946615",
+    "563117729724",
+    "951794811935",
+    "374598961236",
+    "922867396683",
+    "765737843639",
+    "175469284871",
+    "231853711778",
+    "662426712668",
+    "711412347158",
+    "753466987363",
+    "513361312532",
+    "712992815957",
+    "971621888444",
+    "829235161526",
+    "585544633356",
+    "582471228164",
+    "678666359123",
+    "557533689478",
+    "632962475133",
+    "484489193824",
+    "489562189822",
+    "589547936288",
+    "363214487524",
+    "244885399387",
+    "431751228368",
+    "433581868192",
+    "486391569221",
+    "185438575221",
+    "126574388585",
+    "741757479784",
+    "529854679937",
+    "996116119839",
+    "616248973917",
+    "763531783491",
+    "955456118295",
+    "364196983365",
+    "195792996468",
+    "151859598873",
+    "399223169721",
+    "938488813964",
+    "961981959227",
+    "183368827562",
+    "533417736566",
+    "786391632558",
+    "665661658354",
+    "693281533643",
+    "475794684356",
+    "652154162978",
+    "753233719644",
+    "668514843129",
+    "819162623892",
+    "941169431859",
+    "877385381798",
+    "752644929761",
+    "881136466196",
+    "275597777299",
+    "731681792655",
+    "961133895172",
+    "864718285734",
+    "963852916563",
+    "319584985416",
+    "563365646341",
+    "811371928234",
+    "837131396371",
+    "267514771964",
+    "944513428457",
+    "117298239631",
+    "158142752582",
+    "252867443568",
+    "839269684865",
+    "612788593128",
+    "145669731981",
+    "121557291859",
+    "245416776926",
+    "799417897197",
+    "997958836435",
+    "892336777248",
+    "158929292238",
+    "581976444672",
+    "897784492783",
+    "492373714791",
+    "512659818733",
+    "881112998642",
+    "619454958782",
+    "431149748713",
+    "624221476921",
+    "125866399464",
+    "339882449689",
+    "186198784585",
+    "943193294691",
+    "955668961269",
+    "232787996724",
+    "215671314196",
+    "286173241916",
+    "745977673725",
+    "556976448182",
+    "599961512792",
+    "766294538337",
+    "934912591213",
+    "295118729589",
+    "529455466433",
+    "196119929397",
+    "379571934299",
+    "251789649997",
+    "564544131355",
+    "244371196654",
+    "384598329253",
+    "887753195844",
+    "364947325679",
+    "655517954651",
+    "673948786567",
+    "857231548835",
+    "816115936673",
+    "644234165531",
+    "182782912224",
+    "234316622259",
+    "421369185549",
+    "434632855397",
+    "921889371893",
+    "415956914763",
+    "598916996413",
+    "773671349113",
+    "952465217972",
+    "117657531962",
+    "729825168745",
+    "691315125346",
+    "768461952319",
+    "664847713559",
+    "953267689786",
+    "886464195129",
+    "824488329416",
+    "837873762491",
+    "532833541879",
+    "669183782449",
+    "941976537588",
+    "739394546916",
+    "267954879268",
+    "637551427887",
+    "217756494954",
+    "524444658383",
+    "117783274348",
+    "138218735276",
+    "814611949491",
+    "711641973413",
+    "499156317423",
+    "515856611931",
+    "454164859837",
+    "345271433112",
+    "462294118988",
+    "511785788222",
+    "497294727353",
+    "866519986723",
+    "334513529294",
+    "549946382131",
+    "284445431422",
+    "396521188476",
+    "421435255895",
+    "133373659361",
+    "322683334381",
+    "228358422847",
+    "291762694874",
+    "143182978129",
+    "511923256573",
+    "327158398268",
+    "879764613759",
+    "564395222747",
+    "451161679736",
+    "538631466654",
+    "221762325616",
+    "218391991184",
+    "322589379462",
+    "876537814263",
+    "152676556624",
+    "332522971941",
+    "884354318946",
+    "513349618943",
+    "116639746413",
+    "635185846287",
+    "993832498489",
+    "813981174797",
+    "438745114173",
+    "983493951323",
+    "724492262421",
+    "622553389126",
+    "889965243135",
+    "364492359246",
+    "154962668224",
+    "179564995814",
+    "418412875665",
+    "718951851413",
+    "699446724178",
+    "624266421831",
+    "815458725125",
+    "455423278865",
+    "393741199486",
+    "328552864359",
+    "211662639865",
+    "218784516525",
+    "762486672996",
+    "142799718159",
+    "858146415154",
+    "767858144912",
+    "571317457151",
+    "635127952696",
+    "116427191984",
+    "268921994538",
+    "523937669294",
+    "165429152138",
+    "739246183345",
+    "591464355756",
+    "212985874612",
+    "191887635211",
+    "967214577653",
+    "119342152414",
+    "946444632795",
+    "618423867817",
+    "228565148417",
+    "729116422489",
+    "527874729936",
+    "739784153482",
+    "387763951128",
+    "331369926711",
+    "562716493614",
+    "739667844957",
+    "562389434565",
+    "256497188281",
+    "859927364588",
+    "417668946583",
+    "357621613582",
+    "438435178228",
+    "485692541169",
+    "825815739116",
+    "342221452223",
+    "697747991249",
+    "716763689965",
+    "141499982867",
+    "818479319499",
+    "336813343298",
+    "594688742928",
+    "472129283475",
+    "514354144759",
+    "349249721685",
+    "546276298359",
+    "353755529131",
+    "315534574435",
+    "523723475786",
+    "215826764872",
+    "367968398551",
+    "569853653352",
+    "389715484387",
+    "293847485454",
+    "714738141818",
+    "178478368922",
+    "581493616981",
+    "589439538674",
+    "846657726193",
+    "722339992679",
+    "138154781148",
+    "757785319772",
+    "492516914298",
+    "919181521716",
+    "985781138935",
+    "476969195485",
+    "313145133463",
+    "758963111966",
+    "147541537162",
+    "557163366873",
+    "144373897488",
+    "522515164754",
+    "724964923582",
+    "284776712475",
+    "375429755114",
+    "181233596124",
+    "948585673431",
+    "243165586174",
+    "396847976144",
+    "997724962668",
+    "558837194455",
+    "163165456396",
+    "378749551722",
+    "161238482259",
+    "754978243758",
+    "195388849133",
+    "229775525672",
+    "262437452884",
+    "441377892146",
+    "451885565366",
+    "981277526855",
+    "762495822823",
+    "368763327262",
+    "757422791351",
+    "636324136426",
+    "214193645583",
+    "412843856172",
+    "179386156569",
+    "756916173536",
+    "892697125149",
+    "625334487352",
+    "941861857715",
+    "887417525236",
+    "649516938598",
+    "717628619782",
+    "438124184139",
+    "547563892268",
+    "856317483891",
+    "313313831273",
+    "371496153876",
+    "587541149322",
+    "265847332563",
+    "449549215429",
+    "163497196769",
+    "861342291298",
+    "268433315926",
+    "774679513717",
+    "851254219729",
+    "583527834464",
+    "488496781997",
+    "556814553861",
+    "482829231639",
+    "618878266619",
+    "147444452794",
+    "949235426629",
+    "357299947518",
+    "175528632226",
+    "645527857972",
+    "186872457894",
+    "552738847828",
+    "626748382482",
+    "921894985642",
+    "943878645871",
+    "859289776479",
+    "614583493135",
+    "933775286797",
+    "332234613346",
+    "325196781219",
+    "142526557681",
+    "356722692178",
+    "449318681694",
+    "687284547244",
+    "947262995132",
+    "893974619684",
+    "797238311233"
+  ],
+  "is_local": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "padding_side": "left",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|unk|>"
+}

granite-4.0-h-tiny/base/train.log

@@ -0,0 +1,3 @@
+2026-03-18 08:23:07,141 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=3.7489
+2026-03-18 08:23:48,548 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=3.2296
+2026-03-18 08:24:29,243 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=2.0376