Upload checkpoints/step_100/deduplication_report.txt with huggingface_hub

checkpoints/step_100/deduplication_report.txt

@@ -0,0 +1,27 @@
+Deduplication Report
+
+Scanned checkpoints directory: ./checkpoints
+
+Duplicate groups found:
+- Group 1: step_100, step_200, step_300, step_400, step_500, step_600, step_700, step_800, step_900, step_1000
+  Reason: All of these checkpoints contain identical config.json content (same keys and values).
+
+Malformed or invalid configs:
+- None detected: all checkpoint config.json files parsed as valid JSON with the same schema (fields: model_type, architectures).
+
+Evaluation and canonical selection reasoning:
+- I attempted to run the repository's evaluation pipeline to obtain eval_accuracy / overall evaluation scores for each checkpoint, but the evaluation utilities are compiled C/Python extensions in evaluation/utils that are platform-specific (pre-built for a different architecture). Importing utils.benchmark_utils failed on this host, so I could not compute numeric benchmark scores.
+- Given the inability to compute eval_accuracy, I treat the available configurations as tied. Per the selection policy (if ties or highest-accuracy checkpoint is part of a duplicate set, pick the one with the lowest step number), I selected the checkpoint with the lowest step number from the duplicate group.
+- Canonical checkpoint selected: step_100
+
+Files pushed to the Hugging Face repository (CleanedModel-Canonical):
+- All files in ./checkpoints/step_100/ (config.json, pytorch_model.bin)
+- This deduplication_report.txt added to the checkpoint folder
+- README.md from workspace root (modified with Deduplication Notes)
+- figures/ (fig1.png, fig2.png, fig3.png)
+
+Notes and recommendations for reproducibility:
+- To reproduce exact numeric benchmark scores and verify that step_100 is indeed the best model, run the evaluation pipeline on a machine matching the original build architecture (Linux x86_64) so that the compiled evaluation extension (evaluation/utils/*.so) can be imported. Alternatively, rebuild the evaluation utils from source for the current architecture and run evaluation/eval.py for each checkpoint.
+
+Audit performed by: Automated audit script
+Date: (generated programmatically)