sync final logs after metric artifact update
Browse files
workspace/outputs/hf_sync/hf_push_daemon_login.out
CHANGED
|
@@ -4501,3 +4501,9 @@ Traceback (most recent call last):
|
|
| 4501 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 4502 |
OSError: I/O error: I/O error: Cannot send after transport endpoint shutdown (os error 108)
|
| 4503 |
[2026-07-04T04:19:22Z] sync cycle failed with status=1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4501 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 4502 |
OSError: I/O error: I/O error: Cannot send after transport endpoint shutdown (os error 108)
|
| 4503 |
[2026-07-04T04:19:22Z] sync cycle failed with status=1
|
| 4504 |
+
[2026-07-04T04:34:22Z] checking Hugging Face auth
|
| 4505 |
+
[2026-07-04T04:34:25Z] sync start repo=anhtld/vla type=model dry_run=0
|
| 4506 |
+
[2026-07-04T04:34:26Z] upload /lustre09/project/6037638/knguy52/vla -> anhtld/vla/workspace
|
| 4507 |
+
/lustre09/project/6037638/knguy52/vla/.venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py:11299: UserWarning: Warnings while validating metadata in README.md:
|
| 4508 |
+
- empty or missing yaml metadata in repo card
|
| 4509 |
+
warnings.warn(f"Warnings while validating metadata in README.md:\n{message}")
|
workspace/outputs/hf_sync/hf_push_every_15m.setsid.log
CHANGED
|
@@ -2158,3 +2158,46 @@ Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 commi
|
|
| 2158 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2159 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2160 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2158 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2159 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2160 |
Uploading... 256/31,202 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2161 |
+
Uploading... 25,856/31,202 files checked, 162/162 uploaded (0.00B transferred), 25,750 committed in 2 commit(s)
|
| 2162 |
+
Upload completed in 8 commits.
|
| 2163 |
+
url=https://huggingface.co/anhtld/vla/commit/52f42ba921bd701ba0bc4720826e314b4f7365fc
|
| 2164 |
+
[2026-07-04T04:33:15Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs -> anhtld/vla/scratch/experiments/dovla_h16_policy_ckpt_runs
|
| 2165 |
+
Found 1,276 files to upload
|
| 2166 |
+
Uploading... 256/1,276 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2167 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2168 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2169 |
+
[2026-07-04T04:33:47Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs -> anhtld/vla/scratch/experiments/dovla_h16_rollout_runs
|
| 2170 |
+
Found 51 files to upload
|
| 2171 |
+
Uploading... 51/51 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2172 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2173 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2174 |
+
[2026-07-04T04:33:52Z] upload /scratch/knguy52/dovla/experiments/h16_merged_dataset -> anhtld/vla/scratch/experiments/h16_merged_dataset
|
| 2175 |
+
Found 49 files to upload
|
| 2176 |
+
Uploading... 49/49 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2177 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2178 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2179 |
+
[2026-07-04T04:33:56Z] upload /scratch/knguy52/dovla/experiments/six_task_h16_collection -> anhtld/vla/scratch/experiments/six_task_h16_collection
|
| 2180 |
+
Found 87 files to upload
|
| 2181 |
+
Uploading... 87/87 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2182 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2183 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2184 |
+
[2026-07-04T04:34:04Z] upload /scratch/knguy52/dovla/experiments/embedding_cache -> anhtld/vla/scratch/experiments/embedding_cache
|
| 2185 |
+
Found 0 files to upload
|
| 2186 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2187 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2188 |
+
[2026-07-04T04:34:04Z] upload /scratch/knguy52/dovla/cache -> anhtld/vla/scratch/cache
|
| 2189 |
+
Found 10 files to upload
|
| 2190 |
+
Uploading... 10/10 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2191 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2192 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2193 |
+
[2026-07-04T04:34:29Z] upload /scratch/knguy52/dovla/caches -> anhtld/vla/scratch/caches
|
| 2194 |
+
Found 3 files to upload
|
| 2195 |
+
Uploading... 3/3 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
|
| 2196 |
+
No files have been modified since last commit. Skipping to prevent empty commit.
|
| 2197 |
+
url=https://huggingface.co/anhtld/vla/commit/f3fd10f6aa3790eb2df5acd9ec39d69ffd767a08
|
| 2198 |
+
[2026-07-04T04:34:42Z] upload /home/knguy52/links/projects/def-yalda/knguy52/vla/outputs/hf_sync/last_manifest.json -> anhtld/vla/manifests/hf_sync_last_manifest.json
|
| 2199 |
+
/lustre09/project/6037638/knguy52/vla/.venv/lib/python3.11/site-packages/huggingface_hub/cli/upload.py:189: UserWarning: Ignoring --exclude since a single file is uploaded.
|
| 2200 |
+
warnings.warn("Ignoring --exclude since a single file is uploaded.")
|
| 2201 |
+
url=https://huggingface.co/anhtld/vla/commit/090513a59075fc17c01bab4375ab722e1c81976f
|
| 2202 |
+
[2026-07-04T04:34:48Z] sync complete
|
| 2203 |
+
[2026-07-04T04:34:48Z] sync cycle succeeded
|
workspace/outputs/hf_sync/hf_sync.log
CHANGED
|
@@ -1324,3 +1324,24 @@
|
|
| 1324 |
[2026-07-04T04:30:21Z] checking Hugging Face auth
|
| 1325 |
[2026-07-04T04:30:24Z] sync start repo=anhtld/vla type=model dry_run=0
|
| 1326 |
[2026-07-04T04:30:25Z] upload /lustre09/project/6037638/knguy52/vla -> anhtld/vla/workspace
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1324 |
[2026-07-04T04:30:21Z] checking Hugging Face auth
|
| 1325 |
[2026-07-04T04:30:24Z] sync start repo=anhtld/vla type=model dry_run=0
|
| 1326 |
[2026-07-04T04:30:25Z] upload /lustre09/project/6037638/knguy52/vla -> anhtld/vla/workspace
|
| 1327 |
+
[2026-07-04T04:33:15Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs -> anhtld/vla/scratch/experiments/dovla_h16_policy_ckpt_runs
|
| 1328 |
+
[2026-07-04T04:33:15Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs -> anhtld/vla/scratch/experiments/dovla_h16_policy_ckpt_runs
|
| 1329 |
+
[2026-07-04T04:33:47Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs -> anhtld/vla/scratch/experiments/dovla_h16_rollout_runs
|
| 1330 |
+
[2026-07-04T04:33:47Z] upload /scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs -> anhtld/vla/scratch/experiments/dovla_h16_rollout_runs
|
| 1331 |
+
[2026-07-04T04:33:52Z] upload /scratch/knguy52/dovla/experiments/h16_merged_dataset -> anhtld/vla/scratch/experiments/h16_merged_dataset
|
| 1332 |
+
[2026-07-04T04:33:52Z] upload /scratch/knguy52/dovla/experiments/h16_merged_dataset -> anhtld/vla/scratch/experiments/h16_merged_dataset
|
| 1333 |
+
[2026-07-04T04:33:56Z] upload /scratch/knguy52/dovla/experiments/six_task_h16_collection -> anhtld/vla/scratch/experiments/six_task_h16_collection
|
| 1334 |
+
[2026-07-04T04:33:56Z] upload /scratch/knguy52/dovla/experiments/six_task_h16_collection -> anhtld/vla/scratch/experiments/six_task_h16_collection
|
| 1335 |
+
[2026-07-04T04:34:03Z] upload /scratch/knguy52/dovla/experiments/embedding_cache -> anhtld/vla/scratch/experiments/embedding_cache
|
| 1336 |
+
[2026-07-04T04:34:04Z] upload /scratch/knguy52/dovla/experiments/embedding_cache -> anhtld/vla/scratch/experiments/embedding_cache
|
| 1337 |
+
[2026-07-04T04:34:04Z] upload /scratch/knguy52/dovla/cache -> anhtld/vla/scratch/cache
|
| 1338 |
+
[2026-07-04T04:34:04Z] upload /scratch/knguy52/dovla/cache -> anhtld/vla/scratch/cache
|
| 1339 |
+
[2026-07-04T04:34:22Z] checking Hugging Face auth
|
| 1340 |
+
[2026-07-04T04:34:25Z] sync start repo=anhtld/vla type=model dry_run=0
|
| 1341 |
+
[2026-07-04T04:34:26Z] upload /lustre09/project/6037638/knguy52/vla -> anhtld/vla/workspace
|
| 1342 |
+
[2026-07-04T04:34:29Z] upload /scratch/knguy52/dovla/caches -> anhtld/vla/scratch/caches
|
| 1343 |
+
[2026-07-04T04:34:29Z] upload /scratch/knguy52/dovla/caches -> anhtld/vla/scratch/caches
|
| 1344 |
+
[2026-07-04T04:34:42Z] upload /lustre09/project/6037638/knguy52/vla/outputs/hf_sync/last_manifest.json -> anhtld/vla/manifests/hf_sync_last_manifest.json
|
| 1345 |
+
[2026-07-04T04:34:42Z] upload /home/knguy52/links/projects/def-yalda/knguy52/vla/outputs/hf_sync/last_manifest.json -> anhtld/vla/manifests/hf_sync_last_manifest.json
|
| 1346 |
+
[2026-07-04T04:34:44Z] sync complete
|
| 1347 |
+
[2026-07-04T04:34:48Z] sync complete
|
workspace/outputs/hf_sync/hf_sync_daemon.log
CHANGED
|
@@ -121,3 +121,4 @@
|
|
| 121 |
[2026-07-04T03:44:08Z] sync cycle succeeded
|
| 122 |
[2026-07-04T04:13:17Z] sync cycle succeeded
|
| 123 |
[2026-07-04T04:19:22Z] sync cycle failed with status=1
|
|
|
|
|
|
| 121 |
[2026-07-04T03:44:08Z] sync cycle succeeded
|
| 122 |
[2026-07-04T04:13:17Z] sync cycle succeeded
|
| 123 |
[2026-07-04T04:19:22Z] sync cycle failed with status=1
|
| 124 |
+
[2026-07-04T04:34:48Z] sync cycle succeeded
|
workspace/outputs/hf_sync/last_manifest.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
{
|
| 2 |
-
"generated_utc": "2026-07-04T04:
|
| 3 |
"repo_id": "anhtld/vla",
|
| 4 |
"project_dir": "/lustre09/project/6037638/knguy52/vla",
|
| 5 |
"scratch_root": "/scratch/knguy52/dovla",
|
|
|
|
| 1 |
{
|
| 2 |
+
"generated_utc": "2026-07-04T04:34:26.018788+00:00",
|
| 3 |
"repo_id": "anhtld/vla",
|
| 4 |
"project_dir": "/lustre09/project/6037638/knguy52/vla",
|
| 5 |
"scratch_root": "/scratch/knguy52/dovla",
|