Create https://huggingface.co/blog/skypilot-hf-storage

#45
by Pq234 - opened
https:/huggingface.co/blog/skypilot-hf-storage ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Found a failure in the smoke test `pytest tests/smoke_tests/test_mount_and_storage.py::test_huggingface_storage_mounts --huggingface --generic-cloud azure`
2
+
3
+ **Blocking bug β€” COPY-mode file_mounts download to a literal `~/` directory, not the user's home.**
4
+
5
+ SkyPilot's file_mount machinery hands callers a *wrapped* destination like `~/.sky/file_mounts/<mount_point>` (from `backend_utils.FileMountHelper.wrap_file_mount`). Every other cloud-store backend handles this because they shell out (`aws s3 sync`, `gsutil rsync`, `rclone`, etc.) and the shell
6
+ expands `~`. `HFCloudStorage` is unique β€” it invokes Python directly via `python -c '...'` calling `HfApi().sync_bucket(source, '~/.sky/file_mounts/<mount>', ...)`. **Python does not expand `~`.** `huggingface_hub` treats `~` as a literal directory name, so files land at
7
+ `/home/<user>/~/.sky/file_mounts/<mount>/…` while the symlink (`/<mount>` β†’ `/home/<user>/.sky/file_mounts/<mount>`) points at the intended, empty location.
8
+
9
+ Net effect: any task using `store: hf, mode: COPY` (or `hf://` as a `source` in COPY mode) lands no files at the mount point. The smoke test in this PR fails on the first `ls /mount_bucket_copy/foo`.
10
+
11
+ Reproduced end-to-end on Azure. Live VM evidence:
12
+
13
+ ```
14
+ $ ls -la /home/azureuser/~/.sky/file_mounts/mount_bucket_copy/ # literal ~ dir (where files actually landed)
15
+ total 8
16
+ -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 foo
17
+ -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 tmp file
18
+ -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 tmp file2
19
+
20
+ $ ls -la /mount_bucket_copy/ # symlink target (intended; empty)
21
+ total 8
22
+ drwxrwxr-x 2 azureuser azureuser 4096 May 22 03:54 .
23
+ drwxrwxr-x 4 azureuser azureuser 4096 May 22 03:54 ..
24
+ ```
25
+
26
+ After manually running `HfApi().sync_bucket(src, os.path.expanduser('~/.sky/file_mounts/mount_bucket_copy'))` on the same VM, the files appeared correctly at the symlink target.
27
+
28
+ **Fix.** Wrap every destination passed into the generated Python with `os.path.expanduser(...)`. `os` is already imported in the generated code. Four call sites in this file:
29
+
30
+ | Method | Branch | Change |
31
+ |---|---|---|
32
+ | `make_sync_dir_command` | bucket | `sync_bucket(src, {destination!r}, ...)` β†’ `sync_bucket(src, os.path.expanduser({destination!r}), ...)` |
33
+ | `make_sync_dir_command` | repo | `local_dir={destination!r}` β†’ `local_dir=os.path.expanduser({destination!r})` |
34
+ | `make_sync_file_command` | bucket | `files=[({path!r}, {destination!r})]` β†’ `files=[({path!r}, os.path.expanduser({destination!r}))]` |
35
+ | `make_sync_file_command` | repo | `shutil.copy2(downloaded, {destination!r})` β†’ `shutil.copy2(downloaded, os.path.expanduser({destination!r}))` |