Pathumlakmal's picture
🏗️ Building on HF

Pathumlakmal

Pq234
·

AI & ML interests

Found a failure in the smoke test `pytest tests/smoke_tests/test_mount_and_storage.py::test_huggingface_storage_mounts --huggingface --generic-cloud azure` **Blocking bug — COPY-mode file_mounts download to a literal `~/` directory, not the user's home.** SkyPilot's file_mount machinery hands callers a *wrapped* destination like `~/.sky/file_mounts/<mount_point>` (from `backend_utils.FileMountHelper.wrap_file_mount`). Every other cloud-store backend handles this because they shell out (`aws s3 sync`, `gsutil rsync`, `rclone`, etc.) and the shell expands `~`. `HFCloudStorage` is unique — it invokes Python directly via `python -c '...'` calling `HfApi().sync_bucket(source, '~/.sky/file_mounts/<mount>', ...)`. **Python does not expand `~`.** `huggingface_hub` treats `~` as a literal directory name, so files land at `/home/<user>/~/.sky/file_mounts/<mount>/…` while the symlink (`/<mount>` → `/home/<user>/.sky/file_mounts/<mount>`) points at the intended, empty location. Net effect: any task using `store: hf, mode: COPY` (or `hf://` as a `source` in COPY mode) lands no files at the mount point. The smoke test in this PR fails on the first `ls /mount_bucket_copy/foo`. Reproduced end-to-end on Azure. Live VM evidence: ``` $ ls -la /home/azureuser/~/.sky/file_mounts/mount_bucket_copy/ # literal ~ dir (where files actually landed) total 8 -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 foo -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 tmp file -rw-rw-r-- 1 azureuser azureuser 0 May 22 03:54 tmp file2 $ ls -la /mount_bucket_copy/ # symlink target (intended; empty) total 8 drwxrwxr-x 2 azureuser azureuser 4096 May 22 03:54 . drwxrwxr-x 4 azureuser azureuser 4096 May 22 03:54 .. ``` After manually running `HfApi().sync_bucket(src, os.path.expanduser('~/.sky/file_mounts/mount_bucket_copy'))` on the same VM, the files appeared correctly at the symlink target. **Fix.** Wrap every destination passed into the generated Python with `os.path.expanduser(...)`. `os` is already imported in the generated code. Four call sites in this file: | Method | Branch | Change | |---|---|---| | `make_sync_dir_command` | bucket | `sync_bucket(src, {destination!r}, ...)` → `sync_bucket(src, os.path.expanduser({destination!r}), ...)` | | `make_sync_dir_command` | repo | `local_dir={destination!r}` → `local_dir=os.path.expanduser({destination!r})` | | `make_sync_file_command` | bucket | `files=[({path!r}, {destination!r})]` → `files=[({path!r}, os.path.expanduser({destination!r}))]` | | `make_sync_file_command` | repo | `shutil.copy2(downloaded, {destination!r})` → `shutil.copy2(downloaded, os.path.expanduser({destination!r}))` |

Recent Activity

Organizations

Tos's profile picture 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's profile picture d9bc68e's profile picture