Update README for model repo with proper frontmatter
Browse files- Add YAML frontmatter so HuggingFace shows clone widget
- Update pack list to current names
- Remove all --repo-type dataset references
- Simplify CLI examples with actual repo ID
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README.md
CHANGED
|
@@ -1,25 +1,41 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# ContentVault
|
| 2 |
|
| 3 |
Content packs for [Lucky Robots](https://luckyrobots.com). Each pack contains a `metadata.yaml`, a `thumbnail.png`, and a `.zip` with the pack contents.
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
## Content Packs
|
| 6 |
|
| 7 |
| Pack | Description |
|
| 8 |
|------|-------------|
|
| 9 |
-
|
|
| 10 |
-
|
|
|
|
|
| 11 |
| MujocoExample | MuJoCo simulation example |
|
|
|
|
| 12 |
| Oscillator | Oscillator environment |
|
| 13 |
| Panda | Franka Panda robot |
|
| 14 |
| Piper | Piper robot |
|
| 15 |
-
|
|
|
|
|
| 16 |
| PiperUnscrewCap | Piper cap unscrewing task |
|
|
|
|
| 17 |
| Skies-Vol-1 | Sky HDRIs volume 1 |
|
| 18 |
-
| TheBungalow | Bungalow environment |
|
| 19 |
-
| TheLoft | Loft environment |
|
| 20 |
-
| TheOffice | Office environment |
|
| 21 |
| UnitreeG1 | Unitree G1 humanoid |
|
| 22 |
-
|
|
| 23 |
| Welcome | Welcome / intro pack |
|
| 24 |
|
| 25 |
## CDN
|
|
@@ -32,17 +48,18 @@ https://contentvault.luckyrobots.com/{PackName}/{filename}
|
|
| 32 |
|
| 33 |
For example:
|
| 34 |
```
|
| 35 |
-
https://contentvault.luckyrobots.com/
|
| 36 |
-
https://contentvault.luckyrobots.com/
|
| 37 |
```
|
| 38 |
|
| 39 |
## How sync works
|
| 40 |
|
| 41 |
-
A Cloudflare Worker (`worker/`) listens for webhooks from HuggingFace. When a push is made to
|
| 42 |
|
| 43 |
-
1.
|
| 44 |
2. Streams each file directly to Cloudflare R2 (no buffering, handles large zip files)
|
| 45 |
-
3.
|
|
|
|
| 46 |
|
| 47 |
## Worker setup
|
| 48 |
|
|
@@ -64,8 +81,6 @@ npx wrangler secret put HF_TOKEN
|
|
| 64 |
npx wrangler deploy
|
| 65 |
```
|
| 66 |
|
| 67 |
-
### Commands
|
| 68 |
-
|
| 69 |
## Setup
|
| 70 |
|
| 71 |
```bash
|
|
@@ -82,119 +97,27 @@ hf auth login
|
|
| 82 |
|
| 83 |
### Upload entire folder
|
| 84 |
```bash
|
| 85 |
-
hf upload
|
| 86 |
```
|
| 87 |
|
| 88 |
### Upload a single pack/subfolder
|
| 89 |
```bash
|
| 90 |
-
hf upload
|
| 91 |
```
|
| 92 |
|
| 93 |
### Upload a single file
|
| 94 |
```bash
|
| 95 |
-
hf upload
|
| 96 |
```
|
| 97 |
|
| 98 |
### Upload large folder (resumable, multi-threaded, auto-retry)
|
| 99 |
```bash
|
| 100 |
-
hf upload-large-folder
|
| 101 |
-
```
|
| 102 |
-
|
| 103 |
-
### Delete files remotely (without re-uploading everything)
|
| 104 |
-
```bash
|
| 105 |
-
hf upload <repo-id> . . --repo-type dataset --include="" --delete="<folder-name>/*" --commit-message "Remove <folder-name>"
|
| 106 |
-
```
|
| 107 |
-
|
| 108 |
-
### Create a new dataset repo
|
| 109 |
-
```bash
|
| 110 |
-
hf repo create <repo-name> --type dataset
|
| 111 |
-
```
|
| 112 |
-
|
| 113 |
-
### Create under an organization
|
| 114 |
-
```bash
|
| 115 |
-
hf repo create <repo-name> --type dataset --organization <org-name>
|
| 116 |
-
```
|
| 117 |
-
|
| 118 |
-
---
|
| 119 |
-
|
| 120 |
-
## Python One-Liners
|
| 121 |
-
|
| 122 |
-
### Upload large folder (resumable)
|
| 123 |
-
```bash
|
| 124 |
-
python -c "from huggingface_hub import HfApi; HfApi().upload_large_folder(repo_id='<repo-id>', repo_type='dataset', folder_path='<local-folder>')"
|
| 125 |
```
|
| 126 |
|
| 127 |
-
###
|
| 128 |
```bash
|
| 129 |
-
|
| 130 |
-
```
|
| 131 |
-
|
| 132 |
-
### Upload a single file
|
| 133 |
-
```bash
|
| 134 |
-
python -c "from huggingface_hub import HfApi; HfApi().upload_file(path_or_fileobj='<local-file>', path_in_repo='<path-in-repo>', repo_id='<repo-id>', repo_type='dataset', commit_message='Your message')"
|
| 135 |
-
```
|
| 136 |
-
|
| 137 |
-
### Delete a folder
|
| 138 |
-
```bash
|
| 139 |
-
python -c "from huggingface_hub import HfApi; HfApi().delete_folder('<folder-name>', repo_id='<repo-id>', repo_type='dataset', commit_message='Remove <folder-name>')"
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
### Delete a single file
|
| 143 |
-
```bash
|
| 144 |
-
python -c "from huggingface_hub import HfApi; HfApi().delete_file('<file-path>', repo_id='<repo-id>', repo_type='dataset', commit_message='Remove <file-path>')"
|
| 145 |
-
```
|
| 146 |
-
|
| 147 |
-
### Create a repo
|
| 148 |
-
```bash
|
| 149 |
-
python -c "from huggingface_hub import HfApi; HfApi().create_repo(repo_id='<repo-id>', repo_type='dataset')"
|
| 150 |
-
```
|
| 151 |
-
|
| 152 |
-
---
|
| 153 |
-
|
| 154 |
-
## Python Script Examples
|
| 155 |
-
|
| 156 |
-
### Upload large folder (full script with options)
|
| 157 |
-
```python
|
| 158 |
-
from huggingface_hub import HfApi
|
| 159 |
-
|
| 160 |
-
api = HfApi()
|
| 161 |
-
api.upload_large_folder(
|
| 162 |
-
repo_id="<repo-id>",
|
| 163 |
-
repo_type="dataset",
|
| 164 |
-
folder_path="<local-folder>",
|
| 165 |
-
)
|
| 166 |
-
```
|
| 167 |
-
|
| 168 |
-
### Upload a folder to a specific path in the repo
|
| 169 |
-
```python
|
| 170 |
-
from huggingface_hub import HfApi
|
| 171 |
-
|
| 172 |
-
api = HfApi()
|
| 173 |
-
api.upload_folder(
|
| 174 |
-
folder_path="<local-folder>",
|
| 175 |
-
path_in_repo="<path-in-repo>",
|
| 176 |
-
repo_id="<repo-id>",
|
| 177 |
-
repo_type="dataset",
|
| 178 |
-
commit_message="Your message",
|
| 179 |
-
ignore_patterns=["*.cache", ".git/*"],
|
| 180 |
-
)
|
| 181 |
-
```
|
| 182 |
-
|
| 183 |
-
### Batch delete + upload in one commit
|
| 184 |
-
```python
|
| 185 |
-
from huggingface_hub import HfApi, CommitOperationAdd, CommitOperationDelete
|
| 186 |
-
|
| 187 |
-
api = HfApi()
|
| 188 |
-
operations = [
|
| 189 |
-
CommitOperationDelete(path_in_repo="<old-folder>/"),
|
| 190 |
-
CommitOperationAdd(path_in_repo="<file-path>", path_or_fileobj="<local-file>"),
|
| 191 |
-
]
|
| 192 |
-
api.create_commit(
|
| 193 |
-
repo_id="<repo-id>",
|
| 194 |
-
repo_type="dataset",
|
| 195 |
-
operations=operations,
|
| 196 |
-
commit_message="Your message",
|
| 197 |
-
)
|
| 198 |
```
|
| 199 |
|
| 200 |
---
|
|
@@ -202,8 +125,7 @@ api.create_commit(
|
|
| 202 |
## Tips
|
| 203 |
|
| 204 |
- **Resumable uploads**: `upload_large_folder` and `hf upload-large-folder` are resumable. If interrupted, run the same command again.
|
| 205 |
-
- **Don't use raw `git push`** for large files
|
| 206 |
- **Write token required**: Tokens default to read-only. Enable write access at https://huggingface.co/settings/tokens.
|
| 207 |
- **50 GB max per file** on HuggingFace. Split larger files before uploading.
|
| 208 |
-
- **Ignore patterns**: Use `--include` and `--exclude`
|
| 209 |
-
- **Delete the `.cache/huggingface/` folder** inside your local folder to reset upload state if something goes wrong with `upload_large_folder`.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- content-packs
|
| 5 |
+
- robotics
|
| 6 |
+
- 3d-environments
|
| 7 |
+
- luckyrobots
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
# ContentVault
|
| 11 |
|
| 12 |
Content packs for [Lucky Robots](https://luckyrobots.com). Each pack contains a `metadata.yaml`, a `thumbnail.png`, and a `.zip` with the pack contents.
|
| 13 |
|
| 14 |
+
## Clone
|
| 15 |
+
|
| 16 |
+
```bash
|
| 17 |
+
git clone https://huggingface.co/luckyrobots/ContentVault
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
## Content Packs
|
| 21 |
|
| 22 |
| Pack | Description |
|
| 23 |
|------|-------------|
|
| 24 |
+
| Bunglalow | Bungalow environment |
|
| 25 |
+
| Go2VelocityTracking | Unitree Go2 velocity tracking |
|
| 26 |
+
| Loft | Loft environment |
|
| 27 |
| MujocoExample | MuJoCo simulation example |
|
| 28 |
+
| Office | Office environment |
|
| 29 |
| Oscillator | Oscillator environment |
|
| 30 |
| Panda | Franka Panda robot |
|
| 31 |
| Piper | Piper robot |
|
| 32 |
+
| PiperBlockStacking | Piper block stacking task |
|
| 33 |
+
| PiperPickPlace | Piper pick and place task |
|
| 34 |
| PiperUnscrewCap | Piper cap unscrewing task |
|
| 35 |
+
| SO100PickAndPlace | SO-100 pick and place task |
|
| 36 |
| Skies-Vol-1 | Sky HDRIs volume 1 |
|
|
|
|
|
|
|
|
|
|
| 37 |
| UnitreeG1 | Unitree G1 humanoid |
|
| 38 |
+
| UnitreeGo2 | Unitree Go2 quadruped |
|
| 39 |
| Welcome | Welcome / intro pack |
|
| 40 |
|
| 41 |
## CDN
|
|
|
|
| 48 |
|
| 49 |
For example:
|
| 50 |
```
|
| 51 |
+
https://contentvault.luckyrobots.com/Loft/metadata.yaml
|
| 52 |
+
https://contentvault.luckyrobots.com/Loft/Loft.zip
|
| 53 |
```
|
| 54 |
|
| 55 |
## How sync works
|
| 56 |
|
| 57 |
+
A Cloudflare Worker (`worker/`) listens for webhooks from HuggingFace. When a push is made to this repo, the Worker:
|
| 58 |
|
| 59 |
+
1. Parses the commit diff to find changed files
|
| 60 |
2. Streams each file directly to Cloudflare R2 (no buffering, handles large zip files)
|
| 61 |
+
3. Deletes removed files from R2
|
| 62 |
+
4. R2 serves the files through Cloudflare's CDN with a 30-day cache at 300+ edge locations worldwide
|
| 63 |
|
| 64 |
## Worker setup
|
| 65 |
|
|
|
|
| 81 |
npx wrangler deploy
|
| 82 |
```
|
| 83 |
|
|
|
|
|
|
|
| 84 |
## Setup
|
| 85 |
|
| 86 |
```bash
|
|
|
|
| 97 |
|
| 98 |
### Upload entire folder
|
| 99 |
```bash
|
| 100 |
+
hf upload luckyrobots/ContentVault <local-folder> . --commit-message "Your message"
|
| 101 |
```
|
| 102 |
|
| 103 |
### Upload a single pack/subfolder
|
| 104 |
```bash
|
| 105 |
+
hf upload luckyrobots/ContentVault <local-folder>/<pack-name> <pack-name> --commit-message "Add <pack-name>"
|
| 106 |
```
|
| 107 |
|
| 108 |
### Upload a single file
|
| 109 |
```bash
|
| 110 |
+
hf upload luckyrobots/ContentVault <local-file> <path-in-repo> --commit-message "Add file"
|
| 111 |
```
|
| 112 |
|
| 113 |
### Upload large folder (resumable, multi-threaded, auto-retry)
|
| 114 |
```bash
|
| 115 |
+
hf upload-large-folder luckyrobots/ContentVault <local-folder>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
```
|
| 117 |
|
| 118 |
+
### Delete files remotely
|
| 119 |
```bash
|
| 120 |
+
hf upload luckyrobots/ContentVault . . --include="" --delete="<folder-name>/*" --commit-message "Remove <folder-name>"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
```
|
| 122 |
|
| 123 |
---
|
|
|
|
| 125 |
## Tips
|
| 126 |
|
| 127 |
- **Resumable uploads**: `upload_large_folder` and `hf upload-large-folder` are resumable. If interrupted, run the same command again.
|
| 128 |
+
- **Don't use raw `git push`** for large files — use the CLI/API instead. They handle LFS/Xet automatically.
|
| 129 |
- **Write token required**: Tokens default to read-only. Enable write access at https://huggingface.co/settings/tokens.
|
| 130 |
- **50 GB max per file** on HuggingFace. Split larger files before uploading.
|
| 131 |
+
- **Ignore patterns**: Use `--include` and `--exclude` to filter what gets uploaded.
|
|
|